Refrigerator and article storage method, module, electronic device and readable medium thereof

Abstract

The present application relates to a refrigerator and its article storage method, module, electronic device and readable medium, wherein the method includes: collecting first image data, first spatial point cloud data and storage location information of an article placed in the refrigerator; generating a first virtual article corresponding to the article based on the first image data and the first spatial point cloud data; displaying the first virtual article in a virtual position corresponding to the storage location information in a virtual refrigerator corresponding to the refrigerator; collecting article information at at least one time point after the article is placed in the refrigerator; generating second display information and / or a second virtual model corresponding to the article information at each time point; updating the first virtual article using the second display information and / or the second virtual model, obtaining a second virtual article and displaying it. By constructing and displaying a virtual article corresponding to the article, and updating the virtual article according to the deterioration status at intervals, the problem of not being able to display the storage status of food to the user in a timely manner is solved.

Description

Technical Field

[0001] The present application relates to the field of smart home technology, and in particular to a refrigerator and an article storage method, module, electronic device and readable medium thereof. Background Art

[0002] With the development of intelligent home appliances, refrigerators have entered every household, bringing a lot of convenience to everyone, but poor management of items in the refrigerator will bring more and more new problems.

[0003] For example: after users put food in the refrigerator for a period of time, they often forget what food is placed in each position of the refrigerator, and they don’t know the condition of the food placed in each position, resulting in some food in the refrigerator being left for too long, and even some food being spoiled and difficult to be discovered in time. In this way, the storage status of the food cannot be intuitively displayed to the user in time, which will cause the food to be forgotten or even spoiled.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The present application provides a refrigerator and its item storage method, module, electronic device and readable medium to solve the above-mentioned technical problem of "not being able to intuitively display the storage status of food to the user in a timely manner".

[0006] According to one aspect of an embodiment of the present application, the present application provides a method for storing items in a refrigerator, comprising: when detecting that an item is placed in the refrigerator, collecting first image data, first spatial point cloud data, and storage location information of the item; generating a first virtual item corresponding to the item based on the first image data and the first spatial point cloud data, wherein the first virtual item includes a first virtual model corresponding to the first spatial point cloud data and first display information associated with the first image data; displaying the first virtual item at a virtual location corresponding to the storage location information in a virtual refrigerator corresponding to the refrigerator; collecting at least one time after the item is placed in the refrigerator

[0007] generating second display information 5 and / or a second virtual model corresponding to the item information at each time point; using the second display information and / or the second virtual model to the first virtual

[0008] The item is updated, and a second virtual item is obtained and displayed.

[0009] Optionally, generating a first virtual item corresponding to the item based on the first image data and the first spatial point cloud data includes: constructing a first virtual item corresponding to the item based on the first spatial point cloud data.

[0010] virtual model; extracting first surface information of the object from the first image data, and using the first surface information as first display information; using the first display information as the appearance of the first virtual model

[0011] The first virtual object is rendered according to the surface rendering parameters.

[0012] Optionally, collecting information about the item at at least one time point after the item is placed in the refrigerator includes:

[0013] Obtain the captured image captured by the image acquisition device in the refrigerator within the field of view; determine the items

[0014] If the object is located in the captured image, collect the second image data of at least one time point after the object is placed in the refrigerator, and use the second image data as the object information.

[0015] Optionally, generating the second display information corresponding to the item information at each time point includes:

[0016] Extract the deteriorated surface information from the second image data; generate second surface information according to the deteriorated surface information, and use the second surface information as the second display information.

[0017] Optionally, generating a second virtual model corresponding to the item information at each time point includes: extracting the deterioration morphology information from the second image data; generating a second virtual model according to the deterioration morphology information;

[0018] 2. Virtual model.

[0019] Optionally, collecting information about the item at at least one time point after the item is placed in the refrigerator includes:

[0020] Get the target deterioration curve of the item; determine the point in the target deterioration curve that corresponds to the current time point

[0021] deterioration surface information and / or deterioration morphology information; using the deterioration surface information and / or deterioration morphology information 5 as item information.

[0022] Optionally, generating the second display information corresponding to the item information at each time point includes: extracting deterioration surface information from the item information; generating second surface information using the deterioration surface information, and using the second surface information as the second display information.

[0023] Optionally, generating a second virtual model corresponding to the item information at each time point includes: extracting deterioration morphology information from the item information; and generating the second virtual model according to the deterioration morphology information.

[0024] According to another aspect of an embodiment of the present application, the present application provides an item storage module for a refrigerator, comprising: a first acquisition module, for acquiring first image data, first spatial point cloud data and storage location information of the item when it is detected that the item is placed in the refrigerator; a generation module, for generating a first virtual item corresponding to the item based on the first image data and the first spatial point cloud data, wherein the first virtual item includes a first virtual model corresponding to the first spatial point cloud data and first display information associated with the first image data; a display module, for displaying the first virtual item at a virtual position corresponding to the storage location information in a virtual refrigerator corresponding to the refrigerator; a second acquisition module, for acquiring item information at at least one time point after the item is placed in the refrigerator; a generation module, for generating second display information and / or a second virtual model corresponding to the item information at each time point; an update module, for updating the first virtual item using the second display information and / or the second virtual model, obtaining and displaying the second virtual item.

[0025] According to another aspect of an embodiment of the present application, the present application provides a refrigerator, comprising: an image sensor, for collecting first image data of the object when detecting that the object is placed in the refrigerator; a radar sensor, for collecting first spatial point cloud data and storage location information of the object when detecting that the object is placed in the refrigerator; a controller, connected to the image sensor and the radar sensor, for generating a first virtual object corresponding to the object based on the first image data and the first spatial point cloud data, and displaying the first virtual object at a virtual position corresponding to the storage location information in a virtual refrigerator corresponding to the refrigerator; the controller is also used to obtain the object information at at least one time point after the object is placed in the refrigerator; generate second display information and / or a second virtual model corresponding to the object information at each time point; use the second display information and / or the second virtual model to update the first virtual object, obtain the second virtual object and display it.

[0026] According to another aspect of an embodiment of the present application, the present application provides an electronic device, including a memory, a processor, a communication interface and a communication bus, wherein the memory stores a computer program that can be run on the processor, the memory and the processor communicate through the communication bus and the communication interface, and the steps of the above method are implemented when the processor executes the computer program.

[0027] According to another aspect of an embodiment of the present application, the present application also provides a computer-readable medium having a non-volatile program code executable by a processor, and the program code enables the processor to execute the above method.

[0028] The above technical solution provided by the embodiment of the present application has the following advantages compared with the related art:

[0029] The present application provides a method for storing items in a refrigerator, including: when an item is detected to be placed in a refrigerator, collecting first image data, first spatial point cloud data and storage location information of the item; generating a first virtual item corresponding to the item based on the first image data and the first spatial point cloud data, wherein the first virtual item includes a first virtual model corresponding to the first spatial point cloud data and first display information associated with the first image data; displaying the first virtual item at a virtual location corresponding to the storage location information in a virtual refrigerator corresponding to the refrigerator; collecting item information at at least one time point after the item is placed in the refrigerator; generating second display information and / or a second virtual model corresponding to the item information at each time point; updating the first virtual item using the second display information and / or the second virtual model, obtaining a second virtual item and displaying it. By constructing and displaying a virtual item corresponding to the item placed in the refrigerator, and then updating the virtual item at intervals according to the deterioration status, the problem of not being able to intuitively display the storage status of food to the user in a timely manner is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0032] Figure 1 A flowchart of an optional method for storing items in a refrigerator according to an embodiment of the present application;

[0033] Figure 2 A block diagram of an optional refrigerator article storage module provided according to an embodiment of the present application;

[0034] Figure 3 A schematic diagram of an optional refrigerator provided according to an embodiment of the present application;

[0035] Figure 4 A schematic diagram of an optional electronic device structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0037] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application and have no specific meaning. Therefore, "module" and "component" can be used interchangeably.

[0038] With the development of intelligent home appliances, refrigerators have entered every household, bringing a lot of convenience to everyone, but poor management of items in the refrigerator will bring more and more new problems.

[0039] For example: after users put food in the refrigerator for a period of time, they often forget what food is placed in each position of the refrigerator, and they don’t know the condition of the food placed in each position, resulting in some food in the refrigerator being left for too long, and even some food being spoiled and difficult to be discovered in time. In this way, the storage status of the food cannot be intuitively displayed to the user in time, which will cause the food to be forgotten or even spoiled.

[0040] In order to solve the problems mentioned in the background technology, according to one aspect of the embodiments of the present application, a method for storing items in a refrigerator is provided, such as Figure 1 As shown, including:

[0041] Step 101, when it is detected that an object is placed in a refrigerator, first image data, first spatial point cloud data and storage location information of the object are collected;

[0042] Step 102, generating a first virtual item corresponding to the item based on the first image data and the first spatial point cloud data, wherein the first virtual item includes a first virtual model corresponding to the first spatial point cloud data and first display information associated with the first image data;

[0043] Step 103, displaying the first virtual item at a virtual position corresponding to the storage position information in the virtual refrigerator corresponding to the refrigerator;

[0044] Step 104, collecting information about the item at at least one time point after the item is placed in the refrigerator;

[0045] Step 105, generating second display information and / or a second virtual model corresponding to the item information at each time point;

[0046] Step 106: Update the first virtual object using the second display information and / or the second virtual model to obtain and display the second virtual object.

[0047] The present application is applied to the field of smart refrigerators, and in particular to the storage of items and food in refrigerators.

[0048] Generating a virtual model of an object based on the object's point cloud data and image data is a common simulation method nowadays, but the embodiment provided in the present application generates a virtual object based on the object's display information and a virtual model. That is to say, the virtual object of the present application is composed of a virtual model that does not contain the display information of the object itself, and the display information, wherein the display information can be used to characterize the characteristics of the object (object color, object texture, object pattern, etc.).

[0049] The steps of generating a virtual refrigerator include: obtaining component structural parameters of the refrigerator, and using the component structural parameters to perform modeling to obtain multiple component models of the refrigerator; connecting the component models according to the connection relationship between the various components of the refrigerator to obtain an initial virtual refrigerator; obtaining existing items in the refrigerator, and rendering the virtual models of the existing items to the initial virtual refrigerator to obtain a virtual refrigerator.

[0050] After generating a virtual item corresponding to an item, the present application will continue to update the virtual item according to the item image or the target deterioration curve of the item after a certain period of time, so that the user can intuitively see the shape of the item and see the latest status of the current item.

[0051] As an optional embodiment, generating a first virtual item corresponding to the item based on the first image data and the first spatial point cloud data 0 includes: constructing a first virtual item corresponding to the item based on the first spatial point cloud data

[0052] A first virtual model corresponding to the object; extracting first surface information of the object from the first image data, and using the first surface information as first display information; rendering the first display information as an outer surface rendering parameter of the first virtual model to obtain a first virtual object.

[0053] Millimeter-wave radar can be used to collect spatial point cloud data.

[0054] 5 Optionally, construct a first virtual model corresponding to the object based on the first spatial point cloud data

[0055] The method comprises: using the first spatial point cloud data to determine the object shape and the object volume of the object; and then using the object shape and the object volume to construct a first virtual model.

[0056] Specifically, the first surface information includes but is not limited to the object color, object texture and object pattern of the object.

[0057] 0 Determine the texture image and material image of the object according to the first display information, and then use

[0058] The texture image and the material image are used to render the first virtual model, thereby obtaining a first virtual object that is highly similar to the object.

[0059] As an optional embodiment, collecting information about the item at at least one time point after the item is placed in the refrigerator includes: obtaining a capture image captured by an image capture device in the refrigerator within a field of view; determining whether the item is located in the capture image; if the item is located in the capture image, collecting second image data at at least one time point after the item is placed in the refrigerator, and using the second image data as the item information.

[0060] In the embodiment of the present application, after the items are placed in the refrigerator, the first virtual item needs to be updated at regular intervals so that the user can intuitively see the storage status of the items through the first virtual item.

[0061] To update the first virtual object, it is necessary to determine the object status at the current moment. If the object is placed on the surface of the storage area, then second image data of at least one time point after the object is placed in the refrigerator can be collected, and the second image data can be used as object information to update the first virtual object.

[0062] Next, the generation of the second display information corresponding to the item information at the current time point based on the second image data will be described.

[0063] As an optional embodiment, generating second display information corresponding to the item information at each time point includes: extracting deteriorated surface information from the second image data; generating second surface information according to the deteriorated surface information, and using the second surface information as the second display information.

[0064] The surface state of the object can be seen from the second image data, and the deteriorated surface information can be obtained. The deteriorated surface information is then used to generate the second display information of the object at the current moment, that is, the current object color, object texture, object pattern, etc. of the object is determined through the deteriorated surface information.

[0065] As an optional embodiment, generating a second virtual model corresponding to the object information at each time point includes: extracting deterioration morphology information from the second image data; and generating the second virtual model according to the deterioration morphology information.

[0066] Specifically, the deterioration morphological information can be simply understood as the surface smoothness or fullness of the object. For example, the surface of a fresh apple is smooth, but after being left for a long time, the surface will become wrinkled. In this case, the corresponding virtual object needs to be updated according to the surface smoothness.

[0067] The second virtual model generated by the metamorphic morphological information can allow the user to see the current surface of the object more intuitively.

[0068] The present application also provides another embodiment, extracting metamorphic morphology information and metamorphic surface information from the second image data, then generating second surface information using the metamorphic surface information, and generating a second virtual model based on the metamorphic morphology information, updating the first surface information using the second surface information, and updating the first virtual model using the second virtual model.

[0069] Optionally, reminder lights can be set for virtual items, with each virtual item corresponding to a light. The longer the storage time of the item, the brighter the light. Alternatively, the freshness of the item can be determined based on surface information, and the less fresh the item, the brighter the light.

[0070] As an optional embodiment, collecting information about the item at at least one time point after the item is placed in the refrigerator includes: obtaining a target deterioration curve of the item; determining in the target deterioration curve the deterioration surface information and / or deterioration morphology information corresponding to the current time point; and using the deterioration surface information and / or deterioration morphology information as the item information.

[0071] Optionally, the name of the item is identified through the image data, and then the target deterioration curve corresponding to the item name is searched in the curve library. Labels can also be set for a large number of items in advance, and the target deterioration curve corresponding to the label is searched in the curve library using the label of the item as an index.

[0072] The curve library includes multiple deterioration curves. The deterioration curve of the item is generated in advance through multiple image acquisitions and data statistics. The generation process of the deterioration curve includes: continuously shooting the same test item, recording the pictures of the test item from intact to deteriorated in all stages, and recording the volume change, weight change, color change, three-dimensional shape change and other information of the item in the process. With time as the horizontal coordinate and the volume, weight, color, three-dimensional shape and surface smoothness of the item as the vertical coordinate, the volume change curve, weight change curve, color change curve and three-dimensional shape change curve of the item are generated. These are the deterioration curves of the test item. A mapping relationship is established between the pictures corresponding to each stage and the stages of all curves, that is, when a certain point on the deterioration curve is found, the picture of the test item at the stage where the point is located can be obtained.

[0073] After users put items in the refrigerator for a period of time, they often forget what items are placed in each position of the refrigerator, and they don't know the status of the items placed in each position. The existing technology obtains images of items in the refrigerator by taking pictures to judge the status of the items, but the items in the refrigerator are stacked, and it is difficult to obtain images of all items by taking pictures. This application introduces a target deterioration curve to obtain deterioration surface information and / or deterioration morphology information, and can still synchronously update virtual items according to the current status of the items when the items are covered.

[0074] As an optional embodiment, generating the second display information corresponding to the item information at each time point includes: extracting the deteriorated surface information from the item information; generating the second surface information using the deteriorated surface information, and using the second surface information as the second display information.

[0075] The item information obtained from the target deterioration curve includes deterioration surface information. The steps of using the deterioration surface information to generate second surface information and using the second surface information as second display information have been described in the above steps and will not be repeated here.

[0076] As an optional embodiment, generating a second virtual model corresponding to the item information at each time point includes: extracting deterioration morphology information from the item information; and generating the second virtual model according to the deterioration morphology information.

[0077] The object information obtained from the target deterioration curve includes deterioration morphology information. The step of generating the second virtual model according to the deterioration morphology information has been described in the above steps and will not be repeated here.

[0078] The present application also provides another embodiment, extracting deterioration morphology information and deterioration surface information from the object information, then using the deterioration surface information to generate second surface information, and generating a second virtual model based on the deterioration morphology information, using the second surface information to update the first surface information, and using the second virtual model to update the first virtual model.

[0079] The present application provides a method for storing items in a refrigerator, including: when an item is detected to be placed in a refrigerator, collecting first image data, first spatial point cloud data and storage location information of the item; generating a first virtual item corresponding to the item based on the first image data and the first spatial point cloud data, wherein the first virtual item includes a first virtual model corresponding to the first spatial point cloud data and first display information associated with the first image data; displaying the first virtual item at a virtual location corresponding to the storage location information in a virtual refrigerator corresponding to the refrigerator; collecting item information at at least one time point after the item is placed in the refrigerator; generating second display information and / or a second virtual model corresponding to the item information at each time point; updating the first virtual item using the second display information and / or the second virtual model, obtaining a second virtual item and displaying it. By constructing and displaying a virtual item corresponding to the item placed in the refrigerator, and then updating the virtual item at intervals according to the deterioration status, the problem of not being able to intuitively display the storage status of food to the user in a timely manner is solved.

[0080] According to another aspect of the embodiment of the present application, the present application provides an article storage module for a refrigerator, such as Figure 2 As shown, including:

[0081] The first acquisition module 201 is used to acquire first image data, first spatial point cloud data and storage location information of the object when it is detected that the object is placed in the refrigerator;

[0082] A generating module 202, configured to generate a first virtual item corresponding to the item based on the first image data and the first spatial point cloud data, wherein the first virtual item includes a first virtual model corresponding to the first spatial point cloud data and first display information associated with the first image data;

[0083] A display module 203, configured to display the first virtual item at a virtual position corresponding to the storage position information in the virtual refrigerator corresponding to the refrigerator;

[0084] The second collection module 204 is used to collect information about the item at at least one time point after the item is placed in the refrigerator;

[0085] A generating module 205, configured to generate second display information and / or a second virtual model corresponding to the item information at each time point;

[0086] The updating module 206 is configured to update the first virtual item by using the second display information and / or the second virtual model to obtain and display the second virtual item.

[0087] It should be noted that the first acquisition module 201 in this embodiment can be used to execute step 101 in the embodiment of the present application, the generation module 202 in this embodiment can be used to execute step 102 in the embodiment of the present application, the display module 203 in this embodiment can be used to execute step 103 in the embodiment of the present application, the second acquisition module 204 in this embodiment can be used to execute step 104 in the embodiment of the present application, the generation module 205 in this embodiment can be used to execute step 105 in the embodiment of the present application, and the update module 206 in this embodiment can be used to execute step 106 in the embodiment of the present application.

[0088] Optionally, the generation module 202 is also used to construct a first virtual model corresponding to the object based on the first spatial point cloud data; extract first surface information of the object from the first image data, and use the first surface information as first display information; and render the first display information as the outer surface rendering parameters of the first virtual model to obtain a first virtual object.

[0089] Optionally, the second acquisition module 204 is also used to obtain the captured image captured by the image acquisition device in the refrigerator within the field of view; determine whether the object is located in the captured image; if the object is located in the captured image, collect second image data of at least one time point after the object is placed in the refrigerator, and use the second image data as the object information.

[0090] Optionally, the generating module 205 is further configured to extract deteriorated surface information from the second image data; generate second surface information according to the deteriorated surface information, and use the second surface information as the second display information.

[0091] Optionally, the generating module 205 is further configured to extract the deterioration morphology information from the second image data; and generate the second virtual model according to the deterioration morphology information.

[0092] Optionally, the second acquisition module 204 is further used to obtain a target deterioration curve of the item; determine the deterioration surface information and / or deterioration morphology information corresponding to the current time point in the target deterioration curve; and use the deterioration surface information and / or deterioration morphology information as item information.

[0093] Optionally, the generating module 205 is further configured to extract the deteriorated surface information from the item information; generate second surface information using the deteriorated surface information, and use the second surface information as the second display information.

[0094] Optionally, the generating module 205 is further configured to extract deterioration morphology information from the item information; and generate a second virtual model according to the deterioration morphology information.

[0095] It should be noted here that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments.

[0096] According to another aspect of the embodiment of the present application, the present application provides a refrigerator, such as Figure 3 As shown, including:

[0097] The image sensor 302 is used to collect first image data of the object when detecting that the object is placed in the refrigerator;

[0098] The radar sensor 304 is used to collect the first spatial point cloud data of the object and store the location information when detecting that the object is placed in the refrigerator;

[0099] The controller 306 is connected to the image sensor 302 and the radar sensor 304, and is used to generate a first virtual object corresponding to the object based on the first image data and the first spatial point cloud data, and display the first virtual object at a virtual position corresponding to the storage position information in the virtual refrigerator corresponding to the refrigerator; the controller is also used to obtain the object information at at least one time point after the object is placed in the refrigerator; generate second display information and / or a second virtual model corresponding to the object information at each time point; use the second display information and / or the second virtual model to update the first virtual object, obtain the second virtual object and display it.

[0100] According to another aspect of the embodiments of the present application, the present application provides an electronic device, such as Figure 4 As shown, it includes a memory 401, a processor 403, a communication interface 405 and a communication bus 407. The memory 401 stores a computer program that can be run on the processor 403. The memory 401 and the processor 403 communicate through the communication interface 405 and the communication bus 407. When the processor 403 executes the computer program, the steps of the above method are implemented.

[0101] The memory and processor in the above electronic device communicate via a communication bus and a communication interface. The communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc.

[0102] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0103] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0104] According to another aspect of the embodiments of the present application, a computer-readable medium having a non-volatile program code executable by a processor is provided.

[0105] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0106] When the embodiments of the present application are specifically implemented, reference may be made to the above-mentioned embodiments, which have corresponding technical effects.

[0107] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0108] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0109] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0110] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0111] In the embodiments provided in the present application, it should be understood that the disclosed modules and methods can be implemented in other ways. For example, the module embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, modules or units, which can be electrical, mechanical or other forms.

[0112] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0113] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0114] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk. It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0115] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A method for storing items in a refrigerator, It is characterized in that include: When it is detected that an object is placed in the refrigerator, first image data, first spatial point cloud data and storage location information of the object are collected; generating a first virtual item corresponding to the item based on the first image data and the first spatial point cloud data, wherein the first virtual item includes a first virtual model corresponding to the first spatial point cloud data and first display information associated with the first image data; Displaying the first virtual item at a virtual position corresponding to the storage position information in a virtual refrigerator corresponding to the refrigerator; Collecting item information at at least one time point after the item is placed in the refrigerator; Generate second display information and / or a second virtual model corresponding to the item information at each of the time points; Using the second display information and / or the second virtual model to update the first virtual item, obtain the second virtual item and display it; Collecting the item information at at least one time point after the item is placed in the refrigerator includes: In the case where the image data of the object cannot be obtained, obtaining a target deterioration curve of the object, wherein the target deterioration curve includes a mapping relationship between images of all stages of the test object from intact to deteriorated and corresponding stages in a volume change curve, a weight change curve, a color change curve, and a three-dimensional behavior change curve; Determining metamorphic surface information and / or metamorphic morphology information corresponding to a current time point in the target metamorphic curve; The deteriorated surface information and / or deteriorated morphology information is used as the item information.

2. The method according to claim 1, It is characterized in that Generating a first virtual item corresponding to the item based on the first image data and the first spatial point cloud data includes: constructing the first virtual model corresponding to the object based on the first spatial point cloud data; extracting first surface information of the object from the first image data, and using the first surface information as first display information; The first display information is used as the outer surface rendering parameter of the first virtual model for rendering to obtain the first virtual object.

3. The method according to claim 1, It is characterized in that Collecting the item information at at least one time point after the item is placed in the refrigerator includes: Acquire the captured image captured by the image acquisition device in the refrigerator within the field of view; Determining whether the object is located in the captured image; If the object is located in the collected image, second image data of at least one time point after the object is placed in the refrigerator is collected, and the second image data is used as the object information.

4. The method according to claim 3, It is characterized in that Generating the second display information corresponding to the item information at each time point includes: extracting deteriorated surface information from the second image data; Second surface information is generated according to the deteriorated surface information, and the second surface information is used as second display information.

5. The method according to claim 3, It is characterized in that Generating a second virtual model corresponding to the item information at each of the time points includes: extracting metamorphic morphology information from the second image data; The second virtual model is generated according to the metamorphic morphology information.

6. The method according to claim 5, It is characterized in that Generating the second display information corresponding to the item information at each time point includes: extracting deteriorated surface information from the item information; The deteriorated surface information is used to generate second surface information, and the second surface information is used as second display information.

7. The method according to claim 5, It is characterized in that Generating a second virtual model corresponding to the item information at each of the time points includes: extracting deterioration morphology information from the item information; A second virtual model is generated according to the metamorphic morphology information.

8. An article storage module for a refrigerator, It is characterized in that include: A first acquisition module, for acquiring first image data, first spatial point cloud data and storage location information of the object when detecting that the object is placed in the refrigerator; a generating module, configured to generate a first virtual item corresponding to the item based on the first image data and the first spatial point cloud data, wherein the first virtual item includes a first virtual model corresponding to the first spatial point cloud data and first display information associated with the first image data; a display module, configured to display the first virtual item at a virtual position corresponding to the storage position information in a virtual refrigerator corresponding to the refrigerator; A second collection module is used to collect item information at at least one time point after the item is placed in the refrigerator; A generating module, configured to generate second display information and / or a second virtual model corresponding to the item information at each of the time points; An updating module, configured to update the first virtual item by using the second display information and / or the second virtual model, obtain the second virtual item and display it; The second acquisition module is also used for: In the case where the image data of the object cannot be obtained, obtaining a target deterioration curve of the object, wherein the target deterioration curve includes a mapping relationship between images of all stages of the test object from intact to deteriorated and corresponding stages in a volume change curve, a weight change curve, a color change curve, and a three-dimensional behavior change curve; Determining metamorphic surface information and / or metamorphic morphology information corresponding to a current time point in the target metamorphic curve; The deteriorated surface information and / or deteriorated morphology information is used as the item information.

9. A refrigerator, It is characterized in that include: The image sensor is used to collect first image data of the object when detecting that the object is placed in the refrigerator; A radar sensor, configured to collect first spatial point cloud data and store location information of an object when detecting that the object is placed in the refrigerator; a controller, together with the image sensor and the radar sensor, configured to generate a first virtual object corresponding to the object based on the first image data and the first spatial point cloud data, and to display the first virtual object at a virtual position corresponding to the storage position information in a virtual refrigerator corresponding to the refrigerator; The controller is further used to obtain item information at at least one time point after the item is placed in the refrigerator; generate second display information and / or a second virtual model corresponding to the item information at each time point; update the first virtual item using the second display information and / or the second virtual model to obtain and display the second virtual item; Collecting the item information at at least one time point after the item is placed in the refrigerator includes: when the image data of the item cannot be obtained, obtaining the target deterioration curve of the item, wherein the target deterioration curve includes a mapping relationship between images of all stages of the test item from intact to deteriorated and corresponding stages in a volume change curve, a weight change curve, a color change curve, and a three-dimensional behavior change curve; determining the deterioration surface information and / or deterioration morphology information corresponding to the current time point in the target deterioration curve; and using the deterioration surface information and / or deterioration morphology information as the item information.

10. An electronic device comprising a memory, a processor, a communication interface and a communication bus, wherein the memory stores a computer program that can be run on the processor, and the memory and the processor communicate through the communication bus and the communication interface. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer readable medium having a non-volatile program code executable by a processor, It is characterized in that The program code enables the processor to execute the method according to any one of claims 1 to 7.

Images