Imaging method for object in drawer, and refrigeration device

Abstract

Disclosed in the present application are an imaging method for an object in a drawer, and a refrigeration device. The method comprises: acquiring a first image when a drawer is at a first position; acquiring a second image when the drawer is at a second position; and, on the basis of the distance between the first position and the second position, fusing images of a target object in the first image and the second image into a fused image.

Description

Imaging method of drawer contents and refrigeration device

[0001] The present application is based on and claims priority to Chinese patent application No. CN202411231762.2, filed on September 3, 2024, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of imaging technology, in particular to a refrigeration device and an imaging method of drawer contents. BACKGROUND

[0003] With the advancement of technology, the functions of refrigeration devices (such as refrigerators) have gradually increased, for example, some refrigeration devices can obtain images of stored foodstuffs to determine the types, freshness, and other information of the foodstuffs. However, for the drawers of refrigeration devices with small space and relatively dark light, the imaging of the contents in the drawers is often more limited, especially when some foodstuffs have packaging bags on their surfaces, which can cause image distortion due to local reflection of the packaging bag surfaces, making it difficult to take clear images. Without clear images or accurate understanding of the contents of the foodstuffs, further determination of the information of the foodstuffs cannot be made, thereby affecting the foodstuff recognition function of the refrigeration device and resulting in poor user experience.

[0004] Any related art mentioned in the specification does not mean that it is recognized or suggested that the related art constitutes part of the common general knowledge in any jurisdiction, or can be reasonably expected to be understood, considered relevant, and / or combined with other related art by those skilled in the art. SUMMARY

[0005] The present application aims to provide an imaging method of drawer contents and a refrigeration device.

[0006] To achieve one of the above purposes, an embodiment of the present application provides an imaging method of drawer contents, comprising the following steps:

[0007] obtaining a first image when the drawer is located at a first position;

[0008] obtaining a second image when the drawer is located at a second position;

[0009] combining the distance between the first position and the second position to fuse the images of target objects in the first image and the second image into a fused image.

[0010] In an embodiment, the imaging method of drawer contents further comprises the steps of:

[0011] establishing a mapping relationship between image pixel size and world size;

[0012] The fusing the images of the target items in the first image and the second image into a fused image according to the distance between the first position and the second position and the mapping relationship comprises:

[0013] Determining a pixel moving distance on an image according to the distance between the first position and the second position and the mapping relationship;

[0014] Fusing the images of the target items in the first image and the second image into a fused image according to the pixel moving distance.

[0015] In an embodiment, the imaging method of the items in the drawer further comprises the steps of:

[0016] Obtaining a real-time video;

[0017] When the drawer is at the second position, if a limb is detected in the real-time video and the intersection-over-union of the limb and the target item exceeds a coverage threshold, recording a start position and an end position of the limb;

[0018] Determining a pixel moving distance on an image according to the start position, the end position, the distance between the first position and the second position, and the mapping relationship.

[0019] In an embodiment, the first position is a closed position in which the drawer is in a closed state, and the second position is a moved-out position in which the drawer is in a moved-out state.

[0020] In an embodiment, the obtaining of the second image when the drawer is at the second position comprises:

[0021] Obtaining distance information, wherein the distance information is a distance between a current position of the drawer and the closed position;

[0022] Determining whether the drawer is moved out and stops moving according to the distance information;

[0023] If yes, obtaining the second image when the drawer is at the second position.

[0024] In an embodiment, the determining whether the drawer is moved out and stops moving according to the distance information comprises:

[0025] When the distance information changes and is greater than a distance error threshold, determining that the drawer is moved out;

[0026] When the distance information stops changing and is still greater than the distance error threshold, determining that the drawer is moved out and stops moving.

[0027] In an embodiment, the acquiring the second image of the drawer in the second position comprises:

[0028] identifying the target item in the first image;

[0029] when the information of the target item cannot be identified, and the area in the first image with pixel brightness greater than the pixel brightness threshold value exceeds the reflection area threshold value, acquiring a second image of the drawer in a second position.

[0030] In an embodiment, the fusing the images of the target item in the first image and the second image into a fused image comprises:

[0031] extracting a reflection area in the first image with pixel brightness greater than the pixel brightness threshold value;

[0032] replacing the reflection area in the second image with the same area as the reflection area to obtain the fused image.

[0033] In an embodiment, the fusing the images of the target item in the first image and the second image into a fused image comprises:

[0034] calculating the area of the reflection area in the first image and the second image with pixel brightness greater than the pixel brightness threshold value;

[0035] judging the size of the area of the reflection area in the first image and the second image respectively;

[0036] replacing the reflection area in the image with smaller area of the reflection area with the same area as the reflection area in the other image to obtain the fused image.

[0037] In an embodiment, the imaging method of the items in the drawer further comprises the steps of:

[0038] identifying the target item in the fused image;

[0039] when the information of the target item cannot be identified, and the area in the fused image with pixel brightness greater than the pixel brightness threshold value exceeds the reflection area threshold value, acquiring a new second image and fusing a new fused image.

[0040] To achieve one of the above purposes, an embodiment of the present application provides a refrigeration device, comprising:

[0041] a drawer;

[0042] an imaging module configured to acquire an image of the drawer;

[0043] a distance measuring module configured to acquire a moving distance of the drawer;

[0044] a storage module, configured to store a computer program;

[0045] a processing module, configured to implement the steps of the imaging method of the article in the drawer when the computer program is executed.

[0046] To achieve one of the above purposes, an embodiment of the present application provides a readable storage medium storing a computer program, which can implement the steps of the imaging method of the article in the drawer when the computer program is executed by a processing module.

[0047] Compared with the prior art, the imaging method of the article in the drawer has the following beneficial effects: the imaging method of the article in the drawer obtains images of the drawer at different positions, combines distance information between different positions, and performs image fusion, thereby reducing or eliminating image distortion caused by reflection, and finally obtaining a clearer and more real article image. For distortion caused by reflection of the surface of the article, since the degree of reflection at different positions is different, the image fusion of different positions can eliminate the adverse effects of reflection. This method does not depend on too many light conditions and does not need to repeatedly adjust the light source. The real image information of the article can be restored well in different light environments, improving the recognition accuracy and fault tolerance of the system. Compared with the recognition method using only a single image, this image fusion method can integrate image information under different angles and light conditions, thereby restoring the real appearance of the article to the greatest extent, especially in a complex light environment. The realness and clarity of the image are improved, which is conducive to subsequent accurate recognition of the information of the article based on the clearer image, and improves the user experience.

[0048] The terms "comprise" and variations of the term, such as "comprises" and "comprising," "comprised of" and "comprising of," "including," "included," "contain" and "containing," as used herein, do not exclude other features, components, elements, or steps. BRIEF DESCRIPTION OF DRAWINGS

[0049] FIG. 1 is a structural schematic diagram of a refrigeration device according to an embodiment of the present application;

[0050] FIG. 2 is a flowchart of an imaging method of an article in a drawer according to an embodiment of the present application;

[0051] FIG. 3 is a schematic diagram of using a checkerboard to establish a mapping relationship between image pixel size and world size according to an embodiment of the present application;

[0052] FIG. 4 is a schematic diagram of a target article being moved out of a drawer according to an embodiment of the present application;

[0053] Fig. 5 is a schematic diagram of the target object being moved by the limb according to an embodiment of the present application;

[0054] Fig. 6 is a schematic diagram of generating a fusion image according to an embodiment of the present application;

[0055] Fig. 7 is a structural block diagram of the electronic components of the refrigeration device according to an embodiment of the present application.

[0056] Wherein, S1, first position; S2, second position; S3, start position; S4, end position; P1, first image; P2, second image; P3, fusion image; 100, refrigeration device; 1, cabinet; 2, door body; 3, refrigeration space; 4, drawer; 10, imaging module; 20, distance measuring module; 30, storage module; 40, processing module; 50, communication bus. DETAILED DESCRIPTION

[0057] The present application will be described in detail below with reference to the specific embodiments shown in the drawings. However, these embodiments do not limit the present application, and the structural, method, or functional changes made by those of ordinary skill in the art based on these embodiments are included in the protection scope of the present application.

[0058] An imaging method of the drawer object and a refrigeration device are provided in an embodiment of the present application, which can reduce or eliminate the image distortion problem caused by reflection.

[0059] Referring to Fig. 1, the refrigeration device 100 of the present embodiment can be a refrigerator, a freezer, a vertical refrigerator, a wine cabinet, etc. The refrigeration device 100 of the following embodiments will be described taking a refrigerator as an example. The refrigerator includes a cabinet 1 and a door body 2, and the door body 2 and the cabinet 1 enclose a refrigeration space 3. A drawer 4 can be arranged in the refrigeration space 3, and the drawer 4 can be used to place objects, such as food materials.

[0060] The imaging module 10 and the distance measuring module 20 are arranged at the drawer 4. The imaging module 10 is used to acquire the image of the drawer 4. The imaging module 10 can include a camera, which is directed to the inside of the drawer 4 to take a photo. The distance measuring module 20 is used to acquire the moving distance of the drawer 4. The distance measuring module 20 can include a distance sensor, which is arranged on the cabinet 1 or the drawer 4 to detect the distance between the drawer 4 and the back plate behind the drawer 4.

[0061] As described in the background section, when the object image is taken, the reflection problem of the local part of the packaging bag, such as the plastic bag, is particularly obvious due to the narrow space and relatively dark light in the drawer 4. The image distortion caused by the reflection problem, or the inability to clearly see the words on the packaging bag, or the inability to clearly see the objects in the packaging bag, in turn causes the inaccurate target recognition problem.

[0062] The embodiment provides an imaging method for combining images in different positions. The method can obtain images in different positions of the drawer 4, and combines distance information between the images to perform image fusion, so that imaging distortion caused by reflection can be significantly reduced.

[0063] The imaging method for the drawer 4 will be described below with reference to FIGS. 1-6. Although the method is provided in the following embodiments or flowcharts, the execution order of steps that do not have necessary causal relationships in logic based on conventional or non-inventive labor is not limited to the execution order provided in the embodiments.

[0064] As shown in FIG. 2, the imaging method for the drawer 4 includes the following steps.

[0065] Step S10: Obtain a first image P1 when the drawer 4 is in a first position S1.

[0066] Step S20: Obtain a second image P2 when the drawer 4 is in a second position S2.

[0067] Step S30: Combine the distance between the first position S1 and the second position S2, and fuse the images of target objects in the first image P1 and the second image P2 into a fused image P3.

[0068] The distance between the first position S1 and the second position S2 can be obtained by a distance sensor of the distance measuring module 20. The first image P1 and the second image P2 can be obtained by a camera of the imaging module 10. The first image P1 and the second image P2 can be integral images or images including only target objects.

[0069] The purpose of the image fusion in different positions in the embodiment is to eliminate defects in images obtained in different positions. For example, the reflection area on the surface of an object is taken as an example. The reflection means that the content of the area cannot be seen, and only a bright spot is seen. In different distances, the reflection area of the same object can be different. The image fusion can effectively remove or weaken the reflection area, select the best image part for fusion, and improve the quality of the obtained image.

[0070] Wherein by combining the distance between the first position S1 and the second position S2, the image fusion can be accurately performed, and the error caused by reflection is reduced. When the drawer 4 moves, the position of the target object changes, and the distance information can be combined to calculate the corresponding pixel movement distance on the image, so as to ensure that the image parts of the same target object are correctly aligned when the image P3 is fused, and the image overlap caused by position deviation is avoided. That is, the fusion only needs to be determined based on the change of the distance, without transforming the light.

[0071] The image fusion method of the embodiment can integrate image information at different angles, and restore the real appearance of the object to the greatest extent, especially in a complex light environment and a surface reflection environment.

[0072] Wherein, the first position S1 is the closing position of the drawer 4 in the closing state, and the second position S2 is the moving-out position of the drawer 4 in the moving-out state.

[0073] When the drawer 4 is closed, the distance d detected by the distance measuring module 20 can be 0, that is, by d=0, it can be judged that the drawer 4 is in the first position S1. When the drawer 4 is moved out, the distance d detected by the distance measuring module 20 at the second position S2 can be d1, that is, by d=d1, it can be judged that the drawer 4 is in the second position S2. In this way, the distance between the first position S1 and the second position S2 is d1, as shown in FIG. 4. Wherein, d1 can be a fixed value, or any value greater than a certain specific value d0.

[0074] By clearly dividing the position state of the drawer 4, the system can respond differently in different states, effectively reducing unnecessary imaging processes and saving system resources. For example, only when d=d1, the image is obtained, or only when d1>d0, the image is obtained. This method not only improves the imaging efficiency, but also reduces the complexity of data processing, optimizes the system performance while ensuring the imaging quality.

[0075] The imaging method of the object in the drawer 4 of the embodiment further comprises the steps of:

[0076] Step S31: Establishing a mapping relationship between the image pixel size and the world size.

[0077] The mapping relationship can be based on a scale, or as shown in FIG. 3, the chessboard pattern is used to associate the image pixel coordinates and the world coordinates. Through this mapping relationship, it can be ensured that the pixel movement distance in the image fusion process corresponds to the actual object movement distance, so that the system can realize more accurate object position judgment, improve the fusion accuracy and the overall imaging effect.

[0078] On this basis, the step S30 comprises:

[0079] S32: determining a corresponding pixel movement distance on the image according to the distance between the first position S1 and the second position S2 and the mapping relationship;

[0080] S33: fusing images of the target item in the first image P1 and the second image P2 into a fused image P3 according to the pixel movement distance.

[0081] For example, if 1 pixel length corresponds to 5 cm in the world coordinate, i.e. 1 pixel: 5 cm, according to the distance d1, the pixel distance moved on the image can be determined, and according to the pixel distance, the same target item in the first image P1 and the second image P2 can be corresponded, as shown in FIG. 4, based on the distance d1, it can be determined that the target item A moves to the target item A'. By this method, the target item A in the first image P1 and the target item A' in the second image P2 are corresponded, which facilitates the fusion of images.

[0082] The imaging method of the items in the drawer 4 further includes the following steps:

[0083] S41: acquiring a real-time video;

[0084] When the drawer 4 is located at the second position S2, if a limb is detected in the real-time video and the intersection over union (IOU) between the limb and the target item exceeds a coverage threshold, the start position S3 and the end position S4 of the limb are recorded.

[0085] S42: determining a corresponding pixel movement distance on the image according to the start position S3, the end position S4, the distance between the first position S1 and the second position S2, and the mapping relationship.

[0086] The "limb" refers to the hand or other body part of the user that may appear in the field of view of the camera when the drawer 4 is opened, which on the one hand interferes with the imaging of the items in the drawer 4, and on the other hand the limb of the user may move the target item, so this situation is handled by detecting the limb target.

[0087] As shown in FIG. 5, when the drawer 4 is in the second position S2 (such as a partially open state), the real-time video captures the inside of the drawer 4, if a limb is detected in this process, and the intersection over union (IOU) between the limb and the target item exceeds a certain preset coverage threshold, it means that the limb may cover or touch the target item, at this time, the start and end positions S4 of the limb are recorded by the real-time video system to determine the movement trajectory of the limb.

[0088] In the absence of the limb, the target object A should move to the position of the target object A', but under the influence of the limb, the target object A moves to the position of the target object A". Since the user cannot stretch into the limb when the drawer 4 is closed at the first position S1, the position (x1, y1) of the target object A' is the starting position S3 of the drawer 4 pulling out the user's limb, and the position of the target object A" (x2, y2) when the user's limb moves away is the end position S4. The movement amount of the object is (Δx, Δy) = (x2, y2) - (x1, y1). Combined with d1 in FIG. 5, the positions of the target object A and the target object A" in the figure can be corresponded.

[0089] The effect of this processing logic is that when the user's hand or other body part appears in the camera picture, the system can recognize this interference and track the movement of the limb, thereby avoiding false image fusion or object recognition due to the intervention of the limb. For example, if the user's hand moves the target object, the system can update the position of the target object according to the recorded starting and end positions S4 to ensure the accuracy of the fused image P3. By combining the limb movement with the position information of the drawer 4, the system can accurately determine whether the object has been moved and to the new position, thereby ensuring the accuracy of image fusion. This method effectively solves the problem of dynamic interference in the imaging process and improves the intelligent level of the system.

[0090] In addition, d1 in FIG. 4 and FIG. 5 can be the same or different.

[0091] In an embodiment, the step S20 comprises:

[0092] Step S21: Obtain distance information, wherein the distance information is the distance between the current position of the drawer 4 and the closed position;

[0093] Step S22: Determine whether the drawer 4 moves out and stops moving according to the distance information;

[0094] If yes, obtain the second image P2 when the drawer 4 is at the second position S2.

[0095] Steps S21 and S22 are used to determine whether the drawer 4 moves out and stops moving, which greatly improves the accuracy and intelligent level of the imaging process. By monitoring the movement state of the drawer 4 in real time, the system can obtain images at the most appropriate time, thereby avoiding image blur or errors caused by the drawer 4 not being completely moved out or still moving. The system can capture images when the drawer 4 stops moving, further improving the stability and accuracy of imaging and reducing the possibility of error accumulation.

[0096] In addition, if the determination in step S22 is negative, it is possible that the drawer 4 is not moved or is closed again without stopping, in which case the second image P2 can not be acquired until the next time the condition is met.

[0097] In an embodiment, the step S22 comprises:

[0098] Step S221: determining that the drawer 4 is moved out when the distance information changes and is greater than the distance error threshold.

[0099] Step S222: determining that the drawer 4 is moved out and stopped moving when the distance information stops changing and is still greater than the distance error threshold.

[0100] Steps S221 and S222 provide more refined determination criteria. In particular, when the distance information changes and is greater than the error threshold, the system determines that the drawer 4 has been moved out. In actual applications, the distance sensor can be affected by the environment and minor errors, and the distance information changes. By the distance error threshold, the error influence is considered when the change in the distance information is greater than the sensor accuracy, and it is considered that the drawer 4 is being moved, which reduces the probability of false triggering imaging and improves the stability of the system.

[0101] In addition, when the distance information stops changing, it is possible that the drawer 4 is closed again or opened and then stopped. By determining that the distance information stops changing and is still greater than the distance error threshold, it is determined that the drawer 4 is moved to a certain opening, and it is determined that the drawer 4 is moved out and stopped moving, which enhances the sensitivity and accuracy of the system. In combination with the change in the distance information, the state of the drawer 4 can be more accurately identified.

[0102] In an embodiment, the step S20 further comprises:

[0103] Step S23: identifying the target item in the first image P1.

[0104] Step S24: acquiring a second image P2 of the drawer 4 when the drawer 4 is located at a second position S2, when the information of the target item cannot be identified, and the area in the first image P1 with pixel brightness greater than a pixel brightness threshold exceeds a reflection area threshold.

[0105] Steps S23 and S24 are further aimed at solving the problem of light reflection on the surface of the object. Some objects can include local key information, and if the key information is blocked, other areas can clearly show and affect the judgment. Conversely, some key information can be clearly identified even if other areas are not clear. Therefore, when the object information can be identified in step S23, the second image P2 can not be acquired, and when the information cannot be identified in step S23, the second image P2 is acquired, thereby improving the efficiency of the judgment.

[0106] In addition, for the judgment of the light reflection area, based on whether the pixel brightness in the first image P1 is greater than the pixel brightness threshold value, the more serious the light reflection area, the greater the pixel value. When it exceeds a certain threshold value, such as the pixel brightness threshold value, it is determined to be light reflection.

[0107] The light reflection area threshold value can be an area parameter or a proportion parameter. When the area in which the pixel brightness in the first image P1 is greater than the pixel brightness threshold value exceeds the light reflection area threshold value, it can be considered that the light reflection area is too large at this time, and it is further determined that the identification at this time is caused by the light reflection problem. This method can automatically acquire a new image in the case of serious light reflection and reduce the influence of light reflection on identification through image fusion, thereby improving the authenticity and clarity of imaging. Especially in a closed and complex light condition environment such as the refrigerator drawer 4, this method can significantly improve the identification rate of the target object and reduce the false judgment caused by light reflection.

[0108] Image fusion is shown in FIG. 6, and a fused image P3 is generated by replacing the light reflection area. Step S30 includes two implementation manners. In one of the implementation manners, step S30 includes:

[0109] Step S34: Extract the light reflection area in which the pixel brightness in the first image P1 is greater than the pixel brightness threshold value.

[0110] Step S35: Replace the light reflection area in the second image P2 with the same area as the light reflection area to obtain the fused image P3.

[0111] In this implementation manner, based on the first image P1, the light reflection area of the first image P1 is replaced by the same position in the second image P2 through comparison and replacement operation to obtain the fused image. This method greatly improves the visual effect of the image, especially in the case of strong light reflection on the surface of the object, which can effectively reduce the influence of the light reflection area on the image quality, so that the final image is more real and accurate.

[0112] In another implementation manner, step S30 includes:

[0113] Step S36: Calculate the area of the reflective region in the first image P1 and the second image P2 whose pixel brightness is greater than the pixel brightness threshold value;

[0114] Step S37: Determine the size of the area of the reflective region in the first image P1 and the second image P2 respectively.

[0115] Step S38: Replace the reflective region in the image with smaller area of the reflective region with the same region in the other image to obtain the fusion image P3.

[0116] In this embodiment, by comparing the areas of the reflective regions in the first image P1 and the second image P2 and selecting the smaller one for replacement, the system can more efficiently reduce the interference of the reflection on the image and ensure the minimization of the reflective region in the image, thereby maximizing the retention of the true details of the object. This method is particularly effective in handling large-area reflective objects and can significantly improve the clarity and recognizability of the image, making the identification of the target object more reliable.

[0117] In an embodiment, the imaging method of the object in the drawer 4 further comprises the steps of:

[0118] Step S50: Identify the target object in the fusion image P3.

[0119] When the information of the target object cannot be identified and the area of the region in the fusion image P3 whose pixel brightness is greater than the pixel brightness threshold value exceeds the reflective region threshold value, a new second image P2 is obtained and a new fusion image P3 is fused.

[0120] That is, when the information of the target object cannot be identified and the area of the region in the fusion image P3 whose pixel brightness is greater than the pixel brightness threshold value exceeds the reflective region threshold value, the above-mentioned steps S20-S30 are re-executed, that is, a new second image P2 of the drawer 4 is obtained when the drawer 4 is located at a new second position S2, and the first image P1 and the new second image P2 of the target object are fused into a new fusion image P3 in combination with the distance between the first position S1 and the new second position S2.

[0121] Since the drawer 4 is not necessarily stopped at the same position each time it is opened, the new second position S2 and the previous second position S2 can not be at the same position, so the position of the reflective region on the target object can change, thereby obtaining a new second image P2 different from the previous image. In this way, the system can obtain optimal imaging through multiple attempts, and through this adaptive imaging strategy, the system can dynamically cope with various lighting changes and object reflection conditions, significantly improving the reliability of the image and the final identification effect.

[0122] The technical features and technical effects of these details work together to ensure that the system can achieve high-quality imaging, improving the accuracy of item identification and the stability of imaging.

[0123] Compared with the prior art, the present embodiment has the following beneficial effects:

[0124] The imaging method of the items in the drawer 4 acquires images of the drawer 4 at different positions and combines distance information between different positions to perform image fusion, thereby reducing or eliminating image distortion caused by reflection, and ultimately obtaining clearer and more realistic images of the items. For distortion caused by reflection of the surface of the items, since the degree of reflection at different positions is different, fusing images at different positions can eliminate the adverse effects of reflection. This method does not rely on excessive light conditions and does not require repeated adjustment of the light source. It can restore the true image information of the items in different light environments, improve the accuracy of identification and the fault tolerance of the system. Compared with the identification method using only a single image, this image fusion method can integrate image information under different angles and lighting conditions to restore the true appearance of the items to the greatest extent, especially in complex light environments. It performs well and ultimately improves the realism and clarity of the image, which is conducive to subsequent accurate identification of the information of the items based on the clearer image, and improves the user experience.

[0125] In one embodiment, a refrigeration device 100 is provided, and the connection mode of various electronic components in the refrigeration device 100 is shown in the structural block diagram of FIG. 7. The refrigeration device 100 includes an imaging module 10 and a ranging module 20. The imaging module 10 can acquire a first image P1 when the drawer 4 is located at a first position S1, and can acquire a second image P2 when the drawer 4 is located at a second position S2. The ranging module 20 can determine the distance between the first position S1 and the second position S2. In addition, the processing module 40 of the refrigeration device 100 can fuse the images of the target items in the first image P1 and the second image P2 into a fused image P3.

[0126] It should be noted that the details of the different roles of the modules in the refrigeration device 100 of the present application are not disclosed, and please refer to the details disclosed in the imaging method of the items in the drawer 4 of the present application.

[0127] Those skilled in the art can understand that the structural block diagram is only an example of the refrigeration device 100 and does not constitute a limitation on the terminal device of the refrigeration device 100. It can include more or fewer components than the diagram, or combine certain components, or different components, for example, the refrigeration device 100 can also include a power supply module, an input / output device, a network access device, etc.

[0128] The device running the imaging method of the items in the drawer 4 can also include a computer, a notebook, a palm computer, a cloud server and other computing devices, and include but not limited to the processing module 40, the storage module 30, and the computer program stored in the storage module 30 and executable on the processing module 40, such as the above-mentioned imaging method of the items in the drawer 4. The processing module 40 executes the computer program to implement the steps in each of the above-mentioned imaging method of the items in the drawer 4, such as the steps shown in FIG. 2.

[0129] The refrigeration device 100 can also include a communication bus 50 for connecting the imaging module 10, the ranging module 20, the processing module 40 and the storage module 30, and the communication bus 50 can include a channel for transmitting information between the above-mentioned imaging module 10, the ranging module 20, the processing module 40 and the storage module 30.

[0130] In addition, the present application also proposes an electronic device including the storage module 30 and the processing module 40, and the processing module 40 can implement the steps in the above-mentioned imaging method of the items in the drawer 4 when executing the computer program, that is, implement the steps in any one of the above-mentioned imaging method of the items in the drawer 4.

[0131] The electronic device can be integrated into part of the refrigeration device 100, or a local terminal device, or part of a cloud server.

[0132] The processing module 40 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or any conventional processor. The processing module 40 is the control center of the refrigeration device 100, and connects various parts of the refrigeration device 100 through various interfaces and lines.

[0133] The storage module 30 can be used to store the computer programs and / or modules, and the processing module 40 can realize various functions of the refrigeration device 100 by running or executing the computer programs and / or modules stored in the storage module 30 and calling the data stored in the storage module 30. The storage module 30 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function, and the like. In addition, the storage module 30 can include a high-speed random access memory, and can also include a non-volatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0134] For example, the computer program can be divided into one or more modules / units, which are stored in the storage module 30 and executed by the processing module 40 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the refrigeration device 100.

[0135] In an embodiment, an example of the present application provides a readable storage medium storing a computer program, which can realize the steps of the above-mentioned imaging method of the items in the drawer 4 when executed by the processing module 40, that is, realize the steps of any one of the above-mentioned imaging method of the items in the drawer 4.

[0136] The modules of the imaging method of the items in the drawer 4, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes of the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer-readable storage medium. The computer program, when executed by the processing module 40, can realize the steps of each method embodiment.

[0137] The computer program includes computer program code, which can be in the form of source code, object code, executable code, or some intermediate form. The computer-readable medium can include any entity or apparatus capable of carrying the computer program code, recording medium, ∪ disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, software distribution medium, and the like. It should be noted that the computer-readable medium contains content that can be appropriately added or deleted according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0138] It should be understood that although the present specification is described in terms of embodiments, not every embodiment contains only one independent technical solution, and the description of the specification is only for the sake of clarity, and those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can be appropriately combined to form other embodiments that those skilled in the art can understand.

[0139] The above series of detailed descriptions are only specific descriptions of the feasible embodiments of the present application, and are not intended to limit the protection scope of the present application. Any equivalent embodiments or changes made without departing from the spirit of the present application shall be included in the protection scope of the present application.

Claims

1. A method of imaging items within a drawer, the method comprising: The method comprises the following steps: acquiring a first image when the drawer is in a first position; acquiring a second image when the drawer is in a second position; combining the distance between the first position and the second position to fuse the images of the target object in the first image and the second image into a fused image.

2. The method of imaging the contents of a drawer of claim 1, wherein, The method further comprises the following steps: establishing a mapping relationship between image pixel size and world size; combining the distance between the first position and the second position to fuse the images of the target object in the first image and the second image into a fused image comprises: determining the corresponding pixel movement distance on the image according to the distance between the first position and the second position and the mapping relationship; fusing the images of the target object in the first image and the second image into a fused image according to the pixel movement distance.

3. The method of imaging the contents of a drawer of claim 2, wherein, The method further comprises the following steps: acquiring a real-time video; when the drawer is in the second position, if a body is detected in the real-time video and the intersection-over-union of the body and the target object exceeds a coverage threshold, recording the start position and the end position of the body; determining the corresponding pixel movement distance on the image according to the start position, the end position, the distance between the first position and the second position, and the mapping relationship.

4. The method of imaging contents of a drawer of claim 1, wherein, The first position is a closed position in which the drawer is in a closed state, and the second position is a moved-out position in which the drawer is in a moved-out state.

5. The method of imaging the contents of a drawer of claim 4, wherein, The acquiring of the second image when the drawer is in the second position comprises: acquiring distance information, wherein the distance information is the distance between the current position of the drawer and the closed position; determining whether the drawer is moved out and stopped moving according to the distance information; if yes, acquiring the second image when the drawer is in the second position.

6. The method of imaging the contents of a drawer of claim 5, wherein, The determination of whether the drawer is moved out and stopped moving according to the distance information comprises: when the distance information changes and is greater than a distance error threshold, determining that the drawer is moved out; when the distance information stops changing and is still greater than the distance error threshold, determining that the drawer is moved out and stopped moving.

7. The method of imaging contents of a drawer of claim 1, wherein, The acquiring of the second image when the drawer is in the second position comprises: identifying the target object in the first image; when the information of the target object cannot be identified and the area in which the pixel brightness in the first image is greater than a pixel brightness threshold exceeds a reflection area threshold, acquiring the second image of the drawer when the drawer is in the second position.

8. The method of imaging the contents of a drawer of claim 7, wherein, The fusing of the images of the target object in the first image and the second image into a fused image comprises: extracting a reflection area in which the pixel brightness in the first image is greater than the pixel brightness threshold; replacing the reflection area in the second image with the same area in the reflection area to obtain the fused image.

9. The method of imaging the contents of a drawer of claim 7, wherein, The fusing of the images of the target object in the first image and the second image into a fused image comprises: calculating the reflection area in which the pixel brightness in the first image and the second image is greater than the pixel brightness threshold; determining the size of the reflection area in the first image and the second image respectively; The reflection region in the image with a smaller reflection region area is replaced by the same region in another image to obtain the fusion image.

10. The method of imaging the contents of a drawer of claim 7, wherein, Further comprising steps of: identifying the target article in the fusion image; when the information of the target article cannot be identified and the area with a pixel brightness greater than the pixel brightness threshold in the fusion image exceeds the reflection region threshold, obtaining a new second image and fusing a new fusion image.

11. A refrigeration appliance characterized in that, comprising: a drawer; an imaging module for obtaining an image inside the drawer; a distance measuring module for obtaining a moving distance of the drawer; a storage module for storing a computer program; a processing module for implementing the steps of the imaging method of the article inside the drawer according to any one of claims 1 to 10 when the computer program is executed.

12. The refrigeration appliance of claim 11, wherein, Further comprising a communication bus for establishing a connection between the imaging module, the distance measuring module, the processing module and the storage module.

13. A readable storage medium storing a computer program, characterized in that, The computer program is executed by the processing module to implement the steps of the imaging method of the article inside the drawer according to any one of claims 1 to 10.

Images