Counting method and counting device for refrigeration equipment, refrigeration equipment and storage medium

Abstract

The invention relates to the technical field of intelligent household appliances, and discloses a counting method for refrigeration equipment, and the method comprises the steps: obtaining an image of a food material storage region in response to a counting request; under the condition that m is greater than or equal to m0, determining a movement track of the target food material; calculating the number of the target food materials in the food material storage area according to the motion trail of the target food materials; wherein m is the frame number of the image of the food material storage area, m0 is an image frame number threshold value, and m0 is larger than or equal to 3. According to the counting method for the refrigeration equipment, the movement track of the target food material is determined by adopting the images of the multiple frames of food material storage areas, so that the movement track of the target food material is prolonged, the situation that the target food material crosses the warning line can still be detected under the condition that the target food material is small or the taking speed is too high, and the counting accuracy is improved. The accuracy of counting the number of the target food materials in the refrigeration equipment is improved. The invention further discloses a counting device for the refrigeration equipment, the refrigeration equipment and a storage medium.

Description

Technical Field

[0001] The present application relates to the technical field of smart home appliances, for example, to a counting method and counting device for refrigeration equipment, refrigeration equipment, and storage medium. Background Art

[0002] At present, with the development of artificial intelligence technology, more and more home appliances are using artificial intelligence technology. For refrigerators, home appliance manufacturers hope to use new technologies such as image recognition and cloud computing to enable refrigerators to automatically identify ingredients, automatically identify storage and access actions, and automatically count.

[0003] In the related art, a common method for counting food in a refrigerator is to collect images through cameras installed on each layer of the refrigerator for image comparison, and then detect, identify, and track the target food, and use the tripwire detection function to achieve counting.

[0004] In the process of implementing the embodiments of the present disclosure, it is found that there are at least the following problems in the related art:

[0005] The existing trip wire detection function is mostly based on the intersection of the target frame and the virtual line. However, for small target frames or when the speed of picking is too fast, it is easy for the target frame to not be detected when it crosses the virtual line. This will affect the accuracy of the target food quantity counting in the refrigerator.

[0006] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0007] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0008] The embodiments of the present disclosure provide a counting method, a technical device, a refrigeration device and a readable storage medium for a refrigeration device, so as to improve the accuracy of counting the number of target ingredients in a refrigerator.

[0009] In some embodiments, the method includes: in response to a counting request, acquiring an image of the food storage area; when m≥m0, determining the movement trajectory of the target food; and calculating the number of target food in the food storage area based on the movement trajectory of the target food; wherein m is the number of frames of the image of the food storage area, m0 is the image frame number threshold and m0≥3.

[0010] Optionally, the image of the food storage area includes: multiple frames of top view images or multiple frames of side view images of the food storage area.

[0011] Optionally, the step of determining the motion trajectory of the target ingredient includes: determining the ingredient with a changed position as the target ingredient based on multiple frames of overhead images or multiple frames of side view images; and generating the motion trajectory of the target ingredient based on the position information of the target ingredient.

[0012] Optionally, the step of determining the food whose position has changed as the target food according to multiple frames of overhead view images or multiple frames of side view images includes: detecting the position information of the food in the multiple frames of overhead view images, and when a position change of the food is detected, determining the food whose position has changed as the target food, and using the multiple frames of overhead view images as the target images; or, detecting the position information of the food in the multiple frames of side view images, and when a position change of the food is detected, determining the food whose position has changed as the target food, and using the multiple frames of side view images as the target images.

[0013] Optionally, the step of generating the movement trajectory of the target food according to the position information of the target food includes: marking the position of the target food in each frame of the target image; and generating the movement trajectory of the target food according to the position of the target food in each frame of the target image.

[0014] Optionally, the step of generating the motion trajectory of the target food according to the position of the target food in each frame of the target image includes: connecting the position of the target food in the current frame of the target image and the position of the target food in the previous m0-1 frames of the target image as the motion trajectory of the target food.

[0015] Optionally, the target food position in the current frame target image and the target food position in the previous m0-1 frame target image are both located in the tracking chain.

[0016] Optionally, a tracking chain is generated as follows: the target ingredient position in the next target image is predicted based on the target ingredient position in the current frame target image and the target ingredient position in the previous frame target image; the target ingredient position in each frame target image and the predicted target ingredient position in the next frame target image are connected in sequence to generate a tracking chain.

[0017] Optionally, the step of calculating the number of target ingredients in the ingredient storage area based on the movement trajectory of the target ingredients includes: calculating the number of target ingredients in the ingredient storage area when the movement trajectory of the target ingredients intersects with a warning line; wherein the warning line is a dividing line of a position that the target ingredients must pass through when being taken out or put back into the ingredient storage area.

[0018] Optionally, the step of calculating the number of target ingredients in the ingredient storage area includes: when the direction of the movement trajectory of the target ingredient is away from the ingredient storage area, determining that the target ingredient is taken out of the ingredient storage area, and reducing the number of target ingredients in the ingredient storage area by 1; or, when the direction of the movement trajectory of the target ingredient is close to the ingredient storage area, determining that the target ingredient is placed in the ingredient storage area, and increasing the number of target ingredients in the ingredient storage area by 1.

[0019] In some embodiments, the apparatus includes: a processor and a memory storing program instructions, and the processor is configured to execute a counting method for refrigeration equipment when running the program instructions.

[0020] In some embodiments, the refrigeration equipment includes a body, and also includes: a first camera, arranged on the top wall of the body, configured to capture a top view image; a second camera, arranged on the side wall of the body, configured to capture a side view image; and a counting device, arranged on the body.

[0021] In some embodiments, the storage medium includes: a stored program, wherein the program executes a counting method for a refrigeration device when the program is run.

[0022] The counting method and counting device for refrigeration equipment, refrigeration equipment, and storage medium provided in the embodiments of the present disclosure can achieve the following technical effects:

[0023] In the embodiment of the present disclosure, since multiple frames of images of the food storage area are used for image comparison, the movement trajectory of the target food is extended, and when the target food is small or the speed of taking is too fast, the target food can still be detected to have crossed the warning line. Therefore, compared with the related art, the counting method for refrigeration equipment provided by the present disclosure can improve the accuracy of counting the number of target food in the refrigerator.

[0024] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] One or more embodiments are exemplarily described by corresponding drawings, which do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements, and the drawings do not constitute a scale limitation, and wherein:

[0026] Figure 1 is a schematic diagram of a counting method for refrigeration equipment provided by an embodiment of the present disclosure;

[0027] Figure 2 is a schematic diagram of a counting method for refrigeration equipment provided by another embodiment of the present disclosure;

[0028] Figure 3 is a schematic diagram of a counting method for refrigeration equipment provided by yet another embodiment of the present disclosure;

[0029] Figure 4 is a schematic diagram of a counting method for refrigeration equipment provided by yet another embodiment of the present disclosure;

[0030] Figure 5 is a schematic diagram of a counting method for refrigeration equipment provided by yet another embodiment of the present disclosure;

[0031] Figure 6 is a schematic diagram of a counting method for refrigeration equipment provided by yet another embodiment of the present disclosure;

[0032] Figure 7 is a schematic diagram of a counting method for refrigeration equipment provided by yet another embodiment of the present disclosure;

[0033] Figure 8 is a schematic diagram of a counting method for refrigeration equipment provided by yet another embodiment of the present disclosure;

[0034] Fig. 9 is a schematic diagram of a counting method for refrigeration equipment provided by yet another embodiment of the present disclosure;

[0035] Fig.10 is a schematic diagram of a counting method for refrigeration equipment provided by yet another embodiment of the present disclosure;

[0036] Fig.11 is a schematic diagram of a counting device for refrigeration equipment provided by an embodiment of the present disclosure;

[0037] Fig.12 It is a structural schematic diagram of a refrigeration device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0038] In order to be able to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0039] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other.

[0040] At present, refrigeration equipment includes refrigerators and freezers for storing food. Home appliance manufacturers hope to use new technologies such as image recognition and cloud computing to enable refrigeration equipment to automatically identify food, automatically identify storage and access actions, and automatically count. In related technologies, a common method for counting food in refrigeration equipment is to collect images through cameras installed on each layer of the refrigeration equipment for image comparison, and then detect, identify, and track the target food, and use the trip wire detection function to achieve counting.

[0041] The tripwire detection function in the related art is mostly based on the intersection of the target frame and the virtual line. However, for small target frames and when the speed of picking is too fast, it is easy for the target frame to not be detected when it crosses the virtual line. This will affect the accuracy of the target food quantity counting.

[0042] The disclosed embodiment can collect multiple frames of top view images or multiple frames of side view images of the food storage area by setting cameras on the top wall and side wall of the refrigeration equipment, and use multiple frames of food storage area images for image comparison. Thus, the movement trajectory of the target food is extended. In this way, even if the target food is small or the speed of taking is too fast, it is still possible to detect that the target food has crossed the warning line. Therefore, the accuracy of counting the number of target food in the refrigeration equipment is improved.

[0043] Combination Figure 1 As shown, the embodiment of the present disclosure provides a counting method for refrigeration equipment, comprising:

[0044] S110: The refrigeration device obtains an image of the food storage area in response to the counting request.

[0045] S120, the refrigeration device determines whether m is greater than or equal to m0. If the determination result is yes, the process proceeds to S130. If the determination result is no, the process continues to execute S120.

[0046] S130, the refrigeration equipment determines the movement trajectory of the target food.

[0047] S140: The refrigeration device calculates the quantity of the target food in the food storage area according to the movement trajectory of the target food.

[0048] Wherein, m is the number of frames of the image of the food storage area, m0 is the image frame number threshold and m0≥3.

[0049] The counting method for refrigeration equipment provided by the embodiment of the present disclosure obtains an image of the food storage area according to a counting request. When the number of frames of the image of the food storage area is m≥m0, it means that the number of frames of the food storage area currently obtained meets the condition for determining the motion trajectory of the target food. That is, when m≥m0, the motion trajectory of the target food is determined by comparing the images of each frame of the food storage area, and then the number of target food in the food storage area is calculated to improve the accuracy of the statistics of the number of target food in the refrigeration equipment.

[0050] The related art is to count the number of target ingredients in the refrigerator based on the intersection of the target frame and the warning line. When the target frame and the warning line intersect, the number of target ingredients in the food storage area is further calculated.

[0051] In the embodiment of the present disclosure, since multiple frames of images of the food storage area are used for image comparison, the movement trajectory of the target food is extended, and when the target food is small or the speed of taking is too fast, the target food can still be detected to have crossed the warning line. Therefore, the counting method for refrigeration equipment provided in the embodiment of the present disclosure can improve the accuracy of counting the number of target food in the refrigeration equipment.

[0052] Optionally, the image of the food storage area includes: multiple frames of top view images or multiple frames of side view images of the food storage area.

[0053] In this embodiment, multiple frames of overhead view images or multiple frames of side view images of the food storage area are acquired to accurately identify the target food, thereby preventing the target food from being unable to be identified due to obstruction or other reasons by comparing and identifying multiple frames of images from only one angle, thereby improving the accuracy of identifying the target food.

[0054] Combination Figure 2 As shown, the embodiment of the present disclosure provides another counting method for refrigeration equipment, comprising:

[0055] S210: The refrigeration device obtains a plurality of top view images or a plurality of side view images of the food storage area in response to the counting request.

[0056] S220, the refrigeration device determines whether m is greater than or equal to m0. If the determination result is yes, the process proceeds to S230. If the determination result is no, the process continues to execute S220.

[0057] S230, the refrigeration equipment determines the movement trajectory of the target food.

[0058] S240: The refrigeration device calculates the quantity of the target food in the food storage area according to the movement trajectory of the target food.

[0059] Wherein, m is the number of frames of the image of the food storage area, m0 is the image frame number threshold and m0≥3.

[0060] Optionally, the step of determining the motion trajectory of the target ingredient includes: determining the ingredient with a changed position as the target ingredient based on multiple frames of overhead images or multiple frames of side view images; and generating the motion trajectory of the target ingredient based on the position information of the target ingredient.

[0061] In this embodiment, multiple frames of top view images or multiple frames of side view images are used to accurately identify the position change of the food between different frames of images, thereby determining the target food and generating the movement trajectory of the target food. This reduces the possibility that the position of the food cannot be identified due to reasons such as the food being blocked when identifying the position of the food through multiple frames of images at only one angle, and the possibility that the target food cannot be confirmed.

[0062] Combination Figure 3 As shown, the embodiment of the present disclosure provides another counting method for refrigeration equipment, comprising:

[0063] S310: The refrigeration device obtains a plurality of top view images or a plurality of side view images of the food storage area in response to a counting request.

[0064] S320, the refrigeration device determines whether m is greater than or equal to m0. If the determination result is yes, the process proceeds to S330. If the determination result is no, the process continues to execute S320.

[0065] S330: The refrigeration device determines the food with changed position as the target food according to the multiple frames of top view images or the multiple frames of side view images.

[0066] S340: The refrigeration device generates a movement trajectory of the target food according to the location information of the target food.

[0067] S350, the refrigeration equipment calculates the quantity of the target food in the food storage area according to the movement trajectory of the target food.

[0068] Wherein, m is the number of frames of the image of the food storage area, m0 is the image frame number threshold and m0≥3.

[0069] Optionally, the step of determining the food whose position has changed as the target food according to multiple frames of overhead view images or multiple frames of side view images includes: detecting the position information of the food in the multiple frames of overhead view images, and when a position change of the food is detected, determining the food whose position has changed as the target food, and using the multiple frames of overhead view images as the target images; or, detecting the position information of the food in the multiple frames of side view images, and when a position change of the food is detected, determining the food whose position has changed as the target food, and using the multiple frames of side view images as the target images.

[0070] In this embodiment, the position information of the food is detected in multiple frames of top view images or multiple frames of side view images to obtain the position change of the food in different frames of top view images or side view images, and then the food with the position change is determined as the target food, and the multiple frames of top view images or multiple frames of side view images are used as the target image. By using the top view image or the side view image for detection, the position of the food in the image can be determined more accurately, and then the target food can be determined more accurately.

[0071] Furthermore, this embodiment may be executed once or multiple times in the method, and the execution steps of this embodiment may be: detecting the location information of the food in multiple frames of overhead view images, and when a change in the position of the food is detected, taking the food with the changed position as the target food, and taking the multiple frames of overhead view images as the target images.

[0072] Alternatively, the position information of the food is detected in multiple frames of side view images, and when a change in the position of the food is detected, the food with the changed position is used as the target food, and the multiple frames of side view images are used as the target image.

[0073] Alternatively, the position information of the ingredients is detected in multiple frames of overhead view images, and when a position change of the ingredients is detected, the ingredients with the changed position are taken as target ingredients, and the multiple frames of overhead view images are taken as target images; when no position change of the ingredients can be detected in the multiple frames of overhead view images, the position information of the ingredients is detected in multiple frames of side view images, and when a position change of the ingredients is detected in the multiple frames of side view images, the ingredients with the changed position are taken as target ingredients, and the multiple frames of side view images are taken as target images.

[0074] Alternatively, the position information of the food is detected in multiple frames of side view images, and when a position change of the food is detected, the food with the changed position is regarded as the target food, and the multiple frames of side view images are regarded as the target image; when no position change of the food is detected in the multiple frames of side view images, the position information of the food is detected in multiple frames of overhead view images, and when a position change of the food is detected in the multiple frames of overhead view images, the food with the changed position is regarded as the target food, and the multiple frames of overhead view images are regarded as the target image.

[0075] Combination Figure 4 As shown, the embodiment of the present disclosure provides another counting method for refrigeration equipment, comprising:

[0076] S410: The refrigeration device obtains a plurality of frames of top view images or a plurality of frames of side view images of the food storage area in response to a counting request.

[0077] S420, the refrigeration device determines whether m is greater than or equal to m0. If the determination result is yes, the process proceeds to S430 or S431. If the determination result is no, the process continues to execute S420.

[0078] S430, the refrigeration device detects the location information of the food in the multiple frames of the overhead view image. When a change in the location of the food is detected, the food with the changed location is taken as the target food, and the multiple frames of the overhead view image are taken as the target image.

[0079] S431, the refrigeration device detects the position information of the food in the multiple frames of side view images. When a change in the position of the food is detected, the food with the changed position is taken as the target food, and the multiple frames of side view images are taken as the target image.

[0080] S440: The refrigeration device generates a movement trajectory of the target food according to the location information of the target food.

[0081] S450, the refrigeration device calculates the quantity of the target food in the food storage area according to the movement trajectory of the target food.

[0082] Wherein, m is the number of frames of the image of the food storage area, m0 is the image frame number threshold and m0≥3.

[0083] Optionally, the step of generating the movement trajectory of the target food according to the position information of the target food includes: marking the position of the target food in each frame of the target image; and generating the movement trajectory of the target food according to the position of the target food in each frame of the target image.

[0084] In this embodiment, the position of the target food in each frame of the target image is marked to generate the movement trajectory of the target food, so as to calculate the number of target food in the food storage area. If the target food is simply marked with a rectangular frame, and the number of target food in the food storage area is calculated by the rectangular frame, it is easy for the rectangular frame to be too small or the food to be taken too fast to be undetected. Therefore, by calculating the number of target food in the food storage area by the movement trajectory of the target food, the number of target food in the food storage area is reduced, thereby improving the accuracy of the target food quantity statistics in the food storage area.

[0085] Combination Figure 5 As shown, the embodiment of the present disclosure provides another counting method for refrigeration equipment, comprising:

[0086] S510: The refrigeration device obtains a plurality of top view images or a plurality of side view images of the food storage area in response to a counting request.

[0087] S520, the refrigeration device determines whether m is greater than or equal to m0. If the determination result is yes, the process proceeds to S530 or S531. If the determination result is no, the process continues to execute S420.

[0088] S530, the refrigeration device detects the location information of the food in the multiple frames of the overhead view image. When a change in the location of the food is detected, the food with the changed location is taken as the target food, and the multiple frames of the overhead view image are taken as the target image.

[0089] S531, the refrigeration device detects the position information of the food in the multiple frames of side view images. When a change in the position of the food is detected, the food with the changed position is taken as the target food, and the multiple frames of side view images are taken as the target image.

[0090] S540: The refrigeration device marks the location of the target food in each frame of the target image.

[0091] S550: The refrigeration device generates a movement trajectory of the target food according to the position of the target food in each frame of the target image.

[0092] S560: The refrigeration device calculates the quantity of the target food in the food storage area according to the movement trajectory of the target food.

[0093] Wherein, m is the number of frames of the image of the food storage area, m0 is the image frame number threshold and m0≥3.

[0094] Furthermore, the method for marking the position of the target food in the target image is to mark the target food in the target image with a rectangular frame, and use the center point of the rectangular frame as the position of the target food.

[0095] In this embodiment, when confirming the target food, since the volume and shape of the food in the food storage area are different, if the motion trajectory of the target food is generated only by the position of the target food in each frame of the target image, the generated motion trajectory of the target food will deviate from the actual motion trajectory due to the influence of the volume and shape of the target food. Therefore, by marking the target food in the target image with a rectangular frame, the center point of the rectangular frame is used as the position of the target food, so as to reduce the influence of the volume and shape of the target food on the generated motion trajectory of the target food, thereby making the generated motion trajectory more accurate.

[0096] Optionally, the step of generating the motion trajectory of the target food according to the position of the target food in each frame of the target image includes: connecting the position of the target food in the current frame of the target image and the position of the target food in the previous m0-1 frames of the target image as the motion trajectory of the target food.

[0097] Among them, the previous m0-1 frame target image refers to the m0-1th frame target image before the current frame target image in the target image, that is, the m0-1th image frame before the current image frame. For example, if m0 is 5, the previous m0-1 frame target image refers to the image of the 4th frame before the current frame image. According to the frame sequence of the image, if the current frame image is the 9th frame image, the previous m0-1 frame target image is the 5th frame image.

[0098] In this embodiment, since m0≥3, the target ingredient position in the current frame target image and the target ingredient position in the previous m0-1 frame target image are connected as the movement trajectory of the target ingredient, so that the movement trajectory of the target ingredient is extended, thereby reducing the situation where the target ingredient is not detected.

[0099] Combination Figure 6 As shown, the embodiment of the present disclosure provides another counting method for refrigeration equipment, comprising:

[0100] S610: The refrigeration device obtains a plurality of top view images or a plurality of side view images of the food storage area in response to a counting request.

[0101] S620, the refrigeration device determines whether m is greater than or equal to m0. If the determination result is yes, the process proceeds to S630 or S631. If the determination result is no, the process continues to execute S620.

[0102] S630, the refrigeration device detects the location information of the food in the multiple frames of the overhead view image. When a change in the position of the food is detected, the food with the changed position is taken as the target food, and the multiple frames of the overhead view image are taken as the target image.

[0103] S631, the refrigeration device detects the position information of the food in the multiple frames of side view images. When a change in the position of the food is detected, the food with the changed position is taken as the target food, and the multiple frames of side view images are taken as the target image.

[0104] S640: The refrigeration device marks the location of the target food in each frame of the target image.

[0105] S650: The refrigeration device connects the target food position in the current frame target image and the target food position in the previous m0-1 frame target image as a movement trajectory of the target food.

[0106] S660: The refrigeration device calculates the quantity of the target food in the food storage area according to the movement trajectory of the target food.

[0107] Wherein, m is the number of frames of the image of the food storage area, m0 is the image frame number threshold and m0≥3.

[0108] Optionally, the target food position in the current frame target image and the target food position in the previous m0-1 frame target image are both located in the tracking chain.

[0109] In this embodiment, synchronous tracking of the target food can be achieved by generating a tracking chain, and tracking failure will not occur due to occlusion of the target food, which has higher reliability. The target food position in the current frame target image and the target food position in the previous m0-1 frame target image are both located in the tracking chain, which can reduce the situation where the target food cannot be identified in a certain frame image because it is blocked.

[0110] Combination Figure 7 As shown, the embodiment of the present disclosure provides another counting method for refrigeration equipment, comprising:

[0111] S710: The refrigeration device obtains a plurality of top view images or a plurality of side view images of the food storage area in response to a counting request.

[0112] S720, the refrigeration device determines whether m is greater than or equal to m0. If the determination result is yes, the process proceeds to S730 or S731. If the determination result is no, the process continues to execute S720.

[0113] S730, the refrigeration device detects the location information of the food in the multiple frames of the overhead view image. When a change in the position of the food is detected, the food with the changed position is taken as the target food, and the multiple frames of the overhead view image are taken as the target image.

[0114] S731, the refrigeration device detects the position information of the food in the multiple frames of side view images. When a change in the position of the food is detected, the food with the changed position is taken as the target food, and the multiple frames of side view images are taken as the target image.

[0115] S740, the refrigeration equipment marks the location of the target food in each frame of the target image.

[0116] S750, the refrigeration equipment connects the target food position in the current frame target image and the target food position in the previous m0-1 frame target image as the movement trajectory of the target food; wherein, the target food position in the current frame target image and the target food position in the previous m0-1 frame target image are both located in the tracking chain.

[0117] S760: The refrigeration device calculates the quantity of the target food in the food storage area according to the movement trajectory of the target food.

[0118] Wherein, m is the number of frames of the image of the food storage area, m0 is the image frame number threshold and m0≥3.

[0119] Optionally, a tracking chain is generated as follows: the target ingredient position in the next target image is predicted based on the target ingredient position in the current frame target image and the target ingredient position in the previous frame target image; the target ingredient position in each frame target image and the predicted target ingredient position in the next frame target image are connected in sequence to generate a tracking chain.

[0120] In this embodiment, the predicted target ingredient position can better reflect the actual movement state of the target ingredient, thereby achieving continuous tracking of the target ingredient and generating a tracking chain. The target ingredient may not be detected in a certain frame due to occlusion or image quality. The target ingredient position in the next frame target image is predicted based on the target ingredient position in the current frame target image and the target ingredient position in the previous frame target image, and the predicted target ingredient position is made to coincide with the actual target ingredient position in the next frame as much as possible. In the case where the target ingredient cannot be detected in a certain frame image, the predicted target ingredient position is used instead of the actual target ingredient position to prevent the target ingredient's movement trajectory from being disconnected, thereby being able to calculate the number of target ingredients in the ingredient storage area.

[0121] Combination Figure 8 As shown, the embodiment of the present disclosure provides another counting method for refrigeration equipment, comprising:

[0122] S810: The refrigeration device obtains a plurality of top view images or a plurality of side view images of the food storage area in response to a counting request.

[0123] S820, the refrigeration device determines whether m is greater than or equal to m0. If the determination result is yes, the process proceeds to S830 or S831. If the determination result is no, the process continues to execute S820.

[0124] S830, the refrigeration device detects the position information of the food in the multiple frames of the overhead view image. When a change in the position of the food is detected, the food with the changed position is taken as the target food, and the multiple frames of the overhead view image are taken as the target image.

[0125] S831, the refrigeration device detects the position information of the food in the multiple frames of side view images. When a change in the position of the food is detected, the food with the changed position is taken as the target food, and the multiple frames of side view images are taken as the target image.

[0126] S840, the refrigeration equipment marks the location of the target food in each frame of the target image.

[0127] S850, the refrigeration device predicts the position of the target ingredient in the next target image frame according to the position of the target ingredient in the current target image frame and the position of the target ingredient in the previous target image frame.

[0128] S860: The refrigeration device sequentially connects the target food position in each frame of the target image and the predicted target food position in the next frame of the target image to generate a tracking chain.

[0129] S870, the refrigeration equipment connects the target food position in the current frame target image and the target food position in the previous m0-1 frame target image as the movement trajectory of the target food; wherein, the target food position in the current frame target image and the target food position in the previous m0-1 frame target image are both located in the tracking chain.

[0130] S880, the refrigeration device calculates the quantity of the target food in the food storage area according to the movement trajectory of the target food.

[0131] Wherein, m is the number of frames of the image of the food storage area, m0 is the image frame number threshold and m0≥3.

[0132] Optionally, the step of calculating the number of target ingredients in the ingredient storage area based on the movement trajectory of the target ingredients includes: calculating the number of target ingredients in the ingredient storage area when the movement trajectory of the target ingredients intersects with a warning line; wherein the warning line is a dividing line of a position that the target ingredients must pass through when being taken out or put back into the ingredient storage area.

[0133] In this embodiment, since the warning line is the dividing line of the position that the target food must pass through when it is taken out or put back into the food storage area, when the movement trajectory of the target food intersects with the warning line, it means that the target food is put into the food storage area or taken out from the food storage area. In this case, the number of target food in the food storage area is calculated to correctly count the target food, thereby improving the accuracy of the target food quantity statistics in the food storage area.

[0134] Combination Fig. 9 As shown, the embodiment of the present disclosure provides another counting method for refrigeration equipment, comprising:

[0135] S910: The refrigeration device obtains a plurality of top view images or a plurality of side view images of the food storage area in response to a counting request.

[0136] S920, the refrigeration device determines whether m is greater than or equal to m0. If the determination result is yes, the process proceeds to S930 or S931. If the determination result is no, the process continues to execute S920.

[0137] S930, the refrigeration device detects the position information of the food in the multiple frames of the overhead view image. When a change in the position of the food is detected, the food with the changed position is taken as the target food, and the multiple frames of the overhead view image are taken as the target image.

[0138] S931, the refrigeration device detects the position information of the food in the multiple frames of side view images. When a change in the position of the food is detected, the food with the changed position is taken as the target food, and the multiple frames of side view images are taken as the target image.

[0139] S940, the refrigeration device marks the location of the target food in each frame of the target image.

[0140] S950, the refrigeration device predicts the position of the target ingredient in the next target image frame according to the position of the target ingredient in the current target image frame and the position of the target ingredient in the previous target image frame.

[0141] S960, the refrigeration device sequentially connects the target food position in each frame of the target image and the predicted target food position in the next frame of the target image to generate a tracking chain.

[0142] S970, the refrigeration equipment connects the target food position in the current frame target image and the target food position in the previous m0-1 frame target image as the movement trajectory of the target food; wherein, the target food position in the current frame target image and the target food position in the previous m0-1 frame target image are both located in the tracking chain.

[0143] S980, the refrigeration equipment determines whether the movement trajectory of the target food intersects with the warning line. If the determination result is yes, the process proceeds to S990. If the determination result is no, the process continues to execute S980.

[0144] S990, the refrigeration device calculates the quantity of the target food in the food storage area.

[0145] Among them, m is the frame number of the image of the food storage area, m0 is the image frame number threshold and m0≥3, and the warning line is the dividing line of the position that the target food must pass through when taking out or putting back into the food storage area.

[0146] Optionally, the step of calculating the number of target ingredients in the ingredient storage area includes: when the direction of the movement trajectory of the target ingredient is away from the ingredient storage area, determining that the target ingredient is taken out of the ingredient storage area, and reducing the number of target ingredients in the ingredient storage area by 1; or, when the direction of the movement trajectory of the target ingredient is close to the ingredient storage area, determining that the target ingredient is placed in the ingredient storage area, and increasing the number of target ingredients in the ingredient storage area by 1.

[0147] In this embodiment, when the direction of the movement trajectory of the target food is away from the food storage area, it means that the target food is taken out from the food storage area, and the number of the target food in the food storage area is reduced by 1 during the counting process.

[0148] When the direction of the target food's movement track is close to the food storage area, it means that the target food is placed in the food storage area. In this case, the number of target food in the food storage area is increased by 1 during the counting process. The number of target food in the food storage area is calculated by the different directions of the target food's movement track, thereby achieving the correct counting of the target food.

[0149] Combination Fig.10 As shown, the embodiment of the present disclosure provides another counting method for refrigeration equipment, comprising:

[0150] S1010: The refrigeration device obtains a plurality of top view images or a plurality of side view images of the food storage area in response to a counting request.

[0151] S1020, the refrigeration device determines whether m is greater than or equal to m0. If the determination result is yes, the process proceeds to S1030 or S1031. If the determination result is no, the process continues to execute S1020.

[0152] S1030, the refrigeration device detects the position information of the food in the multiple frames of the overhead view image, and when it is detected that the position of the food changes, the food with the changed position is taken as the target food, and the multiple frames of the overhead view image are taken as the target image.

[0153] S1031, the refrigeration device detects the position information of the food in the multiple frames of side view images. When a change in the position of the food is detected, the food with the changed position is taken as the target food, and the multiple frames of side view images are taken as the target image.

[0154] S1040: The refrigeration device marks the location of the target food in each frame of the target image.

[0155] S1050: The refrigeration device predicts the position of the target ingredient in the next target image frame according to the position of the target ingredient in the current target image frame and the position of the target ingredient in the previous target image frame.

[0156] S1060: The refrigeration device sequentially connects the target food position in each frame of the target image and the predicted target food position in the next frame of the target image to generate a tracking chain.

[0157] S1070, the refrigeration equipment connects the target food position in the current frame target image and the target food position in the previous m0-1 frame target image as the movement trajectory of the target food; wherein, the target food position in the current frame target image and the target food position in the previous m0-1 frame target image are both located in the tracking chain.

[0158] S1080, the refrigeration equipment determines whether the movement trajectory of the target food intersects with the warning line. If the judgment result is yes, enter S1081. If the judgment result is no, continue to execute S1080.

[0159] S1081, the refrigeration equipment determines whether the direction of the movement trajectory of the target food is away from the food storage area. If the judgment result is yes, enter S1090; if the judgment result is no, enter S1091.

[0160] S1090: The refrigeration device determines that the target food is taken out from the food storage area, and the quantity of the target food in the food storage area is reduced by 1.

[0161] S1091: The refrigeration device determines that the target food is placed in the food storage area, and the number of the target food in the food storage area is increased by 1.

[0162] Among them, m is the frame number of the image of the food storage area, m0 is the image frame number threshold and m0≥3, and the warning line is the dividing line of the position that the target food must pass through when taking out or putting back into the food storage area.

[0163] More specifically, the value of m0 is m0=3, 5 or 7.

[0164] Preferably, m0=5.

[0165] Table 1

[0166] Technology Type Detection number Number of missed detections Correct rate (%) Response time(s) 3 frames 118 9 92.91 0.127 5 frames 126 1 99.21 0.273 7 frames 126 1 99.21 0.665

[0167] Combined with the method shown in Table 1, the apples taken / put from the food storage area of ​​the refrigeration equipment are automatically detected and counted according to the method described in the above embodiment. The detection data of 127 times when the movement trajectory of the apple intersects with the warning line are manually counted. m0=3, 5 and 7 are set respectively for test experiment comparison. The detection accuracy and response time of each experiment are calculated.

[0168] When m0=3, the number of frames of the top view or side view of the food storage area is small. In this case, the speed of generating the motion trajectory of the apple can be improved, and the response time used by the refrigeration equipment to determine whether the motion trajectory of the apple intersects with the warning line is short, only 0.127s, so that the refrigeration equipment can timely determine the intersection of the motion trajectory with the warning line.

[0169] When m0=5, obtaining a top view or side view image of the food storage area with an appropriate number of frames can increase the speed of generating the motion trajectory of the apple while extending the motion trajectory, and reduce the response time used by the refrigeration equipment to judge, which is 0.273s. This allows the refrigeration equipment to promptly determine the intersection of the motion trajectory and the warning line, and improve the accuracy of the target food quantity statistics. In this case, the apple was taken out or put back 127 times, the number of detections was 126, and the number of missed detections was only 1.

[0170] When m0=7, the number of frames of the top view or side view images of the food storage area is relatively large, which can extend the motion trajectory of the target food. In this case, due to the extension of the motion trajectory of the apple, the apple was taken out or put back 127 times, the number of detected items was 126, and the number of missed items was only 1. This can improve the accuracy of the target food quantity statistics.

[0171] Combination Fig.11 As shown, the embodiment of the present disclosure provides a counting device 200 for refrigeration equipment, including a processor 100 and a memory 101. Optionally, the device may also include a communication interface 102 and a bus 103. The processor 100, the communication interface 102, and the memory 101 may communicate with each other through the bus 103. The communication interface 102 may be used for information transmission. The processor 100 may call the logic instructions in the memory 101 to execute the counting method for refrigeration equipment of the above embodiment.

[0172] In addition, the logic instructions in the memory 101 described above may be implemented in the form of software functional units and when sold or used as independent products, may be stored in a computer-readable storage medium.

[0173] The memory 101 is a computer-readable storage medium that can be used to store software programs and computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor 100 executes the functional application and data processing by running the program instructions / modules stored in the memory 101, that is, the counting method for refrigeration equipment in the above embodiment is implemented.

[0174] The memory 101 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 101 may include a high-speed random access memory and may also include a non-volatile memory.

[0175] Combination Fig.12As shown, an embodiment of the present disclosure provides a refrigeration device 100, including: a main body, a first camera (not shown in the figure), a second camera (not shown in the figure) and the above-mentioned counting device 200 for the refrigeration device. The first camera is arranged on the top wall of the main body, and is configured to collect a top view image. The second camera is arranged on the side wall of the main body, and is configured to collect a side view image. The counting device 200 for the refrigeration device is installed on the product body. The installation relationship described here is not limited to placement inside the product, but also includes installation connections with other components of the product, including but not limited to physical connection, electrical connection or signal transmission connection, etc. Those skilled in the art can understand that the counting device 200 for the refrigeration device can be adapted to a feasible product body, thereby realizing other feasible embodiments.

[0176] An embodiment of the present disclosure provides a storage medium, wherein the storage medium includes a stored program, wherein the program executes a counting method for a refrigeration device when the program is run.

[0177] The above-mentioned storage medium may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0178] The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in the embodiment of the present disclosure. The aforementioned storage medium may be a non-transient storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes, or a transient storage medium.

[0179] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent only possible changes. Unless explicitly required, separate components and functions are optional, and the order of operation may vary. The parts and features of some embodiments may be included in or replace the parts and features of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.

[0180] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods for each specific application to implement the described functions, but such implementations should not be considered to exceed the scope of the embodiments of the present disclosure. The technicians may clearly understand that, for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above may refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

Claims

1. A counting method for refrigeration equipment, characterized in that: include: In response to the counting request, acquiring an image of the food storage area; When m≥m0, determine the movement trajectory of the target food; Calculate the quantity of target ingredients in the food storage area according to the movement trajectory of the target ingredients; Wherein, m is the number of frames of the image of the food storage area, m0 is the image frame number threshold and m0≥3.

2. The counting method for refrigeration equipment according to claim 1, characterized in that: Images of food storage areas, including: Multiple top view images or multiple side view images of the food storage area.

3. The counting method for refrigeration equipment according to claim 2, characterized in that: The steps of determining the movement trajectory of the target food material include: According to the multiple frames of top view images or the multiple frames of side view images, determine the food with changed position as the target food; According to the location information of the target food, the movement trajectory of the target food is generated.

4. The counting method for refrigeration equipment according to claim 3, characterized in that: The step of determining the food with changed position as the target food according to the multiple frames of top view images or the multiple frames of side view images includes: Detecting the position information of the food in the multiple frames of overhead view images, and when a change in the position of the food is detected, taking the food with the changed position as the target food, and taking the multiple frames of overhead view images as the target image; or, The position information of the food is detected in the multiple frames of side view images. When a change in the position of the food is detected, the food with the changed position is taken as the target food, and the multiple frames of side view images are taken as the target image.

5. The counting method for refrigeration equipment according to claim 4, characterized in that: The step of generating a movement trajectory of the target food according to the location information of the target food comprises: Mark the location of the target food in each frame of the target image; According to the position of the target food in each frame of the target image, the movement trajectory of the target food is generated.

6. The counting method for refrigeration equipment according to claim 5, characterized in that: The step of generating a motion trajectory of the target food according to the position of the target food in each frame of the target image comprises: The target food position in the current frame target image and the target food position in the previous m0-1 frame target image are connected as the movement trajectory of the target food.

7. The counting method for refrigeration equipment according to claim 6, characterized in that: The step of calculating the quantity of the target ingredients in the ingredient storage area according to the movement trajectory of the target ingredients comprises: When the movement trajectory of the target food intersects with the warning line, the number of the target food in the food storage area is calculated; wherein the warning line is the dividing line of the position that the target food must pass through when being taken out or put back into the food storage area.

8. A counting device for refrigeration equipment, comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to execute the counting method for refrigeration equipment according to any one of claims 1 to 7 when running the program instructions.

9. A refrigeration device, comprising a body, characterized in that: Also includes: A first camera, disposed on the top wall of the body, configured to capture a top-view image; A second camera, disposed on a side wall of the body, configured to capture a side view image; The counting device according to claim 8 is arranged on the main body.

10. A storage medium, characterized in that: The storage medium includes a stored program, wherein the counting method for refrigeration equipment according to any one of claims 1 to 7 is executed when the program is run.

Images