Detection method and equipment for refrigerator area images and refrigerator

Abstract

The present invention discloses a method, device, and refrigerator for detecting images of refrigerator-specific areas. The method comprises: cropping a fixed area of ​​an image to be detected, obtaining an image area of ​​the specific area where yogurt is placed, and obtaining a first image area; comparing the first image area with an image area of ​​an empty bottle holder to obtain a changed second image area; performing shape detection on the second image area, and passing the detected shape area into a YOLO algorithm for recognition detection to obtain a detection result. Compared with the prior art, the method for detecting images of refrigerator-specific areas of the present invention crops out an image area of ​​the refrigerator-specific area, then compares the image area of ​​the refrigerator-specific area with an image area of ​​an empty bottle holder without yogurt, thereby quickly discovering changed areas, performing circular and rectangular detection on the changed areas, and passing the detected circular and rectangular areas into the YOLO algorithm, thereby increasing the YOLO algorithm's image detection efficiency by 2 to 3 times.

Description

Technical Field

[0001] The present invention relates to the field of household appliances, and in particular to a method and device for detecting images of a refrigerator area, and a refrigerator. Background Art

[0002] With the advancement of technology, the intelligentization of refrigerators has become an inevitable trend. A smart refrigerator is one that offers intelligent control and food management. Specifically, it can automatically switch refrigerator modes to keep food in optimal storage conditions. Users can access information about food quantity, freshness, and quality anytime, anywhere via their phone or computer. It can also provide healthy recipes and nutritional recommendations, and remind users to restock their food regularly.

[0003] In order to realize the above functions, it is necessary to install a camera in the refrigerator to view the food in the refrigerator and detect and identify the food in the refrigerator.

[0004] In existing technologies, object detection is a relatively practical method for obtaining useful information from images. It identifies objects in a given image and determines their locations. In recent years, object detection algorithms have achieved significant breakthroughs, with the YOLO algorithm being a popular one. The YOLO algorithm transforms the object detection problem into a single regression problem, directly extracting bounding boxes and class probabilities from an image. This allows the detection of object categories and locations with a single look (YOLO).

[0005] Compared to traditional object detection methods, this unified model offers many advantages: YOLO's prediction process is simple and fast, enabling real-time detection. It uses full image information for prediction, reducing the background prediction error rate by half compared to Fast R-CNN. It also learns a generalizable representation of the target, offering a degree of universality and significantly higher accuracy than other object detection methods (DPM and R-CNN). Therefore, existing methods for identifying food items in refrigerators typically feed the entire photo into the YOLO algorithm for recognition.

[0006] However, since the YOLO algorithm needs to calculate the entire photo, the calculation amount is large and the detection time is long. Therefore, how to improve the detection speed of food in the refrigerator is a technical problem that needs to be solved urgently. Summary of the Invention

[0007] The object of the present invention is to provide a method and device for detecting refrigerator area images and a refrigerator.

[0008] To achieve one of the above-mentioned objectives, an embodiment of the present invention provides a method for detecting an image of a refrigerator section, wherein the refrigerator section includes a bottle holder for storing yogurt, and the method comprises:

[0009] Perform fixed region cropping on the image to be detected to obtain a special image region where yogurt is placed, thereby obtaining a first image region;

[0010] Comparing the first image region with the empty bottle seat image region to obtain a changed second image region;

[0011] Perform shape detection on the second image area, and pass the detected shape area into the Yolo algorithm for recognition detection to obtain a detection result.

[0012] As a further improvement of an embodiment of the present invention, the “performing shape detection on the second image area and passing the detected shape area into the Yolo algorithm for recognition detection” specifically includes:

[0013] performing circle detection and rectangle detection on the second image area;

[0014] The detected circular area or rectangular area is passed into the Yolo algorithm for recognition and detection.

[0015] As a further improvement of an embodiment of the present invention, the step of “comparing the first image region with the empty bottle holder image region to obtain a changed second image region” specifically includes:

[0016] The RGB value of the first image area is compared with the RGB value of the empty bottle seat image area to obtain an area where the RGB value changes, that is, the second image area.

[0017] As a further improvement of one embodiment of the present invention, the method further includes:

[0018] A fixed area is cropped for the image without yogurt to obtain the empty bottle seat image area.

[0019] As a further improvement of one embodiment of the present invention, the method further includes:

[0020] The detection results include the location and label of each bottle of yogurt.

[0021] As a further improvement of an embodiment of the present invention, the label of the yogurt is the type of yogurt.

[0022] As a further improvement of one embodiment of the present invention, the method further includes:

[0023] Obtain a large number of yogurt images, train the YOLO algorithm, and obtain a yogurt recognition and detection model.

[0024] As a further improvement of an embodiment of the present invention, the “transferring the detected shape area into the YOLO algorithm for recognition and detection” specifically includes:

[0025] The detected multiple shape areas are passed into the yogurt recognition detection model of the YOLO algorithm;

[0026] The yogurt recognition and detection model simultaneously recognizes and detects the multiple shape areas.

[0027] To achieve one of the above-mentioned purposes of the invention, one embodiment of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, it implements the steps in any one of the above-mentioned refrigerator area image detection methods.

[0028] To achieve one of the above-mentioned objects of the invention, one embodiment of the present invention provides a refrigerator, which includes the electronic device as described above.

[0029] Compared with the prior art, the method for detecting refrigerator area pictures of the present invention crops out the refrigerator area picture, and then compares the refrigerator area picture area with the empty bottle holder picture area without yogurt, so as to quickly discover the changed area, and perform circular and rectangular detection on the changed area, and pass the detected circular and rectangular areas into the YOLO algorithm, thereby improving the detection efficiency of the YOLO algorithm for the image by 2 to 3 times. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flow chart of a method for detecting refrigerator zone images of the present invention.

[0031] Figure 2 It is a specific example of the image to be detected in the present invention.

[0032] Among them, 1. the entire area of ​​the image to be detected; 2. the bottle base area; 3. the yogurt placed in the bottle base area. DETAILED DESCRIPTION

[0033] The present invention will be described in detail below with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional changes made by those skilled in the art based on these embodiments are all within the scope of protection of the present invention.

[0034] In the present invention, the refrigerator dedicated area refers to a dedicated space in the refrigerator for storing various yogurts, and a fixed camera is set on the top of the refrigerator dedicated area to take pictures of yogurt. The yogurt in the yogurt picture can then be identified and detected using the Yolo algorithm.

[0035] In order to speed up the detection speed, the present invention provides a detection method for a refrigerator area picture, wherein the refrigerator area includes a bottle holder for storing yogurt. The method cuts out the refrigerator area picture area and then compares the refrigerator area picture area with the empty bottle holder picture area without yogurt, so as to quickly discover the changed area, perform circular and rectangular detection on the changed area, and pass the detected circular and rectangular areas into the YOLO algorithm, thereby improving the detection efficiency of the YOLO algorithm for the image by 2 to 3 times.

[0036] like Figure 1 As shown, the method includes:

[0037] Step S100: performing fixed region cropping on the image to be detected, obtaining a special image region where yogurt is placed, and obtaining a first image region.

[0038] like Figure 2 As shown, the image to be detected is the overall picture 1 of the refrigerator area taken by the corresponding camera, which includes the bottle seat area 2 and the area around the bottle seat. In order to eliminate unnecessary data as much as possible, the image to be detected is cropped in a fixed area, and only the ROI area (that is, the area where yogurt is placed) is retained to obtain the first image area.

[0039] It should be noted that, since the camera's shooting angle is fixed, the cropping of the first image area can be a fixed cropping (i.e., the same area is cropped each time). In addition, for images in which no objects are placed in the bottle holder, a fixed cropping can be performed to obtain a corresponding empty bottle holder image area for comparison with the first image area.

[0040] Step S200: Compare the first image region with the empty bottle holder image region to obtain a changed second image region.

[0041] Since the background of the special picture area where yogurt is placed is the same (both are empty bottle holders), in order to quickly detect the location of the yogurt, the picture with yogurt (first picture area) can be compared with the picture without yogurt (empty bottle holder picture area) to obtain the changed area, which is the second picture area.

[0042] It should be noted that a fixed area of ​​the image without yogurt can be cropped to obtain the empty bottle seat image area.

[0043] In a preferred embodiment, the step of “comparing the first image region with the empty bottle holder image region to obtain a changed second image region” specifically includes:

[0044] The RGB value of the first image area is compared with the RGB value of the empty bottle seat image area to obtain an area where the RGB value changes, that is, the second image area.

[0045] The RGB color model is an industry color standard that creates a wide variety of colors by varying and superimposing the three color channels: red (R), green (G), and blue (B). RGB represents the colors of the three channels, red, green, and blue. This standard encompasses nearly every color perceived by human vision and is one of the most widely used color systems. Each pixel in each image has a corresponding value in the red, green, and blue channels, known as its RGB value. Therefore, for pixels located in the same position in the first image area and the empty bottle seat image area, their RGB values ​​are compared. The area containing pixels whose RGB values ​​have changed is the second image area.

[0046] Step S300: Perform shape detection on the second image area, and pass the detected shape area into the Yolo algorithm for recognition detection to obtain a detection result.

[0047] The shape detection refers to detecting whether there are various shapes in the second image area, such as circle, rectangle, triangle, etc. Since the top shape of yogurt on the market is mostly round or rectangular (such as Figure 2 As shown, the top shape of the yogurt placed in the bottle holder is circular and rectangular), so the shape detection here is preferably circular detection and rectangular detection.

[0048] Directly performing shape detection on the changed area can greatly improve the detection efficiency.

[0049] Through circle detection and rectangle detection, the image area that meets the requirements is passed to the YOLO algorithm for recognition and detection to obtain the detection results.

[0050] The detection results include the location and label of each bottle of yogurt. The label of the yogurt indicates the type of yogurt, such as Youyi C, Yakult, etc.

[0051] It should be noted that before the YOLO algorithm can recognize and detect yogurt images, it needs to obtain a large number of yogurt images (mainly yogurt top images), and then use these images to train the YOLO algorithm to obtain a yogurt recognition and detection model. When the detected shape area is passed to the YOLO algorithm for recognition and detection, the detected multiple shape areas are passed to the yogurt recognition and detection model of the YOLO algorithm, and then the yogurt recognition and detection model simultaneously recognizes and detects the multiple shape areas.

[0052] In a specific embodiment, a large number of pictures of yogurt tops are obtained, and the yolo algorithm is trained with these pictures to obtain a yogurt recognition and detection model. A picture of the refrigerator area of ​​the refrigerator without yogurt is obtained, and it is fixedly cropped to obtain an empty bottle seat picture area. The picture to be detected is obtained, aligned and fixedly cropped to obtain a first picture area. The RGB values ​​of the corresponding pixels in the first picture area and the empty bottle seat picture area are compared to find the area composed of pixels whose RGB values ​​have changed to obtain a second image area. The second image area is detected using circle detection and rectangle detection, and the detected picture area including the circle or rectangle is passed into the yolo algorithm for recognition and detection. The yogurt recognition and detection model in the Yolo algorithm detects the passed-in picture area to obtain the label and position of the yogurt corresponding to each circle or rectangle. The label of the yogurt refers to the type of yogurt, such as Youyi C or Yakult on the market.

[0053] The detection method of refrigerator zone picture provided by the present invention, according to the specific scene of refrigerator zone in refrigerator, uses specific mode to carry out picture preprocessing before detection, that is, in refrigerator zone, because background is fixed, therefore the image to be detected after cropping is compared with the empty bottle seat image, so as to obtain the picture area of ​​change; In addition, since in refrigerator zone, yogurt appearance is relatively fixed (top is generally circular or rectangular), therefore, directly adopt the mode of first carrying out circular detection and rectangular detection, finally the area of ​​the protection circular or rectangular detected is passed into yolo algorithm. Through picture preprocessing before detection, the detection time of yolo algorithm is greatly shortened.

[0054] Specifically, experimental data shows that the refrigerator area image detection method provided by the present invention can improve the image detection efficiency of the YOLO algorithm by 2 to 3 times. The test process is as follows:

[0055] (1) The image to be detected is directly passed to the YOLO algorithm for detection without preprocessing, which takes a total of 1500 milliseconds.

[0056] (2) A fixed area of ​​the image to be detected is cropped and compared with the corresponding empty bottle seat area to obtain the changed image area. This process takes 15 milliseconds.

[0057] (3) Perform circular and rectangular detection on the changed image area to obtain the circular or rectangular area. This detection method is very time-saving, generally only taking 5 milliseconds.

[0058] (4) The image area pre-processed in steps (2) and (3) is passed to the YOLO algorithm for detection, which takes 500 milliseconds.

[0059] According to the above experimental results, the detection method provided by the present invention only takes 520 milliseconds, while the detection method without the present invention takes 1500 milliseconds. The overall detection time of the detection method provided by the present invention is reduced by nearly 1000 milliseconds, and the detection efficiency is improved by 2 to 3 times.

[0060] The present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, it implements any one of the steps in the above-mentioned image detection method for the refrigerator area, that is, it implements the steps in any one of the technical solutions in the above-mentioned image detection method for the refrigerator area.

[0061] The present invention also provides a refrigerator, which includes the electronic device described above.

[0062] It should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each implementation method can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

[0063] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. Any equivalent implementation methods or changes that do not deviate from the technical spirit of the present invention should be included in the scope of protection of the present invention.

[0064] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architectures, functions and operations of the systems, methods and computer program products according to the embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of the code, and the module, program segment or part of the code contains one or more executable instructions for implementing the specified logical functions. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, which can depend on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in an order different from that disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A method for detecting refrigerator area pictures, characterized in that: The refrigerator area is used to store yogurt and includes a bottle holder. The method includes: Perform fixed region cropping on the image to be detected to obtain a special image region where yogurt is placed, thereby obtaining a first image region; Comparing the first image region with the empty bottle seat image region to obtain a changed second image region; Perform shape detection on the second image area, and pass the detected shape area into the YOLO algorithm for recognition detection to obtain a detection result; The step of "comparing the first image region with the empty bottle seat image region to obtain a changed second image region" specifically includes: Compare the RGB values ​​of the first image area with the RGB values ​​of the empty bottle holder image area to obtain an area where the RGB values ​​have changed, namely, the second image area; The “transferring the detected shape area into the YOLO algorithm for recognition detection” specifically includes: The detected multiple shape areas are passed into the yogurt recognition detection model of the YOLO algorithm; The yogurt recognition and detection model simultaneously recognizes and detects the multiple shape areas.

2. The method for detecting refrigerator area images according to claim 1, characterized in that: The “performing shape detection on the second image area and passing the detected shape area into the YOLO algorithm for recognition detection” specifically includes: performing circle detection and rectangle detection on the second image area; The detected circular area or rectangular area is passed into the Yolo algorithm for recognition and detection.

3. The method for detecting refrigerator area images according to claim 1, characterized in that: The method further comprises: A fixed area is cropped for the image without yogurt to obtain the empty bottle seat image area.

4. The method for detecting refrigerator area images according to claim 1, characterized in that: The method further comprises: The detection results include the location and label of each bottle of yogurt.

5. The method for detecting refrigerator area images according to claim 4, characterized in that: The label of the yogurt is the type of yogurt.

6. The method for detecting refrigerator area images according to claim 1, characterized in that: The method further comprises: Obtain a large number of yogurt images, train the YOLO algorithm, and obtain a yogurt recognition and detection model.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the steps in the method for detecting the refrigerator area image according to any one of claims 1 to 6 are implemented.

8. A refrigerator, characterized in that: The refrigerator comprises the electronic device according to claim 7.

Images