A smart refrigerator, a method, equipment, and medium for determining the door angle.

Abstract

This application discloses a smart refrigerator, a method, device, and medium for determining the door angle. In the embodiments of this application, the first edge corresponding to the bottom region in an image containing the smart refrigerator door area can be identified, as well as the second edge of the smart refrigerator door in the door area of ​​the image can be identified. The opening angle of the smart refrigerator door is determined based on the angle between the first edge and the second edge, thereby ensuring the accuracy of the determined door opening angle and improving the user experience.

Description

Technical Field

[0001] This application relates to the field of smart refrigerator technology, and in particular to a smart refrigerator, a method for determining the door angle, a device and a medium. Background Technology

[0002] With the rapid development of information technology, smart refrigerators have emerged. Smart refrigerators can automatically identify the food stored inside and its location, thus bringing unprecedented convenience to users by managing food in their refrigerators.

[0003] In existing technologies, dynamic computer vision solutions are generally used to monitor in real time what food the user puts in and takes out. In other words, by recognizing the user's dynamic actions, the smart refrigerator automatically identifies the food being stored and retrieved. However, this dynamic computer vision solution requires setting a detection area to determine the food handling operation. But because the smart refrigerator door has a relatively large range of motion, it is difficult to accurately determine this detection area. To accurately determine the detection area, the opening angle of the refrigerator door needs to be accurately determined. Based on this opening angle, the position of the door's compartment in space is determined, which is the detection area, thus determining the food handling operation. Existing technologies typically use angle sensors installed on the smart refrigerator to determine the door's opening angle. However, angle sensors are prone to aging or dulling, affecting the accuracy of determining the door angle and thus reducing the user experience. Summary of the Invention

[0004] This application provides a smart refrigerator, a method, device, and medium for determining the door angle, in order to solve the problem that the accuracy of the determined door opening angle cannot be guaranteed in the prior art, which affects the user experience.

[0005] This application provides a smart refrigerator, the smart refrigerator comprising:

[0006] A data acquisition unit configured to acquire images of the door area of ​​the smart refrigerator;

[0007] The controller is configured to:

[0008] Determine the first edge corresponding to the bottom region of the image; determine the second edge of the smart refrigerator door in the door region of the image; determine the first included angle between the first edge and the second edge, and determine the first included angle as the opening angle of the smart refrigerator door at the current time.

[0009] This application provides a method for determining the door angle, the method being applied to a smart refrigerator, the method comprising:

[0010] Acquire an image containing the door area of ​​the smart refrigerator;

[0011] Determine the first edge corresponding to the bottom region of the image, and determine the second edge of the smart refrigerator door in the door region of the image;

[0012] Determine the first included angle between the first edge and the second edge, and set the first included angle as the angle at which the door of the smart refrigerator is currently open.

[0013] Thirdly, this application provides an electronic device, which includes a processor and a memory. The memory is used to store program instructions, and the processor is used to execute the computer program stored in the memory to implement the steps of the above-described door angle determination method.

[0014] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described door angle determination method.

[0015] In this embodiment, an image containing the door area of ​​a smart refrigerator is acquired, a first edge corresponding to the bottom region of the image is identified, a second edge of the smart refrigerator door within the door area of ​​the image is identified, a first angle between the first edge and the second edge is determined, and this first angle is determined as the current opening angle of the smart refrigerator door. Because this embodiment can identify the first edge corresponding to the bottom region of the acquired image containing the smart refrigerator door area, and the second edge of the smart refrigerator door within the door area of ​​the image, the opening angle of the smart refrigerator door is determined based on the angle between the first edge and the second edge, thus ensuring the accuracy of the determined door opening angle and improving the user experience. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This application provides a schematic diagram of the structure of a smart refrigerator according to some embodiments;

[0018] Figure 2a A schematic diagram of an image of a door area of ​​a smart refrigerator, provided for some embodiments of this application;

[0019] Figure 2bA schematic diagram of an image of a door area of ​​a smart refrigerator, provided for some embodiments of this application;

[0020] Figure 3 A schematic diagram illustrating the process of determining a door angle provided in some embodiments of this application;

[0021] Figure 4 This application provides a schematic diagram of the specific structure of a smart refrigerator according to some embodiments;

[0022] Figure 5 A schematic diagram illustrating a method for determining a door angle provided in some embodiments of this application;

[0023] Figure 6 This is a schematic diagram of an electronic device structure provided for some embodiments of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] In this embodiment, an image containing the door area of ​​a smart refrigerator is acquired, a first edge corresponding to the bottom region of the image is identified, a second edge of the smart refrigerator door within the door area of ​​the image is identified, a first angle between the first edge and the second edge is determined, and this first angle is determined as the current opening angle of the smart refrigerator door. Because this embodiment can identify the first edge corresponding to the bottom region of the acquired image containing the smart refrigerator door area, and the second edge of the smart refrigerator door within the door area of ​​the image, the opening angle of the smart refrigerator door is determined based on the angle between the first edge and the second edge, thus ensuring the accuracy of the determined door opening angle and improving the user experience.

[0026] In order to accurately determine the opening angle of a smart refrigerator door and improve user experience, this application provides a smart refrigerator, a method for determining the door angle, a device, and a medium.

[0027] Figure 1 This application provides a schematic diagram of the structure of a smart refrigerator 100 according to some embodiments. The smart refrigerator 100 includes:

[0028] Acquisition unit 101, the acquisition unit is configured to acquire images of the door area of ​​the smart refrigerator;

[0029] Controller 102, the controller being configured to:

[0030] Determine the first edge corresponding to the bottom region of the image; determine the second edge of the smart refrigerator door in the door region of the image; determine the first included angle between the first edge and the second edge, and determine the first included angle as the opening angle of the smart refrigerator door at the current time.

[0031] To determine the opening angle of the smart refrigerator door, this application includes a data acquisition unit within the smart refrigerator. This unit acquires images of the area encompassing the smart refrigerator door, which are used to determine the door's opening angle. The acquisition unit includes an image acquisition device, which can be an RGB image acquisition device. In this application, the image acquisition device is installed at a designated location on the smart refrigerator, typically at the center of the upper edge of the refrigerator cabinet, with its optical axis pointing downwards to facilitate image acquisition of the area encompassing the smart refrigerator door. The smart refrigerator door area refers to the region including the door's edge. Since the smart refrigerator door can open in real-time, and the opening angle may vary at different times, the position and size of the door area in the image will differ depending on the opening angle.

[0032] In addition, since a large field of view is required to determine the opening angle of the door, an RGB image acquisition device with a large field of view (FOV) is generally selected. However, the image acquired by the RGB image acquisition device with a large FOV will be distorted. Therefore, the distorted image can be corrected in advance. The specific process of image correction for the distorted image is existing technology and will not be described in detail here.

[0033] Because the edges of the door, bottom, and drawers of this smart refrigerator are all straight lines, when the door is opened to a certain angle, the angle is the angle between the door edge and the bottom edge of the refrigerator, or the angle between the door edge and the drawer edge. Generally, the drawer edge and the bottom edge of the refrigerator are horizontal.

[0034] Specifically, since the smart refrigerator may have one door or two doors, and each drawer in the smart refrigerator may have one or two drawers, if the smart refrigerator has two doors and each drawer has two drawers, then when determining the door angle, for each door, the angle between the second edge of the door and the corresponding first edge of the bottom of the refrigerator is determined, or the angle between the second edge of the door and the corresponding first edge of the opened drawer is determined.

[0035] To determine the opening angle of the smart refrigerator door, this application, after acquiring an image containing the door area of ​​the smart refrigerator, can identify a first edge corresponding to the bottom region in the image. This bottom region can be either the area where the bottom edge of the smart refrigerator might exist, or the area where the edge corresponding to the drawer of the smart refrigerator might exist. The first edge can be either the bottom edge of the smart refrigerator or the edge corresponding to the drawer of the smart refrigerator; the edge corresponding to the drawer is the edge of the open drawer of the smart refrigerator that is parallel to the bottom edge of the smart refrigerator.

[0036] Specifically, the first edge corresponding to the bottom region of the image can be identified based on a contour detection method. The method of identifying the first edge corresponding to the bottom region of the image based on contour detection is existing technology and will not be elaborated upon here.

[0037] In this application, to determine the opening angle of the smart refrigerator door, it is necessary to identify the second edge of the smart refrigerator door in the door area of ​​the image. Specifically, the second edge of the smart refrigerator door in the door area of ​​the image can be identified based on contour detection and Hough line detection, etc. However, identifying the second edge of the smart refrigerator door in the door area of ​​the image based on contour detection and Hough line detection is prior art and will not be elaborated upon here.

[0038] In this application, in order to determine the opening angle of the smart refrigerator door, after determining the first edge corresponding to the bottom area of ​​the image and the second edge of the smart refrigerator door in the door area of ​​the image, the first included angle between the first edge and the second edge is determined, and the first included angle is determined as the opening angle of the smart refrigerator door at the current time.

[0039] In this embodiment of the application, the first edge corresponding to the bottom region in the image containing the smart refrigerator door area can be identified, as well as the second edge of the smart refrigerator door in the door area of ​​the image. The opening angle of the smart refrigerator door is determined based on the angle between the first edge and the second edge, thereby ensuring the accuracy of the determined opening angle of the door and improving the user experience.

[0040] In order to determine the first edge corresponding to the bottom region of the image, based on the above embodiments, in this application, the controller 102 is used to perform:

[0041] Determine whether the bottom edge of the smart refrigerator is obstructed; if the bottom edge of the smart refrigerator is obstructed, determine the first edge corresponding to the opened drawer of the smart refrigerator; if the bottom edge of the smart refrigerator is not obstructed, determine the first edge corresponding to the bottom edge of the smart refrigerator.

[0042] Since the first edge corresponding to the bottom area can be either the bottom edge of the smart refrigerator or the edge corresponding to a drawer, to determine whether the first edge is the bottom edge or the edge corresponding to a drawer, we can check if the bottom edge of the smart refrigerator is obscured, that is, whether the drawer is open. If the bottom edge of the smart refrigerator is obscured, it means the drawer is open; therefore, we determine the first edge corresponding to the open drawer. If the bottom edge of the smart refrigerator is not obscured, it means the drawer is not open; therefore, we determine the first edge corresponding to the bottom edge.

[0043] Figure 2a and Figure 2b This application provides a schematic diagram of an image of a smart refrigerator door area captured according to some embodiments. The image is now being discussed. Figure 2a and Figure 2b Explanation:

[0044] Figure 2a This is a schematic diagram of the drawer when it is not open. Feature area 2 and feature area 3 are the door body areas, and feature area 1 is the bottom area. At this time, the bottom edge is not covered.

[0045] Figure 2b This is a schematic diagram of the drawer when it is open. Feature area 2 and feature area 3 are the door body areas, and feature area 1 is the bottom area. At this time, the bottom edge is covered.

[0046] To determine whether the bottom edge is occluded, based on the above embodiments, in this application, the controller 102 is used to perform:

[0047] The image is input into a pre-trained bottom edge occlusion determination model to obtain feature values ​​indicating whether the image contains bottom edge occlusion; based on the feature values, it is determined whether the bottom edge is occluded.

[0048] To determine whether the bottom area of ​​the smart refrigerator is occluded, this application pre-trains a bottom edge occlusion determination model. When the smart refrigerator acquires an image containing its door area, the pre-trained bottom edge occlusion determination model processes the image to obtain feature values ​​indicating whether the image contains bottom edge occlusion. In this application, the feature value for an occluded bottom edge can be set to 0, and the feature value for an unoccluded bottom edge can be set to 1; alternatively, the feature value for an occluded bottom edge can be set to 1, and the feature value for an unoccluded bottom edge can be set to 0. After obtaining the feature values ​​indicating whether the image contains bottom edge occlusion, the occlusion status of the bottom edge is determined based on these feature values.

[0049] To determine whether the bottom edge is occluded, based on the above embodiments, in this application, the controller 102 is used to perform:

[0050] Determine whether the length of the bottom edge of the image is less than a preset length threshold.

[0051] Because the installation position of the image acquisition device is fixed, the length of the identified bottom edge is fixed when the bottom edge is not obstructed. However, when the bottom edge is obstructed, the length of the identified bottom edge will be less than the corresponding length when the bottom edge is not obstructed, and may even be zero. Therefore, to determine whether the bottom edge is obstructed, this application can determine whether the length of the bottom edge is less than a preset length threshold after identifying the bottom edge of the image. If so, it indicates that the bottom edge is obstructed; otherwise, the bottom edge is not obstructed. The preset length threshold is set according to requirements and is less than the corresponding length of the bottom edge when it is not obstructed.

[0052] In order to identify the first edge corresponding to the opened drawer of the smart refrigerator, based on the above embodiments, in this application, the controller 102 is used to perform:

[0053] Identify at least one third edge corresponding to the opened drawer of the smart refrigerator in the image, and identify a first straight line corresponding to the at least one third edge; based on Hough line detection, obtain the first polar angle and the first polar diameter corresponding to the first straight line; for each first straight line, determine whether the first polar angle of the first straight line is within a preset range of first polar angles, and determine whether the first polar diameter of the first straight line is within a preset range of first polar diameters; if so, then identify the third edge corresponding to the first straight line as the first edge.

[0054] When identifying the third edge corresponding to an open drawer of a smart refrigerator in an image, the identified third edge may or may not be the first edge corresponding to the open drawer. Furthermore, the identified third edge may be one or more. To improve the accuracy of identifying the first edge corresponding to an open drawer of a smart refrigerator, this application can determine at least one third edge corresponding to an open drawer in the image, and then filter out the first edge corresponding to the open drawer from these third edges.

[0055] Specifically, since the maximum extent to which each drawer of the smart refrigerator can be opened is known in the world coordinate system, the polar angle range and polar radius range that the first edge corresponding to the opened drawer of the smart refrigerator can be determined in the pixel coordinate system based on the pre-saved correspondence between the world coordinate system and the pixel coordinate system. In other words, the first polar angle range and the first polar angle range are pre-set. The pre-set first polar angle range is the polar angle range that the first edge can be in the pixel coordinate system, and the pre-set first polar radius range is the polar radius range that the first edge can be in the pixel coordinate system.

[0056] When identifying the first edge corresponding to the opened drawer of the smart refrigerator, for each determined third edge, a first straight line corresponding to that third edge can be determined first, and the first polar angle and first polar radius corresponding to the first straight line can be obtained. In this application, the first polar angle and first polar radius corresponding to the first straight line can be obtained based on Hough line detection. The position of the pole in the polar radius coordinate system can be set according to requirements. In this application, the position of the pole can be set at the vertex of the upper left corner of the image.

[0057] The method of obtaining the first polar angle and first polar diameter corresponding to the first straight line based on Hough line detection is existing technology and will not be elaborated here.

[0058] After determining the first polar angle and first polar diameter corresponding to the first straight line, in order to determine the first edge corresponding to the opened drawer of the smart refrigerator, in this application, for each first straight line, it can be determined whether the first polar angle of the first straight line is within a preset range of first polar angles, and whether the first polar diameter of the first straight line is within a preset range of first polar diameters. If so, the third edge corresponding to the first straight line is determined as the first edge. The first edge can be one or more. If there are multiple first edges, the multiple first edges are parallel or lie on a horizontal line.

[0059] In order to determine the first edge corresponding to the bottom edge of the smart refrigerator, based on the above embodiments, in this application, the controller 102 is used to perform:

[0060] Determine at least one fourth edge corresponding to the bottom edge of the smart refrigerator in the image, and determine the second straight line corresponding to the at least one fourth edge; based on Hough line detection, determine the second polar angle and the second polar diameter corresponding to the second straight line; for each second straight line, determine whether the third polar angle of the second straight line is within a preset range of the second polar angle, and determine whether the second polar diameter of the second straight line is within a preset range of the second polar diameter. If so, then determine the fourth edge corresponding to the second straight line as the first edge.

[0061] When identifying the fourth edge corresponding to the bottom edge of a smart refrigerator in an image, the identified fourth edge may or may not be the first edge corresponding to the bottom edge of the smart refrigerator. Furthermore, there may be one or more identified fourth edges. To improve the accuracy of identifying the first edge corresponding to the bottom edge of the smart refrigerator, this application can determine at least one fourth edge corresponding to the bottom edge of the smart refrigerator in the image, and then filter out the first edge corresponding to the bottom edge of the smart refrigerator from these fourth edges.

[0062] Specifically, since the position information of the bottom edge of the smart refrigerator is known in the world coordinate system, the polar angle range and polar radius range that the first edge corresponding to the bottom edge of the smart refrigerator can be determined in the pixel coordinate system based on the pre-saved correspondence between the world coordinate system and the pixel coordinate system. That is, the pre-set second polar angle range and second polar angle range, wherein the pre-set second polar angle range is the polar angle range that the first edge can be corresponding to in the pixel coordinate system, and the pre-set second polar radius range is the polar radius range that the first edge can be corresponding to in the pixel coordinate system.

[0063] When filtering out the fourth edge corresponding to the bottom edge of the smart refrigerator, for each determined fourth edge, the second straight line corresponding to the fourth edge can be determined first, and the second polar angle and the second polar diameter corresponding to the second straight line can be obtained. In this application, the second polar angle and the second polar diameter corresponding to the second straight line can be obtained based on Hough line detection.

[0064] The method of obtaining the second polar angle and second polar diameter corresponding to the second straight line based on Hough line detection is existing technology and will not be elaborated here.

[0065] After determining the second polar angle and second polar diameter corresponding to the second straight line, in order to determine the first edge corresponding to the bottom edge of the smart refrigerator, in this application, for each second straight line, it can be determined whether the second polar angle of the second straight line is within a preset range, and whether the second polar diameter of the second straight line is within a preset range. If so, the fourth edge corresponding to the second straight line is determined as the first edge. The first edge can be one or more. If there are multiple first edges, they are parallel or lie on a horizontal line.

[0066] To improve the accuracy of determining the door angle, based on the above embodiments, in this application, the controller 102 is used to perform:

[0067] If the bottom edge of the smart refrigerator is blocked, it is determined whether the first polar angle is a preset angle. If so, the operation is performed for each first straight line to determine whether the first polar angle of the first straight line is within the preset range of the first polar angle.

[0068] If the bottom edge of the smart refrigerator is not obstructed, it is determined whether the second polar angle is a preset angle. If so, the operation is performed for each second straight line to determine whether the second polar angle of the second straight line is within the preset range of the second polar angle.

[0069] In this application, since the image acquisition device is pre-installed at a set position, namely the middle position of the upper edge of the smart refrigerator cabinet, and the bottom edge and drawer edge of the smart refrigerator are both straight lines, the first edge corresponding to the bottom area of ​​the image in the acquired image containing the door area of ​​the smart refrigerator should be a horizontal straight line.

[0070] However, in real-world scenarios, the image acquisition device may be accidentally moved, causing the bottom region of the captured image containing the door area of ​​the smart refrigerator to have a non-horizontal straight line as the first edge. Therefore, in this case, the corresponding preset first polar diameter range, preset first polar angle range, preset second polar diameter range, preset second polar angle range, preset third polar diameter range, and preset third polar angle range will also have deviations.

[0071] To determine whether there is a deviation, this application first determines whether the first edge corresponding to the bottom area of ​​the image containing the door area of ​​the smart refrigerator is a horizontal straight line. Specifically, since the first edge corresponding to the bottom area is different when the bottom edge of the smart refrigerator is occluded and when it is not occluded, the application will explain the cases where the bottom edge of the smart refrigerator is occluded and the cases where the bottom of the smart refrigerator is not occluded.

[0072] If the bottom edge of the smart refrigerator is obscured, after determining the first polar angle and first polar diameter corresponding to the first straight line, and before determining whether the first polar angles of all the first straight lines are within the preset range of first polar angles, it is pre-determined whether the first edge corresponding to the bottom region of the image containing the door area of ​​the smart refrigerator is a horizontal straight line. That is, it is determined whether the first polar angle is a preset angle. If so, then for each first straight line, the operation of determining whether the first polar angle of the first straight line is within the preset range of first polar angles is performed. Here, the preset angle is related to the setting of the polar point position. For example, if the position of the polar point is located at the vertex of the upper left corner of the image, then the preset angle is 90 degrees; if the position of the polar point is located at the center of the image, then the preset angle is 0 degrees.

[0073] If the bottom edge of the smart refrigerator is not obstructed, after determining the second polar angle and second polar diameter corresponding to the second straight line, before determining whether the second polar angle of each second straight line is within a preset range, it is pre-determined whether the first edge corresponding to the bottom region of the image containing the door area of ​​the smart refrigerator is a horizontal straight line. That is, it is determined whether the second polar angle is a preset angle. If so, the operation of determining whether the second polar angle of each second straight line is within a preset range is performed. The preset angle is related to the setting of the polar point position. For example, if the polar point is located at the top left corner of the image, the preset angle is 90 degrees; if the polar point is located at the center of the image, the preset angle is 0 degrees.

[0074] To improve the accuracy of determining the door angle, based on the above embodiments, in this application, the controller 102 is used to perform:

[0075] If it is determined that the first polar angle or the second polar angle is not a preset angle, then the second difference between the target polar angle (which is not a preset angle) and the preset angle is determined; based on the correspondence between the second difference and the pre-saved world coordinate system and pixel coordinate system, the preset first polar radius range, the preset first polar angle range, the preset second polar radius range, the preset second polar angle range, the preset third polar radius range, and the preset third polar angle range are updated.

[0076] If it is determined that the first polar angle or the second polar angle is not a preset angle, then the first edge corresponding to the bottom area of ​​the image in the acquired image containing the door area of ​​the smart refrigerator is a non-horizontal straight line. In other words, at this time, the corresponding preset first polar diameter range, preset first polar angle range, preset second polar diameter range, preset second polar angle range, preset third polar diameter range, and preset third polar angle range also have deviations.

[0077] To avoid the influence of deviations, the preset first polar radius range, preset first polar angle range, preset second polar radius range, preset second polar angle range, preset third polar radius range, and preset third polar angle range need to be updated. Specifically, during the update process, the polar angle that is not a preset angle in the first or second polar angle is first determined as the target polar angle. That is, if the first polar angle is not a preset angle, then the first polar angle is determined as the target polar angle; if the second polar angle is not a preset angle, then the second polar angle is determined as the target polar angle. Then, the second difference between the target polar angle and the preset angle is determined.

[0078] Specifically, when determining the correspondence between the pre-saved world coordinate system and the pixel coordinate system, different rotation matrices are obtained by rotating around the x-axis, y-axis, and z-axis by different angles. Therefore, when updating the pre-set first polar radius range, pre-set first polar angle range, pre-set second polar radius range, pre-set second polar angle range, pre-set third polar radius range, and pre-set third polar angle range based on the correspondence between the second difference and the pre-saved world coordinate system and the pixel coordinate system, since the deviation in the x-axis and y-axis directions will not cause the straight line of the first edge to be non-horizontal, it is only necessary to update the angle parameter in the rotation matrix corresponding to the z-axis to the sum of the original angle parameter and the second difference. The preset first polar radius range, preset first polar angle range, preset second polar radius range, preset second polar angle range, preset third polar radius range, and preset third polar angle range are updated based on the rotation matrices corresponding to the x-axis, y-axis, and z-axis. Specifically, the prior art of updating the preset first polar radius range, preset first polar angle range, preset second polar radius range, preset second polar angle range, preset third polar radius range, and preset third polar angle range based on the rotation matrices corresponding to the x-axis, y-axis, and z-axis is not elaborated here.

[0079] Identify at least one fifth edge corresponding to the door edge of the smart refrigerator in the image, and determine the third straight line corresponding to the at least one fifth edge; based on Hough line detection, determine the third polar angle and third polar radius corresponding to the third straight line;

[0080] For each third line, determine whether the third polar angle of the third line is within the preset range of the third polar angle, and determine whether the third polar diameter of the third line is within the preset range of the third polar diameter. If so, then the fifth edge corresponding to the third line is determined as the second edge.

[0081] When identifying the fifth edge corresponding to the door edge of a smart refrigerator in an image, the identified fifth edge may or may not be the second edge corresponding to the door edge of the smart refrigerator. Furthermore, the identified fifth edge may be one or more. To improve the accuracy of identifying the second edge corresponding to the bottom edge of the smart refrigerator, this application can determine at least one fifth edge corresponding to the door edge of the smart refrigerator in the image, and then filter out the second edge corresponding to the door edge of the smart refrigerator from these fifth edges.

[0082] Specifically, since the maximum extent to which the door of the smart refrigerator can be opened is known in the world coordinate system, the polar angle range and polar radius range that the second edge corresponding to the edge of the smart refrigerator door can be determined in the pixel coordinate system based on the pre-saved correspondence between the world coordinate system and the pixel coordinate system. In other words, a third polar angle range and a third polar angle range are pre-set. The pre-set third polar angle range is the polar angle range that the second edge can be in the pixel coordinate system, and the pre-set third polar radius range is the polar radius range that the second edge can be in the pixel coordinate system.

[0083] When filtering out the fifth edge corresponding to the door edge of the smart refrigerator, for each determined fifth edge, the third straight line corresponding to the fifth edge can be determined first, and the third polar angle and the third polar diameter corresponding to the third straight line can be obtained. In this application, the third polar angle and the third polar diameter corresponding to the third straight line can be obtained based on Hough line detection.

[0084] The method of obtaining the third polar angle and third polar diameter corresponding to the third line based on Hough line detection is an existing technology and will not be elaborated here.

[0085] After determining the third polar angle and third polar diameter corresponding to the third straight line, in order to determine the second edge corresponding to the door edge of the smart refrigerator, in this application, for each third straight line, it can be determined whether the third polar angle of the third straight line is within a preset range of third polar angles, and whether the third polar diameter of the third straight line is within a preset range of second polar diameters. If so, the fifth edge corresponding to the third straight line is determined as the second edge. The second edge can be one or more. If there are multiple second edges, they are horizontal or connected by a horizontal line.

[0086] Figure 3 This application provides a schematic diagram of a process for determining a door angle, which is now being discussed in some embodiments. Figure 3 Explanation:

[0087] First, the RGB image acquisition device acquires RGB images and pre-stores the correspondence between the world coordinate system and the pixel coordinate system. That is, the first polar radius range, the first polar angle range, the second polar radius range, the second polar angle range, the third polar radius range, and the third polar angle range are pre-set. It then determines whether the drawer is open, that is, whether the bottom edge of the smart refrigerator is obstructed. If it is obstructed, it identifies the first edge corresponding to the open drawer of the smart refrigerator. Then, it identifies the second edge of the smart refrigerator door in the door area of ​​the image. Based on the angle between the first edge and the second edge corresponding to the open drawer of the smart refrigerator, it determines the door angle.

[0088] If it is not obstructed, the first edge corresponding to the bottom edge of the smart refrigerator is identified, and then the second edge of the smart refrigerator door in the door area of ​​the image is identified. Based on the angle between the first edge and the second edge corresponding to the bottom edge of the smart refrigerator, the door angle is determined.

[0089] Figure 4 This is a schematic diagram of the specific structure of a smart refrigerator provided in some embodiments of this application. Now, regarding... Figure 4 Explanation:

[0090] The smart refrigerator has two doors. There are two image acquisition devices in the middle of the upper edge of the smart refrigerator. One image acquisition device is used to acquire RGB images, and the other image acquisition device is used to obtain the coordinates of each position of the smart refrigerator in the world coordinate system. This is to facilitate the subsequent setting of the first polar radius range, the first polar angle range, the second polar radius range, the second polar angle range, the third polar radius range, and the third polar angle range based on the pre-saved correspondence between the world coordinate system and the pixel coordinate system.

[0091] To accurately determine the opening angle of a smart refrigerator door and improve user experience, this application provides a method for determining the door angle, applicable to smart refrigerators. Figure 5This application provides a schematic diagram of a method for determining the angle of a door, which includes the following steps:

[0092] S501: Acquire an image containing the door area of ​​the smart refrigerator.

[0093] S502: Determine the first edge corresponding to the bottom region of the image, and determine the second edge of the smart refrigerator door in the door region of the image.

[0094] S503: Determine the first included angle between the first edge and the second edge, and determine the first included angle as the angle of the door of the smart refrigerator at the current time.

[0095] In one possible implementation, determining the first edge corresponding to the bottom region of the image includes:

[0096] Determine whether the bottom edge of the smart refrigerator is obstructed; if the bottom edge of the smart refrigerator is obstructed, determine the first edge corresponding to the opened drawer of the smart refrigerator; if the bottom edge of the smart refrigerator is not obstructed, determine the first edge corresponding to the bottom edge of the smart refrigerator.

[0097] In one possible implementation, determining whether the bottom edge of the smart refrigerator is obstructed includes:

[0098] The image is input into a pre-trained bottom edge occlusion determination model to obtain feature values ​​indicating whether the image contains bottom edge occlusion; based on the feature values, it is determined whether the bottom edge is occluded.

[0099] In one possible implementation, determining whether the bottom edge of the smart refrigerator is obstructed includes:

[0100] Determine whether the length of the bottom edge of the image is less than a preset length threshold.

[0101] In one possible implementation, determining the first edge corresponding to the bottom region of the image includes:

[0102] Identify at least one third edge corresponding to the opened drawer of the smart refrigerator in the image, and determine the first straight line corresponding to the at least one third edge; based on Hough line detection, obtain the first polar angle and the first polar radius corresponding to the first straight line; for each first straight line, determine whether the first polar angle of the first straight line is within a preset range of first polar angles, and determine whether the first polar radius of the first straight line is within a preset range of first polar radii; if so, then determine the third edge corresponding to the first straight line as the first edge; or

[0103] Determine at least one fourth edge corresponding to the bottom edge of the smart refrigerator in the image, and determine the second straight line corresponding to the at least one fourth edge; based on Hough line detection, determine the second polar angle and the second polar diameter corresponding to the second straight line; for each second straight line, determine whether the third polar angle of the second straight line is within a preset range of the second polar angle, and determine whether the second polar diameter of the second straight line is within a preset range of the second polar diameter. If so, then determine the fourth edge corresponding to the second straight line as the first edge.

[0104] In one possible implementation, if the bottom edge of the smart refrigerator is obstructed, after determining the first polar angle and first polar diameter corresponding to the first straight line, and before determining whether the first polar angle of each first straight line is within a preset range of first polar angles, the method further includes:

[0105] Determine whether the first polar angle is a preset angle. If so, perform the operation of determining whether the first polar angle of each first straight line is within the preset range of the first polar angle.

[0106] In one possible implementation, if the bottom edge of the smart refrigerator is not obstructed, after determining the second polar angle and second polar diameter corresponding to the second straight line, and before determining whether the second polar angle of each second straight line is within a preset range of second polar angles, the method further includes:

[0107] Determine whether the second polar angle is a preset angle. If so, perform the operation of determining whether the second polar angle of each second straight line is within the preset range of the second polar angle.

[0108] In one possible implementation, the method further includes:

[0109] If it is determined that the first polar angle or the second polar angle is not a preset angle, then the second difference between the target polar angle (which is not a preset angle) and the preset angle is determined; based on the correspondence between the second difference and the pre-saved world coordinate system and pixel coordinate system, the preset first polar radius range, the preset first polar angle range, the preset second polar radius range, the preset second polar angle range, the preset third polar radius range, and the preset third polar angle range are updated.

[0110] In one possible implementation, the second edge of the smart refrigerator door in the door area where the image is recognized includes:

[0111] Identify at least one fifth edge corresponding to the door edge of the smart refrigerator in the image, and determine the third straight line corresponding to the at least one fifth edge; based on Hough line detection, determine the third polar angle and the third polar diameter corresponding to the third straight line; for each third straight line, determine whether the third polar angle of the third straight line is within a preset range of third polar angles, and determine whether the third polar diameter of the third straight line is within a preset range of third polar diameters. If so, then the fifth edge corresponding to the third straight line is determined as the second edge.

[0112] This method is applied to smart refrigerators. For details on how smart refrigerators execute this door angle determination method, please refer to the other embodiments mentioned above. The specific details will not be repeated here.

[0113] Figure 6 This application provides a schematic diagram of an electronic device structure based on some embodiments. In addition to the above embodiments, this application also provides an electronic device, such as... Figure 6 As shown, it includes: processor 601, communication interface 602, memory 603 and communication bus 604, wherein processor 601, communication interface 602 and memory 603 communicate with each other through communication bus 604.

[0114] The memory 603 stores a computer program, which, when executed by the processor 601, causes the processor 601 to perform the following steps:

[0115] Acquire images of the door area of ​​the smart refrigerator;

[0116] Determine the first edge corresponding to the bottom region of the image;

[0117] Determine the second edge of the smart refrigerator door within the door region of the image;

[0118] Determine the first included angle between the first edge and the second edge, and set the first included angle as the angle at which the door of the smart refrigerator is currently open.

[0119] Furthermore, the processor 601 is also used to determine whether the bottom edge of the smart refrigerator is obstructed; if the bottom edge of the smart refrigerator is obstructed, the first edge corresponding to the opened drawer of the smart refrigerator is determined; if the bottom edge of the smart refrigerator is not obstructed, the first edge corresponding to the bottom edge of the smart refrigerator is determined.

[0120] Furthermore, the processor 601 is also configured to input the image into a pre-trained bottom edge occlusion determination model to obtain feature values ​​indicating whether the image contains bottom edge occlusion; and to determine whether the bottom edge is occluded based on the feature values.

[0121] Furthermore, the processor 601 is also used to determine whether the length of the bottom edge of the image is less than a preset length threshold.

[0122] Further, the processor 601 is also configured to determine at least one third edge corresponding to the opened drawer of the smart refrigerator in the image, and determine a first straight line corresponding to the at least one third edge; based on Hough line detection, obtain a first polar angle and a first polar radius corresponding to the first straight line; for each first straight line, determine whether the first polar angle of the first straight line is within a preset range of first polar angles, and determine whether the first polar radius of the first straight line is within a preset range of first polar radii; if so, then determine the third edge corresponding to the first straight line as a first edge; or, determine at least one fourth edge corresponding to the bottom edge of the smart refrigerator in the image, and determine a second straight line corresponding to the at least one fourth edge; based on Hough line detection, determine the second polar angle and the first polar radius corresponding to the second straight line. For each second straight line, determine whether the third polar angle of the second straight line is within a preset range of second polar angles, and determine whether the second polar diameter of the second straight line is within a preset range of second polar diameters. If so, then determine the fourth edge corresponding to the second straight line as the first edge, or determine at least one fifth edge corresponding to the door edge of the smart refrigerator in the image, and determine the third straight line corresponding to the at least one fifth edge; based on Hough line detection, determine the third polar angle and third polar diameter corresponding to the third straight line; for each third straight line, determine whether the third polar angle of the third straight line is within a preset range of third polar angles, and determine whether the third polar diameter of the third straight line is within a preset range of third polar diameters. If so, then determine the fifth edge corresponding to the third straight line as the second edge.

[0123] Furthermore, the processor 601 is also configured to determine whether the first polar angle is a preset angle if the bottom edge of the smart refrigerator is blocked, and if so, to perform an operation to determine whether the first polar angle of each first straight line is within the preset range of the first polar angle.

[0124] Furthermore, the processor 601 is also configured to determine whether the second polar angle is a preset angle if the bottom edge of the smart refrigerator is not obstructed, and if so, to perform an operation to determine whether the second polar angle of each second straight line is within the preset range of the second polar angle.

[0125] Furthermore, the processor 601 is also configured to, if it is determined that the first polar angle or the second polar angle is not a preset angle, determine a second difference between the target polar angle that is not a preset angle and the preset angle; and update the preset first polar radius range, the preset first polar angle range, the preset second polar radius range, the preset second polar angle range, the preset third polar radius range, and the preset third polar angle range according to the correspondence between the second difference and the pre-saved world coordinate system and pixel coordinate system.

[0126] The communication interface 602 is used for communication between the above-mentioned electronic device and other devices.

[0127] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0128] The processors mentioned above can be general-purpose processors, including central processing units, network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0129] Based on the above embodiments, this application provides a computer-readable storage medium storing a computer program executable by a processor. When the program is run on the processor, the processor executes the following steps:

[0130] Acquire an image containing the door area of ​​the smart refrigerator;

[0131] Determine the first edge corresponding to the bottom region of the image, and determine the second edge of the smart refrigerator door in the door region of the image;

[0132] Determine the first included angle between the first edge and the second edge, and set the first included angle as the angle at which the door of the smart refrigerator is currently open.

[0133] In one possible implementation, determining the first edge corresponding to the bottom region of the image includes:

[0134] Determine whether the bottom edge of the smart refrigerator is obstructed; if the bottom edge of the smart refrigerator is obstructed, determine the first edge corresponding to the opened drawer of the smart refrigerator; if the bottom edge of the smart refrigerator is not obstructed, determine the first edge corresponding to the bottom edge of the smart refrigerator.

[0135] In one possible implementation, determining whether the bottom edge of the smart refrigerator is obstructed includes:

[0136] The image is input into a pre-trained bottom edge occlusion determination model to obtain feature values ​​indicating whether the image contains bottom edge occlusion; based on the feature values, it is determined whether the bottom edge is occluded.

[0137] In one possible implementation, determining whether the bottom edge of the smart refrigerator is obstructed includes:

[0138] Determine whether the length of the bottom edge of the image is less than a preset length threshold.

[0139] In one possible implementation, determining the first edge corresponding to the bottom region of the image includes:

[0140] Identify at least one third edge corresponding to the opened drawer of the smart refrigerator in the image, and determine the first straight line corresponding to the at least one third edge; based on Hough line detection, obtain the first polar angle and the first polar radius corresponding to the first straight line; for each first straight line, determine whether the first polar angle of the first straight line is within a preset range of first polar angles, and determine whether the first polar radius of the first straight line is within a preset range of first polar radii; if so, then determine the third edge corresponding to the first straight line as the first edge; or

[0141] Determine at least one fourth edge corresponding to the bottom edge of the smart refrigerator in the image, and determine the second straight line corresponding to the at least one fourth edge; based on Hough line detection, determine the second polar angle and the second polar diameter corresponding to the second straight line; for each second straight line, determine whether the third polar angle of the second straight line is within a preset range of the second polar angle, and determine whether the second polar diameter of the second straight line is within a preset range of the second polar diameter. If so, then determine the fourth edge corresponding to the second straight line as the first edge.

[0142] In one possible implementation, if the bottom edge of the smart refrigerator is obstructed, after determining the first polar angle and first polar diameter corresponding to the first straight line, and before determining whether the first polar angle of each first straight line is within a preset range of first polar angles, the method further includes:

[0143] Determine whether the first polar angle is a preset angle. If so, perform the operation of determining whether the first polar angle of each first straight line is within the preset range of the first polar angle.

[0144] In one possible implementation, if the bottom edge of the smart refrigerator is not obstructed, after determining the second polar angle and second polar diameter corresponding to the second straight line, and before determining whether the second polar angle of each second straight line is within a preset range of second polar angles, the method further includes:

[0145] Determine whether the second polar angle is a preset angle. If so, perform the operation of determining whether the second polar angle of each second straight line is within the preset range of the second polar angle.

[0146] In one possible implementation, the method further includes:

[0147] If it is determined that the first polar angle or the second polar angle is not a preset angle, then the second difference between the target polar angle (which is not a preset angle) and the preset angle is determined; based on the correspondence between the second difference and the pre-saved world coordinate system and pixel coordinate system, the preset first polar radius range, the preset first polar angle range, the preset second polar radius range, the preset second polar angle range, the preset third polar radius range, and the preset third polar angle range are updated.

[0148] In one possible implementation, the second edge of the smart refrigerator door in the door area where the image is recognized includes:

[0149] Identify at least one fifth edge corresponding to the door edge of the smart refrigerator in the image, and determine the third straight line corresponding to the at least one fifth edge; based on Hough line detection, determine the third polar angle and the third polar diameter corresponding to the third straight line; for each third straight line, determine whether the third polar angle of the third straight line is within a preset range of third polar angles, and determine whether the third polar diameter of the third straight line is within a preset range of third polar diameters. If so, then the fifth edge corresponding to the third straight line is determined as the second edge.

[0150] In this embodiment of the application, the first edge corresponding to the bottom region in the image containing the smart refrigerator door area can be identified, as well as the second edge of the smart refrigerator door in the door area of ​​the image. The opening angle of the smart refrigerator door is determined based on the angle between the first edge and the second edge, thereby ensuring the accuracy of the determined opening angle of the door and improving the user experience.

[0151] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0152] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0153] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0154] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0155] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A smart refrigerator, characterized in that, The smart refrigerator includes: A data acquisition unit configured to acquire images of the door area of ​​the smart refrigerator; The controller is configured to: Determine the first edge corresponding to the bottom region of the image; determine the second edge of the smart refrigerator door in the door region of the image; determine the first included angle between the first edge and the second edge, and determine the first included angle as the opening angle of the smart refrigerator door at the current time; The controller is used to perform: Identify at least one fifth edge corresponding to the door edge of the smart refrigerator in the image, and determine the third straight line corresponding to the at least one fifth edge; based on Hough line detection, determine the third polar angle and the third polar diameter corresponding to the third straight line; for each third straight line, determine whether the third polar angle of the third straight line is within a preset range of third polar angles, and determine whether the third polar diameter of the third straight line is within a preset range of third polar diameters. If so, then the fifth edge corresponding to the third straight line is determined as the second edge.

2. The intelligent refrigerator according to claim 1, characterized in that, The controller is used to perform: Determine whether the bottom edge of the smart refrigerator is obstructed; if the bottom edge of the smart refrigerator is obstructed, determine the first edge corresponding to the opened drawer of the smart refrigerator. If the bottom edge of the smart refrigerator is not obstructed, then the first edge corresponding to the bottom edge of the smart refrigerator is determined.

3. The intelligent refrigerator according to claim 2, characterized in that, The controller is used to perform: The image is input into a pre-trained bottom edge occlusion determination model to obtain feature values ​​indicating whether the image contains bottom edge occlusion. Based on the feature value, determine whether the bottom edge is occluded.

4. The intelligent refrigerator according to claim 2, characterized in that, The controller is used to perform: Determine whether the length of the bottom edge of the image is less than a preset length threshold.

5. The intelligent refrigerator according to claim 2, characterized in that, The controller is used to perform: Identify at least one third edge corresponding to the opened drawer of the smart refrigerator in the image, and determine the first straight line corresponding to the at least one third edge; based on Hough line detection, obtain the first polar angle and the first polar radius corresponding to the first straight line; for each first straight line, determine whether the first polar angle of the first straight line is within a preset range of first polar angles, and determine whether the first polar radius of the first straight line is within a preset range of first polar radii; if so, then determine the third edge corresponding to the first straight line as the first edge; or Determine at least one fourth edge corresponding to the bottom edge of the smart refrigerator in the image, and determine the second straight line corresponding to the at least one fourth edge; based on Hough line detection, determine the second polar angle and the second polar diameter corresponding to the second straight line; for each second straight line, determine whether the third polar angle of the second straight line is within a preset range of the second polar angle, and determine whether the second polar diameter of the second straight line is within a preset range of the second polar diameter. If so, then determine the fourth edge corresponding to the second straight line as the first edge.

6. The intelligent refrigerator according to claim 5, characterized in that, The controller is used to perform: If the bottom edge of the smart refrigerator is blocked, it is determined whether the first polar angle is a preset angle. If so, the operation is performed for each first straight line to determine whether the first polar angle of the first straight line is within the preset range of the first polar angle. If the bottom edge of the smart refrigerator is not obstructed, it is determined whether the second polar angle is a preset angle. If so, the operation is performed for each second straight line to determine whether the second polar angle of the second straight line is within the preset range of the second polar angle.

7. The intelligent refrigerator according to claim 6, characterized in that, The controller is used to perform: If it is determined that the first polar angle or the second polar angle is not a preset angle, then the second difference between the target polar angle (which is not a preset angle) and the preset angle is determined; based on the correspondence between the second difference and the pre-saved world coordinate system and pixel coordinate system, the preset first polar radius range, the preset first polar angle range, the preset second polar radius range, the preset second polar angle range, the preset third polar radius range, and the preset third polar angle range are updated.

8. A method for determining the angle of a door, characterized in that, The method includes: Acquire an image containing the door area of ​​the smart refrigerator; Determine the first edge corresponding to the bottom region of the image, and determine the second edge of the smart refrigerator door in the door region of the image; Determine the first included angle between the first edge and the second edge, and determine the first included angle as the angle of the door of the smart refrigerator at the current time; The second edge of the smart refrigerator door in the door area of ​​the image includes: Identify at least one fifth edge corresponding to the door edge of the smart refrigerator in the image, and determine the third straight line corresponding to the at least one fifth edge; based on Hough line detection, determine the third polar angle and the third polar diameter corresponding to the third straight line; for each third straight line, determine whether the third polar angle of the third straight line is within a preset range of third polar angles, and determine whether the third polar diameter of the third straight line is within a preset range of third polar diameters. If so, then the fifth edge corresponding to the third straight line is determined as the second edge.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory being used to store program instructions, and the processor being used to execute the computer program stored in the memory to implement the steps of the door angle determination method of claim 8.

10. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the steps of the door angle determination method described in claim 8.

Images