Image recognition method, image recognition device and refrigerator

Abstract

The present application relates to the technical field of image processing, and discloses an image recognition method. The method is applied to a refrigerator and includes: when it is detected that the door of the refrigerator is opened, photographing the compartments of the refrigerator to obtain a first video image; when it is detected that there is a heat source in the compartment, photographing the movement of the heat source to obtain a second video image; selecting a video image corresponding to the second video image from the first video image as a third video image; performing image recognition on the third video image, and obtaining relevant information of the ingredients to be put in according to the image recognition result. This method can perform image recognition only on the video image corresponding to the second video image in the first video image. Therefore, the amount of computation is reduced. The present application also discloses an image recognition device and a refrigerator.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and for example, relates to an image recognition method, an image recognition device, and a refrigerator. Background Art

[0002] With the development of computer technology and the application of computer vision principles, accurately recognizing the food ingredients in products such as refrigerators using image recognition technology has wide application value. Currently, the recognition of the food ingredients in the refrigerator is mainly based on the video images captured by a camera installed on the refrigerator door body. For example, the camera starts shooting after the refrigerator door body is opened at a certain angle, and stops shooting after the refrigerator door body is closed.

[0003] In the process of implementing the embodiments of the present disclosure, it is found that there are at least the following problems in the related technologies: Since the camera captures video images and can only obtain the start time and end time of the video images, when performing image recognition on the food ingredients to be placed, it is necessary to perform image recognition on each frame of the captured video images, resulting in a large amount of computation. Summary of the Invention

[0004] To provide a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. The summary is not a general review, nor is it intended to identify key / important elements or delineate the scope of protection of these embodiments. Instead, it serves as a preamble to the following detailed description.

[0005] The embodiments of the present disclosure provide an image recognition method, an image recognition device, and a refrigerator to solve the technical problem that when performing image recognition on the food ingredients to be placed, it is necessary to perform image recognition on each frame of the captured video images, resulting in a large amount of computation.

[0006] In some embodiments, the image recognition method, applied to a refrigerator, includes: when detecting that the door body of the refrigerator is opened, photographing the compartments of the refrigerator to obtain a first video image; when detecting a heat source in the compartment, photographing the movement of the heat source to obtain a second video image; selecting a video image corresponding to the second video image from the first video image as a third video image; performing image recognition on the third video image, and obtaining relevant information of the food ingredients to be placed according to the image recognition result.

[0007] In some embodiments, an image recognition device is applied to a refrigerator and includes: a first detection module configured to capture a first video image of a compartment of the refrigerator when it detects that the door of the refrigerator is opened; a second detection module configured to capture the movement of a heat source when it detects the heat source in the compartment to obtain a second video image; a selection module configured to select a video image corresponding to the second video image from the first video image as a third video image; and an identification module configured to perform image recognition on the third video image and obtain relevant information of the food ingredients to be placed according to the image recognition result.

[0008] In some embodiments, the image recognition device includes a processor and a memory storing program instructions, wherein the processor is configured to execute the image recognition method as described above when executing the program instructions.

[0009] In some embodiments, the refrigerator includes the image recognition device as described above.

[0010] The image recognition method, image recognition device, and refrigerator provided by the embodiments of the present disclosure can achieve the following technical effects:

[0011] By capturing a first video image of the compartment of the refrigerator when it detects that the door of the refrigerator is opened, capturing the movement of the heat source when it detects the heat source in the compartment to obtain a second video image, selecting a video image corresponding to the second video image from the first video image as a third video image, performing image recognition on the third video image, and obtaining relevant information of the food ingredients to be placed according to the image recognition result, it is possible to perform image recognition only on the video image corresponding to the second video image in the first video image. Therefore, the amount of computation is reduced.

[0012] The above general description and the following description are only exemplary and explanatory and are not used to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] One or more embodiments are exemplarily illustrated by corresponding drawings. These exemplary illustrations and the drawings do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements. The drawings do not constitute a scale limitation, and wherein:

[0014] Figure 1 is a schematic diagram of a scene of a compartment of a refrigerator captured by a camera in the related art;

[0015] Figure 2 is a schematic diagram of an implementation environment of the embodiments of the present disclosure;

[0016] Figure 3 is a schematic flowchart of the image recognition method provided by the embodiments of the present disclosure;

[0017] Figure 4 It is a schematic diagram of the hand trajectory of a user in an embodiment of the present disclosure;

[0018] Figure 5 It is a schematic flowchart of an image recognition method provided by an embodiment of the present disclosure;

[0019] Figure 6 It is a schematic structural diagram of an image recognition device provided by an embodiment of the present disclosure;

[0020] Figure 7 It is a schematic structural diagram of an image recognition device provided by an embodiment of the present disclosure. Detailed implementation manners

[0021] In order to be able to understand the features and technical content of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. The attached drawings are only for reference and explanation purposes, and are not used to limit the embodiments of the present disclosure. In the following technical description, for the sake of explanation, multiple details are provided to provide a full understanding of the disclosed embodiments. However, one or more embodiments can still be implemented without these details. In other cases, well-known structures and devices can be shown in a simplified manner to simplify the drawings.

[0022] Figure 1 It is a schematic diagram of the scene of a compartment of a refrigerator taken by a camera in the related art. As Figure 1 shown, a refrigerator generally includes a box body, a compartment housed in the box body, a door body provided outside the compartment, and a drawer and a shelf housed in the compartment for placing food ingredients. In addition, the refrigerator further includes an angle sensor and a camera. Among them, the angle sensor can be provided at the hinge of the door body to detect the opening and closing states of the door body so as to activate the camera; the camera can be provided at the top of the compartment to take pictures of the inside of the compartment.

[0023] When the angle sensor detects that the door body of the refrigerator is opened by a certain angle, the camera is activated to continuously take pictures of the inside of the compartment, and four video images as Figure 1 shown are obtained. It can be seen from the above video images that they record the whole process of the user from opening the refrigerator to putting the lemon into the refrigerator. By performing image recognition on the rectangular area in the above video images, it can be determined that the food ingredient to be put into the refrigerator is a lemon.

[0024] However, there are at least the following problems in the related art: Since it is impossible to determine when the ingredients to be placed enter the rectangular area in the above video image, it is necessary to identify each frame of the image captured after the camera is started. Here, taking a high-speed camera as an example, assuming its acquisition rate is 30 frames per second and the shooting time from start to stop is 2 minutes, for the captured video image, the computational amount of image recognition is 2×60×30 = 3,600 frames. That is to say, 3,600 times of image recognition are required for a 2-minute video image, resulting in a large computational amount.

[0025] Figure 2 is an environmental schematic diagram of the implementation environment of the embodiments of the present disclosure. As Figure 2 shown, the implementation environment includes a storage device 210, a terminal device 220, a server 230, and a network device 240.

[0026] The storage device 210 can be a device for refrigerating, preserving freshness, and / or freezing ingredients, such as a refrigerator, a freezer, etc.; the terminal device 220 can be called a user device, a terminal device, a mobile station, or a mobile terminal, etc. For example, the terminal device 220 can be a mobile phone or a computer with a mobile terminal, or can also be a portable, handheld, computer-integrated, or vehicle-mounted mobile device, or can also be a display device with a display function integrated on the storage device 210; the server 230 can be a single server, or can also be a server cluster composed of several servers, or can also be a cloud computing service center, and the embodiments of the present disclosure do not limit this.

[0027] The storage device 210, the terminal device 220, and the server 230 communicate through the network device 240. The network device 240 can be a wired network device, such as a router, a switch, etc.; or can also be a wireless network device, such as a wireless router, a wireless access point (Access Point, AP) device, etc., and the embodiments of the present disclosure do not limit this. The server 230 stores relevant information of the ingredients, is used to receive the recognition request sent by the storage device 210, and send the recognition result to the terminal device 220.

[0028] It should be noted that the embodiments of the present disclosure can also be implemented in an offline implementation environment.

[0029] Figure 3 is a flowchart of the image recognition method provided by the embodiments of the present disclosure, and Figure 4 is a schematic diagram of the hand trajectory of the user in the embodiments of the present disclosure. Figure 3 The image recognition method can be executed by Figure 2 any one of the storage device 210, the terminal device 220, and the server 230, as Figure 3 shown, the image recognition method, applied to a refrigerator, includes:

[0030] S310, when it is detected that the door of the refrigerator is opened, take pictures of the compartments of the refrigerator to obtain a first video image;

[0031] S320, when a heat source in the compartment is detected, take pictures of the movement of the heat source to obtain a second video image;

[0032] S330, select a video image corresponding to the second video image from the first video image as a third video image;

[0033] S340, perform image recognition on the third video image, and obtain relevant information of the ingredients to be put in according to the image recognition result.

[0034] Specifically, when the angle sensor detects that the door of the refrigerator is opened by a certain angle, start the RGB camera and the infrared thermal imaging camera arranged at the top of the compartment of the refrigerator. Among them, the RGB camera is used to continuously take pictures of the interior of the compartment of the refrigerator to obtain a first video image; the infrared thermal imaging camera is used to detect the heat source in the compartment, and when the heat source in the compartment is detected, take pictures of the movement of the heat source to obtain a second video image.

[0035] Here, the RGB camera is a three-color camera of red (R), green (G), and blue (B) (i.e., RGB camera), which is the most commonly used camera in related technologies. The image taken by it is an RGB image. That is to say, the first video image is an RGB image; the infrared thermal imaging camera uses infrared thermal imaging technology, that is, uses an infrared detector and an optical imaging objective lens to receive the infrared radiation energy distribution pattern of the measured target and reflect it onto the photosensitive element of the infrared detector, so as to obtain an infrared thermal imaging image. This thermal imaging image corresponds to the thermal distribution field on the surface of the object. The heat source usually refers to the hand of the user holding the ingredients to be put in, and its surface temperature is significantly higher than the ambient temperature or the temperature of other objects in the captured picture.

[0036] It should be noted that the RGB camera and the infrared thermal imaging camera are not limited to being arranged at the top of the compartment as described above. For example, they can also be arranged at the upper left corner or upper right corner or lower left corner or lower right corner of the compartment, or they can be respectively arranged at the upper left corner and upper right corner or lower left corner and lower right corner of the compartment. The embodiments of the present disclosure do not limit this. In addition, the number of RGB cameras and infrared thermal imaging cameras can be one, two, or three or more. The embodiments of the present disclosure do not limit this. It should be understood that multiple RGB cameras and multiple infrared thermal imaging cameras are beneficial to simultaneously obtaining multiple images of different angles in the compartment, so as to more accurately determine the relevant information of the ingredients in the compartment.

[0037] In addition, the acquisition rates of the RGB camera and the infrared thermal imaging camera can be the same or different. However, for the convenience of synchronization between the RGB camera and the infrared thermal imaging camera, RGB cameras and infrared thermal imaging cameras with the same acquisition rate are usually adopted. For example, assuming that the acquisition rates of both the RGB camera and the infrared thermal imaging camera are 30 frames per second, then the second frame of the infrared thermal imaging camera corresponds to the second frame of the RGB camera; assuming that the acquisition rate of the RGB camera is 60 frames per second and the acquisition rate of the infrared thermal imaging camera is 30 frames per second, then the third frame of the infrared thermal imaging camera may correspond to the fifth frame of the RGB camera; assuming that the acquisition rate of the RGB camera is 30 frames per second and the acquisition rate of the infrared thermal imaging camera is 25 frames per second, then the 13th frame of the infrared thermal imaging camera may correspond to the 15th frame or the 16th frame of the RGB camera.

[0038] Next, an interception operation is performed on the obtained first video image to obtain multiple video images, and the video image corresponding to the second video image is selected from the multiple video images as the third video image. Further, an image recognition algorithm is used to perform image recognition on the third video image, and relevant information about the ingredients to be placed is obtained according to the image recognition result.

[0039] Here, image recognition refers to the technology of using a computer to process, analyze, and understand an image in order to identify various different patterns of targets and objects; the image recognition algorithm can include, for example, the Fast Region-based Convolutional Neural Network (Fast R-CNN), the Residual Network (ResNet), the Scale-Invariant Feature Transform (SIFT), etc., and the embodiments of the present disclosure are not limited thereto. In addition, an image processing module (not shown) integrated in the refrigerator can be used to perform image recognition on the third video image, or the third video image can be transmitted to a server for image recognition, and the embodiments of the present disclosure are not limited thereto.

[0040] The relevant information about the ingredients to be placed can include, but is not limited to, the name, type, quantity, etc. of the ingredients to be placed. In addition, the relevant information about the ingredients to be placed can be directly displayed on the display module (for example, a display screen) of the refrigerator, or can be sent to a terminal device (for example, a mobile phone, a tablet computer, etc.) through the wireless transmission function built in the refrigerator itself, so as to facilitate the user to view at any time.

[0041] According to the technical solution provided by the embodiments of the present disclosure, by taking a picture of the compartment of the refrigerator when the door of the refrigerator is detected to be opened to obtain a first video image, taking a picture of the movement of the heat source when the heat source in the compartment is detected to obtain a second video image, selecting the video image corresponding to the second video image from the first video image as the third video image, and performing image recognition on the third video image and obtaining relevant information of the ingredients to be placed according to the image recognition result, it is possible to perform image recognition only on the video image corresponding to the second video image in the first video image. Therefore, the amount of computation is reduced.

[0042] In addition, by continuously taking video of the compartments of the refrigerator using an RGB camera and an infrared thermal imaging camera respectively, the entire process of placing the ingredients can be recorded. Therefore, the integrity of the recording is ensured.

[0043] In some embodiments, selecting the video image corresponding to the second video image from the first video image as the third video image includes: performing binarization processing on the second video image to obtain multiple binarized images; obtaining the position coordinates of the highest point of the heat source in each binarized image of the multiple binarized images, and determining the optimal binarized image according to the position coordinates of the highest point; selecting the video image corresponding to the optimal binarized image from the first video image as the third video image.

[0044] Specifically, after obtaining the second video image, perform binarization processing on the second video image to obtain multiple binarized images. Here, image binarization is the process of setting the grayscale value of the pixel points on the image to 0 or 255, that is, presenting the entire image with an obvious black and white effect. The binarization of the image is beneficial to the further processing of the image, making the image simpler, and the data volume is reduced, which can highlight the outline of the target of interest.

[0045] Then, obtain the position coordinates of the highest point of the heat source in each binarized image of the multiple binarized images, and determine the optimal binarized image according to the position coordinates of the highest point; further, select the video image corresponding to the optimal binarized image from the first video image as the third video image. Here, the position coordinates of the highest point of the heat source refer to the coordinates of the position where the highest point of the heat source in each binarized image is located, represented by "(x, y)".

[0046] According to the technical solution provided by the embodiments of the present disclosure, by obtaining the position coordinates of the highest point of the heat source, it is possible to accurately determine the distance from the highest point of the heat source to the center point of the binarized image, thereby determining the optimal binarized image. Therefore, the accuracy of selecting the third video image is improved.

[0047] In some embodiments, the position coordinates of the highest point of the heat source in each of a plurality of binary images are obtained, and an optimal binary image is determined based on the position coordinates of the highest point, including: obtaining the position coordinates of the highest point of the heat source in each of a plurality of binary images to obtain an array consisting of the position coordinates of the plurality of highest points; calculating the minimum value of the array, and taking the binary image corresponding to the minimum value as the optimal binary image.

[0048] Specifically, the position coordinates of the highest point of the heat source in each of the multiple binary images are obtained to obtain an array consisting of the position coordinates of the multiple highest points; the minimum value of the array is calculated, and the binary image corresponding to the minimum value is used as the optimal binary image. Here, an array is a collection used to store multiple data of the same type, for example, it can be expressed as "(x1, y1), (x2, y2), ..., (x i ,y i )", where i is an integer.

[0049] Taking the user's hand trajectory as an example, assuming that the second video image captured by the infrared thermal imaging camera is binarized, the following is obtained: Figure 4 Six binary images are shown. It can be seen from the above binary images that they record the whole process of the user putting the food to be put into the refrigerator, and the position coordinates of the highest point of the hand in each of the six binary images are obtained to obtain the position coordinates of the six highest points (x1, y1), (x2, y2), (x3, y3), (x4, y4), (x5, y5), (x6, y6). The array composed of these six position coordinates can be used to describe the user's hand trajectory; further, the distance from the highest point of the hand to the center point of the binary image is calculated according to the above position coordinates, and the binary image corresponding to the minimum value in the calculation result is used as the optimal binary image, that is, the position of the food to be put into the optimal binary image is the easiest to perform image recognition, and there is only one optimal binary image.

[0050] According to the technical solution provided by the embodiment of the present disclosure, by obtaining the position coordinates of the highest point of the heat source and calculating the distance between the highest point of the heat source and the center point of the binary image, the optimal binary image can be accurately determined, thereby improving the accuracy of selecting the optimal binary image.

[0051] In some embodiments, the minimum value of an array is calculated by the following formula: Among them, x i Indicates the horizontal coordinate of the i-th highest point, y i represents the ordinate of the i-th highest point, w represents the width of the binary image, and h represents the height of the binary image. Indicates The value of variable i when the minimum value is obtained, where i is an integer, and w and h are positive integers.

[0052] Specifically, argminf(x i ,y i ) refers to all independent variables x i ,y i ) that make the function f(x i and y i obtain the minimum value. Here, where x i represents the abscissa of the i-th highest point, y i represents the ordinate of the i-th highest point, w represents the width of the binary image, h represents the height of the binary image, where i is an integer, and w and h are positive integers. Through the formula the binary image where the position coordinates of the highest point with the shortest distance from the center point of the binary image can be calculated, that is, through the above formula, a unique binary image can be calculated.

[0053] According to the technical solution provided by the embodiments of the present disclosure, by performing image recognition on a binary image, the amount of computation for image recognition can be reduced, and thus, the image recognition efficiency is improved.

[0054] In some embodiments, selecting a video image corresponding to the optimal binary image from the first video image as the third video image includes: obtaining the timestamp corresponding to the optimal binary image; selecting the video image corresponding to the timestamp from the first video image as the third video image.

[0055] Specifically, since the RGB camera captures video images, the material for image recognition of the ingredients to be placed is intercepted from the video images, that is, each frame of the video images is intercepted in chronological order to obtain multiple video images; each video image in the multiple video images carries a timestamp, and this timestamp can be used to distinguish different video images, that is, each video image contains a different timestamp. Further, after the optimal binary image is selected, the video image corresponding to the timestamp of the optimal binary image can be selected from the first video image captured by the RGB camera as the third video image.

[0056] According to the technical solution provided by the embodiments of the present disclosure, by selecting the video image corresponding to the timestamp based on the timestamp, the third video image to be recognized can be quickly found, and thus, the speed of image selection is improved.

[0057] In some embodiments, image recognition is performed on the third video image, and relevant information of the ingredients to be placed is obtained according to the image recognition result, including: performing image recognition on the third video image through machine learning, and obtaining relevant information of the ingredients to be placed according to the image recognition result.

[0058] Specifically, image recognition can be performed on the third video image through machine learning, and relevant information of the ingredients to be placed is obtained according to the image recognition result. Here, generally speaking, machine learning is a method that can endow a machine with the ability to complete functions that cannot be completed by direct programming; but in a practical sense, machine learning is a method of training a model by using data and then using the model for prediction. The algorithms of machine learning can include but are not limited to Decision Trees algorithm, Naive Bayes Classification algorithm, Ordinary Least Squares Regression algorithm, Logistic Regression algorithm, Support Vector Machines algorithm, Randomforest algorithm, Artificial Neural Network (ANN) algorithm, Principal Component Analysis algorithm, etc.

[0059] Further, if the ingredients to be placed in the third video image can be recognized, the relevant information of the ingredients to be placed can be directly displayed on the display module (e.g., display screen) of the refrigerator, or the relevant information of the ingredients to be placed can be sent to a terminal device (e.g., mobile phone, tablet computer, etc.) through the wireless transmission function built in the refrigerator itself, so as to facilitate the user to view at any time; if the ingredients to be placed in the third video image cannot be recognized, a prompt message such as "ingredients cannot be recognized" can be directly displayed on the display screen, a prompt sound such as a beep can also be emitted, or a prompt message such as "ingredient recognition failed" can also be sent to the terminal device through the wireless transmission function built in the refrigerator itself to remind the user. The embodiments of the present disclosure are not limited thereto.

[0060] According to the technical solution provided by the embodiments of the present disclosure, by using a machine learning algorithm to perform image recognition on the third video image, the relevant information of the ingredients to be placed can be determined quickly and accurately. Therefore, the accuracy rate of the image recognition result is improved, and the false alarm rate is reduced.

[0061] All the above optional technical solutions can be combined arbitrarily to form optional embodiments of the present application, which will not be elaborated herein one by one.

[0062] Figure 5 It is a schematic flowchart of the image recognition method provided by an embodiment of the present disclosure. Figure 5 The image recognition method can be executed by any one of Figure 2 the storage device 210, the terminal device 220, and the server 230 shown in Figure 5 As shown, when applied to a refrigerator, the image recognition method includes:

[0063] S510, when detecting that the door of the refrigerator is opened, photograph the compartments of the refrigerator to obtain a first video image;

[0064] S520, when detecting a heat source in the compartment, photograph the movement of the heat source to obtain a second video image;

[0065] S530, perform binarization processing on the second video image to obtain multiple binarized images;

[0066] S540, obtain the position coordinates of the highest point of the heat source in each binarized image among the multiple binarized images, and obtain an array composed of multiple position coordinates;

[0067] S550, calculate the minimum value of the array, and use the binarized image corresponding to the minimum value as the optimal binarized image;

[0068] S560, select the video image corresponding to the optimal binarized image from the first video image as the third video image;

[0069] S570, perform image recognition on the third video image through machine learning, and obtain relevant information of the ingredients to be placed according to the image recognition result.

[0070] According to the technical solution provided by the embodiment of the present disclosure, by photographing the compartments of the refrigerator when detecting that the door of the refrigerator is opened to obtain a first video image, photographing the movement of the heat source when detecting a heat source in the compartment to obtain a second video image, performing binarization processing on the second video image to obtain multiple binarized images, obtaining the position coordinates of the highest point of the heat source in each binarized image among the multiple binarized images to obtain an array composed of multiple position coordinates, calculating the minimum value of the array, using the binarized image corresponding to the minimum value as the optimal binarized image, selecting the video image corresponding to the optimal binarized image from the first video image as the third video image, performing image recognition on the third video image through machine learning, and obtaining relevant information of the ingredients to be placed according to the image recognition result, it is possible to perform image recognition only on the video image corresponding to the second video image in the first video image. Therefore, the amount of computation is reduced.

[0071] The following is an embodiment of the device of the present application, which can be used to execute the method embodiment of the present application. For details not disclosed in the device embodiment of the present application, please refer to the method embodiment of the present application.

[0072] Figure 6 It is a schematic structural diagram of an image recognition device provided by an embodiment of the present disclosure. As Figure 6 shown, the image recognition device is applied to a refrigerator and includes:

[0073] A first detection module 610, configured to take a picture of the compartments of the refrigerator when it detects that the door of the refrigerator is opened, so as to obtain a first video image;

[0074] A second detection module 620, configured to take a picture of the movement of the heat source when it detects the heat source in the compartment, so as to obtain a second video image;

[0075] A selection module 630, configured to select the video image corresponding to the second video image from the first video image as the third video image;

[0076] An identification module 640, configured to perform image recognition on the third video image and obtain relevant information of the ingredients to be put in according to the image recognition result.

[0077] According to the technical solution provided by the embodiment of the present disclosure, by taking a picture of the compartments of the refrigerator when it detects that the door of the refrigerator is opened to obtain a first video image, taking a picture of the movement of the heat source when it detects the heat source in the compartment to obtain a second video image, selecting the video image corresponding to the second video image from the first video image as the third video image, and performing image recognition on the third video image and obtaining relevant information of the ingredients to be put in according to the image recognition result, it is possible to perform image recognition only on the video image corresponding to the second video image in the first video image. Therefore, the amount of computation is reduced.

[0078] In some embodiments, Figure 6 the selection module 630 of performs binarization processing on the second video image to obtain multiple binarized images, obtains the position coordinates of the highest point of the heat source in each binarized image among the multiple binarized images, determines the optimal binarized image according to the position coordinates, and selects the video image corresponding to the optimal binarized image from the first video image as the third video image.

[0079] In some embodiments, Figure 6 the selection module 630 of obtains the position coordinates of the highest point of the heat source in each binarized image among the multiple binarized images, obtains an array composed of multiple position coordinates, calculates the minimum value of the array, and uses the binarized image corresponding to the minimum value as the optimal binarized image.

[0080] In some embodiments, the minimum value of the array is calculated by the following formula: where x i represents the abscissa of the i-th highest point, y i represents the ordinate of the i-th highest point, w represents the width of the binary image, h represents the height of the binary image, represents the value of variable i when obtains the minimum value, where i is an integer, and w and h are positive integers.

[0081] In some embodiments, Figure 6 the selection module 630 of obtains the timestamp corresponding to the optimal binary image, and selects the video image corresponding to the timestamp from the first video images as the third video image.

[0082] In some embodiments, the first video images are obtained by a RGB camera, and the second video images are obtained by an infrared thermal imaging camera, wherein the RGB camera and the infrared thermal imaging camera have the same acquisition rate.

[0083] In some embodiments, Figure 6 the recognition module 640 of performs image recognition on the third video image through machine learning, and obtains the relevant information of the ingredients to be placed according to the image recognition result.

[0084] In some embodiments, if the ingredients to be placed in the third video image are recognized, then Figure 6 the recognition module 640 of sends the relevant information of the ingredients to be placed; if the ingredients to be placed in the third video image cannot be recognized, then Figure 6 the recognition module 640 of issues a prompt message.

[0085] Figure 7 is a schematic structural diagram of an image recognition device provided by an embodiment of the present disclosure. As Figure 7 shown, the image recognition device includes: a processor 700 and a memory 701, and may further include a communication interface 702 and a bus 703. Among them, the processor 700, the communication interface 702, and the memory 701 can complete mutual communication through the bus 703. The communication interface 702 can be used for information transmission. The processor 700 can call the logical instructions in the memory 701 to execute the image recognition method of the above embodiments.

[0086] In addition, when the logical instructions in the above-mentioned memory 701 are implemented in the form of a software functional unit and sold or used as an independent product, they can be stored in a computer-readable storage medium.

[0087] The memory 701, being a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as the program instructions / modules corresponding to the methods in the embodiments of the present disclosure. The processor 700 executes functional applications and data processing by running the program instructions / modules stored in the memory 701, that is, implements the image recognition method in the above method embodiments.

[0088] The memory 701 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the terminal device, etc. In addition, the memory 701 may include high-speed random access memory and may also include non-volatile memory.

[0089] The embodiments of the present disclosure provide a refrigerator including the above image recognition device.

[0090] The embodiments of the present disclosure provide a computer-readable storage medium storing computer-executable instructions, and the computer-executable instructions are configured to execute the above image recognition method.

[0091] The embodiments of the present disclosure provide a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions that, when executed by a computer, cause the computer to execute the above image recognition method.

[0092] The above computer-readable storage medium can be a transient computer-readable storage medium or a non-transient computer-readable storage medium.

[0093] The technical solutions of the embodiments of the present disclosure can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes one or more instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present disclosure. The foregoing storage medium can be a non-transient storage medium, including: various media that can store program codes such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs, or can also be a transient storage medium.

[0094] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure, enabling those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, process, and other changes. The embodiments merely represent possible variations. Unless explicitly required, individual components and functions are optional, and the order of operations may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. The scope of the embodiments of the present disclosure includes the entire scope of the claims and all available equivalents of the claims. When used in this application, although terms such as "first", "second", etc. may be used in this application to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without changing the meaning of the description, the first element may be called the second element, and similarly, the second element may be called the first element, as long as all occurrences of the "first element" are consistently renamed and all occurrences of the "second element" are consistently renamed. The first element and the second element are both elements, but they may not be the same element. Moreover, the terms used in this application are only used to describe the embodiments and do not limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to also include the plural forms. Similarly, as used in this application, the term "and / or" refers to any and all possible combinations including one or more of the associated listed items. Additionally, when used in this application, the term "comprise" and its variants "comprises" and / or "comprising" etc. mean the presence of the stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or groups of these. Without further limitation, an element defined by the statement "comprising one..." does not exclude the presence of additional identical elements in the process, method, or device comprising the element. Herein, each embodiment may focus on the differences from other embodiments, and the same or similar parts among the embodiments may be referred to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, the relevant parts may refer to the description of the method part.

[0095] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner can depend on the specific application and design constraints of the technical solution. The skilled person can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the embodiments of the present disclosure. The skilled person can clearly understand that for the convenience and conciseness of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0096] In the embodiments disclosed herein, the disclosed methods, products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units can be merely a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Additionally, the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of devices or units can be electrical, mechanical, or other forms. The units described as separate components can be or can not be physically separated. The components displayed as units can be or can not be physical units, that is, they can be located in one place or can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to implement this embodiment. Additionally, in the embodiments of the present disclosure, the functional units can be integrated in one processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0097] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of code, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions noted in the blocks may occur in an order different from that noted in the accompanying drawings. For example, two consecutive blocks may in fact be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in an order different from that disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may in fact be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. Each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. An image recognition method, applied to a refrigerator, characterized in that, including: When it is detected that the door of the refrigerator is opened, photograph the compartments of the refrigerator to obtain a first video image; When it is detected that there is a heat source in the compartment, photograph the movement of the heat source to obtain a second video image; Perform binarization processing on the second video image to obtain multiple binarized images; Obtain the position coordinates of the highest point of the heat source in each of the multiple binarized images, and determine the optimal binarized image according to the position coordinates; Obtain the timestamp corresponding to the optimal binarized image; Select the video image corresponding to the timestamp from the first video image as the third video image; Perform image recognition on the third video image, and obtain relevant information of the ingredients to be placed according to the image recognition result.

2. The method according to claim 1, characterized in that, The obtaining the position coordinates of the highest point of the heat source in each of the multiple binarized images, and determining the optimal binarized image according to the position coordinates includes: Obtain the position coordinates of the highest point of the heat source in each of the multiple binarized images, and obtain an array composed of multiple such position coordinates; Calculate the minimum value of the array, and use the binarized image corresponding to the minimum value as the optimal binarized image; wherein, calculating the minimum value of the array refers to calculating the minimum value of the distances from the position coordinates of the highest points of multiple heat sources to the center point of the binarized image.

3. The method according to claim 2, characterized in that, Calculate the minimum value of the array through the following formula: Among them, x i represents the abscissa of the i-th highest point, y i represents the ordinate of the i-th highest point, w represents the width of the binarized image, and h represents the height of the binarized image, represents making the value of variable i when taking the minimum value, where i is an integer, and w and h are positive integers.

4. The method according to any one of claims 1 to 3, characterized in that The first video image is obtained through a RGB camera, and the second video image is obtained through an infrared thermal imaging camera, wherein the acquisition rates of the RGB camera and the infrared thermal imaging camera are the same.

5. The method according to any one of claims 1 to 4, characterized in that, The performing image recognition on the third video image, and obtaining relevant information of the ingredients to be placed according to the image recognition result includes: Perform image recognition on the third video image through machine learning, and obtain relevant information of the ingredients to be placed according to the image recognition result.

6. An image recognition device, applied to a refrigerator, characterized in that including: A first detection module configured to, when it is detected that the door of the refrigerator is opened, photograph the compartments of the refrigerator to obtain a first video image; A second detection module configured to, when it is detected that there is a heat source in the compartment, photograph the movement of the heat source to obtain a second video image; A selection module configured to perform binarization processing on the second video image to obtain multiple binarized images; Obtain the position coordinates of the highest point of the heat source in each of the multiple binarized images, and determine the optimal binarized image according to the position coordinates; obtain the timestamp corresponding to the optimal binarized image; Select the video image corresponding to the timestamp from the first video image as the third video image; An identification module configured to perform image recognition on the third video image, and obtain relevant information of the ingredients to be placed according to the image recognition result.

7. An image recognition device, comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to execute the method according to any one of claims 1 to 5 when executing the program instructions.

8. A refrigerator, characterized in that, including the device according to claim 6 or 7.

Images