[0043]In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.
[0044] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary, only used to explain the present invention, and should not be construed as a limitation of the present invention.
[0045] For the convenience of description, the term used to describe the relative position in space, such as "upper", "lower", "rear", "front", etc., is used to describe one unit or feature shown in the drawings relative to another A unit or feature relationship. The term spatially relative position may include different orientations of the device in use or operation other than the orientation shown in the figures. For example, if the device in the figures is turned over, elements described as "below" or "above" other elements or features would then be oriented "below" or "above" the other elements or features. Thus, the exemplary term "below" can encompass both a spatial orientation of below and above.
[0046] like figure 1 As shown, the present invention provides a method for judging the target identity of items in a refrigerator, which is used to detect and track the information of items in the refrigerator, including the steps:
[0047] S11: Acquire a plurality of pictures of the storage space in the refrigerator continuously shot at a fixed position, which are the first image to the nth image, wherein n is an integer greater than 1.
[0048] Shooting at the same point as mentioned here means that the camera set at a fixed point keeps relatively stable between it and the viewing area to obtain multiple images continuously, or continuously captures multiple frames of images from the captured images. In this way, the position of the object in each image is kept the same to the maximum extent, which is convenient for subsequent detection and identification.
[0049] Exemplarily, in this embodiment, the camera is provided at the bottom of the bottle holder of the refrigerator, and is used to capture the information of the food in the bottle holder below the camera. In other embodiments of the present invention, the camera can also be installed in other storage spaces in the refrigerator, such as the refrigerator compartment or the freezer compartment, and the location of the camera can be adjusted according to the structure of the refrigerator compartment or the freezer compartment.
[0050] After receiving the instruction to start shooting, the camera starts to work. Usually, this instruction can be issued by opening the refrigerator door. At this time, since the refrigerator door has just been opened, the objects in the shooting area remain untouched by the user. state, the recognition degree is high, and this is used as the judgment reference image, and the recognition accuracy is high.
[0051] S12: Detect the item information in the first image, and generate target detection frames at each item, which are target detection frames B respectively 1 to the target detection frame B n , and assign ID information to the items in each target detection frame, corresponding to target 1 to target n respectively.
[0052] Specifically, based on the detection algorithm model, the ingredients in the image are obtained, for example, the items can be detected by common target detection algorithms such as YOLOv4. The target detection frame is usually a rectangular or quadrilateral detection frame that encloses a single item.
[0053] Exemplary as figure 2 As shown, it is a schematic diagram of an image. In order to explain the working logic, an image in the process of the user starting to adjust the items in the refrigerator is selected as the first image, which includes 5 target detection frames, and the items in it are correspondingly marked as Goal 1 to Goal 5.
[0054] S13: Detect item information in the second image, and generate target detection frames at each item.
[0055] Exemplary as image 3 shown, for figure 2 The schematic diagram of the latter image, for the convenience of description, is designated as the second image, which includes 5 target detection frames, and has not yet assigned an ID to the item.
[0056] S14: Compare the target detection frame of each item in the second image with the target detection frame B 1 The degree of coincidence between the two, and the target detection frame B with the highest degree of coincidence is obtained. max , detect the similarity between the two, if the similarity meets the minimum requirements, the target detection frame B max The content of the item is assigned as target 1, otherwise, the target 1 information will be deleted after review.
[0057] S15: Compare the coincidence degree and similarity of the target 2 to the target n with each target detection frame in the second image in sequence, and update the item ID information in the second image.
[0058] Specifically, in step S14, the target detection frame of each item in the second image is compared with the target detection frame B 1 The degree of overlap between, including steps:
[0059] S141: Calculate the target detection frame and target detection frame B of each item in the second image 1 The intersection and comparison between.
[0060] S142: For the intersection ratio greater than the first threshold V 1 The target detection frame is determined to meet the minimum requirements for coincidence.
[0061] In image detection, the intersection ratio is used to describe the overlapping area between two frames. It can be used to measure the degree of coincidence between the two target detection frames. The procedure for calculating the intersection ratio is the prior art, which will not be repeated here. .
[0062] First, through the relatively simple calculation of the intersection and union ratio, the target detection frame with a high degree of coincidence with the target detection frame B1 is selected as the candidate target for detection and recognition, thereby reducing the workload required for subsequent recognition.
[0063] Specifically, in step S14, the target detection frame B with the highest coincidence degree is obtained max ,include:
[0064] S143: Calculate the target detection frame B 1 Euclidean distance from the center point of each target detection frame that meets the minimum requirements for coincidence.
[0065] S144: Select the target detection frame with the shortest Euclidean distance and determine it as the target detection frame B with the highest degree of coincidence max.
[0066] In actual situations, the influence of the size and distribution position of the target detection frame on the intersection ratio may lead to errors in the final coincidence judgment. Therefore, the step of testing the Euclidean distance is added, which is the straight-line distance between two points. , when the Euclidean distance is the shortest, it can be considered that the two target detection frames are actually the most similar, and the intersection ratio is greater than the first threshold V 1 In the case of , it is selected as the target detection frame with the highest coincidence degree, and the judgment reliability is high.
[0067] Specifically, in step S14, detecting the similarity between the two includes:
[0068] S145: When the target detection frame B max with the target detection frame B 1 The Euclidean distance is less than or equal to the second threshold V 2 When the target detection frame B is max The contents are assigned target 1.
[0069] Here, for the Euclidean distance less than or equal to the second threshold V 2 In the case of time, it can be determined that the objects in the actual two target detection frames have not moved, so they can be directly assigned, thereby eliminating the subsequent detection steps, saving time and efficiency.
[0070] S146: When the target detection frame B max with the target detection frame B 1 The Euclidean distance between is greater than the second threshold V 2 When , extract the target detection frame B max with the target detection frame B 1 The multi-dimensional features of the inner screenshot area are used to calculate the cosin similarity of the two.
[0071] S147: If the cosin similarity is greater than the set third threshold V 3 , the target detection frame B max The content of the item is assigned as target 1, otherwise, the target 1 information will be deleted after review.
[0072] Here, for Euclidean distances greater than the second threshold V 2 When the situation, the target detection frame B max Relative to the target detection frame B 1 The displacement has actually occurred. At this time, in order to improve the accuracy of recognition, the cosin similarity between the two is compared. Compared with the Euclidean distance, the cosin similarity is more about distinguishing the difference in direction, and for the absolute value It is not sensitive, so for the target detection frame whose position has changed, the detection of cosin similarity has a higher accuracy.
[0073] Specifically, in step S147, delete target 1 information after review, including:
[0074] When there is no target detection frame B in the second image 1 When the target detection frame with the minimum similarity requirement is met, the target detection frame B is 1 Save the internal information, continue to participate in the detection of the item information in the third image, when there is a target detection frame that meets the minimum requirements for coincidence and similarity, assign the item to the target 1, otherwise repeat the above steps until the mth image is in After there is no target detection frame that meets the minimum similarity requirement, delete the target 1 information, where 3≤m≤n.
[0075] Here, considering that there may be misjudgment and occlusion of the latter, when the target detection frame B is not found only in the second image 1 When the target detection frame meets the similarity requirement, the deletion of the information of target 1 is suspended, and after it is saved, it is compared again in the subsequent detection of image information, until the information without target 1 is detected for many times in a row, it can be determined. It has been taken away by the user, thereby improving the accuracy of detection and determination.
[0076] In this embodiment, the value of m is 7, that is, the target detection frame B is not recognized in 7 consecutive images. 1 After the target detection frame that meets the similarity requirements, delete the information of target 1.
[0077] By repeating the above steps, the item ID information in the first image is assigned and updated to the second image.
[0078] Exemplary as Figure 4 As shown, item 1 to item 4 did not move, directly figure 2 The item ID information in is assigned to it, and item 5 is moved by the user, but it meets the minimum similarity requirement, so it is assigned a value.
[0079] Further, step S14 also includes:
[0080] After updating the item ID information, when there is still an item without an ID assigned in the xth image, a new ID is assigned to it. At this time, it is determined that the user has placed a new item in the refrigerator.
[0081] S16: Repeat the above steps, and update the item ID information for the x th image by using the x-1 th image in sequence, where 2≤x≤n.
[0082] further, as Figure 5 As shown, in some embodiments of the present invention, for the situation that the items in the refrigerator are temporarily blocked by the user's hand, the step of detecting and identifying is also included:
[0083]S21: When a hand is detected in the a-th image, and the target j in the a-1-th image is detected but not detected in the a-th image, compare the corresponding target detection frame B j The target detection frame B with the hand in the a-th image h The degree of coincidence between , where 2≤a≤n, 1≤j≤n.
[0084] S22: When the minimum coincidence requirement is met, the target detection frame B is j with the target detection frame B h Information is stored in association.
[0085] Specifically, in step S21, compare the corresponding target detection frame B j The target detection frame B with the hand in the a-th image h The degree of overlap between, including:
[0086] Calculate the target detection frame B j with the target detection frame B h The cross-union ratio between, when the cross-union ratio is greater than the fourth threshold V 4 , it is determined that the minimum coincidence requirement is met.
[0087] Here, in the detection process, when an item is not detected and the hand information is detected at the same time, the item has a high possibility of being blocked by the hand, or picked up by the user to move it. At this time, the two The target detection frame between the two has a high degree of coincidence, so by detecting the intersection ratio between the two, it is judged whether the object is blocked by the hand. And temporarily save the item information for subsequent continued testing.
[0088] Exemplary as Image 6 and Figure 7 shown, where, Figure 7 for Image 6 next image of , in Image 6 Items 1 to 6 are detected in the Figure 7 , the information of item 1 to item 5 and hand is detected and identified, item 6 is picked up by the user's hand, and its information is blocked, Image 6 Medium Items 6 and Figure 7 The intersection ratio of the target detection frame in the middle hand meets the minimum requirement of coincidence, and the contents in the two target detection frames are associated and temporarily saved.
[0089] S23: Continue to perform subsequent image item information detection, when there is an item j2 of the same type as item j, and its corresponding target detection frame B j2 with the target detection frame B h When the minimum coincidence requirements are met, the detection target detection frame B is j2 and target detection frame B j Content similarity, if the similarity meets the minimum requirements, the target detection frame B is j2 The content is assigned j, otherwise, the target detection frame B is j2 Assign a new ID to the inner item.
[0090] Specifically, in step S23, by calculating the target detection frame B j2 with the target detection frame B h Whether the cross-union ratio is greater than the sixth threshold V 6 to determine whether the minimum coincidence requirements are met.
[0091] Specifically, in step S23, the target detection frame B is detected j2 and target detection frame B j Content similarity, including:
[0092] Intercept target detection frame B j2 with the target detection frame B j Inner image, extract its 512-dimensional features respectively, calculate the cosin similarity, if the cosin similarity is greater than the fifth threshold V 5 , then update and assign ID to item j2 as j, otherwise, assign a new ID to item j2.
[0093] Exemplary as Figure 8 shown, it is Figure 7 In the next image, the same type of item 6 is detected and recognized, and its target detection frame is the same as Figure 7 The sixth threshold V of target detection in the middle hand 6 The frames meet the minimum requirements for coincidence, therefore, continue to extract the 512-dimensional features of the images within them. Image 6 The cosin similarity of the image in the target detection frame of item 6 is greater than the fifth threshold V 5 , which is assigned item 6.
[0094] Further, when a hand is detected in the a-th image, and there are multiple objects in the a-1-th image that are not detected in the a-th image after detection, subsequent detection is performed for the multiple objects respectively.
[0095] To sum up, the present invention can detect and track the change of the information and position of each item in a continuous image with high accuracy through the above method, and can effectively detect and track that the tracking is temporarily occluded by the hand or held by the user in multiple continuous images. Items that start to move, less tracking loss and high tracking accuracy.
[0096] The present invention also provides a refrigerator, comprising: a camera, a memory and a processor, and the camera is configured to capture an image of a storage space in the refrigerator. The memory stores a computer program that can run on the processor, and when the processor executes the program, the steps of the above-mentioned method for judging the object identity of the items in the refrigerator are implemented.
[0097] Further, the refrigerator includes a plurality of bottle bases, the plurality of bottle bases are arranged on the refrigerator door body from top to bottom in sequence, and the camera is set at the bottom of the bottle base, vertically downward, and is used for photographing the space of the bottle base below the bottle base. image.
[0098] The present invention also provides a computer storage medium, in which a computer program is stored, and when the computer program runs, the device where the computer storage medium is located executes the steps of the above method for judging the object identity of items in a refrigerator.
[0099] It should be understood that although this specification is described in terms of embodiments, not every embodiment only includes an independent technical solution, and this description in the specification is only for the sake of clarity, and those skilled in the art should take the specification as a whole, and each The technical solutions in the embodiments can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.
[0100] The series of detailed descriptions listed above are only specific descriptions for the feasible embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any equivalent embodiments or changes made without departing from the technical spirit of the present invention All should be included within the protection scope of the present invention.
