A region determination method, apparatus, device, and medium

Abstract

Embodiments of the present application provide a region determination method and device, equipment and medium, which are used to identify a region containing food in an image. Since the region containing food will change when the food is accessed, and the background region without food will not change, in the embodiments of the present application, when determining a first region containing food in a current frame image, if a difference between a parameter value of a pixel point in the current frame image and a parameter value of a pixel point at a corresponding position in a previous frame image exceeds a preset threshold, and the number of pixel points in the first region whose determined differences exceed the preset threshold exceeds a preset number threshold, the pixel point is determined as a pixel point in the first region, so that the first region containing food in the current frame image can be accurately determined in the embodiments of the present application.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, device and medium for determining a region. Background Technology

[0002] With the development and breakthroughs in computer vision technology, smart refrigerators based on computer vision technology have become a research hotspot in the home appliance industry due to their necessity in daily life. Intelligent and precise food management in smart refrigerators is gradually becoming a key technology highlighting core competitiveness in this field. A major prerequisite for intelligent and precise food management is the ability to accurately capture images containing the stored food, facilitating the identification of information such as the food's storage location and type.

[0003] Existing technologies primarily rely on static detection to identify food information. Static detection involves placing an image acquisition device inside the refrigerator and comparing two images of the refrigerator's interior to determine the food's storage and retrieval status. However, static detection suffers from the problem of food items obscuring each other, making it difficult to accurately identify food information. Furthermore, since image acquisition devices cannot be installed in the freezer compartment of a smart refrigerator, static detection cannot achieve full-space food detection within the entire refrigerator.

[0004] Existing technologies have proposed acquiring images of ingredients when they are stored or retrieved, and using these images to identify relevant information about the ingredients. However, these images often contain background areas that are not part of the ingredients, leading to inaccurate identification. Therefore, determining how to identify the areas containing the ingredients within the acquired images is crucial. Summary of the Invention

[0005] This application provides a region determination method, apparatus, device, and medium for identifying regions containing food ingredients in an image.

[0006] In a first aspect, embodiments of this application provide a method for determining a region, the method comprising:

[0007] Based on the received current frame image, determine the difference in parameter value between each pixel in the current frame image and the corresponding pixel in the previous frame image, wherein the current frame image is a color image or a depth image;

[0008] For each pixel in the current frame image, if the difference corresponding to the pixel is greater than a preset threshold, and the number of pixels with a difference greater than the preset threshold among the pixels adjacent to the pixel is greater than a preset number threshold, then the pixel is determined to be a target pixel in the first region containing food in the current frame image.

[0009] Based on each target pixel in the current frame image, a first region containing food ingredients is determined in the current frame image.

[0010] Secondly, embodiments of this application also provide a region determination device, the device comprising:

[0011] The determining module is used to determine the difference between the parameter values ​​of each pixel in the current frame image and the corresponding pixel in the previous frame image, based on the received current frame image, wherein the current frame image is a color image or a depth image;

[0012] The processing module is configured to, for each pixel in the current frame image, determine that if the difference corresponding to the pixel is greater than a preset threshold, and the number of pixels adjacent to the pixel whose difference is greater than the preset threshold is greater than a preset number threshold, then the pixel is determined to be a target pixel in the first region containing food in the current frame image; and determine the first region containing food in the current frame image based on each target pixel in the current frame image.

[0013] Thirdly, embodiments of this application also provide an electronic device, which includes at least a processor and a memory, wherein the processor is configured to execute a computer program stored in the memory to implement the steps of the region determination method as described in any of the preceding claims.

[0014] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the region determination method as described in any of the preceding claims.

[0015] In this embodiment, the difference in parameter values ​​between each pixel in the current frame image and the corresponding pixel in the previous frame image is determined. For each pixel in the current frame image, if the difference is greater than a preset threshold, and the number of pixels adjacent to the pixel whose difference is greater than the preset threshold is greater than a preset number threshold, then the pixel is determined to be a target pixel in the first region containing food in the current frame image. Since the region containing food changes when accessing food, while the background region (not food) remains unchanged, in this embodiment, when determining the first region containing food in the current frame image, if the difference in parameter values ​​between a pixel in the current frame image and the corresponding pixel in the previous frame image exceeds a preset threshold, and the number of pixels adjacent to the pixel whose difference exceeds the preset threshold exceeds a preset number threshold, then the pixel is determined to be a pixel in the first region. Therefore, in this embodiment, the first region containing food in the current frame image can be accurately determined. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of a region determination method provided in an embodiment of this application;

[0018] Figure 2 This application provides a schematic diagram illustrating a process for determining an area containing food ingredients in an image.

[0019] Figure 3 This application provides a schematic diagram illustrating a process for determining a region containing food ingredients in a current frame image, as provided in an embodiment of the present application.

[0020] Figure 4 A schematic diagram of a world coordinate system provided in an embodiment of this application;

[0021] Figure 5a This is a schematic diagram illustrating the process of storing ingredients as provided in an embodiment of this application.

[0022] Figure 5b This is a schematic diagram illustrating the process of removing ingredients, provided in an embodiment of this application.

[0023] Figure 6 This application provides a schematic diagram of adjacent pixels of a certain pixel in a current frame image, as provided in an embodiment of the present application.

[0024] Figure 7 A schematic diagram illustrating the position of a pixel in the current frame image and the corresponding position of a pixel in the previous frame image, provided in an embodiment of this application;

[0025] Figure 8 A schematic diagram illustrating a process for determining relevant information about ingredients, provided in an embodiment of this application;

[0026] Figure 9 This is a schematic diagram of a region determination device provided in an embodiment of this application;

[0027] Figure 10 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation

[0028] The present application will now be described in further detail with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present application.

[0029] In this embodiment, after receiving the current frame image, the smart refrigerator determines the difference in parameter values ​​between each pixel in the current frame image and the corresponding pixel in the previous frame image, where the current frame image is a color image or a depth image. For each pixel in the current frame image, if the difference determined for that pixel is greater than a preset threshold, and the number of pixels adjacent to that pixel whose differences are greater than the preset threshold is greater than a preset number threshold, then that pixel is determined to be the pixel corresponding to the food sampled during food storage and retrieval. This pixel is then determined to be a target pixel in the first region containing the food in the current frame image, and the first region containing the food in the current frame image is determined based on each target pixel in the current frame image.

[0030] Figure 1 This application provides a schematic diagram of a region determination method, which includes the following steps:

[0031] S101: Based on the received current frame image, determine the difference in parameter values ​​between each pixel in the current frame image and the corresponding pixel in the previous frame image, wherein the current frame image is a color image or a depth image.

[0032] The region determination method provided in this application is applied to a smart refrigerator.

[0033] In this embodiment, the smart refrigerator can first receive the current frame image captured by the image acquisition device. To determine the changed areas between the current frame image and the previous frame image, the smart refrigerator can, after receiving the current frame image, determine the difference in parameter values ​​between each pixel in the current frame image and the corresponding pixel in the previous frame image. Based on the determined differences for each pixel in the current frame image, it can then determine whether the object corresponding to that pixel has moved.

[0034] The previous frame of the current frame image was captured by the same image acquisition device as the current frame image. The current frame image can be a color image or a depth image. If the current frame image is a color image, the pixel value is used to determine the difference between the parameter values ​​of each pixel in the current frame image and the corresponding pixel in the previous frame image. If the current frame image is a depth image, the depth value is used to determine the difference between the parameter values ​​of each pixel in the current frame image and the corresponding pixel in the previous frame image.

[0035] S102: For each pixel in the current frame image, if the difference corresponding to the pixel is greater than a preset threshold, and the number of pixels with a difference greater than the preset threshold among the pixels adjacent to the pixel is greater than a preset number threshold, then the pixel is determined to be a target pixel in the first region containing food ingredients in the current frame image.

[0036] In this embodiment, to determine the first region containing food ingredients in the current frame image, after determining the difference corresponding to each pixel in the current frame image, for each pixel in the current frame image, it is determined whether the difference corresponding to that pixel is greater than a preset threshold. If the difference corresponding to that pixel is greater than the preset threshold, then the pixel is further determined to be a target pixel in the first region containing food ingredients in the current frame image by using the pixels adjacent to that pixel. Specifically, if the difference corresponding to that pixel is greater than the preset threshold, then the pixels with differences greater than the preset threshold among the pixels adjacent to that pixel are obtained, and it is determined whether the number of obtained pixels is greater than a preset number threshold. If the number of pixels with differences greater than the preset threshold among the pixels adjacent to that pixel is greater than the preset number threshold, then the pixel is determined to be a target pixel in the first region containing food ingredients in the current frame image.

[0037] In this embodiment, when determining the difference in parameter values ​​between each pixel in the current frame and the corresponding pixel in the previous frame, pixels in the first row, first column, last row, and last column of the current frame can be ignored. For each pixel, the pixels adjacent to it refer to the eight pixels surrounding it. Since there are eight pixels adjacent to each pixel, the preset threshold can be a value such as 5 or 6, which is not less than 0 and not greater than 8.

[0038] Furthermore, in this embodiment, after obtaining each pixel whose difference from its neighboring pixels is greater than a preset threshold, it is further determined whether the number of neighboring pixels exceeds a preset threshold. If it exceeds the preset threshold, the pixel is determined to be a target pixel in the first region containing food in the current frame image. For example, if the difference corresponding to a certain pixel is 5, and each pixel adjacent to that pixel is, in sequence: the first pixel adjacent to the upper left corner of the pixel, the second pixel adjacent to the upper side of the pixel, the third pixel adjacent to the upper right corner of the pixel, the fourth pixel adjacent to the right side of the pixel, the fifth pixel adjacent to the lower right corner of the pixel, the sixth pixel adjacent to the lower side of the pixel, the seventh pixel adjacent to the lower left corner of the pixel, and the eighth pixel adjacent to the left side of the pixel. The determined differences for each pixel are 2, 5, 9, 4, 3, 5, 3, 5. Taking a preset threshold of 3 and a preset quantity threshold of 3 as an example, the pixels whose differences exceed the quantity threshold for each pixel adjacent to this pixel are the second pixel, the third pixel, the fourth pixel, the sixth pixel, and the eighth pixel, with corresponding differences of 5, 9, 4, 5, and 5, respectively. Since the number of pixels adjacent to this pixel that exceed the preset threshold exceeds the preset quantity threshold, and among the pixels adjacent to this pixel that exceed the preset threshold, the adjacent pixels are the second pixel, the third pixel, and the fourth pixel, and the number of adjacent pixels is 3, it means that among the pixels adjacent to this pixel that exceed the preset threshold, the number of adjacent pixels does not exceed the preset quantity threshold. Therefore, this pixel is not a target pixel in the first region containing food in the current frame image.

[0039] S103: Based on each target pixel in the current frame image, determine the first region in the current frame image that contains the food ingredients.

[0040] In this embodiment, since each target pixel in the current frame image is determined, the region containing each target pixel is defined as the first region of the current frame image containing food. Because the target pixels are determined based on their corresponding differences, the determined target pixels may not be continuous; that is, there may be more than one first region of the current frame image containing food. In this embodiment, the first region of the current frame image containing food can be a region of interest (ROI) containing food in the current frame image.

[0041] Since the area containing the food changes when accessing the food, while the background area that is not the food remains unchanged, in this embodiment of the application, when determining the first region containing the food in the current frame image, if the difference between the parameter value of a certain pixel in the current frame image and the corresponding pixel in the previous frame image exceeds a preset threshold, and the number of pixels whose difference exceeds the preset threshold among the adjacent pixels of that pixel exceeds a preset number threshold, then that pixel is determined to be a pixel in the first region. Therefore, in this embodiment of the application, the first region containing the food in the current frame image can be accurately determined.

[0042] To accurately determine the first region containing food in the current frame image, based on the above embodiments, in this embodiment, after determining that the pixel is a target pixel in the first region containing food in the current frame image, and before determining the first region containing food in the current frame image based on each target pixel in the current frame image, the method further includes:

[0043] Remove target pixels located outside the preset area, and for the remaining target pixels, perform the subsequent operation of determining the first region containing the food in the current frame image based on each target pixel in the current frame image.

[0044] In this embodiment, since the first region containing food in the initially determined current frame image may contain noise information, it is necessary to filter the pixels within this first region. Furthermore, when storing or retrieving food, the smart refrigerator can set a preset region for each frame image captured by the image acquisition device that captures the current frame image, indicating the possible presence of food in the image captured when the food is stored or retrieved. Target pixels outside this preset region are determined not to be pixels within the region containing food. This preset region can be determined based on the size of the smart refrigerator; specifically, how this preset region is determined is not limited here.

[0045] Therefore, in this embodiment, after determining that the pixel is a target pixel in the first region containing food in the current frame image, it is determined whether the target pixel is outside the preset region. If the target pixel is outside the preset region, the target pixel is removed. After removing target pixels outside the preset region for each target pixel, for the remaining target pixels, the subsequent operation of determining the first region containing food in the current frame image based on each target pixel in the current frame image is performed.

[0046] To accurately determine the first region containing food in the current frame image, based on the above embodiments, in this embodiment, after determining that the pixel is a target pixel in the first region containing food in the current frame image, and before determining the first region containing food in the current frame image based on each target pixel in the current frame image, the method further includes:

[0047] Based on the coordinate information of each target pixel in the current frame image, the first target coordinate information of the pixel center point of the first region is determined. Target pixels that are more than a preset distance threshold away from the first target coordinate information are removed. For the remaining target pixels, the subsequent operation of determining the first region containing food ingredients in the current frame image based on each target pixel in the current frame image is performed.

[0048] In this embodiment, if no error occurs in identifying the first region containing food in the current frame image, since the food is a whole in the application scenario, the distance between a pixel on the food and any other location is generally within a certain range. Therefore, the smart refrigerator stores a preset distance threshold. After determining that the pixel is a target pixel in the first region containing food in the current frame image, the smart refrigerator can determine the first target coordinate information of the pixel center point of each target pixel based on the coordinate information of each target pixel in the current frame image. After determining the first target coordinate information, the distance between each target pixel and the pixel center point is determined based on the coordinate information of each target pixel and the first target coordinate information. For each target pixel, if the distance between the target pixel and the pixel center point exceeds the preset distance threshold, it is determined that the target pixel is not a pixel in the first region containing food in the current frame image, and the target pixel is removed. For the remaining target pixels after removing the target pixels whose distance from the determined first target coordinate information exceeds the preset distance threshold, the subsequent operation of determining the first region containing food in the current frame image based on each target pixel in the current frame image is performed.

[0049] In this embodiment of the application, the first target coordinate information of the pixel center point in each target pixel is determined in the following way:

[0050]

[0051] Where, x w The x-coordinate of the pixel center point is y. w The ordinate of the pixel center point is x. i Let y be the x-coordinate of the i-th target pixel in the coordinate information. i y is the ordinate of the i-th target pixel in the coordinate information, and n is the number of target pixels.

[0052] In this embodiment, if too many target pixels are removed, it may indicate an error in the determined first region containing the food ingredient. Therefore, the first region containing the food ingredient in the current frame image is re-determined. This can be done by comparing the current frame image with pre-captured images before or after food ingredient storage or retrieval. Specifically, the condition for re-determining the first region containing the food ingredient in the current frame image can be: determining the ratio of the number of remaining target pixels to the number of each determined target pixel. If this ratio is less than or equal to a preset ratio, the first region containing the food ingredient in the current frame image is re-determined. For example, if the number of target pixels in the current frame image is N'... d The number of remaining target pixels is N. d And the preset ratio is 2 / 3, then if N d / N' d If the ratio is ≤2 / 3, then discard the first region containing the ingredients as determined by the current frame image.

[0053] To accurately determine the first region containing food ingredients in the current frame image, based on the above embodiments, the method in this application embodiment further includes:

[0054] Based on the coordinate information of each target pixel in the current frame image, determine the second target coordinate information of the pixel center point of the first region; determine the first distance between the second target coordinate information and the coordinate information of the pixel center point of the first region containing the food in the previous frame image of the current frame image;

[0055] Determine whether the first distance exceeds a preset distance threshold. If so, acquire a target image of the same scene captured at the same time as the current frame image. Wherein, if the current frame image is a color image, the target image is a depth image; if the current frame image is a depth image, the target image is a color image. Based on the parameter values ​​of each pixel in the target image and the parameter values ​​of the corresponding pixel in the previous frame image of the target image, determine the second region containing food in the target image, and determine the third target coordinate information of the pixel center point of the second region. Determine the second distance between the third target coordinate information and the coordinate information of the pixel center point of the region containing food in the previous frame image of the target image. If the second distance does not exceed the preset distance threshold, determine the fourth target coordinate information corresponding to the coordinate information of each pixel in the region containing food in the target image in the current frame image, and update the target pixels in the first region containing food in the current frame image using the pixel at the fourth target coordinate information.

[0056] Since users typically don't add food items quickly, if the distance between the center points of each pixel in the food-containing region exceeds a preset threshold in two consecutive frames of images containing food, it indicates an error in the food-containing region determined in the latter frame. Therefore, in this embodiment, the smart refrigerator pre-stores a preset distance threshold. Based on the coordinates of each target pixel in the current frame image, it determines the second target coordinates of the center points of the first region containing food in the current frame image, determines the coordinates of the center points of the food-containing region in the previous frame image, and determines the first distance between the second target coordinates and the coordinates of the center points determined in the previous frame image. It then determines whether this first distance exceeds the preset distance threshold. If the first distance exceeds the preset distance threshold, the first region containing food in the current frame image can be re-determined based on a target image of the same scene captured at the same time as the current frame image. Wherein, if the current frame image is a color image, the acquired target image is a depth image; if the current frame image is a depth image, the acquired target image is a color image. Specifically, how to determine the coordinates of the pixel center point of the region containing food in an image has been described in the above embodiments and will not be repeated here.

[0057] In this embodiment of the application, the process of re-determining the first region containing food in the current frame image based on the target image of the same scene acquired at the same time as the current frame image can be as follows: the smart refrigerator acquires the target image of the same scene acquired at the same time as the current frame image. After acquiring the target image, the smart refrigerator determines the second region containing food in the target image based on the parameter value of each pixel in the target image and the parameter value of the corresponding pixel in the previous frame target image.

[0058] Specifically, the smart refrigerator determines the difference in parameter values ​​between each pixel in the target image and the corresponding pixel in the previous frame of the target image. For each pixel in the target image, if the difference is greater than a preset threshold, and the number of neighboring pixels with differences greater than the preset threshold is greater than a preset number threshold, then that pixel is determined to be a pixel in the second region containing food in the target image. Based on each pixel in the determined second region containing food, the second region containing food in the target image is further defined. Specifically, the pixels in the second region containing food in the target image are each of the determined pixels that meet the requirements.

[0059] Based on the pixels in each second region containing food in the target image, the third target coordinate information of the pixel center point of that second region is determined. The coordinate information of the pixel center point of the region containing food in the previous frame of the target image is also determined. After determining the third target coordinate information and the coordinate information of the pixel center point of the region containing food in the target image, a second distance is determined between the third target coordinate information and the coordinate information of the pixel center point of the region containing food in the previous frame of the target image. If the determined second distance does not exceed a preset distance threshold, it indicates that the region containing food in the target image is accurate. Since the target image and the current frame image are images of the same scene captured at the same time, the region containing food in the target image is used to determine the region containing food in the current frame image.

[0060] Specifically, the process of determining the region containing food in the current frame image by using the second region containing food in the target image can be as follows: based on the pre-saved correspondence between the depth pixel coordinate system and the world coordinate system, and the correspondence between the color pixel coordinate system and the world coordinate system, determine the fourth target coordinate information corresponding to the coordinate information of each pixel in the region containing food in the target image in the current frame image. After determining the corresponding fourth target coordinate information, update the target pixel in the first region containing food in the current frame image using the pixel at the fourth target coordinate information, thereby updating the first region containing food in the current frame image.

[0061] To accurately determine the first region containing food ingredients in the current frame image, based on the above embodiments, the method in this application embodiment further includes:

[0062] If both the first distance and the second distance exceed the preset distance threshold, it is determined that the food tracking has failed. Then, the first image with the closest time interval to the current frame image and which has successfully tracked the food is found. The first region containing the food in the first image is used to update the first region containing the food in the current frame image.

[0063] When determining the region containing food in the current frame image, if the user's movements when storing or retrieving food are large or too fast, the current frame image may track objects other than food. Therefore, in this embodiment, if both the first distance determined based on the current frame image and the second distance determined based on the target image exceed a preset distance threshold, it is determined that tracking of food based on the current frame image has failed. Since the time interval between adjacent video frames is short, the difference in the corresponding regions of moving objects in adjacent video frames is not significant. Therefore, if tracking of food based on the current frame image fails, the image with the closest time interval to the current frame image that successfully tracked food is found as the first image, and the region containing food in the first image is used to update the first region containing food in the current frame image. This allows the smart refrigerator to use the updated first region from the current frame image to determine the region containing food in the next frame image.

[0064] Figure 2 This is a schematic diagram illustrating a process for determining a region containing food ingredients in an image, as provided in an embodiment of this application.

[0065] Determine the coordinate information of the pixel center points of the regions containing food in the color image and the depth image; determine the first distance and the second distance between the pixel center points of the regions containing food in the color image and the depth image and the pixel center points of the regions containing food determined in the corresponding previous frame image.

[0066] If the first distance corresponding to the color image exceeds a preset distance threshold, the pixels in the region containing the food in the depth image are mapped to the color image to correct the region containing the food in the color image; if the second distance corresponding to the depth image exceeds a preset distance threshold, the pixels in the region containing the food in the color image are mapped to the depth image to correct the region containing the food in the depth image.

[0067] When both the first distance and the second distance exceed the preset distance threshold, the most recently successfully tracked depth image and color image are searched. Based on the most recently successfully tracked depth image and color image, the regions containing food in the depth image and color image are determined.

[0068] Figure 3 This is a schematic diagram illustrating a process for determining a region containing food ingredients in the current frame image, as provided in an embodiment of this application.

[0069] S301: Get the current frame image.

[0070] S302: Determine the difference in parameter value between each pixel in the current frame image and the corresponding pixel in the previous frame image.

[0071] S303: Based on the difference determined by each pixel in the current frame image, determine the first region in the current frame image that contains the food.

[0072] S304: Remove target pixels located outside the region, and / or determine the first target coordinate information of the pixel center point of the first region based on the coordinate information of each target pixel in the current frame image, and remove target pixels whose distance from the first target coordinate information exceeds a preset distance threshold.

[0073] In this embodiment of the application, if the number of target pixels removed is too large, the target pixels contained in the current frame image can be re-determined.

[0074] S305: Determine the coordinate information of the center point of the pixel in the first region of the current frame image, and the first distance between the coordinate information of the center point of the pixel in the region containing the food in the previous frame image.

[0075] S306: Determine whether the first distance exceeds the preset distance threshold. If yes, proceed to S307; otherwise, proceed to S308.

[0076] S307: Obtain the target image from the same scene captured at the same time as the current frame image, and determine the coordinates of the center point of the pixel in the region containing food in the target image, and the second distance between the coordinates of the center point of the pixel in the region containing food in the previous frame image of the target image. Then execute S309.

[0077] S308: Obtain the target image corresponding to the current frame image, and determine the target area containing the food in the current frame image based on the areas containing the food in the current frame image and the target image.

[0078] S309: Determine whether the second distance exceeds the preset distance threshold. If yes, execute S310; otherwise, execute S311.

[0079] S310: Find the first image that has the shortest time interval before the current frame image and has successfully tracked the food ingredients. Based on the area containing the food ingredients in the first image, determine the area containing the food ingredients in the current frame image. Then execute S308.

[0080] S311: Determine the region containing food in the current frame image based on the region containing food in the target image.

[0081] If the image contains areas containing food ingredients, in order to determine whether the food ingredients contained in the current frame image are the ingredients to be retrieved or the ingredients to be stored, based on the above embodiments, in this embodiment of the application, the method further includes:

[0082] If a preset number of consecutive images fail to track the food, the corresponding tracking process will end.

[0083] Find the second image in each frame of the tracking process corresponding to the current frame image, the first image in which the region containing food was identified, and the third image in which the region containing food was identified last; obtain the first coordinate information of the pixel center point of the region containing food in the second image, and the second coordinate information of the pixel center point of the region containing food in the third image;

[0084] The first coordinate information and the second coordinate information are converted into first world coordinate information and second world coordinate information in the world coordinate system; the target feature vector corresponding to the first world coordinate information and the second world coordinate information is determined; based on the target symbol of the component with the preset direction in the target feature vector and the correspondence between the pre-saved symbol and the access method, the target access method corresponding to the target symbol is determined as the access method corresponding to the food contained in the current frame image.

[0085] In this embodiment, if a predetermined number of consecutive images fail to track the food, the corresponding tracking process is determined to have ended, meaning the process of the user storing or retrieving food is considered complete. In this embodiment, based on the second image in each frame of the tracking process corresponding to the current frame that first identifies a region containing food, and the third image in each frame of the tracking process corresponding to the current frame that last identifies a region containing food, it is determined whether the food contained in the current frame image is food that has been stored or food that has been retrieved.

[0086] If the current frame image is not an image frame where food tracking failed, the method for determining the tracking process corresponding to the current frame image can be as follows: In each frame image preceding the current frame image, find the image with the closest time interval to the current frame image among the consecutive preset number of images where food tracking failed. The first image to identify a region containing food after the searched image is designated as the first image. In each frame image following the current frame image, find the image with the closest time interval to the current frame image among the consecutive preset number of images where food tracking failed. The frame image preceding the searched image is designated as the second image. Each frame image between the first and second images is considered an image in the tracking process corresponding to the current frame image. If the current frame image is an image where food tracking failed, it is not necessary to determine whether the food contained in the current frame image is the food that was stored or the food that was retrieved.

[0087] Specifically, the process of determining whether the food contained in the current frame image is the food to be stored or the food to be retrieved can be as follows: First coordinate information of the pixel center points of the region containing the food in the second image, and second coordinate information of the pixel center points of the region containing the food in the third image. After obtaining the first and second coordinate information, based on the pre-saved correspondence between the depth pixel coordinate system and the world coordinate system, and the correspondence between the color pixel coordinate system and the world coordinate system, the first coordinate information is converted into first world coordinate information in the world coordinate system, and the second coordinate information is converted into second world coordinate information in the world coordinate system. After determining the first and second world coordinate information, based on the first and second world coordinate information, the target feature vector corresponding to the second world coordinate information is determined. In this embodiment, the world coordinate information is three-dimensional coordinate information.

[0088] When determining the target feature vector, we can first determine the first feature vector corresponding to the first world coordinate information, and then determine the second feature vector corresponding to the second world coordinate information. Subtracting the second feature vector from the first feature vector gives the target feature vector pointing from the first world coordinate information to the second world coordinate information. For example, if the first world coordinate information is (x1, y1, z1) and the second world coordinate information is (x2, y2, z2), then the first feature vector corresponding to the first world coordinate information is (x1, y1, z1), the second feature vector corresponding to the second world coordinate information is (x2, y2, z2), and the target feature vector pointing from the first world coordinate information to the second world coordinate information is (x2-x1, y2-y1, z2-z1).

[0089] Since storing food involves movement from outside the smart refrigerator to inside, and retrieving food involves movement from inside the smart refrigerator to outside, the signs of the components in the preset direction of the corresponding feature vectors are opposite. Therefore, in this embodiment, the smart refrigerator pre-stores the correspondence between signs and retrieval methods. After obtaining the target feature vector, it obtains the target sign of the component in the preset direction of the target feature vector. Based on the target sign and the pre-stored correspondence between signs and retrieval methods, it determines the target retrieval method corresponding to the target sign. Specifically, if the target sign is positive, the food in the current frame image is determined to be the food that was retrieved; if the target sign is negative, the food in the current frame image is determined to be the food that was stored.

[0090] Figure 4 This is a schematic diagram of a world coordinate system provided in an embodiment of this application.

[0091] Depend on Figure 4 It can be seen that the origin of this world coordinate system is the position of the image acquisition device, the x-axis is the direction parallel to the horizontal plane of the smart refrigerator, the y-axis is the direction perpendicular to the ground, and the z-axis is the direction perpendicular to both the x-axis and the y-axis. If we consider... Figure 4 Taking the world coordinate system established in the example, the component of the target feature vector with the preset direction is the component of the z-direction.

[0092] Figure 5a This is a schematic diagram illustrating the process of storing ingredients as provided in an embodiment of this application.

[0093] Depend on Figure 5a It can be seen that when food is stored in a smart refrigerator, the system obtains the first world coordinate information of the pixel center point of the region containing the food in the second image and the second world coordinate information of the pixel center point of the region containing the food in the third image, and determines the target feature vector obtained by pointing the first world coordinate information to the second world coordinate information. Depend on Figure 5a It can be seen that when the target feature vector In Figure 4 When the z-axis points towards the smart refrigerator, the action at this time is "storage," meaning that the food contained in the current frame image is the food to be stored.

[0094] Figure 5b This is a schematic diagram illustrating the process of removing ingredients, as provided in an embodiment of this application.

[0095] Depend on Figure 5bIt can be seen that when food is taken out of the smart refrigerator, the system obtains the first world coordinate information of the pixel center point of the region containing food in the second image in the world coordinate system, and the second world coordinate information of the pixel center point of the region containing food in the third image in the world coordinate system. The system then determines that the first world coordinate information points to the second world coordinate information, thus obtaining the target feature vector. Depend on Figure 5b It can be seen that when the target is in the vector exist Figure 4 When the direction of the z-axis is away from the smart refrigerator, the action at this time is to take out the food, that is, the food contained in the current frame image is the food that is taken out.

[0096] In this application embodiment, the mainstream tracking methods in the prior art mainly include correlation filtering algorithms and deep learning-based methods. Regardless of the above-mentioned tracking methods or other tracking methods, when determining the object to be tracked within an image, the method for identifying regions containing food ingredients in an image provided in this application embodiment can be used.

[0097] To determine the location of ingredients contained in the current frame image, based on the above embodiments, the method in this application embodiment further includes:

[0098] If it is determined that the food contained in the current frame image is the food to be stored, then a first candidate image of the same scene captured at the same time as the third image is obtained, and the depth images in the third image and the first candidate image are determined as target recognition images; if it is determined that the food contained in the current frame image is the food to be taken out, then a second candidate image of the same scene captured at the same time as the second image is obtained, and the depth images in the second image and the second candidate image are determined as target recognition images.

[0099] The storage location of the food is determined based on the depth value corresponding to the pixel center point of the region containing the food in the target recognition image, and the correspondence between the pre-saved storage location and the actual distance range of the preset direction.

[0100] During the process of storing ingredients, the last image captured in each frame is the image closest to the storage location of the ingredients; during the process of retrieving ingredients, the first image captured in each frame is the image closest to the storage location of the ingredients.

[0101] Therefore, in this embodiment, if it is determined that the food contained in the current frame image is the food to be stored, then a third image is obtained in each frame of the tracking process corresponding to the current frame image, which is the first image in which the area containing the food is identified. In order to determine the location where the food is stored, a first candidate image is obtained in the same scene at the same time as the third image. The image that is a depth image in the third image and the first candidate image is obtained as the target recognition image. If it is determined that the food contained in the current frame image is the food to be removed, then a second image is obtained in each frame of the tracking process corresponding to the current frame image, which is the last image in which the area containing the food is identified. The image that is a second candidate image is obtained in the same scene at the same time as the second image. The image that is a depth image in the second image and the second candidate image is obtained as the target recognition image.

[0102] In this embodiment of the application, if the second image or the third image is a depth image, then the first candidate image and the second candidate image may not need to be acquired.

[0103] After acquiring the target recognition image, based on each pixel in the region containing the food in the acquired target recognition image, the center point of the pixel in the region containing the food is determined. After acquiring the corresponding center point, the depth value at the center point is obtained. This depth value is the distance between the object and the image acquisition device in a certain direction in the actual application, corresponding to that center point. Figure 4 For example, the depth value represents the distance between the object captured at that pixel and the image acquisition device along the y-axis. After obtaining the depth value at the pixel center point, the storage location of the food in the current frame image is determined based on the obtained depth value and the pre-saved correspondence between the storage location and the actual distance range in the preset direction. For example, if the pre-saved distance range for the first layer of the smart refrigerator is (0, y1) and the distance range for the second layer is [y1, y2), and the determined depth value at the pixel center point belongs to [y1, y2), then the storage location of the food in the current frame image is determined to be the second layer of the smart refrigerator.

[0104] To accurately determine the ingredients contained in the current frame image, based on the above embodiments, the method in this application embodiment further includes:

[0105] The coordinate information of each target pixel in the first region containing the food in the current frame image is determined in the third world coordinate information in the world coordinate system, and the coordinate information of each pixel in the second region containing the food in the target image is determined in the fourth world coordinate information in the world coordinate system.

[0106] For each target pixel in the current frame image, determine whether there is fourth coordinate information that matches the third coordinate information corresponding to the target pixel. If so, determine that the target pixel is a pixel in the food identification area of ​​the current frame image.

[0107] The food identification regions in the current frame image are determined based on the pixels in each food identification region;

[0108] The target food ingredient contained in the food ingredient recognition region of the current frame image is obtained by using a pre-trained food ingredient recognition model.

[0109] Since the current frame image and the target image are images of the same scene captured at the same time, in order to improve the accuracy of determining the region containing food in the current frame image, in this embodiment of the application, based on the pre-saved correspondence between the depth pixel coordinate system and the world coordinate system, and the pre-saved correspondence between the color pixel coordinate system and the world coordinate system, the coordinate information of each target pixel in the first region containing food in the current frame image is determined in the third world coordinate information in the world coordinate system, and the coordinate information of each pixel in the second region containing food in the target image is determined in the fourth world coordinate information in the world coordinate system.

[0110] After determining the third-world and fourth-world coordinate information, the smart refrigerator checks for each target pixel in the current frame image whether there exists a fourth coordinate information that completely matches the third coordinate information corresponding to that target pixel. If a fourth coordinate information matching the third coordinate information of the target pixel exists, then it is determined that the target pixel has a corresponding pixel in the target image, and thus the target pixel is determined to be a pixel in the food recognition area of ​​the current frame image. Based on each pixel in the food recognition area, the food recognition area in the current frame image is determined.

[0111] After determining the food identification region in the current frame image, in order to determine the food contained in the current frame image, the food identification region in the current frame image is input into a pre-trained food identification model, the output of the food identification model is obtained, and the output of the food identification model is determined to be the target food contained in the food identification region in the current frame image.

[0112] Specifically, food identification models can be traditional feature extractors and classifiers, or they can be deep learning models. Feature extractors mainly include Histogram of Oriented Gradients (HOG), Local Binary Patterns (LBP), and Deformable Parts Model (DPM), while classifiers mainly include Support Vector Machine (SVM), Adaboost, decision trees, and Bayesian networks. Deep learning models mainly include YOLO (You Only Look Once) and Fast-CNN (Fast-CNN). The specific techniques for determining the target food in the current frame image's identification region using traditional feature extractors and classifiers, and for determining the target food in the current frame image's identification region using deep learning models, are existing technologies and will not be elaborated upon here.

[0113] In this embodiment, the food storage and retrieval result includes the food storage and retrieval action, the storage and retrieval location, and the target food being stored. The storage and retrieval action includes storing the food and retrieving the food. The above method identifies the food storage and retrieval action, the storage and retrieval location, and the target food. For each food storage and retrieval result, methods such as voice broadcasting, sending to the smart refrigerator screen, or a mobile app can be used, allowing users to view the food in the smart refrigerator in real time, thereby achieving intelligent and precise management of the food in the smart refrigerator. When an error occurs, the incorrect identification result can be corrected through voice interaction, realizing a closed loop for the dynamic identification function of food storage and retrieval in the smart refrigerator.

[0114] In this embodiment of the application, in order to improve the efficiency of recognition, a description label can be added to each pixel in the current frame image. The description label can be called the descriptor of the pixel, and the parameter value of each pixel in the adjacent pixels is recorded in the description label.

[0115] Figure 6 This is a schematic diagram of adjacent pixels of a certain pixel in the current frame image provided in an embodiment of this application.

[0116] Depend on Figure 6 It can be seen that, ignoring the boundaries of the current frame image (i.e., the first row, first column, last row, and last column), each pixel in the current frame image is surrounded by eight neighboring pixels. In this embodiment, for each pixel, the parameter values ​​of its eight neighboring pixels can be used as the descriptor corresponding to that pixel, denoted as P. (x,y) =[p1,p2,…,p8]. From Figure 6It can be seen that the parameter values ​​of these 8 adjacent pixels can be recorded in a clockwise direction, starting from the pixel adjacent to the top left corner. That is, p1 is the pixel adjacent to the top left of the pixel, p2 is the pixel adjacent to the top of the pixel, p3 is the pixel adjacent to the top right of the pixel, p4 is the pixel adjacent to the right of the pixel, p5 is the pixel adjacent to the bottom right of the pixel, p6 is the pixel adjacent to the bottom of the pixel, p7 is the pixel adjacent to the bottom left of the pixel, and p8 is the pixel adjacent to the left of the pixel.

[0117] Figure 7 This is a schematic diagram illustrating the position of a pixel in the current frame image and the corresponding pixel in the previous frame image, as provided in an embodiment of this application.

[0118] Depend on Figure 7 It can be seen that pk and pk+1 can be the parameter values ​​at (x, y) of the k-th frame image and the (k+1)-th frame image, respectively. The (k+1)-th frame image can be the current frame image, and the k-th frame image can be the previous frame image. The parameter value of the pixel at (x, y) and the difference between the parameter values ​​of the pixel and its 8 neighboring pixels are calculated respectively. When the difference between the parameter value at (x, y) and the difference between the parameter values ​​of the pixel and the parameter values ​​of the 8 neighboring pixels are all greater than a preset threshold, the pixel at (x, y) in the current frame image is determined to be the target pixel in the region containing food in the current frame image.

[0119] Figure 8 This is a schematic diagram illustrating a process for determining relevant information about ingredients, as provided in an embodiment of this application.

[0120] Depend on Figure 8 As can be seen, in this embodiment of the application, the smart refrigerator determines the area containing food in each frame of image captured during the user's natural food storage and retrieval process, namely the area where the user is holding food or empty-handed, and identifies the food, the food storage and retrieval action, and the location of the food storage for each frame of image captured.

[0121] Figure 9 This application provides a schematic diagram of a region determination device, which includes:

[0122] The determining module 901 is used to determine the difference between the parameter values ​​of each pixel in the current frame image and the corresponding pixel in the previous frame image, based on the received current frame image, wherein the current frame image is a color image or a depth image;

[0123] The processing module 902 is configured to, for each pixel in the current frame image, determine that if the difference corresponding to the pixel is greater than a preset threshold, and the number of pixels adjacent to the pixel whose difference is greater than the preset threshold is greater than a preset number threshold, then the pixel is determined to be a target pixel in the first region containing food in the current frame image; and determine the first region containing food in the current frame image based on each target pixel in the current frame image.

[0124] In one possible implementation, the processing module 902 is further configured to remove target pixels located outside a preset area.

[0125] In one possible implementation, the processing module 902 is further configured to determine the first target coordinate information of the pixel center point of the first region based on the coordinate information of each target pixel point in the current frame image, and remove target pixel points whose distance from the first target coordinate information exceeds a preset distance threshold.

[0126] In one possible implementation, the processing module 902 is further configured to: determine second target coordinate information of the pixel center point of the first region based on the coordinate information of each target pixel point in the current frame image; determine a first distance between the second target coordinate information and the coordinate information of the pixel center point of the first region containing food in the previous frame image of the current frame image; determine whether the first distance exceeds a preset distance threshold, and if so, acquire a target image of the same scene acquired at the same time as the current frame image; wherein, if the current frame image is a color image, the target image is a depth image, and if the current frame image is a depth image, the target image is a color image; based on the coordinate information of each target pixel point in the target image... The parameter values ​​of the pixels are compared with the parameter values ​​of the corresponding pixels in the previous frame of the target image to determine the second region containing food in the target image, and the third target coordinate information of the pixel center point of the second region is determined. The second distance between the third target coordinate information and the coordinate information of the pixel center point of the region containing food in the previous frame of the target image is determined. If the second distance does not exceed the preset distance threshold, the fourth target coordinate information corresponding to the coordinate information of each pixel in the region containing food in the target image is determined in the current frame image, and the target pixels in the first region containing food in the current frame image are updated using the pixel at the fourth target coordinate information.

[0127] In one possible implementation, the processing module 902 is further configured to determine that the food tracking has failed if both the first distance and the second distance exceed the preset distance threshold, and to find the first image that has the closest time interval to the current frame image and has successfully tracked the food, and to update the first region of the current frame image containing the food using the region of the first image containing the food.

[0128] In one possible implementation, the processing module 902 is further configured to: determine the end of the corresponding tracking process if a predetermined number of consecutive images fail to track the food; search for the second image in each frame of the tracking process corresponding to the current frame image, the first image in which the region containing the food was identified, and the third image in which the region containing the food was identified for the last time; obtain the first coordinate information of the pixel center point of the region containing the food in the second image, and the second coordinate information of the pixel center point of the region containing the food in the third image; convert the first coordinate information and the second coordinate information into first world coordinate information and second world coordinate information in the world coordinate system; determine the target feature vector corresponding to the second world coordinate information pointed to by the first world coordinate information; and determine the target access method corresponding to the target symbol as the access method corresponding to the food contained in the current frame image based on the target symbol of the component with a predetermined direction in the target feature vector and the pre-saved correspondence between the symbol and the access method.

[0129] In one possible implementation, the processing module 902 is further configured to: if it is determined that the food contained in the current frame image is the food to be stored, then acquire a first candidate image of the same scene captured at the same time as the third image, and determine the depth images in the third image and the first candidate image as target recognition images; if it is determined that the food contained in the current frame image is the food to be retrieved, then acquire a second candidate image of the same scene captured at the same time as the second image, and determine the depth images in the second image and the second candidate image as target recognition images; and determine the storage location of the food based on the depth value corresponding to the pixel center point of the region containing the food in the target recognition image, and the pre-saved correspondence between the storage location and the actual distance range of the preset direction.

[0130] In one possible implementation, the processing module 902 is further configured to determine the third world coordinate information in the world coordinate system for the coordinate information corresponding to each target pixel in the first region containing food in the current frame image, and to determine the fourth world coordinate information in the world coordinate system for the coordinate information corresponding to each pixel in the second region containing food in the target image; for each target pixel in the current frame image, determine whether there is fourth coordinate information that matches the third coordinate information corresponding to the target pixel; if so, determine that the target pixel is a pixel in the food identification region of the current frame image; determine the food identification region in the current frame image based on the pixels in each food identification region; and obtain the target food contained in the food identification region of the current frame image through a pre-trained food identification model.

[0131] Based on the above embodiments, this application also provides an electronic device, such as... Figure 10 As shown, it includes: processor 1001, communication interface 1002, memory 1003 and communication bus 1004, wherein processor 1001, communication interface 1002 and memory 1003 communicate with each other through communication bus 1004.

[0132] The memory 1003 stores a computer program, which, when executed by the processor 1001, causes the processor 1001 to perform the following steps:

[0133] Based on the received current frame image, determine the difference in parameter value between each pixel in the current frame image and the corresponding pixel in the previous frame image, wherein the current frame image is a color image or a depth image;

[0134] For each pixel in the current frame image, if the difference corresponding to the pixel is greater than a preset threshold, and the number of pixels with a difference greater than the preset threshold among the pixels adjacent to the pixel is greater than a preset number threshold, then the pixel is determined to be a target pixel in the first region containing food in the current frame image.

[0135] Based on each target pixel in the current frame image, a first region containing food ingredients is determined in the current frame image.

[0136] In one possible implementation, after determining that the pixel is a target pixel in a first region containing food in the current frame image, and before determining the first region containing food in the current frame image based on each target pixel in the current frame image, the method further includes:

[0137] Remove target pixels located outside the preset area, and for the remaining target pixels, perform the subsequent operation of determining the first region containing the food in the current frame image based on each target pixel in the current frame image.

[0138] In one possible implementation, after determining that the pixel is a target pixel in a first region containing food in the current frame image, and before determining the first region containing food in the current frame image based on each target pixel in the current frame image, the method further includes:

[0139] Based on the coordinate information of each target pixel in the current frame image, the first target coordinate information of the pixel center point of the first region is determined. Target pixels that are more than a preset distance threshold away from the first target coordinate information are removed. For the remaining target pixels, the subsequent operation of determining the first region containing food ingredients in the current frame image based on each target pixel in the current frame image is performed.

[0140] In one possible implementation, the method further includes:

[0141] Based on the coordinate information of each target pixel in the current frame image, determine the second target coordinate information of the pixel center point of the first region; determine the first distance between the second target coordinate information and the coordinate information of the pixel center point of the first region containing the food in the previous frame image of the current frame image;

[0142] Determine whether the first distance exceeds a preset distance threshold. If so, acquire a target image of the same scene captured at the same time as the current frame image. Wherein, if the current frame image is a color image, the target image is a depth image; if the current frame image is a depth image, the target image is a color image. Based on the parameter values ​​of each pixel in the target image and the parameter values ​​of the corresponding pixel in the previous frame image of the target image, determine the second region containing food in the target image, and determine the third target coordinate information of the pixel center point of the second region. Determine the second distance between the third target coordinate information and the coordinate information of the pixel center point of the region containing food in the previous frame image of the target image. If the second distance does not exceed the preset distance threshold, determine the fourth target coordinate information corresponding to the coordinate information of each pixel in the region containing food in the target image in the current frame image, and update the target pixels in the first region containing food in the current frame image using the pixel at the fourth target coordinate information.

[0143] In one possible implementation, the method further includes:

[0144] If both the first distance and the second distance exceed the preset distance threshold, it is determined that the food tracking has failed. Then, the first image with the closest time interval to the current frame image and which has successfully tracked the food is found. The first region containing the food in the first image is used to update the first region containing the food in the current frame image.

[0145] In one possible implementation, the method further includes:

[0146] If a preset number of consecutive images fail to track the food, the corresponding tracking process will end.

[0147] Find the second image in each frame of the tracking process corresponding to the current frame image, the first image in which the region containing food was identified, and the third image in which the region containing food was identified last; obtain the first coordinate information of the pixel center point of the region containing food in the second image, and the second coordinate information of the pixel center point of the region containing food in the third image;

[0148] The first coordinate information and the second coordinate information are converted into first world coordinate information and second world coordinate information in the world coordinate system; the target feature vector corresponding to the first world coordinate information and the second world coordinate information is determined; based on the target symbol of the component of the preset direction in the target feature vector and the correspondence between the pre-saved symbol and the access method, the target access method corresponding to the target symbol is determined as the access method corresponding to the food contained in the current frame image.

[0149] In one possible implementation, the method further includes:

[0150] If it is determined that the food contained in the current frame image is the food to be stored, then a first candidate image of the same scene captured at the same time as the third image is obtained, and the depth images in the third image and the first candidate image are determined as target recognition images; if it is determined that the food contained in the current frame image is the food to be taken out, then a second candidate image of the same scene captured at the same time as the second image is obtained, and the depth images in the second image and the second candidate image are determined as target recognition images.

[0151] The storage location of the food is determined based on the depth value corresponding to the pixel center point of the region containing the food in the target recognition image, and the correspondence between the pre-saved storage location and the actual distance range of the preset direction.

[0152] In one possible implementation, the method further includes:

[0153] The coordinate information of each target pixel in the first region containing the food in the current frame image is determined in the third world coordinate information in the world coordinate system, and the coordinate information of each pixel in the second region containing the food in the target image is determined in the fourth world coordinate information in the world coordinate system.

[0154] For each target pixel in the current frame image, determine whether there is fourth coordinate information that matches the third coordinate information corresponding to the target pixel. If so, determine that the target pixel is a pixel in the food identification area of ​​the current frame image.

[0155] The food identification regions in the current frame image are determined based on the pixels in each food identification region;

[0156] The target food ingredient contained in the food ingredient recognition region of the current frame image is obtained by using a pre-trained food ingredient recognition model.

[0157] The communication bus mentioned in the above server can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0158] The communication interface 1002 is used for communication between the above-mentioned electronic device and other devices.

[0159] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0160] The processors mentioned above can be general-purpose processors, including central processing units, network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0161] Based on the above embodiments, this application also provides a computer-readable storage medium storing a computer program executable by an electronic device. When the program is run on the electronic device, the electronic device performs the following steps:

[0162] The memory stores a computer program that, when executed by the processor, causes the processor to perform the following steps:

[0163] Based on the received current frame image, determine the difference in parameter value between each pixel in the current frame image and the corresponding pixel in the previous frame image, wherein the current frame image is a color image or a depth image;

[0164] For each pixel in the current frame image, if the difference corresponding to the pixel is greater than a preset threshold, and the number of pixels with a difference greater than the preset threshold among the pixels adjacent to the pixel is greater than a preset number threshold, then the pixel is determined to be a target pixel in the first region containing food in the current frame image.

[0165] Based on each target pixel in the current frame image, a first region containing food ingredients is determined in the current frame image.

[0166] In one possible implementation, after determining that the pixel is a target pixel in a first region containing food in the current frame image, and before determining the first region containing food in the current frame image based on each target pixel in the current frame image, the method further includes:

[0167] Remove target pixels located outside the preset area, and for the remaining target pixels, perform the subsequent operation of determining the first region containing the food in the current frame image based on each target pixel in the current frame image.

[0168] In one possible implementation, after determining that the pixel is a target pixel in a first region containing food in the current frame image, and before determining the first region containing food in the current frame image based on each target pixel in the current frame image, the method further includes:

[0169] Based on the coordinate information of each target pixel in the current frame image, the first target coordinate information of the pixel center point of the first region is determined. Target pixels that are more than a preset distance threshold away from the first target coordinate information are removed. For the remaining target pixels, the subsequent operation of determining the first region containing food ingredients in the current frame image based on each target pixel in the current frame image is performed.

[0170] In one possible implementation, the method further includes:

[0171] Based on the coordinate information of each target pixel in the current frame image, determine the second target coordinate information of the pixel center point of the first region; determine the first distance between the second target coordinate information and the coordinate information of the pixel center point of the first region containing the food in the previous frame image of the current frame image;

[0172] Determine whether the first distance exceeds a preset distance threshold. If so, acquire a target image of the same scene captured at the same time as the current frame image. Wherein, if the current frame image is a color image, the target image is a depth image; if the current frame image is a depth image, the target image is a color image. Based on the parameter values ​​of each pixel in the target image and the parameter values ​​of the corresponding pixel in the previous frame image of the target image, determine the second region containing food in the target image, and determine the third target coordinate information of the pixel center point of the second region. Determine the second distance between the third target coordinate information and the coordinate information of the pixel center point of the region containing food in the previous frame image of the target image. If the second distance does not exceed the preset distance threshold, determine the fourth target coordinate information corresponding to the coordinate information of each pixel in the region containing food in the target image in the current frame image, and update the target pixels in the first region containing food in the current frame image using the pixel at the fourth target coordinate information.

[0173] In one possible implementation, the method further includes:

[0174] If both the first distance and the second distance exceed the preset distance threshold, it is determined that the food tracking has failed. Then, the first image with the closest time interval to the current frame image and which has successfully tracked the food is found. The first region containing the food in the first image is used to update the first region containing the food in the current frame image.

[0175] In one possible implementation, the method further includes:

[0176] If a preset number of consecutive images fail to track the food, the corresponding tracking process will end.

[0177] Find the second image in each frame of the tracking process corresponding to the current frame image, the first image in which the region containing food was identified, and the third image in which the region containing food was identified last; obtain the first coordinate information of the pixel center point of the region containing food in the second image, and the second coordinate information of the pixel center point of the region containing food in the third image;

[0178] The first coordinate information and the second coordinate information are converted into first world coordinate information and second world coordinate information in the world coordinate system; the target feature vector corresponding to the first world coordinate information and the second world coordinate information is determined; based on the target symbol of the component of the preset direction in the target feature vector and the correspondence between the pre-saved symbol and the access method, the target access method corresponding to the target symbol is determined as the access method corresponding to the food contained in the current frame image.

[0179] In one possible implementation, the method further includes:

[0180] If it is determined that the food contained in the current frame image is the food to be stored, then a first candidate image of the same scene captured at the same time as the third image is obtained, and the depth images in the third image and the first candidate image are determined as target recognition images; if it is determined that the food contained in the current frame image is the food to be taken out, then a second candidate image of the same scene captured at the same time as the second image is obtained, and the depth images in the second image and the second candidate image are determined as target recognition images.

[0181] The storage location of the food is determined based on the depth value corresponding to the pixel center point of the region containing the food in the target recognition image, and the correspondence between the pre-saved storage location and the actual distance range of the preset direction.

[0182] In one possible implementation, the method further includes:

[0183] The coordinate information of each target pixel in the first region containing the food in the current frame image is determined in the third world coordinate information in the world coordinate system, and the coordinate information of each pixel in the second region containing the food in the target image is determined in the fourth world coordinate information in the world coordinate system.

[0184] For each target pixel in the current frame image, determine whether there is fourth coordinate information that matches the third coordinate information corresponding to the target pixel. If so, determine that the target pixel is a pixel in the food identification area of ​​the current frame image.

[0185] The food identification regions in the current frame image are determined based on the pixels in each food identification region;

[0186] The target food ingredient contained in the food ingredient recognition region of the current frame image is obtained by using a pre-trained food ingredient recognition model.

[0187] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0188] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0189] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0190] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0191] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for determining a region, characterized in that, The method includes: Based on the received current frame image, the difference in parameter value between each pixel in the current frame image and the corresponding pixel in the previous frame image is determined. This difference is used to determine whether the object corresponding to the pixel has moved. The current frame image is a color image or a depth image; the previous frame image and the current frame image are images acquired by the same image acquisition device; and the parameter value is a depth value. For each pixel in the current frame image, if the difference corresponding to the pixel is greater than a preset threshold, and the number of pixels with a difference greater than the preset threshold among the pixels adjacent to the pixel is greater than a preset number threshold, then the pixel is determined to be a target pixel in the first region containing food in the current frame image. Based on each target pixel in the current frame image, determine the first region in the current frame image that contains the food ingredients; The method further includes: Based on the coordinate information of each target pixel in the current frame image, determine the second target coordinate information of the pixel center point of the first region; determine the first distance between the second target coordinate information and the coordinate information of the pixel center point of the first region containing the food in the previous frame image of the current frame image; Determine whether the first distance exceeds a preset distance threshold. If so, acquire a target image of the same scene captured at the same time as the current frame image. Wherein, if the current frame image is a color image, the target image is a depth image; if the current frame image is a depth image, the target image is a color image. Based on the parameter values ​​of each pixel in the target image and the parameter values ​​of the corresponding pixel in the previous frame image of the target image, determine the second region containing food in the target image, and determine the third target coordinate information of the pixel center point of the second region. Determine the second distance between the third target coordinate information and the coordinate information of the pixel center point of the region containing food in the previous frame image of the target image. If the second distance does not exceed the preset distance threshold, determine the fourth target coordinate information corresponding to the coordinate information of each pixel in the region containing food in the target image in the current frame image, and update the target pixels in the first region containing food in the current frame image using the pixel at the fourth target coordinate information.

2. The method according to claim 1, characterized in that, After determining that the pixel is a target pixel in the first region containing food in the current frame image, and before determining the first region containing food in the current frame image based on each target pixel in the current frame image, the method further includes: Remove target pixels located outside the preset area, and for the remaining target pixels, perform the subsequent operation of determining the first region containing the food in the current frame image based on each target pixel in the current frame image.

3. The method according to claim 1, characterized in that, After determining that the pixel is a target pixel in the first region containing food in the current frame image, and before determining the first region containing food in the current frame image based on each target pixel in the current frame image, the method further includes: Based on the coordinate information of each target pixel in the current frame image, the first target coordinate information of the pixel center point of the first region is determined. Target pixels that are more than a preset distance threshold away from the first target coordinate information are removed. For the remaining target pixels, the subsequent operation of determining the first region containing food ingredients in the current frame image based on each target pixel in the current frame image is performed.

4. The method according to claim 1, characterized in that, The method further includes: If both the first distance and the second distance exceed the preset distance threshold, it is determined that the food tracking has failed. Then, the first image with the closest time interval to the current frame image and which has successfully tracked the food is found. The first region containing the food in the first image is used to update the first region containing the food in the current frame image.

5. The method according to claim 4, characterized in that, The method further includes: If a preset number of consecutive images fail to track the food, the corresponding tracking process will end. Find the second image in each frame of the tracking process corresponding to the current frame image, the first image in which the region containing food was identified, and the third image in which the region containing food was identified last; obtain the first coordinate information of the pixel center point of the region containing food in the second image, and the second coordinate information of the pixel center point of the region containing food in the third image; The first coordinate information and the second coordinate information are converted into first world coordinate information and second world coordinate information in the world coordinate system; the target feature vector corresponding to the first world coordinate information and the second world coordinate information is determined; based on the target symbol of the component of the preset direction in the target feature vector and the correspondence between the pre-saved symbol and the access method, the target access method corresponding to the target symbol is determined as the access method corresponding to the food contained in the current frame image. The method further includes: If it is determined that the food contained in the current frame image is the food to be stored, then a first candidate image of the same scene captured at the same time as the third image is obtained, and the depth images in the third image and the first candidate image are determined as target recognition images; if it is determined that the food contained in the current frame image is the food to be taken out, then a second candidate image of the same scene captured at the same time as the second image is obtained, and the depth images in the second image and the second candidate image are determined as target recognition images. The storage location of the food is determined based on the depth value corresponding to the pixel center point of the region containing the food in the target recognition image, and the correspondence between the pre-saved storage location and the actual distance range of the preset direction.

6. The method according to claim 1, characterized in that, The method further includes: The coordinate information of each target pixel in the first region containing the food in the current frame image is determined in the third world coordinate information in the world coordinate system, and the coordinate information of each pixel in the second region containing the food in the target image is determined in the fourth world coordinate information in the world coordinate system. For each target pixel in the current frame image, determine whether there is fourth coordinate information that matches the third coordinate information corresponding to the target pixel. If so, determine that the target pixel is a pixel in the food identification area of ​​the current frame image. The food identification regions in the current frame image are determined based on the pixels in each food identification region; The target food ingredient contained in the food ingredient recognition region of the current frame image is obtained by using a pre-trained food ingredient recognition model.

7. A region determination device, characterized in that, The device includes: The determining module is used to determine, based on the received current frame image, the difference in parameter values ​​between each pixel in the current frame image and the corresponding pixel in the previous frame image, so as to determine whether the object corresponding to the pixel has moved based on the determined difference in each pixel in the current frame image. The current frame image is a color image or a depth image; the previous frame image and the current frame image are images acquired by the same image acquisition device; and the parameter values ​​are depth values. The processing module is configured to, for each pixel in the current frame image, determine that if the difference corresponding to the pixel is greater than a preset threshold, and the number of pixels with a difference greater than the preset threshold among the adjacent pixels is greater than a preset number threshold, then determine that the pixel is a target pixel in the first region containing food in the current frame image; and determine the first region containing food in the current frame image based on each target pixel in the current frame image. The processing module is further configured to: determine second target coordinate information of the pixel center point of the first region based on the coordinate information of each target pixel point in the current frame image; determine a first distance between the second target coordinate information and the coordinate information of the pixel center point of the first region containing food in the previous frame image of the current frame image; determine whether the first distance exceeds a preset distance threshold, and if so, acquire a target image of the same scene captured at the same time as the current frame image; wherein, if the current frame image is a color image, the target image is a depth image, and if the current frame image is a depth image, the target image is a color image; and determine the second target coordinate information of the first region containing food in the current frame image based on the coordinate information of each target pixel point in the current frame image. Based on the parameter values ​​of the corresponding pixels in the previous frame of the target image, a second region containing food ingredients in the target image is determined, and third target coordinate information of the pixel center point of the second region is determined. A second distance is determined between the third target coordinate information and the coordinate information of the pixel center point of the region containing food ingredients in the previous frame of the target image. If the second distance does not exceed the preset distance threshold, then the fourth target coordinate information corresponding to the coordinate information of each pixel in the region containing food ingredients in the target image is determined in the current frame image, and the target pixels in the first region containing food ingredients in the current frame image are updated using the pixel at the fourth target coordinate information.

8. An electronic device, characterized in that, The electronic device includes at least a processor and a memory, the processor being configured to implement the steps of the region determination method as described in any one of claims 1-6 when executing a computer program stored in the memory.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the region determination method as described in any one of claims 1-6.

Images