Volume estimation in liquid handling

Abstract

In one embodiment, a computer-implemented method (100) of estimating a volume of a treated liquid product in a container for a liquid treatment device is described. The method includes receiving (102) first image data corresponding to a view of the treated liquid product in the container after a first ingredient has been treated by the liquid treatment device. The method also includes segmenting (104) a portion of the first image data corresponding to a surface level of the treated liquid product in the container from a background of the treated liquid product. The method also includes determining (106) a dimension of the surface level of the treated liquid product in the container from the segmented portion of the first image data. The method also includes estimating a first volume of the treated liquid product based on the dimension. The first volume is estimated according to a predetermined relationship between a dimension of a given surface level and a known liquid volume held by the container at the given surface level.

Description

Technical Field

[0001] The present invention relates to a computer-implemented method for estimating the volume of a processed liquid product, a non-transitory machine-readable medium, and a liquid processing apparatus. Background Technology

[0002] A growing trend in kitchen appliances is towards enhanced (so-called "smart") functionality to meet consumers' increasing demand for improved monitoring and control of food and liquid preparation. In the case of liquid handling devices such as blenders or juicers, consumers may require improved monitoring during use. Some smart liquid handling devices may have a degree of capability to analyze the contents of the liquid. These devices can rely entirely on artificial intelligence (AI) technology to perform this analysis. However, the computational cost of running such analyses is high, increasing the cost for consumers to purchase and operate the liquid handling device. Furthermore, AI models require extensive training, increasing development costs. Some liquid handling devices may include hardware components as part of a digital scale to measure the weight of contents added by the consumer. However, such hardware components can increase the complexity of the liquid handling device.

[0003] US11478766B2 discloses a mixing system including a mixer base and a container. The mixer base includes a housing that houses a motor. The container is attachable to the mixer base. The mixing system includes a user device that communicates with the mixer base. The user device can communicate with a remote computing device. The user device generates instructions and recipes for the mixer base.

[0004] US2022 / 160160A1 discloses a smart juicer system including a juicer configured to produce green vegetable juice by extracting juice from input ingredients. The juicer is also configured to generate juice extraction information including at least one of the following: time information regarding juice extraction, identification information regarding the input ingredients undergoing juice extraction, a juice extraction-based formula, and juice extraction preparation conditions. A terminal processing device includes: a user interface for receiving input regarding whether a user has consumed green vegetable juice; a communication module for data communication with a juicer; and a terminal processor configured to generate information about a user's history of green vegetable juice consumption based on intake information and juice extraction information obtained from the juicer via the communication module.

[0005] CN111568180A discloses a water dispenser display control method, a water dispenser, and a computer-readable storage medium. The method includes the following steps: acquiring a container image of a container placed below the outlet of the water dispenser; controlling the water dispenser to release water based on the container image, and acquiring information on changes in the water volume within the container; generating a real-time water volume image based on the water volume change information, and projecting the real-time water volume image onto a display screen.

[0006] JPH10132641A discloses a method for inspecting the liquid volume in a transparent container having an approximately conical shoulder. The method includes determining the boundary of the liquid level from an image obtained by photographing the shoulder of the bottle, binarizing the image based on a threshold, obtaining the area of ​​the liquid level from the image, and determining whether the area is within a specific allowable range. If the area is not within the allowable range, the bottle is deemed defective. Summary of the Invention

[0007] There are situations where it is necessary to measure the quantity of contents in a container of a liquid handling device. While components such as digital scales can be incorporated into the liquid handling device to measure the weight of the contents, this can increase the complexity of the liquid handling device. Certain aspects or embodiments described herein relate to improving how the quantity of contents in a container of a liquid handling device is measured. Certain aspects or embodiments can reduce or eliminate certain problems associated with existing solutions.

[0008] In a first aspect of the invention, a computer-implemented method is provided for estimating the volume of a treated liquid product in a container of a kitchen liquid handling apparatus. The method includes receiving first image data corresponding to a view of the top surface of the treated liquid product in the container after a first component has been processed by the liquid handling apparatus. The method further includes segmenting a portion of the first image data corresponding to the top surface of the treated liquid product in the container from the background of the treated liquid product.

[0009] The method further includes determining the dimensions of the segmented portions of the first image data. The method also includes estimating a first volume of the processed liquid product based on these dimensions, wherein the first volume is estimated according to a predetermined relationship between the dimensions of a given top surface and a known liquid volume held by a container on the given top surface.

[0010] The following describes some embodiments related to the first aspect and other aspects.

[0011] In some embodiments, dimensions include length or area derived from the segmented portion.

[0012] In some embodiments, the segment is segmented by: determining image parameter values ​​of pixels in the region of interest of the first image data; comparing the image parameter values ​​with a threshold indicating the presence of the processed liquid product; and in response to the comparison indicating that the pixel corresponds to the processed liquid product, indicating that the pixel is mapped to the segmented portion.

[0013] In some embodiments, the threshold is identified based on one or more of the following: the color and texture of the processed liquid product.

[0014] In some embodiments, the threshold is based on the average image parameter value in the region of interest of the first image data.

[0015] In some embodiments, the region of interest is identified in a reference image acquired when there is no ingredient in the container. The region of interest may correspond to a set of pixels that is mapped to a reference component of the liquid handling apparatus, which is no longer visible when the container contains the processed liquid product.

[0016] In some embodiments, the reference component includes one or more of the following: a seal for sealing a container, a blade for a liquid handling device, or a support module for holding the blade in the container.

[0017] In some embodiments, the method further includes: estimating a first volume of the processed liquid product; receiving input indicating the type of a first component; identifying nutritional information of the first component from a database; and determining the nutrient content of the processed liquid product based on the identified nutritional information and the estimated first volume of the processed liquid product.

[0018] In some embodiments, the received input includes additional image data corresponding to a view of the first component acquired before processing the first component, and wherein machine vision is used to determine the type of the first component.

[0019] In some embodiments, the received input includes user input indicating the type of the first component.

[0020] In some embodiments, the method further includes receiving second image data corresponding to a view of the top surface of a treated liquid product in a container after the second component has been processed by a liquid handling apparatus. The method may further include segmenting a portion of the second image data corresponding to the top surface of the treated liquid product in the container from the background of the treated liquid product. The method may further include determining the size of the segmented portion of the second image data. The method may further include estimating a second volume of the treated liquid product based on the size, wherein the second volume is estimated according to a predetermined relationship.

[0021] In some embodiments, the method further includes determining the nutrient content of the treated liquid product based on the nutritional information of the second component and the estimated second volume of the treated liquid product.

[0022] In some embodiments, the nutrient content of the processed liquid product is determined based on: nutrient information of a first component and an estimated first volume of the processed liquid product; and nutrient information of a second component and the volume difference between the estimated second volume and the estimated first volume.

[0023] In a second aspect of the invention, a non-transitory machine-readable medium is provided. This non-transitory machine-readable medium stores instructions that can be read and executed by a processor to implement the methods of any of the first and related embodiments.

[0024] In a third aspect of the invention, a liquid processing apparatus is provided for determining the nutrient content of a processed liquid product. The liquid processing apparatus includes: a container for receiving ingredients; a blade for processing the ingredients in the container; a motor system for driving a cutting element; a camera for capturing images of the ingredients; and a controller. The controller is configured to implement the methods of any of the first and related embodiments.

[0025] Certain aspects or embodiments described herein may provide various technical benefits, such as: improved techniques for estimating the quantity, particularly volume, of a treated liquid product in a container; increased accuracy in estimating the volume of a treated liquid product while reducing the burden on consumers (e.g., in terms of user input levels); reduced complexity in estimating the volume of a treated liquid product; and other technical benefits as discussed herein.

[0026] These and other aspects of the invention will become apparent and will be elucidated with reference to one or more embodiments described below. Attached Figure Description

[0027] Exemplary embodiments of the invention will now be described by way of example only with reference to the following figures, wherein:

[0028] Figure 1 Refers to a method for estimating the volume of a processed liquid product according to one embodiment;

[0029] Figure 2 This is a schematic diagram of a liquid treatment ecosystem according to one embodiment;

[0030] Figure 3 This is a schematic diagram of a liquid handling apparatus according to one embodiment;

[0031] Figure 4Figures (A) to (D) are schematic diagrams depicting the stages of using a liquid handling device;

[0032] Figure 5 Figures (A) to (C) are schematic diagrams of the interior of a container in a liquid handling apparatus, with or without contents.

[0033] Figure 6 Figures (A) to (D) are schematic diagrams of segmented image data showing views of the interior of a container in a liquid handling apparatus, wherein the contents of the container increase in level continuously;

[0034] Figure 7 Refers to a method for determining the nutrient content of a processed liquid product according to one embodiment;

[0035] Figure 8 Refers to a method for estimating the volume and nutrient content of a processed liquid product according to one embodiment;

[0036] Figure 9 These are schematic diagrams of a machine-readable medium for implementing various embodiments; and

[0037] Figure 10 This is a schematic diagram of an apparatus for implementing various embodiments. Detailed Implementation

[0038] Consumers may require improved measurement of the amount of processed liquid product in containers of liquid handling devices. However, measuring this amount can be highly complex once the ingredients are in the container. For example, hardware components, such as a digital scale, can be integrated with the liquid handling device. In another example, AI technology can be used to perform the analysis. However, these example technologies involve complex hardware and / or software implementations.

[0039] Therefore, there is a need for an improved technique for measuring the amount of processed liquid product in a container of a liquid handling device.

[0040] Figure 1 This refers to a computer-implemented method 100 for estimating the volume of a processed liquid product in a container for a liquid processing apparatus, according to one embodiment. The liquid processing is performed using a liquid processing apparatus, such as a blender, juicer, etc. Such liquid processing apparatuses can be freestanding or handheld. Ingredients with high liquid content, such as fruits and vegetables, can be liquid-processed to release their liquid content, thereby producing a processed liquid product. Examples of such processed liquid products include smoothies, juices, or soups.

[0041] Method 100 can be implemented by, for example, a (electronic) processor within the liquid handling device itself or by another processor located remotely from the liquid handling device, as described in more detail below. The blocks of method 100 are described in more detail below.

[0042] Method 100 includes, in block 102, receiving first image data corresponding to a view of a processed liquid product in a container after the first component has been processed by a liquid processing device.

[0043] As described in more detail below, image data can be acquired by a camera. Image data may include a single image or a sequence of images. In some cases, image data may include raw imaging data acquired by the camera. In some cases, image data may be processed in some way (e.g., compressed or converted to another color space format). In some cases, the processor implementing method 100 may receive image data directly from the camera implementing method 100. In some cases, image data may be stored in a memory communicatively coupled to the processor (either within the liquid handling device itself or in another entity remote from the liquid handling device). In such cases, the processor receives the image data from the memory before implementing method 100.

[0044] The view of a processed liquid product can refer to the camera's field of view. The inner walls of the container and other components of the liquid handling device (such as blades, seals, etc.) are visible in the view at certain times. The camera is configured to make changes in the contents of the container visible in the view. That is, image data can be recorded as the consumer fills the container with ingredients during use.

[0045] Image data may include a set of image parameter values. Each image parameter value refers to a value such as intensity recorded by a pixel of the camera. Each pixel maps to a location in the view. Therefore, a camera may have a set of pixels, and each of these pixels may record an image parameter value representing the view at the time the image is acquired. Image parameter values ​​may refer to the intensity and / or color recorded by the pixel. Image parameter values ​​may represent image parameter values ​​under a color model such as Red-Green-Blue (RGB) or YUV color models or any other suitable color model, where the parameter may be a color component (or color axis) of the color model.

[0046] In the use of the liquid processing device, a consumer can add a first ingredient or have the first ingredient added to a container. The ingredient added to the container can be solid (e.g., fruit or vegetable) or liquid (e.g., juice, water, soup, etc.), although this disclosure states throughout that the ingredient added to the container is initially in solid form. After the first ingredient has been processed from solid to liquid form, an image of the liquid-processed first ingredient, visible in view, can be acquired by a camera. The corresponding first image data is then received according to box 102. The term "first" does not imply that no other images were acquired prior to the first image data. In some cases, other image data may be acquired prior to the first image data, as described below.

[0047] Method 100 further includes, in block 104, segmenting a portion of first image data corresponding to the surface level of the treated liquid product in the container from the background of the treated liquid product.

[0048] When the first component has been processed into a liquid form, the surface of the processed liquid product can be horizontal (or substantially horizontal, depending on the surface texture caused by any present bubbles, fibers, etc.). A portion of the first image data (corresponding to a subset of the camera's pixel group) is mapped to the area of ​​the processed liquid product visible in the view (i.e., horizontal on the surface). A segmentation procedure can be implemented to distinguish the subset of pixels mapped to that portion (i.e., the processed liquid product) from the remaining subset of pixels mapped to the background (e.g., any visible inner wall of the container or any other component of the liquid processing device). Such a segmentation procedure can be based on non-AI-based techniques, such as Otsu-based methods (for automatic image thresholding) or AI-based techniques. Some implementations of a segmentation procedure using image thresholding are described below.

[0049] Method 100 further includes, in block 106, determining the dimensions of the surface level of the processed liquid product in the container from the segmented portion of the first image data.

[0050] The size of the segmented portion (or the size of the image surrounding the segmented portion) indicates how much processed liquid product is in the container. As the container is filled with the processed liquid product, the angular size of the segmented portion changes in the image data (i.e., if the container contains a different volume after the last image is acquired). Therefore, it is possible to determine how much processed liquid product is in the container based on the size of the segmented portion recorded from the image data.

[0051] In some embodiments, dimensions include length or area derived from the segmented portion. For example, if the segmented portion is circular, the length may refer to the diameter of the segmented portion, or another straight line measured from one edge of the segmented portion to the other. The area may refer to the total area of ​​the segmented portion. In any case, the number of pixels mapped to the length or area may indicate the size of the segmented portion. Thus, in some cases, determining the dimension may refer to determining the number of pixels (length or area) in the image data that indicate the angular size of the surface level of the processed liquid product as apparent to the camera.

[0052] Method 100 further includes, in block 108, estimating a first volume of the processed liquid product based on the dimensions. The first volume is estimated according to a predetermined relationship between the dimensions of a given surface level and a known liquid volume held by a container of the given surface level.

[0053] A predetermined relationship can be established based on knowledge of the volumetric capacity of a container at a given surface level, which can be determined theoretically or experimentally. A transfer function corresponding to the predetermined relationship can be established, indicating the volume held by the container at a given surface level. In other words, the dimensions of the surface level of the processed liquid product in the container are mapped to a given volume. As the container is filled, the dimensions change, thus indicating the volume held by the container according to the predetermined relationship.

[0054] In this way, an improved technique is provided for measuring the amount (in this case, volume) of processed liquid product in a container of a liquid handling apparatus.

[0055] The method 100 described herein and certain other embodiments may provide one or more technical benefits, such as those described below, and may be understood with reference to the entire disclosure.

[0056] The technical benefits of method 100 and / or related embodiments may include providing an improved technique for estimating the quantity and volume of processed liquid products (e.g., due to the robustness and / or simplicity of the technique, such as compared to existing solutions that rely on all technologies or require complex hardware components). Another technical benefit of method 100 and / or related embodiments may be improved accuracy in estimating the volume of processed liquid products while reducing the burden on consumers (e.g., in terms of user input levels). Yet another technical benefit of method 100 and / or related embodiments may be reduced hardware and / or software complexity required for estimating the volume of processed liquid products compared to existing solutions such as those based on AI technology or involving the use of additional hardware components. Kitchen appliances are becoming increasingly intelligent to meet growing consumer demands. Such kitchen appliances may already include cameras for acquiring images. Method 100 and / or related embodiments can utilize the already available image acquisition capabilities provided by intelligent liquid handling devices to estimate volume without requiring modification to the hardware of the liquid handling device. Therefore, method 100 and / or related embodiments can be implemented in deployed liquid handling devices that are already in use by consumers (e.g., by implementing software or firmware updates to install the functionality of method 100 and / or related embodiments).

[0057] Potential applications of estimating the volume of a treated liquid product could include facilitating nutrient tracking and / or flavor personalization. Method 100 and / or related embodiments can facilitate such nutrient tracking and / or flavor personalization. The ability to determine the amount of ingredients added to a container can be useful to consumers, as this reduces the burden on consumers to track quantities themselves (e.g., in the case of consumers using their own scales). Consumers can expect to know the nutrient content of the treated liquid product. Method 100 and / or related embodiments can facilitate accurate estimation of the nutrient content of the treated liquid product based on nutritional information of the added ingredients (e.g., per unit volume). Nutritional information can be obtained from food composition databases, such as the USDA National Nutrient Database, as a standard reference. Therefore, in some cases, Method 100 and / or related embodiments can facilitate the estimation of the amount of treated liquid product in a container, allowing nutrient content to be calculated without the need for weighing, which can reduce the burden on consumers.

[0058] The following describes some embodiments related to method 100.

[0059] Figure 2This is a schematic diagram of a liquid handling ecosystem 200 according to one embodiment. Certain embodiments described herein (e.g., method 100 and related embodiments) can be implemented in certain parts of the liquid handling ecosystem 200. The liquid handling ecosystem 200 depicts a variety of devices and entities that can be deployed as part of the liquid handling ecosystem 200. As described below, not every device or entity depicted is required in all scenarios.

[0060] Ecosystem 200 includes a liquid processing device 202 for processing component 204 (e.g., the first component and any other components subsequently added). Liquid processing device 202 includes a controller 206 for monitoring and controlling the liquid processing. For example, controller 206 can control the blades (not shown) of liquid processing device 202 (e.g., controlling the cutting / stirring / extraction process of liquid processing device 202). Controller 206 is communicatively coupled to a camera 208 for capturing images. Camera 208 is positioned such that the region of interest associated with component 204 is within the field of view of camera 208. This particular configuration is an example. For example, camera 208 may or may not be inside liquid processing device 202, but even if camera 208 is outside liquid processing device 202, component 204 is still within its field of view.

[0061] In some cases, the liquid handling ecosystem 200 includes a cloud computing service 210 communicatively coupled to the controller 206. The cloud computing service 210 can provide data storage and / or data processing services. The cloud computing service 210 can provide computing resources when there are insufficient available computing resources in any connected device. In some cases, the cloud computing service 210 can provide updates and other services to the liquid handling unit 202.

[0062] In some cases, the liquid handling ecosystem 200 includes user equipment 212 communicatively coupled to controller 206. User equipment 212 can refer to any computing device associated with a user (e.g., a user of liquid handling device 202). Examples of user equipment 212 include smartphones, smartwatches, tablets, Internet of Things (IoT) devices, etc. In some cases, user equipment 212 may be communicatively coupled to cloud computing service 210.

[0063] Any or a combination of controller 206, cloud computing service 210, and user equipment 212 can be used to implement method 100 and other embodiments described herein. For example, in some cases, controller 206 can implement method 100 and related embodiments. In this respect, controller 206 may include a (electronic) processor (not shown) for implementing computer-implemented method 100 and related embodiments. In other cases, processors associated with various devices and entities of the liquid handling ecosystem 200 can implement method 100 and related embodiments.

[0064] Figure 3 This is a schematic diagram of a liquid processing apparatus 300 for processing components according to one embodiment. The liquid processing apparatus 300 may implement the functions of some of the embodiments described herein, such as regarding... Figure 1 The method 100 describes certain features of the liquid handling apparatus 300, which may correspond to... Figure 2 The liquid handling device 202 has the characteristics or has the same as Figure 2 The liquid handling device 202 has similar functions.

[0065] The liquid handling apparatus 300 includes a container 302 for receiving component 304. The liquid handling apparatus 300 also includes blades 306 for processing (e.g., cutting / stirring / extracting) the component 304 in the container 302. The liquid handling apparatus 300 also includes a motor system 308 for driving the blades 306. The liquid handling apparatus 300 can function as a blender, juicer, etc. The liquid handling apparatus 300 also includes a camera 310 for capturing images of the component 304 (i.e., images of a “view” associated with the component 304). In some cases, the camera 310 may be integrated into the lid (not shown) of the container 302, such that the camera 310 is positioned to image the interior of the container 302. In some cases, the camera 310 is positioned externally to the container 302 (e.g., mounted in the handle of the container 302 or externally to the container 302). In any case, the camera 310 is configured to image the interior of the container 302.

[0066] The liquid handling device 300 also includes a controller 312, such as a controller corresponding to... Figure 2 The controller 206. In this embodiment, the controller 312 is configured to implement method 100. In another embodiment, the controller 312 is configured to implement an embodiment related to method 100.

[0067] Therefore, in the implementation of method 100, the controller 312 is configured to receive first image data corresponding to a view of the processed liquid product in the container after the first component has been processed by the liquid processing device 300.

[0068] The controller 312 is also configured to segment a portion of first image data corresponding to the surface level of the processed liquid product in container 302 from the background of the processed liquid product.

[0069] The controller 312 is also configured to determine the dimensions of the surface level of the processed liquid product in the container 302 based on the segmented portion of the first image data.

[0070] The controller 312 is also configured to estimate a first volume of the processed liquid product based on this dimension. The first volume is estimated according to a predetermined relationship between the dimension of a given surface level and the known liquid volume held by the container 302 at the given surface level.

[0071] although Figure 3 The method 100 describes the implementation of the controller 312 of the liquid handling apparatus 300, but in some cases, other devices or entities (such as...) Figure 2 The described method (and related embodiments) can implement at least some of the functions of method 100 (and related embodiments).

[0072] Figure 4 Figures (A) to (D) are schematic diagrams depicting the stages of using a liquid handling apparatus for method 100 and related embodiments. References are made in the following description. Figure 3 The liquid handling apparatus 300 is depicted. The four stages (labeled A to D) of using the liquid handling apparatus 300 indicate the following processes: (A) adding a first component to the container, (B) performing liquid handling, (C) adding a second component to the container, and (D) performing liquid handling again. Although two components are added, the liquid handling apparatus can handle fewer or more components depending on the consumer's needs.

[0073] In stage (A), the consumer adds the first ingredient to container 302. In some cases, the type of the first ingredient can be input by the user (e.g., via the user interface of the liquid handling device or other user equipment associated with the consumer). In other cases, the type of the first ingredient can be determined by acquiring an image of the first ingredient (before the first image data described in method 100) and performing image recognition (e.g., using AI techniques such as convolutional neural networks trained to identify ingredient types).

[0074] In stage (B), the first component is liquid-treated until the (top) surface of the treated liquid product is level (i.e., flat or substantially flat). Camera 310 acquires an image corresponding to a view of the surface of the treated liquid product in container 302. The acquired image corresponds to the first image data involved in method 100. The (first) volume of the treated liquid product can then be calculated according to method 100. The nutrient content of the treated liquid product can be determined based on the type of the first component (i.e., nutrient content per unit volume) and the nutrient information of the first volume.

[0075] Repeat the above steps for the second component.

[0076] In stage (C), the consumer adds the second ingredient to container 302. The amount and type of the second ingredient can be determined as described in stage (A).

[0077] In stage (D), the second component is liquid-processed in the manner described in stage (B), and the total volume of the resulting processed liquid product (i.e., the second volume) can be estimated. The volume corresponding to the second component is determined by calculating the difference between the second volume estimated in stage D and the first volume estimated in stage B. Since the volume corresponding to each component of the processed liquid product has been estimated and the nutrient information (nutrient content per unit volume) is known, the cumulative nutrient content of the processed liquid product can be determined. For example, the cumulative nutrient content can be determined by summing the nutrient content per unit volume for each added component. The same principle applies to any number of components, provided that an image is acquired and the volume is estimated after each different component is added to container 302.

[0078] Figure 5 Figures (A) to (C) are schematic diagrams of the interior of a container in a liquid processing apparatus, with or without contents (i.e., processed liquid product). The image data referred to herein represents, for example... Figure 5 The view shown. (Compared to) Figure 3 The reference numerals for features similar to or corresponding to those of the liquid handling device 300 are increased by 200.

[0079] exist Figure 5In each of Figures (A) to (C), the inner wall of container 502 is visible because a camera (not shown) is positioned within the lid (not shown) of container 502 and configured to image the bottom interior region and inner wall of container 502. The depth of field of the camera can be well controlled so that objects from the bottom to the top of container 502 are clearly distinguishable in the image data. In this case, the wall of container 502 includes corrugations, such that the internal cross-section of container 502 in a plane perpendicular to the imaging axis of the camera is non-circular. This container shape is an example; other container shapes are possible. For example, container 502 can be cylindrical, truncated conical, or any other shape, as long as the camera can image changes in volume maintained by container 502.

[0080] exist Figure 5 In (A), container 502 is empty, making the components of the liquid handling apparatus at the bottom of container 502 visible. In this case, reference component 514 (e.g., a seal) at the bottom of container 502 is visible. Reference component 514 may have a color (such as blue) that facilitates identification of the area of ​​interest, as described in more detail below. As an example, reference component 514 may include any component, such as a seal for sealing container 502, the blade 506 itself, or a support module (not shown, but may include the bottom interior region of container 502 when coupled to a motor system (not shown) for holding the blade 506 in container 502 and coupling the blade 506 to the motor system (not shown) of the liquid handling apparatus.

[0081] exist Figure 5 In (B), the treated liquid product 516 is present in container 502. Assuming the treated liquid product 516 is opaque, components at the bottom of container 502 are no longer visible because they are obscured by the treated liquid product 516 at its surface level. The dashed line corresponds to the location of reference component 514.

[0082] exist Figure 5 In (C), a portion 518 of the first image data corresponding to the surface level of the processed liquid product 516 in container 502 is segmented from the background of the processed liquid product 516. In this way, remaining features of container 502, such as ripples, are no longer visible in the segmented image data. Pixels mapped to the segmented portion 518 can be analyzed to determine dimensions (e.g., length or area). In the area example, the total number of pixels corresponding to the segmented portion 518 corresponds to the area. Therefore, the number of pixels can be used to estimate the volume based on a predetermined relationship of container 502.

[0083] Now for reference Figure 5 This describes an implementation method for estimating the volume of a processed liquid product.

[0084] The internal area at the bottom of container 502 (see) Figure 5 (A)) Within the view captured by the camera. A region of interest (ROI) can be defined in the image data, which maps to the interior region at the bottom of the container. The visibility of a reference component (such as a colored seal) within the ROI can indicate the absence of the processed liquid product in container 502 (i.e., container 502 is empty). When the processed liquid product 516 is present, the visibility of the reference component is reduced or completely blurred, and the pixel intensity values ​​of the set of pixels mapped to the ROI are used to determine a threshold (e.g., a threshold image parameter value, such as pixel intensity value). Therefore, the characteristics of the processed liquid product in the ROI, such as color, texture, etc., can be used to determine the segmentation threshold. Thus, the set of pixels mapped to the reference component can define the ROI within the image data. When the processed liquid product 516 is present in container 502, an appropriate threshold for the processed liquid product 516 can be determined based on the characteristics of the processed liquid product 516 (e.g., color, texture). For example, the average pixel intensity value of the pixels in the ROI of the first image data can provide the threshold. In this way, the threshold can be appropriately selected in response to processed liquid products that have different colors or other appearance characteristics (due to being made from different components with different colors).

[0085] After each component is added (once the liquid processing is completed after the component is added), the acquired image data is segmented (e.g., using the thresholds described above).

[0086] A transfer function (representing a predetermined relationship) between the size of the segmented portion 518 (e.g., the number of pixels corresponding to the size of the segmented portion 518) and the volume held by the container 502 with a given surface level can be obtained. The surface level refers to the height of the processed liquid product 516 within the container 502. Therefore, the surface height of the processed liquid product 516 within the container 502 is related to the volume of the processed liquid product 516. In some cases, the transfer function can be obtained based on theoretical knowledge of the internal shape of the container 502 and the expected size of the surface level for a given volume. In other cases, the transfer function can be obtained experimentally by taking images of liquids with different known volumes in the container 502 and fitting a curve to the resulting data (i.e., volume as a function of the number of pixels corresponding to the segmented portion 518).

[0087] The volume of the processed liquid product 516 in container 502 can then be estimated based on the dimensions derived from the segmented portion 518 of the image data and the transfer function (i.e., the number of pixels corresponding to the length or area).

[0088] Assuming a clear segmentation of the treated liquid product 516 (e.g., if the sides of container 502 are not coated with the treated liquid product 516 and / or the surface of the treated liquid product 516 is uneven), the volume can be estimated with high accuracy based on the transfer function (accurately with an error of less than 3% according to experiments). It is assumed that the camera position is fixed. It is also assumed that an appropriate threshold has been obtained for performing the segmentation.

[0089] Another embodiment for estimating the volume of the processed liquid product 516 will now be described. Depending on the embodiment, some steps may be omitted.

[0090] The received image data is in the YUV color space or has been converted to the YUV color space. A color channel (e.g., the U channel) is selected to obtain a monochrome channel image (i.e., img-U). Other color spaces such as RGB, Lab, and those for texture information can be used in a similar manner to perform the same type of analysis.

[0091] If necessary, the resulting image img-U is filtered (e.g., using a Gaussian filter) to reduce noise (e.g., using a kernel of size 5×5 pixels) to obtain a filtered image (i.e., img-gs).

[0092] The resulting image (img-gs) is binarized using an appropriate threshold (e.g., based on the characteristics of the processed liquid product 516 itself) to obtain the surface-level contour of the processed liquid product 516. A binarized image is obtained where the pixel intensity value of the foreground pixels (corresponding to pixels mapped to the processed liquid product 516) is 255, and the pixel intensity value of the background pixels (corresponding to pixels mapped to the background of the processed liquid product) is 0. Since the threshold is related to features of the processed liquid product 516 such as color or texture, the threshold is determined based on the appearance of the processed liquid product. For example, color can affect the accuracy of segmentation, so an average value based on the actual appearance (i.e., color) of the processed liquid product 516 can be used to determine an appropriate threshold. The region of interest corresponding to the bottom interior region of the liquid processing device can be predefined (e.g., by the manufacturer of the liquid processing device) or determined based on the appearance of a reference component as described above. The average value of image parameter values ​​registered in the region of interest, for example, in the U channel, can be used as the threshold. The surface-level contour of the processed liquid product 516 can be obtained through threshold-based segmentation.

[0093] Some embodiments related to these implementations will now be described.

[0094] Therefore, in some embodiments, the segmented portion 518 is segmented by: determining image parameter values ​​of pixels in the region of interest of the first image data; comparing the image parameter values ​​with a threshold indicating the presence of the processed liquid product 516; and, in response to the comparison indicating that the pixel corresponds to the processed liquid product 516, indicating that the pixel is mapped to the segmented portion 518. The image parameter values ​​can be any component from the YUV color space or any other suitable color space. The classification of a pixel as corresponding to the surface or background of the processed liquid product 516 can be based on thresholding techniques, such as Otsu's method.

[0095] In some embodiments, the threshold is identified based on one or more of the following: the color and texture of the processed liquid product.

[0096] In some embodiments, the threshold is based on the average image parameter value in the region of interest of the first image data. The term "average" may refer to "mean", "median", or "mode".

[0097] In some embodiments, the region of interest is identified in a reference image acquired when there is no component in container 502. The region of interest corresponds to a set of pixels that is mapped to a reference component of the liquid handling apparatus, which is no longer visible when the processed liquid product 516 is present in container 502.

[0098] In some embodiments, the reference component includes one or more of the following: a seal for sealing the container 502, a blade 506 of a liquid handling device, or a support module for holding the blade 506 in the container 502.

[0099] Figure 6 Figures (A) to (D) are schematic diagrams of segmented image data showing views of the interior of a container of a liquid processing apparatus, wherein the level of the processed liquid product in the container of the liquid processing apparatus increases continuously. Figure 6 (A) shows a view of the bottom interior area of ​​the container before any ingredients are added. Figure 6 (A) corresponds to things like about Figure 5 (A) An image of the reference component 614 as described. Figure 6 Views (B) to (D) correspond to the views taken when the first, second, and third components have been added to the container and the liquid is processed after each addition of each component. That is, Figure 6 (B) to (D) correspond to images of the processed liquid product after image segmentation. The level of the processed liquid product increases after each component is added, thereby increasing the size of the segmented portion 618, as shown below. Figure 6The image sequence shown in (B) to (D) depicts the process. The size of the segmented portion 618 can be used to derive the dimensions for estimating the volume based on a predetermined relationship (i.e., a transfer function). The predetermined relationship can be verified by taking a set of images of a container filled with a known volume of liquid and checking the accuracy using the determined dimensions (e.g., the number of pixels corresponding to the segmented portion 618). As described above, using the above technique to establish the predetermined relationship, the (relative) error was found to be less than 3%.

[0100] Some other embodiments are described below.

[0101] Figure 7 Refers to method 700 for determining the nutrient content of a processed liquid product according to one embodiment. Method 700 can be implemented in the same manner as described with respect to method 100 (i.e., using the same processor). Where appropriate, certain blocks of method 700 can be implemented in any order, and are not limited thereto. Figure 7 The order described.

[0102] Method 700 includes, in block 108, estimating a first volume of the processed liquid product (i.e., according to method 100).

[0103] Method 700 also includes, in block 702, receiving input indicating the type of the first component. As described above, this type of input can be based on user input or machine vision.

[0104] Method 700 also includes, in box 704, identifying nutritional information of the first component from a database. This database can provide nutritional information per unit volume of the first component, such as energy, fat, protein, vitamin and / or mineral content. A processor implementing method 700 can access the database, for example, via the Internet.

[0105] Method 700 further includes, in block 706, determining the nutrient content of the treated liquid product based on the identified nutrient information and the estimated first volume of the treated liquid product.

[0106] In some embodiments, the received input includes additional image data corresponding to a view of the first component acquired prior to processing the first component. Machine vision can be used to determine the type of the first component.

[0107] In some embodiments, the received input includes user input indicating the type of the first component. Such user input can be entered via the user interface of the liquid handling device or another user device such as a smartphone application (app).

[0108] Figure 8Refers to method 800 for estimating the volume and nutrient content of a treated liquid product according to one embodiment. Method 800 can be implemented in the same manner as described with respect to method 100 (e.g., using the same processor). Where appropriate, certain blocks of method 800 can be implemented in any order and are not limited thereto. Figure 8 The order shown. Method 800 can be executed after method 100.

[0109] Method 800 includes, at block 802, receiving second image data corresponding to a view of a treated liquid product in a container after the second component has been processed by a liquid handling apparatus.

[0110] Method 800 further includes, in block 804, segmenting a portion of second image data corresponding to the surface level of the treated liquid product in the container from the background of the treated liquid product.

[0111] Method 800 further includes, in block 806, determining the dimensions of the surface level of the treated liquid product in the container from the segmented portion of the second image data.

[0112] Method 800 further includes, at block 808, estimating a second volume of the processed liquid product based on the size. The second volume is estimated according to a predetermined relationship.

[0113] In some embodiments, method 800 further includes, at block 810, determining the nutrient content of the treated liquid product based on nutritional information of the second ingredient and an estimated second volume of the treated liquid product. Block 810 can be implemented using the same functionality (relative to the first ingredient) described in method 700 to determine the nutrient content of the treated liquid product after the addition of the second ingredient.

[0114] In some embodiments, the nutrient content of the treated liquid product is determined based on: nutrient information of the first component and an estimated first volume of the treated liquid product; and nutrient information of the second component and the volume difference between the estimated second volume (i.e., the total volume obtained from processing the first and second components) and the estimated first volume (obtained only from processing the first component).

[0115] The embodiments associated with the segmentation procedure for the first image data can also be applied to the segmentation procedure for the second image data (and any subsequent images). The principles of these embodiments can be extended to adding any number of components, as long as each component is added one by one, and volume estimation is performed based on the fresh image obtained after liquid processing is performed after each addition of each component.

[0116] Figure 9This is a schematic diagram of a non-transitory machine-readable medium 900 for implementing the various embodiments described herein. As used herein, the term "non-transitory" does not include transient propagation signals. The machine-readable medium 900 stores instructions 902 that can be read and executed by a processor 904 to implement methods (e.g., method 100 and / or related embodiments) of any of the embodiments described herein. The machine-readable medium 900 and / or processor 904 may be... Figure 2 or Figure 3 It may be implemented by any one of the controller 206, cloud computing service 210, user equipment 212 and / or controller 312.

[0117] Figure 10 This is a schematic diagram of an apparatus 1000 for implementing the various embodiments described herein. The apparatus 1000 may be composed of... Figure 2 or Figure 3 It may be implemented by any one of the controller 206, cloud computing service 210, user equipment 212 and / or controller 312.

[0118] Device 1000 includes processor 1002. Processor 1002 is configured to communicate with interface 1004. Interface 1004 can be any interface (wireless or wired) implementing a communication protocol to facilitate data exchange (e.g., image data, control commands for the liquid handling device, etc.) with other devices, such as another part of the liquid handling ecosystem 200.

[0119] The apparatus 1000 also includes a memory 1006 (e.g., non-transitory or otherwise) storing instructions 1008 that can be read and executed by the processor 1002 to implement the various embodiments described herein (e.g., method 100 or any of the associated embodiments).

[0120] Any model described herein can be implemented by a processing circuitry device for implementing the methods described herein. Therefore, certain blocks of the method may involve using such models to provide the stated functionality. The model can be (machine learning) ML-based or non-ML-based. However, some embodiments described herein involve using non-ML-based models, which avoids the need for large amounts of computational resources and / or local processing.

[0121] Although the invention has been described in detail in the accompanying drawings and the foregoing description, such descriptions are to be considered illustrative or exemplary, and not restrictive; the invention is not limited to the disclosed embodiments.

[0122] One or more features described in one embodiment may be combined with or replaced with features described in another embodiment.

[0123] The embodiments in this disclosure may be provided as a method, a system, or as a combination of machine-readable instructions and processing circuitry. Such machine-readable instructions may be included on a non-transitory machine (e.g., computer) readable storage medium (including, but not limited to, disk storage, CD-ROM, optical storage, flash memory, etc.) having computer-readable program code thereon or thereon.

[0124] This disclosure is described with reference to flowchart illustrations and block diagrams of methods, apparatus, and systems according to embodiments of this disclosure. Although the flowchart illustrations show a specific execution sequence, the execution sequence may differ from the depicted sequence. A block described in one flowchart may be combined with a block in another flowchart. It should be understood that each block in a flowchart and / or block diagram, as well as combinations of blocks in flowcharts and / or block diagrams, can be implemented using machine-readable instructions.

[0125] Machine-readable instructions can be executed, for example, by a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device, to implement the functions described in the specification and figures. Specifically, a processor or processing circuitry device, or a module thereof, can execute machine-readable instructions. Therefore, the functional modules of the devices and other apparatuses described herein can be implemented by a processor that executes machine-readable instructions stored in memory or by a processor that operates according to instructions in an embedded logic circuitry device. The term 'processor' should be interpreted broadly to include CPUs, processing units, ASICs, logic units, or programmable gate arrays, etc. These methods and functional modules can be executed by a single processor or divided among several processors.

[0126] Such machine-readable instructions can also be stored in a computer-readable storage device that can instruct a computer or other programmable data processing device to operate in a specific mode.

[0127] Such machine-readable instructions can also be loaded onto a computer or other programmable data processing device to cause the computer or other programmable data processing device to perform a series of operations to produce computer-implemented processing. Thus, the instructions that execute on the computer or other programmable device implement the functions specified by the boxes (one or more) in the flowchart and / or block diagram.

[0128] Furthermore, the teachings herein can be implemented in the form of a computer program product stored in a storage medium and including a plurality of instructions for causing a computer device to implement the methods described in the embodiments of this disclosure.

[0129] The following embodiments are also disclosed:

[0130] Example 1. A computer-implemented method (100) for estimating the volume of a processed liquid product in a container of a liquid processing apparatus, the method comprising:

[0131] Receive (102) first image data, which corresponds to a view of the processed liquid product in the container after the first component has been processed by the liquid processing device;

[0132] A portion of the first image data corresponding to the surface level of the processed liquid product in the container is segmented from the background of the processed liquid product (104);

[0133] (106) Determine (108) the size of the surface level of the processed liquid product in the container from the segmented portion of the first image data; and estimate (109) the first volume of the processed liquid product based on the size, wherein the first volume is estimated according to a predetermined relationship between the size of the given surface level and the known liquid volume held by the container of the given surface level.

[0134] Example 2. A computer-implemented method according to Example 1, wherein dimensions include lengths or areas derived from the segmented portions.

[0135] Example 3. A computer-implemented method according to any one of Examples 1 to 2, wherein the portion is divided in the following manner:

[0136] Determine the image parameter values ​​of pixels in the region of interest of the first image data;

[0137] Compare the image parameter values ​​with thresholds indicating the presence of the processed liquid product; and

[0138] In response to a comparison indicating that the pixel corresponds to the processed liquid product, the pixel is indicated to be mapped to the segmented portion.

[0139] Example 4. A computer-implemented method according to Example 3, wherein the threshold is identified based on one or more of the following: the color and texture of the processed liquid product.

[0140] Example 5. A computer-implemented method according to any one of Examples 3 to 4, wherein the threshold is based on the average image parameter value in the region of interest of the first image data.

[0141] Example 6. A computer-implemented method according to any one of Examples 3 to 5, wherein a region of interest is identified in a reference image acquired when there is no ingredient in the container, and wherein the region of interest corresponds to a set of pixels that is mapped to a reference component of a liquid processing apparatus, which is no longer visible when the container contains the processed liquid product.

[0142] Example 7. A computer-implemented method according to Example 6, wherein the reference component includes one or more of the following: a seal for sealing a container (502), a blade (506) of a liquid handling device, or a support module for holding the blade in the container.

[0143] Example 8. A computer-implemented method (700) according to any one of Examples 1 to 7, comprising:

[0144] Estimate (108) the first volume of the processed liquid product;

[0145] Receive (702) an input indicating the type of the first component;

[0146] Nutritional information of the first component identified from the database (704); and

[0147] Based on the identified nutritional information and the estimated first volume of the processed liquid product, the nutrient content of the processed liquid product is determined (706).

[0148] Example 9. A computer-implemented method according to Example 8, wherein the received input includes additional image data corresponding to a view of the first component acquired before processing the first component, and wherein machine vision is used to determine the type of the first component.

[0149] Example 10. A computer-implemented method according to Example 8, wherein the received input includes user input indicating the type of a first component.

[0150] Example 11. A computer-implemented method (800) according to any one of Examples 1 to 10, comprising:

[0151] Receive (802) second image data, which corresponds to a view of the processed liquid product in the container after the second component has been processed by the liquid processing device;

[0152] A portion of the second image data corresponding to the surface level of the processed liquid product in the container is segmented from the background of the processed liquid product (804);

[0153] (806) Determine the surface level dimension of the processed liquid product in the container from the segmented portion of the second image data; and estimate (808) a second volume of the processed liquid product based on the dimension, wherein the second volume is estimated according to a predetermined relationship.

[0154] Example 12. A computer-implemented method according to Example 11, comprising determining (810) the nutrient content of the processed liquid product based on nutritional information of a second component and an estimated second volume of the processed liquid product.

[0155] Example 13. A computer-implemented method according to Example 12, wherein the nutrient content of the processed liquid product is determined based on the following:

[0156] Nutritional information of the first component and the estimated first volume of the processed liquid product; and

[0157] Nutritional information of the second component and the volume difference between the estimated second volume and the estimated first volume.

[0158] Example 14. A non-transitory machine-readable medium (900) storing instructions (902) that can be read and executed by a processor (904) to implement the method according to any one of Examples 1 to 13.

[0159] Example 15. A liquid processing apparatus (300) for determining the nutrient content of a processed liquid product, the liquid processing apparatus comprising:

[0160] Container (302) for receiving ingredients (304);

[0161] Blade (306) for processing ingredients in a container;

[0162] A motor system (308) for driving the cutting element;

[0163] Camera (310), used to capture images of the components; and

[0164] Controller (312), which is configured to:

[0165] Receive first image data, which corresponds to a view of the processed liquid product in the container after the first component has been processed by the liquid processing device;

[0166] A portion of the first image data corresponding to the surface level of the processed liquid product in the container is segmented from the background of the processed liquid product;

[0167] The dimensions of the surface level of the processed liquid product in the container are determined from the segmented portion of the first image data; and

[0168] The first volume of the processed liquid product is estimated based on the size, wherein the first volume is estimated according to a predetermined relationship between the size of a given surface level and the known liquid volume held by a container of the given surface level.

[0169] Elements or steps described with respect to one embodiment may be combined with or replaced by elements or steps described with respect to another embodiment. Other variations of the disclosed embodiments can be understood and implemented by those skilled in the art in practicing the claimed invention by studying the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude multiple. A single processor or other unit may perform the functions of several items listed in the claims. The fact that certain measures are recited only in mutually different dependent claims does not indicate that combinations of these measures cannot be used advantageously. Computer programs may be stored or distributed on suitable media, such as optical storage media or solid-state media provided with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems. No reference numerals in the claims should be construed as limiting the scope.

Claims

1. A computer-implemented method (100) for estimating the volume of a processed liquid product in a container of a kitchen liquid handling apparatus, the method comprising: Receive (102) first image data, the first image data corresponding to a view of the top surface of the treated liquid product in the container after the first component has been treated by the liquid processing device; A portion of the first image data corresponding to the top surface of the processed liquid product in the container is segmented from the background of the processed liquid product (104). Determine (106) the size of the segmented portion of the first image data; as well as The first volume of the processed liquid product is estimated based on the size estimate (108), wherein the first volume is estimated according to a predetermined relationship between the size of the given top surface and the known liquid volume held by the container of the given top surface.

2. The computer-implemented method of claim 1, wherein the dimension includes the length or area of ​​the segmented portion.

3. The computer-implemented method according to any one of claims 1 to 2, wherein the portion is divided in the following manner: Determine the image parameter values ​​of pixels in the region of interest of the first image data; The image parameter values ​​are compared with thresholds indicating the presence of the processed liquid product; and In response to the comparison indicating that the pixel corresponds to the processed liquid product, the pixel is indicated to be mapped to the segmented portion.

4. The computer-implemented method of claim 3, wherein the threshold is identified based on one or more of the following: the color and texture of the processed liquid product.

5. The computer-implemented method of claim 3, wherein the threshold is based on the average image parameter value in the region of interest of the first image data.

6. The computer-implemented method of claim 3, wherein the region of interest is identified in a reference image acquired when there is no ingredient in the container, and wherein the region of interest corresponds to a set of pixels mapped to a reference component of the liquid processing apparatus, the reference component being no longer visible when the processed liquid product is present in the container.

7. The computer-implemented method according to claim 6, wherein the reference component comprises one or more of the following: a seal for sealing the container (502), a blade (506) of the liquid handling device, or a support module for holding the blade in the container.

8. The computer-implemented method (700) according to any one of claims 1, 2, 4, and 7, comprising: Estimate (108) the first volume of the processed liquid product; Receive (702) an input indicating the type of the first component; Nutritional information of the first component is obtained from the database identifier (704); as well as Based on the identified nutritional information and the estimated first volume of the treated liquid product, the nutrient content of the treated liquid product is determined (706).

9. The computer-implemented method of claim 8, wherein the received input includes additional image data corresponding to a view of the first component acquired prior to processing the first component, and wherein machine vision is used to determine the type of the first component.

10. The computer-implemented method of claim 8, wherein the received input includes user input indicating the type of the first component.

11. The computer-implemented method (800) according to any one of claims 1, 2, 4, 7, 9, and 10, comprising: Receive (802) second image data, the second image data corresponding to a view of the top surface of the treated liquid product in the container after the second component has been processed by the liquid processing device; A portion of the second image data corresponding to the top surface of the processed liquid product in the container is segmented from the background of the processed liquid product (804). Determine (806) the size of the segmented portion of the second image data; as well as The second volume of the processed liquid product is estimated based on the size estimate (808), wherein the second volume is estimated according to the predetermined relationship.

12. The computer-implemented method of claim 11, comprising determining (810) the nutrient content of the processed liquid product based on nutritional information of the second component and an estimated second volume of the processed liquid product.

13. The computer-implemented method of claim 12, wherein the nutrient content of the processed liquid product is determined based on: Nutritional information of the first component and the estimated first volume of the processed liquid product; and The nutritional information of the second component and the volume difference between the estimated second volume and the estimated first volume.

14. A non-transitory machine-readable medium (900) storing instructions (902) that can be read and executed by a processor (904) to implement the method according to any one of claims 1 to 13.

15. A kitchen liquid handling device (300), the liquid handling device comprising: Container (302), the container being used to receive ingredients (304); Blade (306), the blade being used to process the components in the container; Motor system (308), the motor system being used to drive the blade; Camera (310), the camera being used to capture images of the components; as well as Controller (312), the controller is configured to: Receive first image data, the first image data corresponding to a view of the top surface of the treated liquid product in the container after the first component has been processed by the liquid processing device; A portion of the first image data corresponding to the top surface of the processed liquid product in the container is segmented from the background of the processed liquid product; Determine the size of the segmented portion of the first image data; and The first volume of the processed liquid product is estimated based on the dimensions, wherein the first volume is estimated according to a predetermined relationship between the dimensions of a given top surface and the known liquid volume held by the container of the given top surface.

Images