Bottle analysis system

Abstract

The bottle analysis system receives image data of the bottle to be analyzed, and this data is processed to identify the shape of the bottle, and optionally, any identifying marks are added. The bottle type is then determined. Image analysis is used to determine the liquid level in the bottle, thereby determining the volume of liquid in the bottle.

Description

Technical Field

[0001] This invention relates to the analysis of bottles, and particularly to bottles for feeding infants or collecting milk when using a breast pump. Background Technology

[0002] Baby bottles come in many different types, shapes, and sizes.

[0003] It is known that when bottle-feeding, mothers are interested in monitoring the amount of milk their infants are drinking and the amount of milk collected during breast pump use. The volume of milk intake and the type of bottle used are important for newborns and toddlers. Overfeeding or underfeeding is a common problem faced by caregivers and parents. Studies have shown that overfeeding leads to vomiting, diarrhea, weight loss, and in some cases, death. Underfeeding leads to excessive jaundice, severe dehydration, and hypoglycemia that can cause brain damage.

[0004] With continued use and washing, the volume markings on baby bottles tend to fade, making it difficult to manually assess the volume. People also often find it difficult to distinguish between different bottle types.

[0005] The applicant has proposed a system for automated milk management using a sleeve that is mounted around a bottle and includes a weighing scale to weigh the bottle before and after feeding the infant, thereby determining the actual amount of milk fed. The system is disclosed in WO2019 / 030029.

[0006] This requires users to use additional components. These components also need to be designed to fit a variety of bottle sizes and shapes.

[0007] US2010 / 097451 discloses a sensor that uses image analysis to sense contents (air bubbles or foreign objects) in a syringe. The image analysis can also detect the syringe type.

[0008] WO2020 / 117780 discloses a system for determining the liquid level in a container based on image analysis. This invention relates to determining the reagent level in a container used for testing patient samples.

[0009] US2011 / 093109 discloses the control of items on a production line and the use of image analysis to determine the volume of products and packaging on the production line.

[0010] A simpler system is needed that can monitor intake or the amount produced during the use of a breast pump. Summary of the Invention

[0011] This invention is defined by the claims.

[0012] According to one aspect of the present invention, a bottle analysis system is provided, comprising:

[0013] Input for receiving image data of the bottle to be analyzed; and

[0014] Processor, wherein the processor is adapted to:

[0015] Image analysis is used to identify at least the shape characteristics of the bottle, and thereby determine the bottle type;

[0016] Retrieve the additional shape characteristics of the bottle from a database that associates different bottle types with other shape characteristics; and

[0017] Image analysis is used to determine the liquid level in the bottle, and the volume of liquid in the bottle is determined based on the bottle type and other shape characteristics.

[0018] Image data, such as images, multiple images, or video streams, is used by the system to identify bottle types through image analysis. In this way, the system does not need to be adapted to a specific bottle, but it can identify the bottle at least from its shape. This shape is obtained, for example, based on aspect ratio and other shape characteristics, and preferably does not depend on absolute dimensions.

[0019] Bottle type is determined, for example, by accessing a pre-stored database of bottle types. For these pre-stored bottles, the bottle volume can be stored as data along with the volume for different liquid levels starting from the bottom of the bottle, or essentially as a function relating the liquid level to the corresponding volume. This information can be considered to include additional shape characteristics. For example, the initially identified shape characteristics can be scale-independent, while other shape characteristics are related to the specific scale of the bottle. Thus, once the bottle type is identified, the liquid level observed by image analysis can be simply converted into a volume based on the known 3D shape of the bottle. The detection of the liquid level is also scale-independent, i.e., it is the relative height between the bottom and top of the bottle, and therefore can be obtained from scale-independent image analysis.

[0020] The processor can also be adapted to use image analysis to identify the presence of any identifier in a predefined set of identifiers. Some bottles have known predefined identifiers, such as brand names or bottle type identifiers. Therefore, image analysis searches for these identifiers as well as the overall bottle shape. However, for the purposes of the system, these identifiers do not need to be applied to the bottle. Instead, they are, for example, existing surface marking features formed as part of a standard bottle design for identification via image analysis.

[0021] If the pre-identified bottle type cannot be identified, bottle shape characteristics can be identified based on 3D shape analysis.

[0022] By determining the volume of a liquid at multiple points in time, the volume of the liquid, particularly milk, can be tracked. Processors utilize computer vision techniques, for example, aided by pattern recognition and neural networks (which can be considered as implementations of artificial intelligence), as well as mathematical and statistical formulas and techniques.

[0023] Therefore, this invention provides simple integration with existing applications and enables a seamless infant feeding monitoring experience. Sharing historical data among multiple users (e.g., multiple caregivers) is straightforward, allowing them to track the amount of milk the infant is ingesting.

[0024] By using image analysis to determine at least one shape of a bottle, no special markings are required, and the bottle does not need to be positioned in any particular orientation relative to the image capture system. Shape information does not change with aging (e.g., surface scratches, or color changes (due to wear)). The same bottle type can be identified even with different caps. Image recognition enables the identification of bottle type and liquid level in a single step.

[0025] If a video stream is received as input to image data, it is, for example, converted into a separate image for subsequent image analysis.

[0026] The processor, for example, is adapted to use a neural network to determine the bottle type. The neural network can be trained using existing bottles and can include bottles from multiple manufacturers. Training is also based on identifier tags.

[0027] The processor can be adapted to use image analysis to:

[0028] Identify liquid surfaces; and optionally

[0029] Mark the liquid level on the surface of the bottle.

[0030] The liquid surface can be used to determine the volume, either as a function of the liquid level based on the known volume for a particular bottle (as described above), or by the alignment of an identifier with a liquid level marker (or based on both of the above).

[0031] The processor can be adapted to process first and second image data and determine changes in liquid volume. This allows intake to be monitored over time, or in practice, allows for the measurement of the amount of expressed milk when using a breast pump.

[0032] In a preferred embodiment, the system is used to monitor bottle feeding of an infant, such that changes in liquid volume correspond to changes in milk intake.

[0033] The system may also include memory for storing historical milk intake. This allows the mother or other caregiver to record the baby's feeding performance over time.

[0034] The processor can be adapted to determine the amount of milk to be fed to the infant based on historical milk intake and output the determined amount. This provides guidance to the mother or caregiver who is feeding the infant on the appropriate amount.

[0035] Processors can be adapted to output the determined quantity by generating an augmented reality image of the bottle representing the determined amount of breast milk. This provides an easy-to-follow guide for the mother or caregiver. In particular, they can fill the bottle to the level represented by the augmented reality.

[0036] The system may include a camera for capturing images or video streams. In one example, the system is implemented as a mobile phone or tablet, thus utilizing existing camera capabilities and processing power.

[0037] The present invention also provides a method for analyzing bottles, comprising:

[0038] Receive image data of the bottle to be analyzed;

[0039] Image analysis is used to identify at least the shape characteristics of the bottle, and thereby determine the bottle type;

[0040] The additional shape characteristics of the bottle are obtained by accessing a database that associates different bottle types with the additional shape characteristics; and

[0041] Image analysis is used to determine the liquid level in the bottle, and the volume of liquid in the bottle is determined based on the bottle type and the additional shape characteristics.

[0042] This method can be applied to various bottle types and sizes. It is simple for users because, if they want to determine changes in liquid volume, they can optionally simply capture images or videos of the bottle at different time points.

[0043] This method may include using image analysis to identify whether any of the identifiers in a predefined set of identifiers exist.

[0044] The method may include:

[0045] Using neural networks to determine the bottle type; and

[0046] Image analysis is used to identify liquid surfaces, and optionally, level markings on the bottle surface are also identified.

[0047] The method may include processing first image data and second image data and determining changes in liquid volume. The method is used, for example, to monitor bottle feeding of an infant, where changes in liquid volume correspond to milk intake, and the method further includes determining the amount of milk to be fed to the infant based on historical milk intake and outputting the determined amount.

[0048] The determined quantity is output by generating an augmented reality image of the bottle representing the determined quantity of milk.

[0049] The present invention also provides a computer program for implementing the above method.

[0050] These and other aspects of the invention will become apparent from the embodiments described below. Attached Figure Description

[0051] To better understand the invention and to more clearly illustrate how to implement it, reference will now be made to the accompanying drawings by way of example only, wherein:

[0052] Figure 1 The image shows a captured image of a nursing bottle and a nursing bottle on a smartphone screen;

[0053] Figure 2 The first method for analyzing the bottle is shown; and

[0054] Figure 3 It shows Figure 2 The development of methods in which the system determines how much milk to provide in the bottle based on the baby’s past feeding data. Detailed Implementation

[0055] The invention will be described with reference to the accompanying drawings.

[0056] It should be understood that while the detailed description and specific examples indicate exemplary embodiments of the apparatus, system, and method, they are for illustrative purposes only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the apparatus, system, and method of the present invention will become more readily apparent from the following description, the appended claims, and the accompanying drawings. It should be understood that the drawings are merely schematic and not drawn to scale. It should also be understood that the same reference numerals are used in all the drawings to denote the same or similar parts.

[0057] This invention provides a bottle analysis system in which image data (one or more images or video streams) of a bottle to be analyzed are received and processed to identify the shape and any markings on the bottle. The bottle type is then determined, and its characteristics are retrieved from a database. Image analysis is used to determine the liquid level in the bottle, thereby determining the volume of liquid in the bottle.

[0058] To use the system, for example, you can take photos or pictures of bottles such as baby bottles before and after feeding by a user. You can also take photos of empty bottles. The system can be implemented, for example, through software provided on a smartphone.

[0059] Figure 1 The image shows a captured image of the feeding bottle 10 on the screen of a smartphone 11. Using any combination of neural networks, computer vision, and mathematical techniques, the bottle type and the actual volume of liquid within it can be determined. This simplifies the dispensing and tracking of infant feedings while avoiding the need for separate, dedicated physical equipment or continuous human monitoring.

[0060] Figure 1 Various regions of interest identified based on image processing are shown, including the overall shape of the bottle in region 12, the shape of the bottom portion of the bottle in region 14, markings such as brand name and / or model number in region 16, and liquid level markings in region 18.

[0061] By acquiring information from all these sources (where available – in many cases, there may be no visible markings), the bottle type can be determined from a database of previously analyzed bottles. This step can be performed, for example, using empty bottles or any amount of liquid. The database provides additional information not obtained from image processing, particularly eliminating the need for scaling the initial image processing. At this stage, the bottle type, total bottle volume, and brand have been established using the database.

[0062] This information can then be provided to the user, for example, so that they know they are using the correct type of bottle before filling the breast milk.

[0063] Liquid levels can also be established by identifying liquid boundaries, or it can be determined whether the bottle is empty at a given stage. Regions of interest can be identified in still images, live video, or a set of images extracted from a video stream.

[0064] This determination is based on object detection techniques using computer vision and pattern recognition, including image preprocessing. Object detection is performed using neural networks trained with internally created and labeled data and image processing.

[0065] Volume calculation involves identifying the liquid boundary within the bottle, followed by post-processing techniques. Based on the known bottle type, the measured segment of the bottle's height corresponds to a known volume.

[0066] If the pre-stored bottle type is unavailable, image processing can determine the bottle's 3D shape (e.g., assuming a given wall thickness) and then mathematically derive the internal volume as a function of the liquid height. Thus, the volume can be determined using post-processing techniques employing image algorithms and / or mathematical formulas.

[0067] This may require additional images to identify pre-stored bottle types. Therefore, if the system cannot identify known bottle types, it can instruct the user to take multiple images from a set of perspectives to enable 3D shape determination.

[0068] However, for volume calculations, scaled information, i.e., absolute dimensions, is required. This can be achieved, for example, by asking the user to apply scaling to the bottle for capture by image processing.

[0069] However, a preferred approach is to train and update the algorithm using any new bottle type. In this case, the system always relies on identifying the bottle type and then deriving volume information from the database.

[0070] Therefore, the liquid volume can be determined by using mathematical techniques, such as boundary techniques and / or line detection algorithms, to calculate the volume based on a measurement of the ratio of the liquid height in the bottle to the actual bottle height. The ratio of liquid height to bottle height is not sensitive to scaling, so an absolute measurement of the liquid level is not required once the bottle type is known.

[0071] Alternatively, template overlay can be used to determine the volume of a feeding bottle by matching the detected volume with a matching template on the frame of the feeding bottle or an image.

[0072] Due to the bottle's transparency and the constantly changing, dynamic background, post-processing steps can be used to increase the accuracy required for the inspection process. If the bottle is held at an angle, the bottle type can still be easily detected, but the determination of volume is not accurate if liquid is present inside.

[0073] Therefore, users can be instructed to perform the analysis when the bottle is standing on a horizontal surface such as a table.

[0074] Once the volume is known, volume information can be presented and stored for users and / or for further research purposes.

[0075] Figure 2 A method for analyzing bottles is illustrated. The method begins in step 20. During this period, an image or video stream of the bottle to be analyzed is received.

[0076] In step 22, it is determined whether the received input includes an image. If so, the method proceeds to preprocessing step 24.

[0077] If the input does not include individual images, it is determined in step 23 whether a video stream has been received. If so, individual frames are extracted in step 26 before proceeding to preprocessing step 24. If the input is also not video, no suitable input has been received, and the method returns to the beginning.

[0078] Preprocessing, for example, involves applying grayscale, removing excessive noise, and extracting the bottle's position relative to an image that includes the nipple and bottom.

[0079] Following preprocessing, image analysis is used in step 28 to determine the bottle type. This involves at least identifying the shape of the bottle and any identifying markings present. For example, a bottle type may have a specific bottom shape and a specific closed edge through which the nipple protrudes. The bottle type is determined by referring to a bottle type database 25.

[0080] The bottle type is further supplemented with additional information from the database, which can be considered as additional shape characteristics that take into account the absolute dimensions of the specific bottle type being identified.

[0081] In step 30, image analysis is used to determine the liquid level in the bottle, thereby determining the liquid volume in the bottle. This involves identifying a line caused by the liquid surface against the inner wall of the bottle. If the bottle is fixed on a flat surface, this line is preferably flat and straight. However, tilt angles can be detected, and the liquid volume can still be determined based on known 3D shapes from a type database. Before the method ends in step 34, the volume information is output to the user in step 32.

[0082] Systems and methods are implemented, for example, as features of bottle-feeding tracking applications. These systems and methods can be used to assist in feeding monitoring and bottle detection when the user lacks the ability to do so correctly. Tracking volume over time can be used as part of infant development monitoring. This can be used in hospital settings to automate and reduce the workload of tracking feeding data.

[0083] Figure 3 It shows Figure 2 The development of methods in which the system determines how much milk a user should provide in a bottle based on the baby’s past feeding data.

[0084] In the specific example shown, augmented reality is used to present the amount of breast milk to the user. This allows for the personalization of feeding monitoring applications for individual infants. As the infant grows, the user changes the bottle, but the bottle identification method described above is adaptive enough to include all bottles and growth stages. This not only helps in recording how much breast milk the infant has consumed and should consume, but can also be used as a supplementary tool to record how much breast milk has been expressed for future use and when that breast milk should be fed to the infant.

[0085] Figure 3 A camera 40, as used in the example above, is shown, as are other sensors of a typical mobile phone that may also be used in the processing method, including a gyroscope 42, an accelerometer 44, and a magnetometer 46.

[0086] All this sensor information is provided to the processor, which executes step 48, which uses the type database 25 to determine the bottle type and volume in the same manner as described above.

[0087] Sensors are used to determine the bottle's position relative to a mobile phone (or other camera device), for example, based on the assumption that the bottle is vertically oriented. However, if the bottle is not vertically oriented, this can also be detected based on identifying the bottle's axis. The orientation of a non-vertical bottle can also be determined by knowing the camera's orientation and the bottle's axis observed in the camera image.

[0088] Magnetometer 46 is used to determine the true magnetic north of the mobile phone.

[0089] Then, accelerometer 44 is used to determine the mobile phone's orientation relative to the gravity vector, such as its longitudinal or lateral position relative to the gravity vector. This allows for the establishment of a foundation for the real world and a virtual world relative to the real world (for augmented reality).

[0090] The gyroscope 42 is used to reduce jitter and noise in the camera sensor, thereby making the positioning between the real-world view and the virtual world more accurate.

[0091] The additional sensor data used in processing step 48 can be used to compensate for the camera angle. For example, because the transparency shows the front and rear milk levels, a transparent bottle and the milk volume within it can result in the detection of multiple volume levels. Knowing the angle at which the image was captured helps determine which level to select during level detection.

[0092] Furthermore, sensors can be used as additional input to the algorithm to improve accurate bottle classification. Therefore, as mentioned above, the use of sensors can improve the ability to accurately detect bottle type and volume, as well as the ability to achieve image overlay in the case of augmented reality systems.

[0093] In step 50, the milk history database 52, which stores historical feeding information about an individual infant, is accessed. This is used to determine the desired amount of milk to be fed to the infant.

[0094] The use of augmented reality, for example, involves capturing a video sequence of a bottle. Then, in step 54, the determined desired amount of milk can be processed into an augmented reality overlay. The moving image of the bottle is then applied to 3D rendering software, with the desired amount of milk determined based on historical data superimposed on the bottle.

[0095] The rendered graphics are then sent for post-processing, where image algorithms provide resolution adaptation for the resolution used to capture the image and provide appropriate image processing to enhance the real-world image or frame.

[0096] Then, a combiner is used to provide real-world frames or images to the graphics rendering.

[0097] Calibration data from mobile device sensors is used in this way for augmented reality rendering. By providing additional contextual information to the image, the use of sensors reduces the complexity of the required image processing.

[0098] Then, the final image 58 is displayed in real time, visually showing how much milk should be filled into the bottle.

[0099] If automatic volume monitoring is not used, users can manually enter drinking information into the milk history database.

[0100] The camera can be mounted, for example, on a solid base such as a table, with the display facing the user and the camera facing the feeding bottle. Alternatively, a separate camera and display can exist, connected via a standard communication device, such as a mobile phone camera and a separate display.

[0101] Visual feedback can include information beyond the optimal amount of milk, such as instructions on maintaining the correct angle of the bottle while feeding the baby.

[0102] The system can be used in hospital settings, where it can replace temporary care provided by young parents. It can record and provide instructions regarding bottle volume and other feeding characteristics. Furthermore, the records can be subsequently analyzed by doctors or other medical personnel.

[0103] The system can be enhanced by adding tags to the bottles. This allows the system to identify individual bottles, and by providing timestamp functionality, it can be further enhanced to track bottles based on the date they were filled. Therefore, the system can have milk management capabilities.

[0104] By studying the accompanying drawings, the disclosure, and the appended claims, those skilled in the art can understand and implement variations of the disclosed embodiments in carrying out the claimed invention. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality.

[0105] As described above, the system utilizes a processor to perform data processing. The processor can be implemented in various ways, using software and / or hardware, to perform a variety of desired functions. The processor typically employs one or more microprocessors that can be programmed using software (e.g., microcode) to perform the desired functions. The processor can be implemented as a combination of dedicated hardware performing certain functions and one or more programmable microprocessors and associated circuitry performing other functions. Those skilled in the art can readily develop processors for performing any of the methods described herein.

[0106] Examples of circuits that may be used in various embodiments of the present invention include (but are not limited to) conventional microprocessors, application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0107] In various implementations, a processor may be associated with one or more storage media, such as volatile and non-volatile computer memories (e.g., RAM, PROM, EPROM, and EEPROM). The storage media may be encoded with one or more programs that, when executed on one or more processors and / or controllers, perform the required functions. The various storage media may be fixed within the processor or controller, or they may be transferable, allowing one or more programs stored thereon to be loaded into the processor.

[0108] A single processor or other unit can implement the functions described in the claims.

[0109] Computer programs can be stored / distributed on suitable media, such as optical or solid-state storage media provided with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems. (Optional)

[0110] If the term "suitable" is used in the claims or description, it should be noted that the term "suitable" is intended to be equivalent to the term "configured as". The fact that certain measures are recited in mutually different dependent claims does not mean that combinations of these measures cannot be used advantageously.

[0111] Any reference numerals in the claims should not be construed as limiting the scope.

Claims

1. A nursing bottle analysis system, comprising: Input used to receive image data of the nursing bottle to be analyzed; as well as Processor, wherein the processor is adapted to: (28) Use image analysis to identify at least the shape characteristics of the nursing bottle, and use the identified shape characteristics to determine the bottle type from a pre-stored database (25) of different nursing bottle types; Additional information about the nursing bottle is obtained from the database, which associates the different nursing bottle types with the additional information, wherein the additional information identifies the volume for different liquid levels starting from the bottom of the nursing bottle; as well as (30) Using image analysis to determine the liquid level in the nursing bottle, and using the determined liquid level, based on the nursing bottle type and the additional information taking into account the absolute size of the identified bottle, wherein the processor is adapted to use scaling-independent image analysis to identify the shape characteristics of the bottle and determine the liquid level, the scaling-independent image analysis not depending on the absolute size.

2. The system of claim 1, wherein the processor is further adapted to use the image analysis to identify whether any of the identifiers in a predefined set of identifiers exists.

3. The system according to claim 1 or 2, wherein the processor is adapted to use a neural network to determine the type of feeding bottle.

4. The system according to any one of claims 1 to 2, wherein the processor is adapted to use image analysis to identify liquid surfaces.

5. The system according to any one of claims 1 to 2, wherein the processor is adapted to process first image data and second image data and to determine changes in liquid volume.

6. The system of claim 5, wherein the system is used to monitor bottle feeding of an infant, wherein the change in liquid volume corresponds to the amount of milk ingested.

7. The system of claim 6, wherein the processor is adapted to determine (50) the amount of milk to be fed to the infant based on historical milk intake, and output the determined amount.

8. The system of claim 7, wherein the processor is adapted to output the determined quantity (54) by generating an augmented reality image of the nursing bottle representing the determined quantity of milk.

9. The system according to any one of claims 1, 2, 6 to 8, comprising a camera (40) for capturing the image data, the image data being captured as one or more image or video streams.

10. The system according to any one of claims 1, 2, 6 to 8, wherein the processor is adapted to determine the liquid volume using a function that associates the liquid level with a corresponding volume for a pre-stored type of nursing bottle.

11. The system of claim 6, wherein the system further comprises a memory (52) for storing historical milk intake.

12. A method for analyzing a nursing bottle, comprising: (26) Receive image data of the nursing bottle to be analyzed; Image analysis was used to at least identify the shape features of the feeding bottle; Access a pre-stored database of different nursing bottle types (25), and use at least the identified shape features to determine (28) the bottle type; The additional information of the nursing bottle is obtained by accessing a database that associates different bottle types with additional information, wherein the additional information identifies the volume for different liquid levels starting from the bottom of the nursing bottle; as well as (30) Using image analysis to determine the liquid level in the nursing bottle, and using the determined liquid level, based on the nursing bottle type and the additional information, the additional information taking into account the absolute size of the nursing bottle, wherein the image analysis used to identify the shape features of the nursing bottle and to determine the liquid level is an image analysis that is independent of absolute size and scaling.

13. The method of claim 12, comprising: Use neural networks to determine the type of feeding bottle; as well as Image analysis is used to identify liquid surfaces.

14. The method of claim 12 or 13, the method for monitoring bottle feeding of an infant, the method comprising processing first image data and second image data and determining a change in liquid volume corresponding to milk intake, wherein the method further comprises determining the amount of milk to be fed to the infant based on historical milk intake and outputting the determined amount.

15. The method of claim 14, wherein the method includes outputting the determined quantity by generating an augmented reality image of the nursing bottle representing the determined quantity of milk.

16. The method of claim 12 or 13, wherein, for a pre-stored type of nursing bottle, the function associates the liquid level with the corresponding volume.

17. A computer program product comprising computer program code, wherein when the program is run on a processor, the computer program code is adapted to implement the method according to any one of claims 12 to 16.

Images