System and method for providing assistance during bottle feeding

Abstract

A system provides assistance during bottle feeding. Video images of a subject bottle feeding an infant are captured and displayed. Using image analysis, reorientations of the bottle and / or the infant are determined that are needed to reach a desired bottle orientation and / or infant orientation. Reorientation instructions are provided in conjunction with the video images to assist the subject in reorienting the bottle and / or the infant to achieve the desired bottle orientation.

Description

Technical Field

[0001] This invention relates to bottle feeding of infants (e.g., babies), and in particular, to systems and methods for helping to properly hold a bottle and / or an infant. Background Technology

[0002] Bottle-feeding is a more complex process than many people realize. The angle at which the baby is fed during bottle feeding is crucial for ensuring the baby's comfort and allowing them to receive the correct amount of milk.

[0003] For example, incorrect feeding techniques, such as the wrong bottle angle, can cause too much air to enter the stomach, and in extreme cases, this can lead to choking or respiratory failure. Proper feeding technique means delivering all the milk correctly, reducing gas and colic, and helping to prevent overfeeding and underfeeding of the infant.

[0004] Colic can be caused, for example, by air in the digestive system. If the liquid level in the feeding bottle drops below a certain level and / or the feeding bottle is provided to the baby in a horizontal position, air can enter the nipple area of ​​the bottle. However, a horizontal or near-horizontal feeding position may be preferred (depending on the baby's orientation) because it more closely mimics the natural feeding position.

[0005] Therefore, the bottle (relative to the baby) has a relatively narrow optimal angle range for the baby, which may make it difficult for the mother or other caregivers to achieve successful feeding. Summary of the Invention

[0006] This invention is defined by the claims.

[0007] According to an example of one aspect of the invention, an image processor is provided for use in a system that provides assistance during bottle feeding, the image processor comprising:

[0008] Input, which is used to receive video images of the subject bottle-feeding the infant;

[0009] Output, for providing video images for display, wherein the image processor is adapted to:

[0010] Identify the baby within the video image;

[0011] Determine the infant's orientation;

[0012] Identify the bottle within the video image;

[0013] Determine the orientation of the bottle;

[0014] Determine the reorientation required for the bottle and / or baby to achieve the desired bottle orientation; and

[0015] Control one or more devices selected from the group consisting of: a display device for outputting reorientation instructions, and an audio output device for outputting reorientation instructions.

[0016] In one embodiment, the image processor is also adapted to control the display device to output the redirection command superimposed on the video image.

[0017] In one embodiment, the display device is selected from a group consisting of a display and an audio output device, and the image processor is also adapted to control the display device to output the redirection command superimposed on the video image.

[0018] The image processor performs image analysis to provide real-time assistance to the subject (e.g., the infant's mother or other caregiver) during bottle feeding. Specifically, the processor determines the desired bottle orientation and / or infant orientation and provides feedback to help the subject maintain the most appropriate bottle angle and / or infant orientation to properly hold the infant and bottle, thereby achieving successful feeding.

[0019] The desired bottle angle depends on the infant's orientation and the angle relative to the horizontal plane. Therefore, it may be necessary to adjust the bottle angle and / or the infant's or subject's orientation to achieve the desired relationship.

[0020] For example, if the baby is in a suitable position for feeding, the processor may only provide instructions to reorient the bottle. If the baby is not in a suitable position, the system may provide instructions to reorient the baby before providing instructions to reorient the bottle.

[0021] Reorientation commands allow users to be guided throughout the feeding process. The system also maintains subject engagement by providing positive reinforcement and constant audio and / or visual feedback.

[0022] In a hospital setting, this system can address staff shortages by allowing a single medical professional to remotely monitor multiple eating processes and provide video images to remote monitoring stations. Additionally, the medical professional can adjust the angle to provide optimal feedback in specific situations.

[0023] The image processor can be adapted to identify the orientation of a subject's hand in an image and determine the orientation of the bottle's long axis perpendicular to the orientation of the palm of the hand.

[0024] Therefore, bottle orientation can be derived from the subject's hand (since the subject's hand may make the bottle itself invisible). The subject can then understand that the system is calibrated based on the assumption that the bottle's long axis is perpendicular to the palm of the hand. This is a natural feeding posture, so the bottom of the bottle is supported in the palm of the hand. Therefore, bottle identification can include presenting orientation based on the identified orientation of the subject's hand holding the bottle.

[0025] The desired bottle orientation can correspond to an angle of inclination relative to the baby's body axis, which is within a predetermined range.

[0026] The processor is also adapted to identify the position of the infant's head and the position of the subject's feeding hand in the image.

[0027] By positioning the infant's head and hands, reorientation information can be provided as augmented reality, showing the subject the most appropriate angle of the bottle next to the subject's hand.

[0028] The processor can also be adapted to recognize the pose of the subject in the image.

[0029] This posture information is based on factors such as head position, arm and shoulder position, sitting or standing posture, and orientation (backward or forward). This posture information can help determine how to hold the baby and thus help determine the baby's body axis. It can also help determine the angle at which the bottle is held.

[0030] Therefore, as much information as possible is reliably extracted from the images to enable the assessment of the current bottle orientation and infant body orientation.

[0031] The processor can be adapted to output redirection instructions as an indication of the desired orientation on the display output.

[0032] This provides visual cues for desired orientation, giving subjects intuitive guidance on how to adjust their grip on the bottle or even their feeding position.

[0033] Reorientation instructions can include arrow indicators located near the bottle. These can indicate how orientation should be adjusted using visual overlays on the video image. Similar instructions can be provided to guide subjects in adjusting their position and how they hold the baby.

[0034] The processor can be adapted to output redirection instructions using audio signals. Audio signals can be used additionally or alternatively, so that subjects do not need to look at a display to benefit from feedback information.

[0035] The present invention also provides a system for providing assistance during bottle feeding, the system comprising:

[0036] Video images of subjects bottle-feeding infants were captured;

[0037] Identify babies within video images;

[0038] Determine the infant's orientation;

[0039] Identify bottles within video images;

[0040] Determine the orientation of the bottle;

[0041] Determine the reorientation required for the bottle and / or baby to achieve the desired bottle orientation; and

[0042] Outputting reorientation instructions by performing one or more activities selected from the group consisting of: displaying output reorientation instructions, and outputting reorientation instructions using audio signal output, to help the subject reorient the bottle and / or baby to the desired bottle and / or baby orientation.

[0043] In this embodiment, the video image is displayed in conjunction with the redirection command.

[0044] In one embodiment, a display is selected from the group consisting of output and display, and the video image is displayed in conjunction with the redirection command.

[0045] The method may include identifying the orientation of the subject's hand in an image and determining that the orientation of the bottle's long axis is perpendicular to the orientation of the hand.

[0046] The desired bottle orientation corresponds, for example, to an angle of inclination relative to the baby's body axis within a predetermined range.

[0047] The method may also include recognizing the subject's pose in the image. The method may include providing an output reorientation instruction as an indication of the desired orientation on the display output.

[0048] The present invention also provides a computer program product, including instructions that, when the program is run on a processor, are used to implement the above-described method.

[0049] The present invention also provides a computer-readable medium comprising the above-described computer program product.

[0050] These and other aspects of the invention will become apparent from the embodiments described below. Attached Figure Description

[0051] To better understand the invention and to more clearly illustrate how to implement it, reference will now be made to the accompanying drawings by way of example only, wherein:

[0052] Figure 1The sequence of image processing steps used in the system of the present invention is shown;

[0053] Figure 2 The system's hardware components are shown; and

[0054] Figure 3 Methods for providing assistance during bottle feeding are shown. Detailed Implementation

[0055] The invention will be described with reference to the accompanying drawings.

[0056] It should be understood that while the detailed description and specific examples indicate exemplary embodiments of the apparatus, system, and method, they are for illustrative purposes only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the apparatus, system, and method of the present invention will become more readily apparent from the following description, the appended claims, and the accompanying drawings. It should be understood that the drawings are merely schematic and not drawn to scale. It should also be understood that the same reference numerals are used in all the drawings to denote the same or similar parts.

[0057] This invention provides a system for providing assistance during bottle feeding. Video images of a subject bottle-feeding an infant are captured and displayed. Using image analysis, the reorientation of the bottle and / or infant is determined as the reorientation required to achieve the desired bottle orientation and / or infant orientation. Reorientation instructions are provided in conjunction with the video images to assist the subject in reorienting the bottle and / or infant to achieve the desired bottle orientation.

[0058] Figure 1 The sequence of image processing steps used in the system of the present invention is shown.

[0059] Video images 10 are captured of subject 12 (mother or caregiver) bottle-feeding an infant, such as baby 13. The images can be captured by a mobile device such as a mobile phone. The process can begin by the caregiver holding the infant in front of the mobile device's camera to initiate the recognition process, after which the subject can assume a feeding posture. Image processing can then include tracking the positions of the infant and subject after the initial recognition. However, image processing can alternatively identify all necessary features without any initialization steps. The mobile device then begins acquiring live video data.

[0060] Then, image preprocessing steps are performed, such as grayscale conversion based on the image.

[0061] In step 14, frame extraction of the image is performed, and in step 16, 3D pose estimation is performed to at least identify the infant's pose. In one example, the subject's pose is also determined. Specifically, at least the orientation of the subject's hand relative to the bottle and the infant's pose are determined.

[0062] The subject's posture information also helps determine the infant's orientation (three dimensions), as infant orientation can be partially derived from the subject's posture.

[0063] However, if image processing can locate and analyze the bottle and the baby with sufficient accuracy, the subject's pose information may not be required, as it is primarily used to deduce the orientation of the bottle, especially since the bottle may be obscured and invisible.

[0064] Pose estimation, for example, involves running frames via a neural network to extract 2D key points from each frame. This allows for the identification of key locations, such as the infant's face and body, and the subject's actual hand holding the bottle.

[0065] For example, subjects are encouraged to hold the bottle in a specific way, allowing the system to hypothesize the relationship between the orientation of the subject's bottle-feeding hand and the long axis of the bottle. For instance, the long axis of the bottle is assumed to be perpendicular to the palm of the hand, i.e., perpendicular to the overall plane of the hand.

[0066] For example, when a subject holds the bottle to the camera, the correct bottle orientation relative to the hand can be shown to the subject (making the system's assumptions valid), so that the user can hold the bottle correctly.

[0067] Key points and postures of the subject and infant can be converted into a map represented by Figure 18. This can include the subject's head, hands, arms, neck, etc., the infant's body orientation, and observed or assumed bottle orientation. Figure 18 schematically shows the infant's long axis and the line between the mother's head and the infant's head at right angles.

[0068] Image 20 shows a magnified view of the region of interest, including the baby's head, the subject's hand, and the bottle.

[0069] Appropriate image processing algorithms for identifying and tracking features of interest in video data are well known and are outside the scope of this application.

[0070] In step 22, the optimal bottle angle and / or orientation for the subject is determined.

[0071] In the simplest case, the subject's posture is the appropriate feeding posture, so the optimal bottle angle is determined for the subject's specific posture.

[0072] In step 24, a visualization of the optimal bottle angle orientation is generated. This visualization is shown as image 26, which represents the reorientation instructions used to position the bottle.

[0073] In more advanced systems, the system can first assess whether the subject's posture (and the infant's posture) is suitable for bottle feeding. If not, reorientation instructions or training information can be presented to the subject to improve how they hold the infant during bottle feeding.

[0074] These instructions, for example, use arrows and dotted lines to indicate how much the bottle should be moved from its current position. Similar instructions can be provided to suit the subject's posture or the infant's orientation, or training videos can be provided for this purpose.

[0075] After post-processing the redirection instructions in step 27 to make them compatible with the combination of video images, the redirection instructions are combined with the original video images in real time to overlay the instructions onto the actual image. This is shown in image 28.

[0076] Reorientation instructions can be based on 3D visualizations using graphics, such as dashed lines showing how the bottle should be held accurately. Audio cues can also be provided to reinforce that the bottle should be moved up, down, or at a different angle if the subject cannot frequently view the screen. Augmented reality can be used, for example, to provide optimal visualization.

[0077] In specialized settings like hospitals, the system can also alert healthcare staff monitoring remotely within the hospital if the feeding location (for the baby and / or bottle) is not optimized within a specific timeframe, allowing for timely action to support and educate parents.

[0078] Figure 2 Hardware components for a system used to assist during bottle feeding, enabling the process described above, are shown. A camera system 30, which may include a single camera, multiple cameras, or a 3D camera, is used to capture video images of the subject bottle-feeding the infant. The camera system can be a fixed camera, or it can be a camera on a mobile device such as a tablet computer or a phone, which the user faces.

[0079] Display 32 is used to display video images. The display can be a display of the same mobile device, or it can be a separate display that sends images to it, for example, via short-range communication such as Bluetooth. An audio output system is also provided, such as including a speaker of the same mobile device.

[0080] Image processor 34 performs the above-described processing. The image processor can simply be a processor in a mobile device that functions as a camera and optionally also as a display. Therefore, the present invention can be implemented as an app to be loaded onto a tablet or telephone. However, the processing can alternatively be performed remotely, for example, using a local device (e.g., a laptop computer) that uses short-range communication or even communicating with a remote device via the Internet.

[0081] The key functions of image processing are:

[0082] Identify the baby within the video image;

[0083] Determine the infant's orientation;

[0084] Identify bottles within video images (either through direct identification or based on hypotheses derived from image information about the subject);

[0085] Determine the orientation of the bottle.

[0086] Based on image processing, the required reorientation of the bottle and / or the infant, or even changes in the subject's posture, can be determined to achieve the desired feeding conditions.

[0087] The desired bottle angle can be simply an angle relative to the horizontal plane, such as 45 degrees. However, this angle can vary depending on the bottle type, and the system can be adjusted to suit the specific type of bottle used. The baby needs to be positioned so that they can drink from the bottle at the desired bottle angle.

[0088] Therefore, it may be necessary to adjust the bottle angle and / or baby orientation. The desired bottle angle provides a downward tilt relative to the horizontal plane to hold the entire nipple in place, and also requires an appropriate angle relative to the baby's body orientation so that the baby can comfortably lock onto the bottle nipple. The baby should be at the appropriate tilt to help with burping, for example, to prevent colic.

[0089] The control display device outputs reorientation commands in conjunction with video images to help the subject reorient the bottle and / or infant to achieve the desired bottle orientation. The system maintains subject engagement by providing positive reinforcement and constant audio and / or visual feedback.

[0090] Figure 3 A method for providing assistance during bottle feeding is shown, the method comprising:

[0091] In step 40, video images of the subject feeding the infant with a bottle are captured;

[0092] In step 42, the baby in the video image is identified;

[0093] In step 44, the infant's orientation is determined;

[0094] In step 46, the bottle within the video image is identified; and

[0095] In step 48, the orientation of the bottle is determined.

[0096] In step 50, the required reorientation of the bottle is determined in order to achieve the desired bottle orientation.

[0097] In step 52, a video image is displayed in conjunction with the output reorientation command to help the subject reorient the bottle to the desired bottle orientation.

[0098] The method can also begin at step 42, for example, when the captured video is already available.

[0099] This system can also be used to track infant feeding patterns and habits. This invention can also be used to study infant feeding behaviors and patterns in hospital and home applications.

[0100] The system can be based entirely on the analysis of natural images. However, it can also use more easily recognizable markers, such as markers on an infant's body or face, and / or markers on the subject's hand holding the bottle, and / or markers on the bottle itself.

[0101] Image recognition systems can automatically obtain all the necessary pose information without requiring any special actions from the subject. However, the subject can alternatively provide input, such as having an initial pose in front of the camera as described above.

[0102] The desired bottle angle, for example, perpendicular to the infant's body axis, is determined as part of the 3D pose information. At a minimum, the system utilizes the infant's body orientation, the infant's head position, and the bottle's orientation (and position) to determine the angle between them. The angle relative to the horizontal plane is also known.

[0103] The desired orientation can be obtained as a 3D vector. This can be represented in a simplified 2D form, so that for a given posture of the subject and infant, all that is needed is the alignment of the bottle with a 2D vector on the display, with real-time feedback to show when alignment is achieved. 3D orientation information can be displayed using a more complex representation of the desired reorientation.

[0104] Augmented reality can be used to transfer a 3D vector in the desired direction to the 2D space on the screen. Alternatively, the 3D representation on the screen can be selected by the user through a viewing options menu in the user interface.

[0105] More accurate estimates of the desired orientation can be created by utilizing additional sensors (such as 3D cameras or multiple cameras) that provide more detailed information about the bottle's location and the baby's orientation. Audio feedback can also be used to give the user reorientation instructions in the 3D domain.

[0106] Those skilled in the art can readily develop processors for performing the methods described. Therefore, Figure 3 Each step of the flowchart can represent a different action performed by the processor, and can be performed by the corresponding module of the processor.

[0107] Processors can be implemented in a variety of ways using software and / or hardware to perform a wide range of required functions. A processor typically employs one or more microprocessors that can be programmed using software (e.g., microcode) to perform the desired functions. A processor can be implemented as a combination of dedicated hardware for performing certain functions and one or more programmable microprocessors and associated circuitry for performing other functions.

[0108] Examples of circuit devices that may be employed in various embodiments of the present invention include, but are not limited to, conventional microprocessors, application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0109] In various implementations, a processor may be associated with one or more storage media, such as volatile and non-volatile computer memories (e.g., RAM, PROM, EPROM, and EEPROM). The storage media may be encoded with one or more programs that, when executed on one or more processors and / or controllers, perform the required functions. The various storage media may be fixed within the processor or controller, or they may be transferable, allowing one or more programs stored thereon to be loaded into the processor.

[0110] By studying the accompanying drawings, the disclosure, and the appended claims, those skilled in the art can understand and implement variations of the disclosed embodiments in practicing the claimed invention. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality.

[0111] The fact that certain measures are described in mutually different dependent claims does not imply that combinations of these measures cannot be used advantageously.

[0112] Computer programs can be stored / distributed on suitable media, such as optical or solid-state media provided with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems.

[0113] If the term “suitable” is used in the claims or specification, it should be noted that the term “suitable” is intended to be equivalent to the term “configured as”.

[0114] Any reference numerals in the claims should not be construed as limiting the scope.

Claims

1. An image processor (34) for use in a system providing assistance during bottle feeding, said image processor comprising: Input, used to receive video images of the subject bottle-feeding the infant; Output, for providing video images for display, wherein the image processor is adapted to: Identify the baby within the video image; Determine the infant's orientation; Identify the bottle within the video image; Determine the orientation of the bottle; Determine the reorientation required for the bottle and / or the baby to achieve the desired bottle orientation; as well as Control one or more devices selected from the group consisting of: a display device for outputting reorientation instructions, and an audio output device for outputting reorientation instructions.

2. The image processor (34) according to claim 1 is further adapted to control the display device to output the redirection command superimposed on the video image.

3. The image processor (34) according to claim 2, wherein the display device is selected from the group consisting of a display and an audio output device.

4. The image processor (34) according to claim 1 is also adapted to identify the orientation of the subject's hand in the image and determine that the orientation of the long axis of the bottle is perpendicular to the orientation of the palm of the hand.

5. The image processor (34) according to any one of claims 3 to 4, wherein the desired bottle orientation corresponds to an angle of inclination relative to the body axis of the infant within a predetermined range.

6. The image processor (34) according to any one of claims 3 to 5 is also adapted to identify the position of the infant's head and the position of the subject's feeding hand in the image.

7. The image processor (34) according to any one of claims 3 to 6 is also adapted to recognize the pose of the subject in the image.

8. A system for providing assistance during bottle feeding, comprising: Camera (30) was used to capture video images of the subject bottle-feeding the infant; The image processor according to any one of claims 1 to 7; as well as One or more devices, wherein the one or more devices are selected from the group consisting of: A display (32) for displaying the video image, and an audio output device for outputting redirection commands.

9. A method for providing assistance during bottle feeding, comprising: (42) Identify the infant in the captured video image of the subject feeding the infant in a bottle; (44) Determine the orientation of the infant; (46) Identify the bottle in the video image; (48) Determine the orientation of the bottle; (50) Determine the reorientation required for the bottle and / or baby to achieve the desired bottle orientation; as well as (52) Output redirection instructions by executing one or more activities selected from a group consisting of: Displays and outputs reorientation instructions, and uses audio signal output to output reorientation instructions to help the subject reorient the bottle and / or baby to the desired bottle and / or baby orientation.

10. The method of claim 9, wherein the video image is displayed in conjunction with the redirection command.

11. The method of claim 10, wherein the display is selected from the group consisting of output and display.

12. The method of claim 11, further comprising identifying the orientation of the subject's hand in the image and determining that the orientation of the long axis of the bottle is perpendicular to the orientation of the palm of the hand.

13. The method according to any one of claims 11 to 12, comprising recognizing the subject's posture in the image.

14. A computer program product comprising instructions for implementing the method according to any one of claims 9 to 13 when the program is run on a processor.

15. A computer-readable medium comprising a computer program product according to claim 14.

Images