Milk volume detection and modeling method and system, storage medium and program product

Abstract

The invention provides a milk quantity detection and modeling method and system, a storage medium and a program product, and belongs to the technical field of milk quantity detection. The milk quantity detection model establishment method comprises the following steps: collecting images; annotating the image to obtain an initial annotation; establishing and training a basic image segmentation model; marking a part of video frames in the image; performing fine-grained adjustment on the basic image segmentation model to obtain an adjusted image segmentation model; predicting the image by using the adjusted image segmentation model, and selecting a correct prediction result as a new label; combining the new annotation and the initial annotation into a data set, and training a basic image segmentation model in combination with an attention mechanism to obtain an advanced image segmentation detection model; outputting the milk bottle detection result, the maximum capacity of the milk bottle and the liquid level position; and performing reasoning result integration post-processing, and outputting a final result. The milk bottle milk quantity detection method is high in accuracy, low in false alarm rate, high in efficiency and extensible.

Description

Technical Field

[0001] The present invention relates to the technical field of milk quantity detection, and in particular to a method, system, storage medium and program product for milk quantity detection and modeling. Background Art

[0002] A feeding bottle is a maternal and infant product that contains breast milk and milk. Generally, there are scales marked on the side of the feeding bottle. In practical applications, especially in the field of maternal and infant care, the amount of breast milk contained in the bottle needs to be manually read from the scale on the bottle body and then entered into the record for storage. However, this method requires manual operation and is more troublesome to record. The manual reading and counting of milk production is prone to errors and is inefficient. Therefore, designing a method that can automatically detect the amount of milk in the feeding bottle and record it can provide great convenience for mothers to record the amount of milk each time. With the development of deep learning in artificial intelligence, especially the significant progress in computer vision technology, it has become possible to recognize and detect specific objects in images by computer.

[0003] However, there are still many difficulties in the details of identifying the amount of milk in a bottle, and the technology is constantly iterating and updating. For example, the Chinese patent with the publication number CN109258479A discloses an "automatic monitoring system for milk production of dairy cows based on image recognition". The liquid level positioning module used in this system is used to divide the image of the milk bottle obtained by the camera into a scale area and a non-scale area and segment it, and realize accurate positioning of the liquid level; the milk bottle image segmentation module is used to accurately separate the scale area, and use projection and Shen algorithm to separate the scale value and scale line; the scale feature extraction and recognition module realizes the proportional positioning of small scale lines according to the equal spacing of large scale lines, analyzes the scale value, and uses the nearest neighbor algorithm to realize the recognition of the scale and obtain the scale value data. The separation of scale values ​​and scale lines based on projection and Shen algorithm is based on the traditional image processing technology to extract the image skeleton by refinement, determine the scale value and scale line by the skeleton, and then use the projection ratio algorithm to identify the milk amount. The disadvantages of this system are: the requirements for the background environment in the image are very high. In the case of a cluttered background, it is difficult for the image backbone extracted by the shen algorithm to accurately find the scale value and scale line; at the same time, the camera's viewing angle must be fixed, and the offset angle will also cause recognition errors; the system is not very practical in daily life. For example, the Chinese patent with publication number CN110661921A discloses "a mobile phone remote control smart milk bottle based on Internet of Things technology". The device is based on a hardware-based milk volume detection method. The key is that the user mixes milk powder according to the data in the way of adding water first and then milk powder. The temperature sensor and weight sensor monitor the water temperature and water volume of the water added first based on the upper and lower limit thresholds of each component of milk powder. The disadvantages of this device are: using hardware to identify the amount of milk makes its application scenario narrow, and it requires full-time personnel to regularly maintain it. The hardware cost used is high, and it is difficult to expand its application in the field of mothers and babies.

[0004] In summary, the existing methods have the following problems: high complexity of equipment operation, low accuracy of milk volume detection, poor generalizability and practicality, the need for full-time personnel to maintain the equipment, high labor costs, and narrow usage scenarios. There is an urgent need for a technical solution that can solve the above problems. Summary of the invention

[0005] In view of the problems existing in the above-mentioned prior art, the present invention proposes a method, system, storage medium and program product for milk amount detection and modeling, which aims to perform pixel-level deep learning detection on milk bottles, improve the accuracy and robustness of milk bottles under different background conditions, different lighting, different angles, and different devices, and improve the accuracy of milk amount detection; detect the position of the milk bottle in the scene according to the model algorithm, and then identify the liquid level in the milk bottle, so as to improve the efficiency of detection, reduce the operation and environmental requirements for detecting milk amount using this method, and make it highly scalable and easier to use.

[0006] In a first aspect, the present invention provides a method for establishing a milk quantity detection model, comprising the following steps:

[0007] Collect images;

[0008] Annotate the image to obtain initial annotations;

[0009] Based on the initial annotation, build and train the basic image segmentation model;

[0010] Annotate some video frames in the image;

[0011] Based on the annotations of some video frames, the basic image segmentation model is fine-grainedly adjusted to obtain an adjusted image segmentation model;

[0012] Use the adjusted image segmentation model to predict the image and select the correct prediction result as the new annotation;

[0013] Combine the new annotations and the initial annotations into a dataset, and train the basic image segmentation model with the attention mechanism to obtain an advanced image segmentation detection model.

[0014] The advanced image segmentation detection model outputs the bottle detection results, the maximum capacity of the bottle, and the liquid level position;

[0015] According to the bottle detection results, the maximum capacity of the bottle and the liquid level position, the reasoning results are integrated and processed to output the final result.

[0016] As a further improvement of the present invention, the content of the image includes a scene containing a milk bottle and a scene not containing a milk bottle;

[0017] The images include pictures and videos;

[0018] The image collection method includes downloading from the network and setting up an environment for collection.

[0019] As a further improvement of the present invention, the labeling of the image includes: performing pixel-level image recognition labeling on the feeding bottle and the amount of milk on the liquid level.

[0020] As a further improvement of the present invention, the advanced image segmentation detection model performs cyclic iteration.

[0021] As a further improvement of the present invention, the reasoning result is integrated and then processed to calculate the amount of milk in the bottle by proportional conversion according to the maximum capacity and liquid level position of the bottle;

[0022] The end result is the amount of milk in the bottle.

[0023] In a second aspect, the present invention provides a computer system, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-mentioned method for establishing a milk quantity detection model.

[0024] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned method for establishing a milk quantity detection model.

[0025] In a fourth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the above-mentioned method for establishing a milk quantity detection model.

[0026] In a fifth aspect, the present invention provides a method for detecting milk quantity, using the above-mentioned method for establishing a milk quantity detection model, comprising the following steps:

[0027] Get video stream input from a video source;

[0028] Input the images in the video stream into the advanced image segmentation detection model in turn, detect each frame of the image, and output the bottle detection result, the maximum capacity of the bottle, and the liquid level position;

[0029] When the bottle detection result is that there is no bottle, directly output the result that there is no bottle and save the picture;

[0030] When the bottle detection result is that there is a bottle, the maximum capacity and liquid level position of the bottle are reasoned and integrated and processed, and the image is saved;

[0031] Manual confirmation and retraining of image annotations for unsuccessful image recognition in the saved images;

[0032] Output and record the inference results and post-processing data.

[0033] In a sixth aspect, the present invention provides a computer system, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-mentioned milk quantity detection method.

[0034] In a seventh aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above-mentioned milk quantity detection method when executed by a processor.

[0035] In an eighth aspect, the present invention provides a computer program product, comprising a computer program, which implements the steps of the above-mentioned milk quantity detection method when executed by a processor.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The present invention provides a method, system, storage medium and program product for milk quantity detection and modeling, and constructs a method for detecting bottle milk quantity with high accuracy, low false alarm rate, high efficiency and scalability.

[0038] Specifically, first, the present invention innovatively adopts the pixels and schemes of deep learning image segmentation, which overcomes the problems brought about by the traditional visual processing schemes adopted in other methods, such as the fixed operator template, the inability to effectively cope with the changes in the image, and the difficulty in extracting edge lines while effectively suppressing the degradation of model detection performance caused by image noise; second, the present invention only manually annotates a small number of pictures for videos of different scenes, and fine-grainedly adjusts multiple models on the basic model to generate a large number of high-quality annotations for subsequent training of the model; third, the present invention does not completely rely on the recognition accuracy of the image segmentation model. By inputting the results of the image segmentation model into another image classification model, post-processing is integrated to reduce segmentation misdetection and reduce the false alarm rate; fourth, the present invention makes full use of the information in the scene, not only using the information of the image in the spatial domain, but also using the information in the time domain, further reducing the possibility of false alarms; fifth, the camera equipment of the present invention does not require manual setting or correction, is widely applicable to daily life, is easy to implement, has low maintenance costs, and is convenient for engineering expansion and implementation. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A flow chart of a method for establishing a milk quantity detection model disclosed in an embodiment of the present invention;

[0040] Figure 2 The present invention discloses a flow chart of a method for detecting milk quantity. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the present invention will combine the accompanying drawings to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments, wherein steps S1, S2... in the embodiments described in the present invention do not limit the only execution steps of the present invention; wherein the various models and mechanisms described in the present invention are not the only limiting methods of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0042] In the present invention, "module", "system", etc. refer to related entities applied to computers, such as hardware, a combination of hardware and software, software or software in execution, etc. In detail, for example, software includes but is not limited to a process running on a processor, a processor, an object, executable software, an execution thread, a program and / or a computer. In addition, an application or script program running on a server, and a server can all be software. One or more software can be in an execution process and / or thread, and the software can be localized on one computer and / or distributed between two or more computers, and can be run by various computer-readable media.

[0043] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0044] Example 1

[0045] In an embodiment of the present invention, a method for establishing a milk quantity detection model is provided. Figure 1 As shown, the following steps are included:

[0046] S101: collect images;

[0047] The content of the image includes a scene containing a milk bottle and a scene not containing a milk bottle;

[0048] The images include pictures and videos;

[0049] The image collection method includes downloading from the network and setting up an environment for collection.

[0050] S102: Annotate the image to obtain initial annotations;

[0051] The labeling of the image includes: performing pixel-level image recognition and labeling of the feeding bottle and the amount of milk on the liquid level.

[0052] Preferably, this step can be mainly based on the annotation of a single picture, supplemented by a small number of video frames captured from the video.

[0053] S103: Based on the initial annotation, establish and train a basic image segmentation model;

[0054] S104: marking some video frames in the image;

[0055] In one embodiment of the present invention, a very small number of video frames collected from each different scene video are annotated. Preferably, the number of annotated video frames is, for example, 50.

[0056] S105: Based on the annotations of some video frames, fine-grained adjustment is performed on the basic image segmentation model to obtain an adjusted image segmentation model;

[0057] In one embodiment of the present invention, a very small number of video frames collected for each different scene video are annotated, preferably, the annotated video frames are, for example, 50. For each different scene video, the annotated video frames are used to fine-tune the basic image segmentation model in step S103 to obtain an adjusted image segmentation model.

[0058] S106: Use the adjusted image segmentation model to predict the image, and select the correct prediction result as a new label;

[0059] In this embodiment, by implementing steps S104 to S106, a large number of high-quality new annotations can be generated through a small amount of manual annotation, that is, through the adjusted image segmentation model, which can be used to solve the problem of time-consuming and labor-intensive pixel-level detection and annotation. It should be noted that since the generalization performance of the model obtained by fine-tuning on the initial segmentation model is limited, different fine-tune models need to be used for videos of different scenes.

[0060] S107: Combine the new annotations and the initial annotations into a data set, and train the basic image segmentation model with the attention mechanism to obtain an advanced image segmentation detection model;

[0061] Preferably, the advanced image segmentation detection model can be iterated as needed, that is, the above steps S104 to S107 can be iterated in a loop.

[0062] Preferably, the image segmentation model in the present invention can adopt the YOLOv5-seg model (English name is You Only Look Once segment). In addition to using YOLOv5seg, the image segmentation model can also use segmentation models including FPN, FCN, U-Net and its variants, LinkNet, PSPNet, PAN, DAN, SANet, DeepLab series and its variants; the image detection model in the present invention can adopt the YOLOv5s model (English name is You Only Look Once-small). In addition to using the YOLOv5 series, the image detection model can also use detection models including ResNeXt, ResNeSt, DenseNet, EfficientNet, etc. It should be noted that the models described in the present invention are not the only limiting methods of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0063] S108: the advanced image segmentation detection model outputs the milk bottle detection result, the maximum capacity of the milk bottle and the liquid level position;

[0064] Among them, the maximum capacity of the bottle is determined by different models of bottles.

[0065] S109: Based on the bottle detection result, the maximum capacity of the bottle and the liquid level position, the reasoning result is integrated and post-processed to output the final result.

[0066] Preferably, the reasoning result is integrated and then processed to calculate the amount of milk in the bottle by proportional conversion according to the maximum capacity and liquid level position of the bottle;

[0067] The end result is the amount of milk in the bottle.

[0068] Example 2

[0069] In an embodiment of the present invention, a computer system is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method for establishing a milk quantity detection model in Embodiment 1.

[0070] Example 3

[0071] In an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for establishing a milk quantity detection model in Embodiment 1 are implemented.

[0072] Example 4

[0073] In an embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps of the method for establishing a milk quantity detection model in Embodiment 1 are implemented.

[0074] Example 5

[0075] In an embodiment of the present invention, a method for detecting the amount of milk provided is provided, wherein a bottle milk amount detection model established by the above-mentioned method for establishing a milk amount detection model is used. Figure 2 As shown, the following steps are included:

[0076] S201: Obtain video stream input from a video source;

[0077] Among them, the video source can be a surveillance camera, an inspection robot, a vehicle-mounted camera, etc.

[0078] S202: Input the images in the video stream into the advanced image segmentation detection model in sequence, detect each frame of the image, and output the milk bottle detection result, the maximum capacity of the milk bottle, and the liquid level position;

[0079] Among them, the maximum capacity of the bottle is determined by different models of bottles.

[0080] S203: When the bottle detection result is that there is no bottle, directly output the result that there is no bottle and save the picture;

[0081] S204: When the bottle detection result is that there is a bottle, perform post-processing on the integration of the inference results of the maximum capacity and the liquid level position of the bottle, and save the image;

[0082] Preferably, the reasoning results are integrated and then processed to calculate the amount of milk in the bottle by proportional conversion according to the maximum capacity and liquid level position of the bottle.

[0083] S205: Manual confirmation, retraining the annotation of the images that have not been successfully recognized in the saved images;

[0084] S206: Output and record the data processed after the reasoning result integration.

[0085] Example 6

[0086] In an embodiment of the present invention, a computer system is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the milk quantity detection method in Embodiment 5.

[0087] Example 7

[0088] In an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the milk quantity detection method in Embodiment 5 are implemented.

[0089] Example 8

[0090] In an embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps of the milk quantity detection method in Embodiment 5 are implemented.

[0091] The present invention provides a method, system, storage medium and program product for milk quantity detection and modeling, and constructs a method for detecting bottle milk quantity with high accuracy, low false alarm rate, high efficiency and scalability.

[0092] Specifically, first, the present invention innovatively adopts the pixels and schemes of deep learning image segmentation, which overcomes the problems brought about by the traditional visual processing schemes adopted in other methods, such as the fixed operator template, the inability to effectively cope with the changes in the image, and the difficulty in extracting edge lines while effectively suppressing the degradation of model detection performance caused by image noise; second, the present invention only manually annotates a small number of pictures for videos of different scenes, and fine-grainedly adjusts multiple models on the basic model to generate a large number of high-quality annotations for subsequent training of the model; third, the present invention does not completely rely on the recognition accuracy of the image segmentation model. By inputting the results of the image segmentation model into another image classification model, post-processing is integrated to reduce segmentation misdetection and reduce the false alarm rate; fourth, the present invention makes full use of the information in the scene, not only using the information of the image in the spatial domain, but also using the information in the time domain, further reducing the possibility of false alarms; fifth, the camera equipment of the present invention does not require manual setting or correction, is widely applicable to daily life, is easy to implement, has low maintenance costs, and is convenient for engineering expansion and implementation.

[0093] According to experiments, the image segmentation model of the present invention can achieve a pixel-level accuracy of more than 95.5% mIoU, the classification model can achieve a classification accuracy of more than 99%, and the final offset detection alarm accuracy reaches more than 95%.

Claims

1. A method for establishing a milk quantity detection model, characterized in that: The following steps are involved: Collect images; Annotate the image to obtain initial annotations; Based on the initial annotation, build and train the basic image segmentation model; Annotate some video frames in the image; Based on the annotations of some video frames, the basic image segmentation model is fine-grainedly adjusted to obtain an adjusted image segmentation model; Use the adjusted image segmentation model to predict the image and select the correct prediction result as the new annotation; Combine the new annotations and the initial annotations into a dataset, and train the basic image segmentation model with the attention mechanism to obtain an advanced image segmentation detection model. The advanced image segmentation detection model outputs the bottle detection results, the maximum capacity of the bottle, and the liquid level position; According to the bottle detection results, the maximum capacity of the bottle and the liquid level position, the reasoning results are integrated and processed to output the final result.

2. The method for establishing a milk quantity detection model according to claim 1, characterized in that: The contents of the image include a scene containing a milk bottle and a scene not containing a milk bottle; The images include pictures and videos; The image collection method includes downloading from the network and setting up an environment for collection.

3. The method for establishing a milk quantity detection model according to claim 1, characterized in that: The labeling of the image includes: performing pixel-level image recognition labeling on the feeding bottle and the amount of milk on the liquid level.

4. The method for establishing a milk quantity detection model according to claim 1, characterized in that: The advanced image segmentation detection model performs cyclic iteration.

5. The method for establishing a milk quantity detection model according to claim 1, characterized in that: The reasoning result is integrated and then processed to calculate the amount of milk in the bottle by proportional conversion according to the maximum capacity and liquid level position of the bottle; The end result is the amount of milk in the bottle.

6. A computer system comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A method for detecting milk quantity, characterized in that: The method for establishing a milk quantity detection model according to any one of claims 1 to 5 comprises the following steps: Get video stream input from a video source; Input the images in the video stream into the advanced image segmentation detection model in turn, detect each frame of the image, and output the bottle detection result, the maximum capacity of the bottle, and the liquid level position; When the bottle detection result is that there is no bottle, directly output the result that there is no bottle and save the picture; When the bottle detection result is that there is a bottle, the maximum capacity and liquid level position of the bottle are reasoned and integrated and processed, and the image is saved; Manual confirmation and retraining of image annotations for unsuccessful image recognition in the saved images; Output and record the inference results and post-processing data.

10. A computer system comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of any one of the methods of claim 9.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 9 are implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to claim 9 are implemented.

Images