Method, device and system for automatically counting grain moistening time and storage medium

Abstract

The application provides a grain moistening time length automatic statistics method, device and system and a storage medium. Video images of a wine factory are collected, and for each frame of the video images, the first positions of each grain pile in the images and the grain pile categories of each grain pile are determined according to the images and a preset grain pile detection model, and the second positions of water guns in the images are determined according to the images and a preset water gun detection model. The grain pile category is a grain moistening category or a non-grain moistening category. The grain moistening start time point and the grain moistening end time point of each grain pile are determined according to the first positions, categories and second positions of the water guns in the continuous multiple frames of images, and the grain moistening time length of each grain pile is determined according to the grain moistening start time point and the grain moistening end time point. That is, the grain moistening time length of each grain pile can be automatically counted by the collected images, manual counting is not required, the work efficiency can be improved, and the labor cost can be reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of brewing, in particular to a method, device and system for automatically counting grain moistening time and a storage medium. BACKGROUND

[0002] At present, the related data in the process of artisanal liquor making are usually recorded manually. Manual recording has problems such as untimely recording, missing recording, and consumption of manpower and material resources. Grain moistening is a key process in the next sand making cycle, and the quality of grain moistening lays the foundation for the whole year's production. The grain moistening time is a key factor. However, it is relatively cumbersome for artisans to manually record the grain moistening time, and it is easy to miss. It is difficult to achieve long-term stable collection, and the cost of recording is large. SUMMARY

[0003] The purpose of the embodiments of the present application is to provide a method, device and system for automatically counting grain moistening time and a storage medium to solve the above technical problems.

[0004] In one aspect, a method for automatically counting grain moistening time is provided, the method comprising:

[0005] collecting video images of a liquor-making workshop;

[0006] For each frame of image of the video image, according to the image and a preset grain pile detection model, the first positions of each grain pile in the image and the grain pile categories of each grain pile are determined, and according to the image and a preset water gun detection model, the second positions of water guns in the image are determined; the categories are grain moistening categories or non-grain moistening categories;

[0007] According to the first positions of each grain pile in the continuous multiple frames of the image, the grain pile categories and the second positions of the water guns in the image, the grain moistening start time point and the grain moistening end time point of the grain pile are determined;

[0008] According to the grain moistening start time point and the grain moistening end time point, the grain moistening time of the grain pile is determined.

[0009] In one of the embodiments, before determining the first positions of each grain pile in the image and the grain pile categories of each grain pile according to the image and a preset grain pile detection model, the method comprises:

[0010] obtaining a first video sample image of a grain moistening stage in the liquor-making workshop;

[0011] determining a first sample picture training set from the first video sample image; the first sample picture training set includes multiple first sample pictures, and each first sample picture is labeled with a corresponding grain pile position and a grain pile category for each grain pile;

[0012] training the grain pile detection model based on the first sample picture training set.

[0013] In one of the embodiments, before determining the second position of the water cannon in the image according to the image and a preset water cannon detection model, the method comprises:

[0014] obtaining a second video sample image at a start stage of grain moistening in the winery;

[0015] determining a second sample picture training set from the second video sample image; the second sample picture training set comprises a plurality of second sample pictures, and each second sample picture is labeled with a water cannon position;

[0016] training the water cannon detection model based on the second sample picture training set.

[0017] In one of the embodiments, determining the start time point of grain moistening of the grain pile according to the first position of each grain pile in the continuous multiple frames of images, the grain pile category, and the second position of the water cannon in the image comprises:

[0018] for each grain pile in the continuous multiple frames of images, when it is determined that the grain pile category of the grain pile changes from a non-grain moistening category to a grain moistening category, and the grain pile is of the grain moistening category within a continuous preset first frame number, setting a value in a first preset indication tag of the grain pile to a first preset value;

[0019] when it is determined that the first position of the grain pile and the second position of the water cannon in the same frame of the image intersect, and the intersection exists within a continuous preset second frame number, setting a value in a second preset indication tag of the grain pile to a second preset value;

[0020] obtaining a first time point when the value in the first preset indication tag of the grain pile is the first preset value, and the value in the second preset indication tag of the grain pile is the second preset value;

[0021] determining the start time point of grain moistening of the grain pile according to the first time point.

[0022] In one of the embodiments, determining the start time point of grain moistening of the grain pile according to the first time point comprises:

[0023] querying a water discharge time record table of the water cannon corresponding to the grain pile;

[0024] The water discharge time point closest to the first time point in the water discharge time record table is taken as the grain moistening start time point of the grain pile; or, the water discharge time point of the last water discharge within a preset time period before and / or after the first time point is taken as the grain moistening start time point of the grain pile.

[0025] In one of the embodiments, the determination of the grain moistening start time point of each grain pile according to the first position of each grain pile, the grain pile category, and the second position of the water gun in the continuous multiple frames of the images comprises:

[0026] When it is determined that the value in the first preset indication label of the grain pile is the third preset value in the continuous preset third frame number, the value in the first preset indication label of the grain pile is set to a fourth preset value;

[0027] When it is determined that the value in the first preset indication label of the grain pile is the fourth preset value in the continuous preset fourth frame number, the value in the second preset indication label of the grain pile is set to the third preset value.

[0028] In one of the embodiments, the determination of the grain moistening end time point of each grain pile according to the first position of each grain pile, the grain pile category, and the second position of the water gun in the continuous multiple frames of the images comprises:

[0029] For each grain pile in the continuous multiple frames of the images, when it is determined that the category of the grain pile changes from the grain moistening category to the non-grain moistening category, and the grain pile is the non-grain moistening category in the continuous preset fifth frame number, the second time point when the value in the first preset indication label of the grain pile is the first preset value and the value in the second preset indication label of the grain pile is the second preset value is acquired;

[0030] The second time point is taken as the grain moistening end time point of the grain pile, the value in the first preset indication label of the grain pile is set to the fourth preset value, and the value in the second preset indication label of the grain pile is set to the third preset value.

[0031] On the other hand, an automatic grain moistening time length statistical device is provided, which comprises:

[0032] The acquisition module is configured to acquire video images of a wine-making workshop.

[0033] The first determination module is configured to, for each frame of the video images, determine the first position of each grain pile and the grain pile category of each grain pile according to the image and a preset grain pile detection model, and determine the second position of the water gun according to the image and a preset water gun detection model; the grain pile category is a grain moistening category or a non-grain moistening category.

[0034] The second determining module is configured to determine a grain moistening start time point and a grain moistening end time point of the grain pile according to the first positions of each grain pile in the continuous multiple frames of images, the categories, and the second position of the water gun in the images.

[0035] The third determining module is configured to determine a grain moistening duration of the grain pile according to the grain moistening start time point and the grain moistening end time point.

[0036] In another aspect, a grain moistening duration automatic statistical system is provided, which includes a processor, a memory, and a camera. The memory stores a computer program, and the processor executes the computer program to implement any of the above methods.

[0037] In another aspect, a computer readable storage medium is provided, which stores a computer program. When the computer program is executed by at least one processor, any of the above methods is implemented.

[0038] The grain moistening duration automatic statistical method, device, system, and storage medium provided by the present application can automatically and statistically determine the grain moistening duration of the grain pile through the collected video images of the winery, and for each frame of image of the video images, the first positions of each grain pile and the categories of each grain pile are determined according to the image and a preset grain pile detection model, and the second position of the water gun in the image is determined according to the image and a preset water gun detection model. The category of the grain pile is a grain moistening category or a non-grain moistening category. The grain moistening start time point and the grain moistening end time point of the grain pile are determined according to the first positions of each grain pile, the categories, and the second position of the water gun in the images in the continuous multiple frames of images. The grain moistening duration of the grain pile is determined according to the grain moistening start time point and the grain moistening end time point. That is, the grain moistening duration of the grain pile can be automatically and statistically determined through the collected images, without manual statistical determination, which can improve the work efficiency and reduce the labor cost. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 A flowchart of a grain moistening duration automatic statistical method provided by Embodiment One of the present application is shown in the figure.

[0040] Figure 2 A training flowchart of a grain pile detection model provided by Embodiment One of the present application is shown in the figure.

[0041] Figure 3 A training flowchart of a water gun detection model provided by Embodiment One of the present application is shown in the figure.

[0042] Figure 4 A flowchart of another grain moistening duration automatic statistical method provided by Embodiment One of the present application is shown in the figure.

[0043] Figure 5A structure schematic diagram of the automatic grain moistening time length statistical device provided for Embodiment Two of the present application is shown in the figure.

[0044] Figure 6 A structure schematic diagram of the automatic grain moistening time length statistical system provided for Embodiment Three of the present application is shown in the figure. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0046] Embodiment One:

[0047] The present application provides an automatic grain moistening time length statistical method, and for details, please refer to Figure 1 The method comprises the following steps as shown in the figure:

[0048] S11: Collecting video images of a wine-making factory building;

[0049] S12: For each frame of image of the video images, determining the first positions of each grain pile in the image and the grain pile categories of each grain pile according to the image and a preset grain pile detection model, and determining the second position of the water gun in the image according to the image and a preset water gun detection model; the grain pile category is a grain moistening category or a non-grain moistening category.

[0050] S13: Determining the grain moistening start time point and the grain moistening end time point of the grain pile according to the first positions, the grain pile categories of each grain pile in the continuous multiple frames of images and the second position of the water gun in the image.

[0051] S14: Determining the grain moistening time length of the grain pile according to the grain moistening start time point and the grain moistening end time point.

[0052] Next, the process of the above steps is described in detail.

[0053] In this embodiment, cameras can be installed inside the brewery to capture video images of the grain-growing environment. In actual production, breweries typically occupy a large area and are quite tall. Therefore, a single camera may not be able to cover the entire brewery from a single angle, failing to capture a panoramic view. Therefore, multiple cameras can be installed at fixed intervals on the top of the brewery, ensuring overlap between cameras. Each camera acquires real-time video images of the brewery, which are then uploaded to a server and stitched together to obtain a real-time panoramic view. This technique requires that the cameras be installed to ensure overlap between the fields of view of adjacent cameras, and that distortion correction be applied to the camera images. Then, a matrix mapping transformation is used to map the images captured by all cameras to the same feature space, and finally, the overlapping fields of view are stitched together to obtain a panoramic video image.

[0054] It should be noted that the grain pile detection model and water gun detection model in the embodiments of this application can be models obtained in advance through model training.

[0055] First, the training process of the grain pile detection model will be introduced. Please refer to [link / reference]. Figure 2 As shown, before step S12, the following steps may be included:

[0056] S21: Obtain the first video sample image of the grain moistening stage in the brewery.

[0057] S22: Determine the first sample image training set from the first video sample images; the first sample image training set includes multiple first sample images, and each first sample image is labeled with the corresponding grain pile location and grain pile category for each grain pile.

[0058] S23: The grain pile detection model is obtained by training based on the first sample image training set.

[0059] In step S21, video images of the grain moistening stage in the brewery can be acquired to obtain the first video sample image.

[0060] In step S22, video samples under different lighting conditions during the grain moistening stage can be selected from the first video sample images. The video samples are output frame by frame. The working area of ​​grain moistening is cropped in the output images to a preset size and saved as a sample set. Next, each image in the sample set is labeled. Specifically, based on the characteristics of moistened and non-moistened grain piles during the grain moistening process, the labelme labeling tool is used to label each grain pile in the image in the form of a label box and assign different labels, such as assigning a label to moistened grain piles or a label to non-moistened grain piles, thereby obtaining the first sample image training set.

[0061] In step S23, part of the pictures in the first sample picture training set can be taken as the training picture set and the other part can be taken as the verification picture set according to the sample quantity, scene complexity and experience value and the like. In the embodiment of the present application, the grain pile detection model can be obtained by training based on the first sample picture training set and the YOLOv5 initial network model, and it can be understood that the training can also be performed based on other initial target detection network models in other embodiments.

[0062] The YOLOv5 network model mainly has four parts, namely, an input end, a Backbone, a Neck and a Prediction. When the YOLOv5 network model is taken as the initial network model for model training in the embodiment of the present application, the existing YOLOv5 network model can be improved, and specifically, an up-sampling layer can be added in the up-sampling module of the PANet feature fusion network of the YOLOv5 network model. The up-sampling layer is a 4-fold up-sampling layer added on the basis of the 8-fold, 16-fold and 32-fold up-sampling layers. In the PANet feature fusion network, a Concatenate fusion layer is added to fuse the 4-fold up-sampled feature map and the feature map of the same size obtained in the backbone network extraction process through the added fusion layer, to generate a 4-fold up-sampled feature map. The 4-fold up-sampled feature map is used for the detection of the grain piles in the grain moistening process and the non-grain moistening grain piles. Based on the improved grain pile target detection model, four kinds of prediction layers of different scales are added for the multi-scale detection of the Head part.

[0063] In the training process of the grain pile detection model, the EIOU can be used as the loss function. The EIOU loss function is improved on the basis of the existing CIOU loss function, wherein the IOU is the ratio of the intersection to the union.

[0064] The CIOU loss function is defined as follows:

[0065] ;

[0066] .

[0067] wherein, is a weight function, and is used to measure the similarity of the length-width ratio, represents the center point of the prediction box, represents the width of the prediction box, represents the height of the prediction box, represents the width of the real box, represents the height of the real box, represents the length of the diagonal of the minimum bounding rectangle of the prediction box and the real box, represents the Euclidean distance between the two center points of the real box and the prediction box.

[0068] The penalty term of EIOU is to separate the aspect factor of the aspect ratio to calculate the length and width of the target box and the anchor box respectively on the basis of the CIOU penalty term. The loss function is divided into three parts: overlap loss, center distance loss and width-height loss. The width-height loss makes the difference between the width and height of the target box and the anchor box minimum, so that the convergence speed is faster. The definition is as follows:

[0069] ;

[0070] wherein, represents the intersection over union of the real box and the predicted box, represents the center point of the target box, and are the width and height of the minimum bounding box covering the predicted box and the real box.

[0071] The setting process of the YOLOv5 initial network model is introduced above. The training process of the first sample image training set and the YOLOv5 initial network model is introduced below.

[0072] It can be understood that first, the training parameters can be set, and the training batch size batch, the learning rate I, and the training iteration number Epoch can be set according to the empirical value. Then, the training image set and the verification image set are input into the YOLOv5 initial network model. It can be understood that in some embodiments, the scaling of the training images and the verification images for adaptive grain detection can be realized according to the input image size set by the network.

[0073] During the training process, the learning rate and the iteration number can be adjusted according to the average precision change and the loss change trend of the cross-validation of the training image set and the verification image set, until the precision change and the loss change gradually tend to be stable, and the final learning rate and iteration number are determined.

[0074] Finally, based on the determined learning rate and iteration number, the training of the grain detection model is completed, and a good convergence grain detection model based on the improved YOLOv5 is obtained. The model with good training results is stored in the disk for subsequent online inference.

[0075] The training process of the water gun detection model is introduced below.

[0076] Please refer to Figure 3 Before step S12, the following steps can be included:

[0077] S31: Obtain a second video sample image of the beginning stage of the grain moistening in the wine-making workshop.

[0078] S32: Determine a second sample picture training set from the second video sample image; the second sample picture training set includes a plurality of second sample pictures, and each second sample picture is labeled with a water gun position.

[0079] S33: Train a water gun detection model based on the second sample picture training set.

[0080] In step S31, a video image of a grain moistening stage in a distillery can be collected to obtain a second video sample image. Preferably, a video image of a grain moistening start stage in the distillery can be collected because the water gun usually appears in the grain moistening start stage.

[0081] Similarly, in step S32, a video sample of the grain moistening start stage can be selected from the second video sample image, the video sample is output frame by frame, the working area of the grain moistening is cropped in the output picture, the working area is cropped to a preset size and saved as a sample set; second, each picture in the sample set is labeled. Specifically, a labelme labeling tool can be used to label the water gun appearing in the picture in the form of a label box, so as to obtain the second sample picture training set.

[0082] In step S33, part of the pictures in the second sample picture training set can be taken as a training picture set and the other part can be taken as a verification picture set according to the number of samples, the complexity of the scene and the experience value. In the embodiment of the application, the grain pile detection model can be obtained by training the second sample picture training set and the maskrcnn initial network model. Of course, it can be understood that the initial target detection network model can also be trained in other embodiments.

[0083] The maskrcnn initial network model mainly includes: Backbone, RPN, ProposalLayer, DetectionTargetLayer, ROIAlign and bbox detection.

[0084] The Backbone adopts a 101-layer ResNet network and FPN, which is used to extract the features of the original picture. Meanwhile, the FPN fuses the bottom features and the high-level features extracted by the neural network, fully utilizes the features extracted at each stage, and facilitates detailed detection.

[0085] The RPN (Region Proposal Network) sets a plurality of anchors and finds the most suitable anchor to identify the target, thereby improving the detection accuracy.

[0086] The input of the ProposalLayer is the output of the RPN network, which corrects the anchors and retains a part of the anchors with a high correct prediction probability.

[0087] DetectionTargetLayer is to classify the ROI into positive and negative samples and correspond to the real bounding box, and adjust the RPN network.

[0088] ROIAlign unifies the different sizes of pictures generated by RPN and ROI, facilitating the input of subsequent classification.

[0089] The bbox detection classifies the unified size feature map obtained to obtain classification and regression values.

[0090] When training based on the second sample picture training set and the maskrcnn initial network model, first, the training parameters can be set, and the training batch size batch, the learning rate I, and the training iteration number Epoch are set according to the experience value; then the training picture set and the verification picture set are sent into the maskrcnn initial network model for training. Similarly, during the training process, the learning rate and the iteration number can be adjusted according to the average precision change and the loss change trend of the cross-validation of the training picture set and the verification picture set, until the precision change and the loss change gradually tend to be stable, and the final learning rate and iteration number are determined. Finally, based on the determined learning rate and iteration number, the training of the water gun detection model is completed, a water gun detection model based on the maskrcnn detection network with good convergence is obtained, and the training model with good convergence is stored to the disk for subsequent online inference.

[0091] For ease of understanding, the process of determining the first positions of the grain piles and the categories of the grain piles in the image according to the image and the preset grain pile detection model in step S12 is introduced.

[0092] In step S12, for each frame of the video image, it can be processed into an input format supported by the grain pile detection model, then it is transmitted into the grain pile detection model for inference, and finally the result after inference is processed to output the position and category of the target frame, that is, the first position of the grain pile and the corresponding category. In this process, the image can be scaled, centered and padded, and the whole process is combined as an affine transformation, and the corresponding affine transformation matrix M is as follows:

[0093] ;

[0094]

[0095] Indicates whether scaling after rotation, default is 1 indicating no scaling, Indicates the source image, Indicates the target image, (x, y) is the coordinate of any pixel point on the source image, and (x', y') is the coordinate of any pixel point on the target image.

[0096] After the equal scaling is performed, the converted picture color channel can be converted from BGR to RGB, normalized, converted from H (height), W (weight), and C (channel) to CHW, and added with a dimension batch. Finally, the input into the training network is a 4-dimensional Tensor (B, C, H, W).

[0097] The processed matrix is then sent to the trained grain pile detection model for inference. The result of the inference is then decoded to map the target image to the original image, which can be achieved by inverse affine transformation. Finally, the optimal result can be selected from the overlapping prediction boxes through NMS (Non-Maximum Suppression), and the coordinate values and categories of the prediction boxes are output, i.e., the first position of the grain pile and the corresponding category.

[0098] The process of determining the second position of the water gun in the image based on the image and the preset water gun detection model will be introduced below.

[0099] Similarly, in step S12, for each frame of the video image, it can be processed into an input format supported by the water gun detection model, then input into the water gun detection model for inference, and finally the result of the inference is processed to output the position of the target box, i.e., the second position of the water gun.

[0100] The water gun detection model based on maskrcnn is trained in a deep learning framework, so in step S12, the input image can be preprocessed, inferred, and the model can be forward propagated and post-processed in the deep learning framework, and finally the second position of the water gun is output.

[0101] Step S13 will be introduced below. In step S13:

[0102] For each grain pile in the continuous multiple frames of images, when it is determined that the grain pile category of a certain grain pile changes from a non-moist grain category to a moist grain category, and the grain pile is a moist grain category within a continuous preset first frame number, the value in the first preset indication label of the grain pile is set to a first preset value;

[0103] When it is determined that the first position of a certain grain pile in the same frame of image intersects with the second position of the water gun, and there is an intersection within a continuous preset second frame number, the value in the second preset indication label of the grain pile is set to a second preset value;

[0104] The first time point when the value in the first preset indication label of the grain pile is the first preset value, and the value in the second preset indication label of the grain pile is the second preset value is obtained;

[0105] Determine the start time of the grain pile according to the first time point.

[0106] Optionally, when it is determined that the value in the second preset indicator tag of the grain pile is the third preset value for a continuous preset third frame number, the value in the first preset indicator tag of the grain pile is set to a fourth preset value; when it is determined that the value in the first preset indicator tag of the grain pile is the fourth preset value for a continuous preset fourth frame number, the value in the second preset indicator tag of the grain pile is set to the third preset value.

[0107] It should be noted that the first preset indicator tag and the second preset indicator tag can be flexibly set by the developer, and specifically, the first preset indicator tag and the second preset indicator tag can be set for each grain pile, for example, can be set to flag1 and flag2 respectively, and the first preset value, the second preset value, the third preset value and the fourth preset value can be flexibly set by the developer, and can be different from each other or partially the same, for example, the first preset value and the second preset value can be both set to T, and the third preset value and the fourth preset value can be both set to F.

[0108] At this time, when the grain pile category changes from the non-grain moistening category to the grain moistening category and continuously remains the grain moistening category for a preset first frame number, the flag1 of the grain pile is marked as T, and within a continuous preset third frame number, the flag2 of the grain pile is still F, and then the flag1 is set to F. When the position of the water gun detected and the position of the grain pile have an intersection and the intersection lasts for a preset second frame number, the flag2 of the grain pile is marked as T, and within a continuous preset fourth frame number, the flag1 of the grain pile is still F, and then the flag2 is set to F. Record the first time point t1 when the flag1 and the flag2 of the grain pile are both T, and determine the start time of the grain moistening of the grain pile according to the first time point.

[0109] In the embodiment of the application, the water discharge time record table of the water gun corresponding to the grain pile can be queried; the water discharge time point closest to the first time point in the water discharge time record table is taken as the start time of the grain moistening of the grain pile; or the water discharge time point of the last water discharge within a preset time length before and / or after the first time point is taken as the start time of the grain moistening of the grain pile.

[0110] Specifically, the time point of the start of the water discharge of the water gun can be queried in the Internet of Things system within N minutes before and after the t1 time point, the water discharge time record table is stored in the Internet of Things system, the last water discharge time point is obtained, and the last water discharge time point is taken as the start time t2 of the grain moistening of the grain pile, or the time point closest to the first time point t1 is queried from the water discharge time record table, and the time point closest to the first time point t1 is taken as the start time t2 of the grain moistening of the grain pile.

[0111] For example, for each grain pile in a plurality of continuous frames of images, when it is determined that the grain pile category of the grain pile changes from the wet grain category to the non-wet grain category, and the grain pile is the non-wet grain category within a continuous preset fifth frame number, the value in the first preset indication tag of the grain pile is the first preset value, and the value in the second preset indication tag is the second preset value, a second time point is obtained, the second time point is taken as the wet grain end time point of the grain pile, the value in the first preset indication tag of the grain pile is set to the fourth preset value, and the value in the second preset indication tag is set to the third preset value.

[0112] That is, when the grain pile category changes from the wet grain category to the non-wet grain category, and the grain pile is the non-wet grain category within a continuous preset fifth frame number, the values in the flag1 and the flag2 of the current grain pile are queried, if the flag1 and the flag2 are both T, a second time point t3 is obtained as the wet grain end time point of the grain pile, and the flag1 and the flag2 are both set to F.

[0113] In the embodiments of the present application, the wet grain end time point can be subtracted from the wet grain start time point to obtain the wet grain duration of the grain pile, that is, t3-t2 can be taken as the wet grain duration of the grain pile.

[0114] It should be noted that the preset first frame number, the preset second frame number, the preset third frame number, the preset fourth frame number, and the preset fifth frame number in the embodiments of the present application can be flexibly set by the developer, for example, the preset first frame number can be set to 20, the preset second frame number can be set to 10, the preset third frame number and the preset fourth frame number can be set to 500, and the preset fifth frame number can be set to 20. Of course, it can be understood that the preset first frame number and the preset second frame number can also be set to 1, at this time, when it is determined that the category of a certain grain pile changes from the non-wet grain category to the wet grain category, the value in the first preset indication tag of the grain pile can be set to the first preset value, and when the first position of a certain grain pile in the same frame of image intersects with the second position of the water gun, the value in the second preset indication tag of the grain pile can be set to the second preset value.

[0115] In order to better understand the wet grain duration automatic statistical method provided in the embodiments of the present application, a specific example is described below. In this example, the corresponding network model can be obtained according to the above model training process, and the network model includes the above-mentioned grain pile detection model and the water gun detection model. The first preset indication tag in the example is flag1, the second preset indication tag is flag2, the first preset value and the second preset value are both T, the third preset value and the fourth preset value are both F, the preset third frame number and the preset fourth frame number are both 500, the preset first frame number and the preset fifth frame number are set to 20, and the preset second frame number is set to 10.

[0116] The grain moistening time automatic statistical method in the example can be seen from Figure 4 as shown, comprising the following steps:

[0117] S41: input each frame image obtained into a network model for inference.

[0118] Each frame image here refers to continuous images obtained after processing each frame image of a video image after video image acquisition of a distillery.

[0119] S42: when it is determined that the category of the grain pile in the image changes from a non-grain moistening category to a grain moistening category, and the grain moistening category is maintained for 20 consecutive frames, flag1=T.

[0120] S43: when it is determined that the grain pile in the image is flag2=F for 500 consecutive frames, the grain pile flag1=F.

[0121] S44: when it is determined that the first position of the grain pile and the second position of the water gun intersect, and the intersection is maintained for 10 consecutive frames, flag2=T.

[0122] S45: when it is determined that the grain pile in the image is flag1=F for 500 consecutive frames, the grain pile flag2=F.

[0123] S46: obtain a time point t1 of the grain pile with flag1=T and flag2=T.

[0124] S47: determine a grain moistening start time point t2 of the grain pile according to t1.

[0125] S48: when it is determined that the category of the grain pile in the image changes from the grain moistening category to the non-grain moistening category, and the non-grain moistening category is maintained for 20 consecutive frames, query the values of flag1 and flag2 of the grain pile.

[0126] S49: when the grain pile flag1 and flag2 are both T, take the current time point as a grain moistening end time point t3 of the grain pile,

[0127] S50: take t3-t2 as the grain moistening time of the grain pile.

[0128] Through the grain moistening time automatic statistical method provided in the example, the automatic statistics of the grain moistening time is realized, which can greatly improve the work efficiency of artisans in the grain moistening process, and the error is relatively low. Table 1 below reflects the error of the grain moistening time statistical method provided in the example:

[0129] Table 1

[0130]

[0131] The grain moistening time automatic statistical method provided by the embodiments of the present application is the first combination of machine vision and liquor brewing grain moistening process, the first practice of deep learning application in the grain moistening process, which helps to improve the scientific nature of data statistics, liberate manpower, improve productivity, and provide scientific data support for yield analysis. In addition, the grain pile detection model based on YOLOv5 in the embodiments of the present application adds an upsampling layer in the upsampling module, which is more sensitive to the identification of small target features. In the grain moistening process, under the imaging environment of the embodiments of the present application, the accuracy of grain pile detection and the accuracy of grain pile and non-grain pile classification can be greatly improved. The grain pile detection based on YOLOv5 in the embodiments of the present application uses EIOU, which reduces the model convergence speed under the premise of ensuring detection accuracy, and the offline training time of the model reaches 2 / 3 of the original.

[0132] In addition, the grain moistening time automatic statistical method provided by the embodiments of the present application combines deep learning and the Internet of Things, combines the intelligence of deep learning and the accuracy of the Internet of Things, and greatly improves the accuracy of the grain moistening time automatic statistical method.

[0133] In summary, the grain moistening time automatic statistical method provided by the embodiments of the present application is a machine vision-based grain moistening time automatic statistical method, which can accurately identify grain piles, water guns and the like in the grain moistening process in real time through a monitoring camera or offline through stored videos, and automatically combine Internet of Things data to accurately count the grain moistening time. It has the characteristics of high accuracy, strong robustness and high efficiency. In the liquor brewing industry, it can replace the traditional manual monitoring and statistical method, improve productivity, liberate labor, and provide scientific and accurate data support for yield analysis.

[0134] It should be understood that, although each step in the above flowchart is displayed in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the above flowchart can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.

[0135] Embodiment two:

[0136] Based on the same inventive concept, the embodiments of the present application provide a grain moistening time automatic statistical device, please refer to Figure 5 It should be understood that the specific functions of the grain moistening time automatic statistical device can be referred to the description in the above, and the detailed description is appropriately omitted here to avoid repetition.

[0137] The grain moistening time automatic counting device comprises at least one software function unit which can be stored in the form of software or firmware in the memory or solidified in the operating system of the device. Specifically, the grain moistening time automatic counting device comprises:

[0138] The acquisition module 501 is configured to acquire video images of the wine factory building.

[0139] The first determination module 502 is configured to, for each frame of image of the video images, determine a first position of each grain pile in the image and a grain pile category of each grain pile in the image according to the image and a preset grain pile detection model, and determine a second position of a water gun in the image according to the image and a preset water gun detection model; the grain pile category is a grain moistening category or a non-grain moistening category.

[0140] The second determination module 503 is configured to determine a grain moistening start time point and a grain moistening end time point of each grain pile according to the first position, the grain pile category and the second position of the water gun in the image of a plurality of continuous frames of the image.

[0141] The third determination module 504 is configured to determine a grain moistening time of each grain pile according to the grain moistening start time point and the grain moistening end time point.

[0142] It should be noted that, for the sake of brevity of description, the content described in the above embodiments will not be repeated here.

[0143] Embodiment three:

[0144] The embodiment provides a grain moistening time automatic counting system, please refer to Figure 6 , the grain moistening time automatic counting system comprises a processor 601, a memory 602 and a camera 603, the memory 602 stores a computer program, the processor 601 and the memory 602 realize communication through a communication bus, the processor 601 executes the computer program to realize each step of each method in the above embodiment one, which will not be repeated here. It can be understood that Figure 6 , the structure shown in the figure is only schematic, the grain moistening time automatic counting system can further comprise more or less components than Figure 6 , or have a different configuration from Figure 6 .

[0145] The processor 601 can be an integrated circuit chip with a signal processing capability. The processor 601 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; or can be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The processor 601 can implement or execute the methods, steps, and logical block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0146] The memory 602 can include, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0147] The embodiment also provides a computer readable storage medium, such as a floppy disk, an optical disk, a hard disk, a flash memory, a U disk, an SD card, an MMC card, etc., in which one or more programs for implementing the above steps are stored. The one or more programs can be executed by the one or more processors 601 to implement the steps of the lubricating time length automatic statistical method in the above embodiment one, which will not be described here.

[0148] It should be noted that the diagrams provided in the embodiments only illustrate the basic concepts of the present application in a schematic manner, and only the components related to the present application are shown in the diagrams, not the number, shape and size of the components when actually implemented. The actual implementation of each component can be randomly changed in shape, number and proportion, and the layout pattern of the components can also be more complex. The structure, proportion, size, etc. shown in the diagrams attached to the present application are only used to cooperate with the content disclosed in the specification, so as to be understood and read by those skilled in the art, and do not limit the conditions that can be implemented by the present application, so they do not have technical significance. Any modification of structure, change of proportion relationship or adjustment of size, without affecting the effect and purpose that can be achieved by the present application, should still fall within the scope of the technical content disclosed by the present application. At the same time, the terms such as "upper", "lower", "left", "right", "middle" and "one" in the present specification are only for the convenience of clear understanding of the description, not for limiting the scope of the present application, and the change or adjustment of the relative relationship, without substantially changing the technical content, is also regarded as the scope of the present application.

[0149] Any combination of the technical features in the above embodiments can be made, and for the sake of brevity, not all possible combinations are described above, however, as long as the combination of the technical features does not exist in contradiction, it shall be considered within the scope of the present disclosure.

[0150] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it shall not be understood as a limitation on the patent scope of the present application. It shall be pointed out that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, and these shall be within the protection scope of the present application. Therefore, the patent protection scope of the present application shall be subject to the appended claims.

Claims

1. A method for automatically counting the duration of grain moistening, characterized in that, The method comprises: collecting video images of a distillery; for each frame of the video images, determining first positions of each grain pile in the images and grain pile categories of each grain pile in the images according to the images and a preset grain pile detection model, and determining a second position of a water gun in the images according to the images and a preset water gun detection model; the grain pile category is a grain moistening category or a non-grain moistening category; determining a grain moistening start time point and a grain moistening end time point of each grain pile according to the first positions of the grain pile in continuous multiple frames of the images, the grain pile categories, and the second position of the water gun in the images; determining a grain moistening duration of the grain pile according to the grain moistening start time point and the grain moistening end time point; the determination manner of the grain moistening start time point comprises: for each grain pile in continuous multiple frames of the images, when it is determined that the grain pile category of the grain pile changes from a non-grain moistening category to a grain moistening category, and the grain pile is of the grain moistening category within a continuous preset first frame number, setting a value in a first preset indication tag of the grain pile to a first preset value; when it is determined that the first position of the grain pile and the second position of the water gun in the same frame of the images intersect, and the intersection exists within a continuous preset second frame number, setting a value in a second preset indication tag of the grain pile to a second preset value; obtaining a first time point when the value in the first preset indication tag of the grain pile is the first preset value, and the value in the second preset indication tag of the grain pile is the second preset value; determining the grain moistening start time point of the grain pile according to the first time point; when it is determined that the value in the second preset indication tag of the grain pile is a third preset value within a continuous preset third frame number, setting the value in the first preset indication tag of the grain pile to a fourth preset value; when it is determined that the value in the first preset indication tag of the grain pile is the fourth preset value within a continuous preset fourth frame number, setting the value in the second preset indication tag of the grain pile to the third preset value; the determination manner of the grain moistening end time point comprises: for each grain pile in continuous multiple frames of the images, when it is determined that the grain pile category of the grain pile changes from a grain moistening category to a non-grain moistening category, and the grain pile is of the non-grain moistening category within a continuous preset fifth frame number, obtaining a second time point when the value in the first preset indication tag of the grain pile is the first preset value, and the value in the second preset indication tag of the grain pile is the second preset value; taking the second time point as the grain moistening end time point of the grain pile, setting the value in the first preset indication tag of the grain pile to the fourth preset value, and setting the value in the second preset indication tag to the third preset value.

2. The method of claim 1, wherein the length of time the food is moistened is automatically calculated. Before the determination of the first positions of each grain pile in the images and the grain pile categories of each grain pile in the images according to the images and a preset grain pile detection model, the method comprises: obtaining a first video sample image of a grain moistening stage in the distillery; Determine a first sample picture training set from the first video sample image; the first sample picture training set includes multiple first sample pictures, and each first sample picture is annotated with a corresponding grain pile position and a grain pile category for each grain pile; Train the grain pile detection model based on the first sample picture training set.

3. The automatic grain soaking time calculation method as described in claim 1, characterized in that, Before determining the second position of the water gun in the image according to the image and a preset water gun detection model, the method comprises: Obtain a second video sample image of a start stage of grain moistening in the winery; Determine a second sample picture training set from the second video sample image; the second sample picture training set includes multiple second sample pictures, and each second sample picture is annotated with a water gun position; Train the water gun detection model based on the second sample picture training set.

4. The method of claim 1, wherein the method further comprises: determining the length of time the food is moistened by the user; and automatically counting the length of time the food is moistened. The method comprises: Query a water discharge time record table of the water gun corresponding to the grain pile; Determine the start time point of grain moistening of the grain pile according to the first time point, which comprises:

5. A device for automatically counting the duration of grain moistening, characterized in that Query a water discharge time record table of the water gun corresponding to the grain pile; Determine the start time point of grain moistening of the grain pile according to the first time point, which comprises: Query a water discharge time record table of the water gun corresponding to the grain pile; Determine the start time point of grain moistening of the grain pile according to the first time point, which comprises: The device comprises: A collection module configured to collect video images of a winery; A first determination module configured to, for each frame of the video images, determine first positions of each grain pile and grain pile categories of each grain pile in the image according to the image and a preset grain pile detection model, and determine a second position of a water gun in the image according to the image and a preset water gun detection model; the grain pile category is a grain moistening category or a non-grain moistening category; The second determining module is configured to determine the grain moistening start time point and the grain moistening end time point of each grain pile in the plurality of continuous images according to the first position corresponding to each grain pile, the grain pile category, and the second position of the water gun in the images; for each grain pile in the plurality of continuous images, when it is determined that the grain pile category of the grain pile changes from the non-grain moistening category to the grain moistening category, and the grain pile is of the grain moistening category within a continuous preset first frame number, the value in the first preset indication tag of the grain pile is set to a first preset value; when it is determined that the first position of the grain pile intersects with the second position of the water gun in the same image, and the intersection exists within a continuous preset second frame number, the value in the second preset indication tag of the grain pile is set to a second preset value; a first time point is obtained when the value in the first preset indication tag of the grain pile is the first preset value, and the value in the second preset indication tag of the grain pile is the second preset value; the grain moistening start time point of the grain pile is determined according to the first time point; when it is determined that the value in the second preset indication tag of the grain pile is a third preset value within a continuous preset third frame number, the value in the first preset indication tag of the grain pile is set to a fourth preset value; when it is determined that the value in the first preset indication tag of the grain pile is the fourth preset value within a continuous preset fourth frame number, the value in the second preset indication tag of the grain pile is set to the third preset value; for each grain pile in the plurality of continuous images, when it is determined that the grain pile category of the grain pile changes from the grain moistening category to the non-grain moistening category, and the grain pile is of the non-grain moistening category within a continuous preset fifth frame number, a second time point is obtained when the value in the first preset indication tag of the grain pile is the first preset value, and the value in the second preset indication tag is the second preset value; the second time point is taken as the grain moistening end time point of the grain pile, the value in the first preset indication tag of the grain pile is set to the fourth preset value, and the value in the second preset indication tag is set to the third preset value; The third determining module is configured to determine the grain moistening duration of the grain pile according to the grain moistening start time point and the grain moistening end time point.

6. A system for automatically counting the duration of grain moistening, characterized in that The computer readable storage medium stores a computer program, and the computer program is executed by at least one processor to implement the method according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by at least one processor to implement the method according to any one of claims 1-4.

Images