Feeding Monitoring Method, Device, Equipment and Storage Medium for Garbage Feeding Port

Abstract

The present application discloses a feeding monitoring method, device, equipment and storage medium for a garbage feeding port. By acquiring each feeding image of the garbage feeding port; using a contour recognition model to process each feeding image to obtain each garbage material bitmap marked with the contour of the garbage pile; for each garbage material bitmap, determining the area ratio occupied by the garbage pile in the feeding port; acquiring the time node corresponding to this garbage material bitmap and determining the time node when the corresponding feeding port is empty; calculating the time interval; according to the time interval and the area ratio corresponding to each garbage material bitmap, predicting the time interval between any time node and the time node when the corresponding feeding port is empty, so as to achieve feeding monitoring. Compared with manual monitoring, the present application can more accurately determine the actual feeding situation of the garbage feeding port, which can help the staff reasonably arrange the time and resources for garbage feeding or other treatment processes, improve the garbage feeding efficiency, and save time and effort.

Description

Technical Field

[0001] The present application relates to the technical field of garbage feeding monitoring, and particularly relates to a feeding monitoring method, device, equipment and storage medium for a garbage feeding port. Background Art

[0002] In recent years, environmental protection has become a hot topic globally, and all sectors of society have given great attention to it. With the progress of technology and the pursuit of sustainable development, a series of innovative industrial environmental protection technologies have emerged continuously, and there are higher requirements for the effect of garbage treatment. Garbage is generally put in from the garbage feeding port, so the monitoring of the garbage feeding port is crucial. For example, paying attention to whether the current feeding is completed in real time, judging when the feeding port is emptied, when the next round of garbage feeding work should start, etc., and these processes are all related to the overall efficiency of garbage treatment.

[0003] However, the existing methods for monitoring the garbage feeding port stay at manual monitoring, which requires staff to continuously monitor the feeding situation of the feeding port. This method is easily affected by human factors and leads to misjudgment, seriously affecting the efficiency of garbage treatment. Summary of the Invention

[0004] In view of this, the present application provides a feeding monitoring method, device, equipment and storage medium for a garbage feeding port, which is used to solve the problem that the existing methods for monitoring the garbage feeding port stay at manual monitoring, which requires staff to continuously monitor the feeding situation of the feeding port. This method is easily affected by human factors and leads to misjudgment, seriously affecting the efficiency of garbage treatment.

[0005] To achieve the above objectives, the following solutions are proposed:

[0006] In a first aspect, a feeding monitoring method for a garbage feeding port includes:

[0007] Obtain each feeding image during the feeding process of the garbage feeding port;

[0008] Process each of the feeding images by using a pre-trained contour recognition model to obtain each garbage material bitmap marked with the contour of the garbage pile;

[0009] For each of the garbage material bitmaps, determine the area ratio occupied by the garbage pile in the feeding port according to the marked contour of the garbage pile;

[0010] Obtain the time node corresponding to the garbage material bitmap, and determine the time node when the feeding port is empty corresponding to this time node;

[0011] Calculate the time interval between the time node corresponding to the garbage material bitmap and the time node when the feeding port is empty;

[0012] Predict the time interval between any time node and the time node when the corresponding feeding port is empty according to the time interval corresponding to each waste material bitmap and the area ratio, so as to realize feeding monitoring.

[0013] Preferably, the contour recognition model includes a backbone network module, a neck network module, and a head network module;

[0014] Among them, the input end of the backbone network module is used as the input end of the contour recognition model, the output end of the backbone network module is connected to the input end of the neck network module, the output end of the neck network module is connected to the input end of the head network module, and the output end of the head network module is used as the output end of the contour recognition model.

[0015] Preferably, the backbone network module includes a first downsampling unit, a second downsampling unit, a third downsampling unit, a fourth downsampling unit, a fifth downsampling unit, a first feature extraction unit, a second feature extraction unit, a third feature extraction unit, a fourth feature extraction unit, and a spatial pyramid pooling unit;

[0016] Among them, the input end of the first downsampling unit is used as the input end of the backbone network module, the output end of the first downsampling unit is connected to the input end of the second downsampling unit, the output end of the second downsampling unit is connected to the input end of the first feature extraction unit, the output end of the first feature extraction unit is connected to the input end of the third downsampling unit, the output end of the third downsampling unit is connected to the input end of the second feature extraction unit, the output end of the second feature extraction unit is connected to the input end of the fourth downsampling unit, the output end of the fourth downsampling unit is connected to the input end of the third feature extraction unit, the output end of the third feature extraction unit is connected to the input end of the fifth downsampling unit, the output end of the fifth downsampling unit is connected to the input end of the fourth feature extraction unit, and the output end of the fourth feature extraction unit is connected to the input end of the spatial pyramid pooling unit;

[0017] The output ends of the second feature extraction unit, the third feature extraction unit, and the spatial pyramid pooling unit are respectively used as the first output end, the second output end, and the third output end of the backbone network module, and are all connected to the input end of the neck network module.

[0018] Preferably, the neck network module includes a first upsampling unit, a second upsampling unit, a first fusion unit, a second fusion unit, a third fusion unit, a fourth fusion unit, a fifth feature extraction unit, a sixth feature extraction unit, a seventh feature extraction unit, a first convolution unit, and a second convolution unit;

[0019] Among them, the input end of the second fusion unit, the input end of the first fusion unit, and the input end of the first upsampling unit are respectively used as the first input end, the second input end, and the third input end of the neck network module;

[0020] The first output end of the second fusion unit is used as the first output end of the neck network module, the first output end of the sixth feature extraction unit is used as the second output end of the neck network module, and the output end of the seventh feature extraction unit is used as the third output end of the neck network module;

[0021] The first input end, the second input end, and the third input end of the neck network module are respectively connected to the first output end, the second output end, and the third output end of the backbone network module;

[0022] The output end of the first upsampling unit and the second output end of the backbone network module are both connected to the second input end of the neck network module, the input end of the fifth feature extraction unit and the input end of the third fusion unit are both connected to the output end of the first fusion unit, the output end of the fifth feature extraction unit is connected to the input end of the second upsampling unit, the output end of the second upsampling unit and the first output end of the backbone network module are both connected to the first input end of the neck network module, the second output end of the second fusion unit is connected to the input end of the first convolutional unit, the output end of the first convolutional unit and the output end of the first fusion unit are both connected to the input end of the third fusion unit, the output end of the third fusion unit is connected to the input end of the sixth feature extraction unit, the second output end of the sixth feature extraction unit is connected to the input end of the second convolutional unit, the output end of the second convolutional unit is connected to the input end of the fourth fusion unit, the input end of the fourth fusion unit is further connected to the input end of the first upsampling unit, and the output end of the fourth fusion unit is connected to the input end of the seventh feature extraction unit.

[0023] Preferably, the structures of the first feature extraction unit, the second feature extraction unit, the third feature extraction unit, the fourth feature extraction unit, the fifth feature extraction unit, the sixth feature extraction unit, and the seventh feature extraction unit are the same, and they all serve as feature extraction units. The feature extraction unit includes: a first CBS layer, a splitting layer, a comprehensive bottleneck layer, a splicing layer, and a second CBS layer;

[0024] Among them, the input end of the first CBS layer is used as the input end of the feature extraction unit;

[0025] The output end of the first CBS layer is connected to the input end of the splitting layer. The third output end of the splitting layer is connected to the input end of the comprehensive bottleneck layer. The first output end of the splitting layer, the second output end of the splitting layer, and the output end of the comprehensive bottleneck layer are all connected to the input end of the splicing layer. The output end of the splicing layer is connected to the input end of the second CBS layer;

[0026] The output end of the second CBS layer serves as the output end of the feature extraction unit.

[0027] Preferably, for determining the area ratio of the garbage heap in the feeding port in the garbage material bitmap according to the marked garbage heap contour, it includes:

[0028] Convert the garbage material bitmap into a binary map;

[0029] Extract the segmentation mask corresponding to the garbage heap contour from the binary map;

[0030] Extract the segmentation contour in the segmentation mask, and calculate the area of the garbage heap contour by using the segmentation contour;

[0031] Obtain the width and height of the garbage material bitmap;

[0032] Calculate the area when the garbage bin is full according to the preset ratio, width, and height;

[0033] Divide the area of the garbage heap contour by the area when the garbage bin is full to obtain the area ratio of the garbage heap in the feeding port.

[0034] Preferably, for predicting the time interval between any time node and the time node when the corresponding feeding port is empty according to the time interval corresponding to each garbage material bitmap and the area ratio, it includes:

[0035] Establish a falling experience function according to the time interval corresponding to each garbage material bitmap and the area ratio;

[0036] For each garbage material bitmap, determine the garbage material bitmaps corresponding to the previous preset number of frames and the garbage material bitmaps corresponding to the next preset number of frames at the time node corresponding to this garbage material bitmap as each adjacent material bitmap;

[0037] Determine the area ratio of each adjacent material bitmap;

[0038] Calculate the garbage falling speed corresponding to this garbage material bitmap based on the area ratio;

[0039] Establish a time interval function according to the falling experience function and the garbage falling speed corresponding to each garbage material bitmap;

[0040] Predict the time interval between any time node and the time node when the corresponding feeding port is empty by using the time interval function.

[0041] In a second aspect, a feeding monitoring device for a garbage feeding port includes:

[0042] An acquisition module for acquiring each feeding image during the feeding process of the garbage feeding port;

[0043] A model processing module for processing each of the feeding images by using a pre-trained contour recognition model to obtain each garbage material bitmap marked with the contour of the garbage pile;

[0044] An area ratio calculation module for, for each of the garbage material bitmaps, determining the area ratio of the garbage pile in the feeding port in the corresponding garbage material bitmap according to the marked contour of the garbage pile;

[0045] A time node determination module for obtaining the time node corresponding to the garbage material bitmap and determining the time node when the corresponding feeding port is empty;

[0046] A time interval calculation module for calculating the time interval between the time node corresponding to the garbage material bitmap and the time node when the feeding port is empty;

[0047] A prediction module for predicting the time interval between any time node and the time node when the corresponding feeding port is empty according to the time interval corresponding to each of the garbage material bitmaps and the area ratio, so as to implement feeding monitoring.

[0048] In a third aspect, a feeding monitoring device for a garbage feeding port includes a memory and a processor;

[0049] The memory is used for storing a program;

[0050] The processor is used for executing the program to implement each step of the feeding monitoring method for the garbage feeding port as described in any item of the first aspect.

[0051] In a fourth aspect, a storage medium stores a computer program, and when the computer program is executed by a processor, each step of the feeding monitoring method for the garbage feeding port as described in any item of the first aspect is implemented.

[0052] As can be seen from the above technical solution, the present application obtains each feeding image of the garbage feeding port during the feeding process; processes each of the feeding images using a pre-trained contour recognition model to obtain each garbage material bitmap marked with the contour of the garbage pile; for each of the garbage material bitmaps, determines the area ratio occupied by the garbage pile in the feeding port according to the marked contour of the garbage pile; obtains the time node corresponding to this garbage material bitmap, and determines the time node when the feeding port is empty corresponding to this time node; calculates the time interval between the time node corresponding to this garbage material bitmap and the time node when the feeding port is empty; predicts the time interval between any time node and the time node when the corresponding feeding port is empty according to the time interval corresponding to each garbage material bitmap and the area ratio, so as to realize feeding monitoring. The present application first obtains each feeding image of the garbage feeding port during the feeding process, which can more accurately determine the actual feeding situation of the garbage feeding port compared with manual monitoring. The pre-trained contour recognition model can process the feeding images to obtain each garbage material bitmap marked with the contour of the garbage pile, reducing the subjectivity of manual judgment. Since it is necessary to determine the time interval information, it is crucial to determine the area ratio occupied by the contour of the garbage pile in each garbage material bitmap in the feeding port. By comparing the time node of each garbage material bitmap with the time node when the corresponding feeding port is empty, the time interval can be accurately obtained, which can reduce the workload of manual recording and judgment, avoid time recording errors caused by human factors, and can predict the garbage feeding situation at any time node, thereby helping the staff to reasonably arrange the time and resources for garbage feeding or other treatment processes, improve the garbage feeding efficiency, save time and effort, and can also reduce the feeding or other treatment delays caused by excessive garbage accumulation. Description of the Drawings

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0054] Figure 1 It is an optional flowchart of a feeding monitoring method for a garbage feeding port provided by an embodiment of the present application;

[0055] Figure 2 It is a schematic diagram of a garbage material marked with the contour of a garbage pile provided by an embodiment of the present application;

[0056] Figure 3 It is a schematic structural diagram of a contour recognition model provided by an embodiment of the present application;

[0057] Figure 4 A structural schematic diagram of a backbone network module provided by an embodiment of the present application;

[0058] Figure 5 A structural schematic diagram of a spatial pyramid pooling unit provided by an embodiment of the present application;

[0059] Figure 6 A structural schematic diagram of a neck network module provided by an embodiment of the present application;

[0060] Figure 7 A structural schematic diagram of a Segment unit provided by an embodiment of the present application;

[0061] Figure 8 A structural schematic diagram of a feature extraction unit provided by an embodiment of the present application;

[0062] Figure 9 A structural schematic diagram of a feeding monitoring device for a garbage feeding port provided by an embodiment of the present application;

[0063] Figure 10 A structural schematic diagram of a feeding monitoring device for a garbage feeding port provided by an embodiment of the present application. Detailed implementation manners

[0064] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0065] The present application can be applied to any scenario including a garbage feeding port, such as a waste power plant. The waste power plant converts waste into electric energy, which not only helps to solve the increasingly serious garbage treatment problem, but also adds new vitality to the renewable energy field.

[0066] The present invention can be used in many general-purpose or special-purpose computing device environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor devices, distributed computing environments including any of the above devices or equipment, and so on.

[0067] An embodiment of the present invention provides a feeding monitoring method for a garbage feeding port. This method can be applied to various computer terminals or intelligent terminals, and its execution subject can be a processor or a server of a computer terminal or an intelligent terminal. The method flow chart of the method is as Figure 1 shown, and specifically includes:

[0068] S1: Obtain each feeding image during the feeding process at the garbage feeding port.

[0069] Nowadays, in various fields, image recognition technology has become a key factor for improvement. This application also applies this idea to monitor the feeding process. First, video data for monitoring the garbage feeding port is obtained. The video data of the garbage feeding port is the video data collected by a camera or monitor for monitoring the garbage feeding port. The field of view of the camera or monitor should cover the garbage feeding port and be able to clearly monitor the entire process of garbage being fed, including the moment of input, the falling process, the process of the garbage completely entering the garbage bin from the feeding port, etc., to ensure the integrity of the feeding process, which can guarantee the monitoring accuracy. Then, by performing image segmentation on the monitoring video data, each feeding image can be obtained.

[0070] It can be understood that the monitoring video data contains multiple frames of images. Each frame of image corresponds to a time node, or multiple frames of images correspond to a time node simultaneously. The length of one frame can be determined according to the duration of the monitoring video data. The monitoring video data may contain multiple rounds of feeding processes or only one round of feeding process. Therefore, the time period for segmenting the monitoring video data can be determined according to the situation. For example, one feeding image is segmented every preset time period.

[0071] By obtaining the monitoring video data and then segmenting the images, the integrity and certainty of the data can be ensured, avoiding the discontinuity and instability of manual monitoring.

[0072] S2: Process each of the feeding images using a pre-trained contour recognition model to obtain garbage material bitmaps marked with the contours of the garbage piles.

[0073] The contour recognition model is trained with a dataset containing countless feeding sample images as the training sample set, and the true contour of the garbage pile in each feeding sample image is used as the sample label. Feeding images for monitoring the garbage feeding port can be collected from various industries or scenarios. In one example, a sample set containing 1300 feeding sample images is collected. To improve the training accuracy of the model, these feeding sample images can be pre-processed, and various data augmentation methods can be adopted to expand the data volume, making the model more adaptable and capable of processing various different feeding images, including flipping, rotating, scaling, translating, cropping, color transformation, mirror symmetry, adding noise, etc. After training is completed, the training set and the test set are split in a ratio of 8:2. The training set is used to train the contour recognition model, and the test set is used to test the trained contour recognition model. If the test passes, the training is completed. If the test fails, training continues to ensure the usability of the model.

[0074] Using a pre-trained contour recognition model can automatically identify the contour of the garbage dump, reducing the subjectivity and error of manual judgment. Moreover, the contour recognition model can quickly and accurately process a large number of feeding images, improving the monitoring efficiency.

[0075] The contour of the garbage dump involved in this application is the contour of the garbage dump itself, so that the calculated area ratio is relatively accurate.

[0076] S3: For each of the said garbage feeding position maps, determine the area ratio of the garbage dump in the feeding port in this garbage feeding position map according to the marked contour of the garbage dump.

[0077] Since the contour of the garbage dump is marked in the garbage feeding position map, and at the same time, the area of the feeding port in the garbage feeding position map can also be determined, the area ratio of the garbage dump in the feeding port can be calculated. According to the area ratio, the stacking situation of the garbage can be quantified, providing accurate data support for the subsequent time interval prediction. This will be more objective than manual visual inspection and avoid a high misjudgment rate.

[0078] Specifically, considering that generally, when monitoring the feeding port, it is not only for the feeding port, but also for the surrounding areas such as the periphery of the garbage bin and the ground around the feeding port. Therefore, the feeding port does not occupy the entire garbage feeding position map. And in the subsequent monitoring process, the focus is on the situation of garbage being put into the feeding port. So the area ratio is calculated according to the area of the garbage dump and the area of the feeding port, which will be relatively accurate.

[0079] S4: Obtain the time node corresponding to this garbage feeding position map, and determine the time node when the feeding port corresponding to this time node is empty.

[0080] Each garbage feeding position map will correspond to a time node. This time node is not the specific time corresponding to the garbage feeding position map (such as accurate to seconds), but an information indicating the time scale. One or more garbage feeding position maps may correspond to the same time node.

[0081] It should be noted that if the time node corresponding to this garbage feeding position map is the first time node, and the time node when the corresponding feeding port is empty is the second time node, then the second time node is equal to or later than the first time node. That is to say, only the time node corresponding to when all the garbage in this garbage feeding position map enters the garbage bin below the feeding port, or the situation where there is no garbage at the feeding port after garbage feeding is considered as the time node when the feeding port corresponding to this time node is empty as stipulated above.

[0082] In this way, the feeding situation can be analyzed through the recording and comparison between time nodes.

[0083] S5: Calculate the time interval between the time node corresponding to this waste material bitmap and the time node when the feeding port is empty.

[0084] By calculating the time interval between two time nodes, the speed and frequency of the garbage pile entering the feeding port or falling can be quantified. Subsequently, the subsequent monitoring can provide data support for the management and scheduling of the garbage feeding or other treatment processes. This method is more accurate and efficient than manual monitoring and can detect and handle abnormal situations in a timely manner.

[0085] Specifically, the time interval can be measured in time dimensions with lower levels such as minutes and seconds. For example, the time node corresponding to a waste material bitmap is 14:23, and the time node when the corresponding feeding port is empty is 14:24, then the time interval is 1 minute / 60 seconds.

[0086] S6: Predict the time interval between any time node and the time node when the corresponding feeding port is empty according to the time interval corresponding to each waste material bitmap and the area ratio to achieve feeding monitoring.

[0087] Since the waste material bitmaps involved in the above steps do not represent the images of the feeding port at every moment / every frame / every second during the feeding process, the determined time interval and area ratio can be used to summarize and analyze the time interval between any time node and the time node when the corresponding feeding port is empty. The time node can be any time dimension representing time, such as frame / second / moment, etc. Therefore, it is possible to determine the situation of each time node during the current garbage feeding work, so as to achieve the purpose of early warning of garbage accumulation, reduce the idle time between two rounds of feeding, improve efficiency, and help the staff reasonably arrange the time and resources of garbage feeding or other treatment processes. For example, prepare for the next round of feeding before the predicted time interval of the current round of feeding ends.

[0088] This application first obtains each feeding image during the feeding process at the garbage feeding port, which can more accurately determine the actual feeding situation of the garbage feeding port compared to manual monitoring. The pre-trained contour recognition model can process the feeding images to obtain each garbage bitmap marked with the contour of the garbage pile, reducing the subjectivity of manual judgment. Since it is necessary to determine the time interval information, it is crucial to determine the area ratio occupied by the garbage pile contour in the feeding port in each garbage bitmap. By comparing the time node of each garbage bitmap with the time node when the feeding port is empty, the time interval can be accurately obtained, reducing the workload of manual recording and judgment, avoiding time recording errors caused by human factors, and being able to predict the garbage feeding situation at any time node. Thus, it can help the staff reasonably arrange the time and resources for garbage feeding or other treatment processes, improve the garbage feeding efficiency, save time and effort, and also reduce the feeding delay or other treatment delays caused by excessive garbage accumulation.

[0089] In one example, a garbage bitmap marked with the contour of the garbage pile is as Figure 2 shown. The area ratio in this figure is 24%. During the process of the contour recognition model identifying the contour of the garbage pile, there is also a process of garbage pile recognition and confirmation. For example, first, the minimum bounding rectangle of the suspected garbage pile is identified, and the probability of the suspected garbage pile (rubbish heap) within this rectangle is obtained as 0.95. Since the probability is relatively high, it is determined as the garbage pile, and then the specific contour of the garbage pile is identified.

[0090] In the method provided by the embodiment of the present invention, for the contour recognition model, its structure is as Figure 3 shown, including a backbone network module, a neck network module, and a head network module;

[0091] Among them, the input end of the backbone network module serves as the input end of the contour recognition model, the output end of the backbone network module is connected to the input end of the neck network module, the output end of the neck network module is connected to the input end of the head network module, and the output end of the head network module serves as the output end of the contour recognition model.

[0092] Specifically, this contour recognition model belongs to a deep learning model. The backbone network module adopts a Backbone network; the neck network (Neck network) module can adopt a feature pyramid network with a structure of PAN-FPN (Path Aggregation Network-Feature Pyramid Network), and the head network module can adopt a Head network.

[0093] The structures of the backbone network module, the neck network module, and the head network module will be described in detail below.

[0094] 1) Backbone network module.

[0095] As Figure 4 shown, the backbone network module includes a first downsampling unit, a second downsampling unit, a third downsampling unit, a fourth downsampling unit, a fifth downsampling unit, a first feature extraction unit, a second feature extraction unit, a third feature extraction unit, a fourth feature extraction unit, and a spatial pyramid pooling unit;

[0096] Among them, the input end of the first downsampling unit serves as the input end of the backbone network module. The output end of the first downsampling unit is connected to the input end of the second downsampling unit. The output end of the second downsampling unit is connected to the input end of the first feature extraction unit. The output end of the first feature extraction unit is connected to the input end of the third downsampling unit. The output end of the third downsampling unit is connected to the input end of the second feature extraction unit. The output end of the second feature extraction unit is connected to the input end of the fourth downsampling unit. The output end of the fourth downsampling unit is connected to the input end of the third feature extraction unit. The output end of the third feature extraction unit is connected to the input end of the fifth downsampling unit. The output end of the fifth downsampling unit is connected to the input end of the fourth feature extraction unit. The output end of the fourth feature extraction unit is connected to the input end of the spatial pyramid pooling unit;

[0097] The output ends of the second feature extraction unit, the third feature extraction unit, and the spatial pyramid pooling unit respectively serve as the first output end, the second output end, and the third output end of the backbone network module, and are all connected to the input end of the neck network module.

[0098] Specifically, the structures of the first downsampling unit, the second downsampling unit, the third downsampling unit, the fourth downsampling unit, and the fifth downsampling unit are the same. Each is a convolutional group (CBS) composed of 1 Convolution layer, 1 BatchNorm layer, and 1 SILU layer. The convolutional kernel k of these downsampling units is 3, the stride s is 2, and the padding p is 1. The backbone network module is mainly used for feature extraction. In one example, the resolution of the input feeding image to the backbone network module is 640×640, and the number of channels is 3. It will first pass through the first downsampling unit for downsampling, and the resolution becomes 320×320, and the number of channels becomes 64. Then it passes through the second downsampling unit and the first feature extraction unit, and the resolution becomes 160×160, and the number of channels becomes 128. During this process, the size of the feeding image does not change. Then it passes through three groups of downsampling units and feature extraction units in sequence. That is to say, the feeding image with a resolution of 160×160 and 128 channels first passes through the third downsampling unit and the second feature extraction unit in sequence to obtain a feeding image P4 with a resolution reduced to 80×80 and 256 channels. The feeding image P4 then passes through the fourth downsampling unit and the third feature extraction unit in sequence to obtain a feeding image P6 with a resolution reduced to 40×40 and 512 channels. The feeding image P6 then passes through the fifth downsampling unit, the fourth feature extraction unit, and the spatial pyramid pooling unit in sequence to obtain a feeding image P9 with a resolution reduced to 20×20 and 1024 channels, thereby outputting three different scales of feeding images from the first output end, the second output end, and the third output end of the backbone network module respectively.

[0099] The spatial pyramid pooling unit (Spatial Pyramid Pooling Fast, SPPF) is an improved structure, and its structure is as Figure 5 shown, which includes two CBS1 layers. The convolutional kernel k of these two CBS1 layers is 3, the stride s is 1, and the padding p is 0. It also includes three MaxPool2d layers (layers for two-dimensional maximum pooling Max Pooling operation) and 1 Concat layer.

[0100] This way of hierarchical downsampling and feature extraction can gradually extract abstract image information. At the same time, combined with multi-scale output, it can provide rich feature representations for the subsequent neck network module, making the performance of the contour recognition model better.

[0101] 2) Neck network module.

[0102] The structure of the neck network module is as Figure 6As shown, it includes a first upsampling unit, a second upsampling unit, a first fusion unit, a second fusion unit, a third fusion unit, a fourth fusion unit, a fifth feature extraction unit, a sixth feature extraction unit, a seventh feature extraction unit, a first convolution unit, and a second convolution unit;

[0103] Among them, the input end of the second fusion unit, the input end of the first fusion unit, and the input end of the first upsampling unit are respectively used as the first input end, the second input end, and the third input end of the neck network module;

[0104] The first output end of the second fusion unit is used as the first output end of the neck network module, the first output end of the sixth feature extraction unit is used as the second output end of the neck network module, and the output end of the seventh feature extraction unit is used as the third output end of the neck network module;

[0105] The first input end, the second input end, and the third input end of the neck network module are respectively connected to the first output end, the second output end, and the third output end of the backbone network module;

[0106] The output end of the first upsampling unit and the second output end of the backbone network module are both connected to the second input end of the neck network module, the input end of the fifth feature extraction unit and the input end of the third fusion unit are both connected to the output end of the first fusion unit, the output end of the fifth feature extraction unit is connected to the input end of the second upsampling unit, the output end of the second upsampling unit and the first output end of the backbone network module are both connected to the first input end of the neck network module, the second output end of the second fusion unit is connected to the input end of the first convolution unit, the output end of the first convolution unit and the output end of the first fusion unit are both connected to the input end of the third fusion unit, the output end of the third fusion unit is connected to the input end of the sixth feature extraction unit, the second output end of the sixth feature extraction unit is connected to the input end of the second convolution unit, the output end of the second convolution unit is connected to the input end of the fourth fusion unit, the input end of the fourth fusion unit is further connected to the input end of the first upsampling unit, and the output end of the fourth fusion unit is connected to the input end of the seventh feature extraction unit.

[0107] Specifically, the neck network module processes the three input images output by the backbone network module. First, P9 is upsampled by a factor of 2 by the first upsampling unit and output to the first fusion unit, where it is fused with P6 to obtain image F1 (with a resolution of 40×40); F1 then enters the second upsampling unit for upsampling by a factor of 2 after passing through the fifth feature extraction unit, and is output to the second fusion unit, where it is fused with P4 to obtain feature map T1 (with a resolution of 80×80), and T1 is output from the first output end of the neck network module; at the same time, T1 is input to the first convolution unit for convolution processing, and the output is given to the third fusion unit to be fused with F1 to obtain image F2 (with a resolution of 40×40); F2 passes through the sixth feature extraction unit to obtain feature map T2 (with a resolution of 40×40), and T2 is output from the second output end of the neck network module; at the same time, feature map T2 is input to the second convolution module for convolution, and then is fused with P9 in the fourth fusion unit to obtain F3 (with a resolution of 40×40); finally, F3 is processed by the seventh feature extraction unit to obtain feature map T3 (with a resolution of 40×40), and then is output from the third output end of the neck network module.

[0108] That is, after being processed by the neck network module, feature maps T1, T2, and T3 are finally output through its first output end, second output end, and third output end respectively, and these three feature maps are input to the head network module for further processing.

[0109] Feature maps of a single scale may not be able to fully represent the complex features of the target. Especially for small targets or occluded targets, it can be understood that images of different scales contain different levels of information. For example, P4 with a resolution of 80×80 belongs to a shallow feature map, which contains rich detail information but weak semantic information, while P9 with a resolution of 20×20, although lacking sufficient spatial details, contains stronger semantic information. Therefore, the neck network module set in this application can enhance the detection ability of the contour recognition model for multi-scale targets by fusing multi-scale image information and simultaneously utilizing rich detail information and deep semantic information, enabling the contour recognition model to better capture the global and local features of the target.

[0110] Among them, the first convolution unit and the second convolution unit have the same structure, both being CBS structures, with a convolution kernel k of 3, a stride s of 2, and a padding p of 1.

[0111] The PAN-FPN feature pyramid network consists of three parts: multi-scale feature extraction, bidirectional feature fusion, and enhanced feature representation. Therefore, after the backbone network module extracts feature maps of different scales, the top-down feature pyramid in the Feature Pyramid Network (FPN) structure in the neck network can be used to achieve the preliminary fusion of multi-scale features. Then, based on the FPN, the bottom-up path of the Path Aggregation Network (PAN) is introduced to gradually transfer the low-level features to the high-level, further enriching the multi-scale features. Through lateral connections, features of different scales are fused to ensure that each layer of features contains rich context information. Finally, through the fusion of bidirectional paths, the fused feature maps will contain richer context information and semantic information, which can enhance the detection ability of the contour recognition model for targets of different scales.

[0112] 3) Head network module.

[0113] The head network module can adopt the structure of the YOLACT (You Only Look At CoefficienTs) model, which can parallelly predict the prototype mask of the image and the mask coefficients of each Bbox instance. In the head network module, upsampling is performed on T1, T2, and T3 respectively to obtain a feature map of size 160×160 as a native mask for network prediction. The prediction head is used to generate the position, category, and mask coefficients of the prediction box. Then, through NMS (Non-Maximum Suppression), two targets are selected. The prediction masks of these two targets are generated by means of the native mask and mask coefficients. Then, the prediction masks of these two targets are cropped according to the prediction box, that is, cropped, and a threshold is set for binarization to obtain the final result.

[0114] This module contains 3 Segment units and 1 Proto (Prototype Unit) unit. The 3 Segment units simultaneously process the three types of feature maps T1, T2, and T3, and at the same time process T1 through the Proto unit. The structure of the Segment unit is as Figure 7 shown, including three branches, each branch contains 2 CBS2 layers, 1 Conv2d layer, and 1 Box.Loss layer (Bounding Box Loss Layer). The convolution kernel k of the CBS2 layer is 3, the stride s is 1, and the padding p is 1. The convolution kernel k of the Conv2d layer is 3, the stride s is 1, and the padding p is 0.

[0115] Optionally, the first feature extraction unit, the second feature extraction unit, the third feature extraction unit, the fourth feature extraction unit, the fifth feature extraction unit, the sixth feature extraction unit, and the seventh feature extraction unit have the same structure. For the convenience of explanation, these seven feature extraction units are all referred to as feature extraction units.

[0116] The feature extraction unit includes: a first CBS layer, a splitting layer, an integrated bottleneck layer, a splicing layer, and a second CBS layer;

[0117] Among them, the input end of the first CBS layer serves as the input end of the feature extraction unit;

[0118] The output end of the first CBS layer is connected to the input end of the splitting layer, the third output end of the splitting layer is connected to the input end of the integrated bottleneck layer, the first output end of the splitting layer, the second output end of the splitting layer, and the output end of the integrated bottleneck layer are all connected to the input end of the splicing layer, and the output end of the splicing layer is connected to the input end of the second CBS layer;

[0119] The output end of the second CBS layer serves as the output end of the feature extraction unit.

[0120] Specifically, the feature extraction unit C2f_Dual in the present application incorporates a dual convolution mechanism into the existing C2f module (the C2f module is a modular design for convolutional neural networks). By utilizing the principles of heterogeneous convolution and group convolution, the core structure Bottleneck of the existing C2f module is lightweight improved, which can reduce the computational complexity of the model without losing feature information. The structure of this feature extraction unit is as Figure 8 shown.

[0121] Taking the feature map T1 as an example for illustration, after T1 is input into the feature extraction unit, it will first pass through the first CBS layer. The first CBS layer includes a conv2d layer (with a convolution kernel size of 1×1), a BatchNorm layer (denoted as BN in the formula), and a SiLU activation function. That is, after T1 passes through the conv2d layer, the feature map is obtained, after passing through the BatchNorm layer, is obtained, and then after passing through the SiLU activation function, the final calculation result is obtained, denotes T1, and the formula is as follows:

[0122] ;

[0123] ;

[0124] ;

[0125] Then As the output of the first CBS layer, it is input to the splitting layer, and the torch.chunk function is introduced to split it, obtaining , two parts, which are compared with The number of channels is halved, that is ( is the number of channels).

[0126] ;

[0127] , These two parts are the two outputs of the splitting layer respectively. At the same time, there is also a third output, and the third output is also , and the comprehensive bottleneck layer needs to process this third output. Among them, there are n improved bottleneck blocks in the comprehensive bottleneck layer. Each time an improved bottleneck block passes, a feature map branch with the number of channels of will be output , that is to say, it needs to pass through n improved bottleneck blocks in sequence. Finally, the comprehensive bottleneck layer will output n feature map branches with the number of channels being half of the number of channels of the previous feature map respectively . Then, these n outputs are input to the splicing layer together with , for splicing, obtaining :

[0128] ;

[0129] The output number of channels of is:

[0130] ;

[0131] Finally, it passes through the second CBS layer (the structures of the first CBS layer and the second CBS layer are the same, the size of the convolution kernel k is 3, the stride s is 2, and the padding p is 1; among them, the k of the conv2d layer is 3, the stride s is 1, and the padding p is 0), that is, it passes through the conv2d layer in sequence to obtain , passes through the BatchNorm layer to obtain , and then passes through the SiLU activation function to obtain the final output graph , and the formula is as follows:

[0132] ;

[0133] ;

[0134] ;

[0135] That is, it is a waste material bit map marked with the outline of the garbage dump, which is the final output of the contour recognition model.

[0136] Furthermore, the improved bottleneck block is a residual module composed of 1 Dualconv layer and 1 third CBS layer, which includes two branches. Taking as an example, it takes as the input and output of one branch without any processing. At the same time, it takes as the input of the other branch. In this branch, successively passes through 1 third CBS layer (the convolution kernel k of this third CBS layer has a size of 1, a stride s of 1, and a padding p of 0) and 1 Dualconv layer, and outputs . Then, the outputs of these two branches are merged and concatenated to obtain:

[0137] .

[0138] Setting two branches can retain the original features of to avoid information loss, and can also accelerate the training process of the model. The Dualconv layer includes two parallel conv2d layers, that is, the feature map entering the Dualconv layer will be processed by the two conv2d layers simultaneously, and then the outputs of the two conv2d layers are fused. This double convolution combines the advantages of group convolution and heterogeneous convolution. The heterogeneous convolution combines the advantages of 3×3 convolution kernels and 1×1 convolution kernels. Among them, the 3×3 convolution kernel can capture more spatial information when extracting features, while the 1×1 convolution kernel can perform interaction and information integration between feature channels without increasing too many parameters and computational complexity; and the group convolution method adopted by the Dualconv layer divides the feature map into multiple groups, and each group of convolutions only processes a part of the corresponding input feature map, thus greatly reducing the complexity of the model.

[0139] The structures of these two conv2d layers are different. One conv2d layer has a convolution kernel k with a size of 3, a stride s of 1, a padding p of 1, and a number of groups g of 4, while the other conv2d layer has a convolution kernel k with a size of 1, a stride s of 1, and a padding p of 0.

[0140] In one instance, the size of the output feature map is , being the width and height of the output feature map. Then, in standard convolution, the input feature map passes through convolution filters of size in the standard convolution layer, where is the size of the convolution kernel, is the number of channels of the convolution kernel. Therefore, the standard convolutional layer performs (Floating Point Operations) with a total number of:

[0141] .

[0142] In the Dualconv layer of the present application, the number of convolution filter banks is used to control the proportion of convolution kernels in the convolution filters. For a given , in all channels, the proportion of the combined synchronous convolution kernels with a size of ( ) is , while the proportion of the remaining convolution kernels is ( ). Therefore, in the dual convolutional layer structure composed of convolution filter banks, the of the combined convolution kernels is:

[0143] ;

[0144] The remaining of the convolution kernels is:

[0145] ;

[0146] Then the total number is:

[0147] ;

[0148] Then, comparing the computational cost of the dual convolutional layer provided in the present application with that of the standard convolutional layer, the computational reduction ratio is:

[0149] .

[0150] Therefore, the dual convolutional layer provided in the present application has improved computational efficiency compared to the standard convolutional layer and significantly reduces the computational cost.

[0151] Through the improvement of some of the above-mentioned modules or units, the present application finally trains a lightweight contour recognition model, which can reduce the complexity and the number of parameters of the model, can significantly reduce the demand for computing resources while maintaining high accuracy, and can be adapted to various fields, especially for deployment in the actual industrial environment.

[0152] Next, the process of determining the area ratio of the garbage pile in the feeding port in the garbage material bitmap according to the marked garbage pile contour in the present application will be described in detail.

[0153] Convert the garbage material bitmap into a binary image;

[0154] Extract the segmentation mask corresponding to the garbage pile contour from the binary image;

[0155] Extract the segmentation contour in the segmentation mask, and calculate the area of the garbage pile contour by using the segmentation contour;

[0156] Obtain the width and height of the garbage material bitmap;

[0157] Calculate the area when the garbage bin is full according to the preset ratio, width and height;

[0158] Divide the area of the garbage pile contour by the area when the garbage bin is full to obtain the area ratio of the garbage pile in the feeding port.

[0159] Specifically, the mask is usually a binary image, in which the garbage pile contour and its internal area are white, the pixel value is 255, the background is white, and the pixel value is 0. The function in OpenCV (Open Source Computer Vision Library) can be used to find the segmentation contour of the garbage pile in the segmentation mask and calculate the area of the garbage pile contour. The area of the garbage pile contour actually refers to the area of the garbage pile, and the calculation formula is:

[0160] ;

[0161] The preset ratio can be determined according to the on-site feeding situation. This embodiment does not limit this. In one example, the area of the feeding port in the whole garbage material bitmap is approximately half of the whole garbage material bitmap. Therefore, the preset ratio is set to 1 / 2, the area of the garbage pile contour is denoted as , the width of the garbage material bitmap is denoted as , the height is denoted as , and thus the area ratio of the garbage pile in the feeding port is obtained, and the calculation formula is:

[0162] .

[0163] The following embodiments specifically explain the process of predicting the time interval between any time node and the time node when the corresponding feeding port is empty according to the time interval corresponding to each garbage material bitmap and the area ratio in the present application.

[0164] Establish a falling experience function based on the time interval corresponding to each of the waste material position maps and the area ratio;

[0165] For each of the waste material position maps, determine the waste material position maps corresponding to the previous preset number of frames and the waste material position maps corresponding to the next preset number of frames at the time node corresponding to this waste material position map as adjacent waste material position maps;

[0166] Determine the area ratio of each of the adjacent waste material position maps;

[0167] Calculate the waste falling speed corresponding to this waste material position map based on the area ratio;

[0168] Establish a time interval function according to the falling experience function and the waste falling speed corresponding to each of the waste material position maps;

[0169] Use the time interval function to predict the time interval between any time node and the time node when the feeding port is empty.

[0170] Specifically, denote the time node as , denote the time interval as , the area ratio corresponding to the time node is , use the and rule to obtain the falling experience function of the waste material level:

[0171] ;

[0172] is the area ratio, then select the waste material position maps of the preset number of frames earlier than the time node corresponding to this waste material position map, such as the first 1 - 3 frames before this waste material position map, and at the same time select the waste material position maps of the preset number of frames later than the time node corresponding to this waste material position map as adjacent waste material position maps, from which the waste falling speed corresponding to this waste material position map can be calculated .

[0173] Perform weighted analysis on the falling experience function and the waste falling speed , then the time interval between the current time node and the time node when the feeding port is empty can be predicted more accurately , where is the viscosity coefficient, is the waste falling rate coefficient of the feeding port, and establish the time interval function:

[0174] .

[0175] Therefore, the time interval function can be used to predict the time interval between any time node and the time node when the corresponding feeding port is empty. Then, by adding the time length corresponding to the time interval to the time node, it is possible to determine when the garbage of this round is completely fed, which is convenient for preparing for the next round of feeding and improving the overall feeding efficiency.

[0176] Corresponding to Figure 1 the method described above, an embodiment of the present invention further provides a feeding monitoring device for a garbage feeding port, which is used to Figure 1 For the specific implementation of the method in Figure 9 , the feeding monitoring device for the garbage feeding port provided by the embodiment of the present invention can be in a computer terminal or various mobile devices. Combining Figure 9 as shown in

[0177] An acquisition module 10, which is used to acquire each feeding image during the feeding process of the garbage feeding port;

[0178] A model processing module 20, which is used to process each of the feeding images by using a pre-trained contour recognition model to obtain each garbage material position map marked with the contour of the garbage pile;

[0179] An area ratio calculation module 30, which is used to determine, for each of the garbage material position maps, the area ratio occupied by the garbage pile in the feeding port according to the marked contour of the garbage pile;

[0180] A time node determination module 40, which is used to obtain the time node corresponding to this garbage material position map and determine the time node when the corresponding feeding port is empty;

[0181] A time interval calculation module 50, which is used to calculate the time interval between the time node corresponding to this garbage material position map and the time node when the feeding port is empty;

[0182] A prediction module 60, which is used to predict the time interval between any time node and the time node when the corresponding feeding port is empty according to the time interval corresponding to each of the garbage material position maps and the area ratio, so as to achieve feeding monitoring.

[0183] This application first obtains each feeding image during the feeding process of the garbage feeding port, which can more accurately determine the actual feeding situation of the garbage feeding port compared to manual monitoring. The pre-trained contour recognition model can process the feeding images to obtain each garbage material map marked with the contour of the garbage pile, reducing the subjectivity of manual judgment. Since it is necessary to determine the time interval information, it is crucial to determine the area ratio of the contour of the garbage pile in each garbage material map to the feeding port. By comparing the time nodes of each garbage material map with the time nodes when the feeding port is empty, the time interval can be accurately obtained, reducing the workload of manual recording and judgment, avoiding time recording errors caused by human factors, and being able to predict the garbage feeding situation at any time node. Thus, it can help the staff reasonably arrange the time and resources for garbage feeding or other treatment processes, improve the garbage feeding efficiency, save time and effort, and also reduce the feeding or other treatment delays caused by excessive garbage accumulation.

[0184] Furthermore, the embodiment of this application provides a feeding monitoring device for a garbage feeding port. Optionally, Figure 10 shows the hardware structure block diagram of the feeding monitoring device for the garbage feeding port. Referring to Figure 10 , the hardware structure of the feeding monitoring device for the garbage feeding port may include: at least one processor 01, at least one communication interface 02, at least one memory 03, and at least one communication bus 04.

[0185] In the embodiment of this application, the number of the processor 01, the communication interface 02, the memory 03, and the communication bus 04 is at least one, and the processor 01, the communication interface 02, and the memory 03 complete mutual communication through the communication bus 04.

[0186] The processor 01 may be a central processing unit CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.

[0187] The memory 03 may include a high-speed RAM memory, and may also include non-volatile memory, such as at least one disk memory.

[0188] Among them, the memory stores a program, and the processor can call the program stored in the memory. The program is used to execute the following feeding monitoring method for the garbage feeding port, including:

[0189] Obtain each feeding image during the feeding process of the garbage feeding port;

[0190] Process each of the feeding images using a pre-trained contour recognition model to obtain waste material bitmaps with the contours of the waste piles marked thereon;

[0191] For each of the waste material bitmaps, determine the area ratio of the waste pile in the feeding port in the waste material bitmap according to the marked contour of the waste pile;

[0192] Obtain the time node corresponding to the waste material bitmap, and determine the time node when the feeding port is empty corresponding to this time node;

[0193] Calculate the time interval between the time node corresponding to the waste material bitmap and the time node when the feeding port is empty;

[0194] Predict the time interval between any time node and the time node when the corresponding feeding port is empty according to the time interval corresponding to each waste material bitmap and the area ratio, so as to achieve feeding monitoring.

[0195] Optionally, the refinement functions and expansion functions of the program may refer to the description of the feeding monitoring method of the waste feeding port in the method embodiments.

[0196] An embodiment of the present application further provides a storage medium, which can store a program suitable for a processor to execute. When the program runs, it controls the device where the storage medium is located to execute the following feeding monitoring method of the waste feeding port, including:

[0197] Obtain each feeding image during the feeding process of the waste feeding port;

[0198] Process each of the feeding images using a pre-trained contour recognition model to obtain waste material bitmaps with the contours of the waste piles marked thereon;

[0199] For each of the waste material bitmaps, determine the area ratio of the waste pile in the feeding port in the waste material bitmap according to the marked contour of the waste pile;

[0200] Obtain the time node corresponding to the waste material bitmap, and determine the time node when the feeding port is empty corresponding to this time node;

[0201] Calculate the time interval between the time node corresponding to the waste material bitmap and the time node when the feeding port is empty;

[0202] Predict the time interval between any time node and the time node when the corresponding feeding port is empty according to the time interval corresponding to each waste material bitmap and the area ratio, so as to achieve feeding monitoring.

[0203] Specifically, the storage medium may be a computer-readable storage medium, and the computer-readable storage medium may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM.

[0204] Optionally, the refinement function and the extension function of the program may refer to the description of the feeding monitoring method of the garbage feeding port in the method embodiment.

[0205] In addition, in each of the embodiments of the present disclosure, the functional modules may be integrated together to form an independent part, or each module may exist alone, or two or more modules may be integrated to form an independent part. If the function is implemented in the form of a software functional module and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present disclosure, in essence, or the part that contributes to the prior art, or a part of the technical solution, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a live broadcast device, or a network device, etc.) to execute all or part of the steps of the methods in the embodiments of the present disclosure.

[0206] Finally, it should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the element.

[0207] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts among the various embodiments may be referred to each other.

[0208] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A feeding monitoring method for a garbage feeding port, characterized in that Including: Obtaining each feeding image of the garbage feeding port during the feeding process; Processing each of the feeding images by using a pre-trained contour recognition model to obtain each garbage material position map marked with the contour of the garbage pile; For each of the garbage material position maps, determining the area ratio occupied by the garbage pile in the feeding port according to the marked contour of the garbage pile; Obtaining the time node corresponding to this garbage material position map, and determining the time node when the feeding port is empty corresponding to this time node; Calculating the time interval between the time node corresponding to this garbage material position map and the time node when the feeding port is empty; Predicting the time interval between any time node and the time node when the corresponding feeding port is empty according to the time interval corresponding to each of the garbage material position maps and the area ratio to achieve feeding monitoring; including: establishing a falling experience function according to the time interval corresponding to each of the garbage material position maps and the area ratio; for each of the garbage material position maps, determining each of the garbage material position maps corresponding to the previous preset number of frames and each of the garbage material position maps corresponding to the next preset number of frames at the time node corresponding to this garbage material position map as each adjacent material position map; determining the area ratio of each of the adjacent material position maps; calculating the garbage falling speed corresponding to this garbage material position map based on the area ratio; establishing a time interval function according to the falling experience function and the garbage falling speed corresponding to each of the garbage material position maps; using the time interval function to predict the time interval between any time node and the time node when the corresponding feeding port is empty.

2. The method according to claim 1, wherein The contour recognition model includes a backbone network module, a neck network module, and a head network module; Wherein, the input end of the backbone network module is used as the input end of the contour recognition model, the output end of the backbone network module is connected to the input end of the neck network module, the output end of the neck network module is connected to the input end of the head network module, and the output end of the head network module is used as the output end of the contour recognition model.

3. The method according to claim 2, wherein The backbone network module includes a first downsampling unit, a second downsampling unit, a third downsampling unit, a fourth downsampling unit, a fifth downsampling unit, a first feature extraction unit, a second feature extraction unit, a third feature extraction unit, a fourth feature extraction unit, and a spatial pyramid pooling unit; Among them, the input end of the first downsampling unit serves as the input end of the backbone network module. The output end of the first downsampling unit is connected to the input end of the second downsampling unit. The output end of the second downsampling unit is connected to the input end of the first feature extraction unit. The output end of the first feature extraction unit is connected to the input end of the third downsampling unit. The output end of the third downsampling unit is connected to the input end of the second feature extraction unit. The output end of the second feature extraction unit is connected to the input end of the fourth downsampling unit. The output end of the fourth downsampling unit is connected to the input end of the third feature extraction unit. The output end of the third feature extraction unit is connected to the input end of the fifth downsampling unit. The output end of the fifth downsampling unit is connected to the input end of the fourth feature extraction unit. The output end of the fourth feature extraction unit is connected to the input end of the spatial pyramid pooling unit; The output end of the second feature extraction unit, the output end of the third feature extraction unit, and the output end of the spatial pyramid pooling unit respectively serve as the first output end, the second output end, and the third output end of the backbone network module, and are all connected to the input end of the neck network module.

4. The method according to claim 3, wherein The neck network module includes a first upsampling unit, a second upsampling unit, a first fusion unit, a second fusion unit, a third fusion unit, a fourth fusion unit, a fifth feature extraction unit, a sixth feature extraction unit, a seventh feature extraction unit, a first convolutional unit, and a second convolutional unit; Among them, the input end of the second fusion unit, the input end of the first fusion unit, and the input end of the first upsampling unit respectively serve as the first input end, the second input end, and the third input end of the neck network module; The first output end of the second fusion unit serves as the first output end of the neck network module. The first output end of the sixth feature extraction unit serves as the second output end of the neck network module. The output end of the seventh feature extraction unit serves as the third output end of the neck network module; The first input end, the second input end, and the third input end of the neck network module are respectively connected to the first output end, the second output end, and the third output end of the backbone network module; The output end of the first upsampling unit and the second output end of the backbone network module are both connected to the second input end of the neck network module. The input end of the fifth feature extraction unit and the input end of the third fusion unit are both connected to the output end of the first fusion unit. The output end of the fifth feature extraction unit is connected to the input end of the second upsampling unit. The output end of the second upsampling unit and the first output end of the backbone network module are both connected to the first input end of the neck network module. The second output end of the second fusion unit is connected to the input end of the first convolutional unit. The output end of the first convolutional unit and the output end of the first fusion unit are both connected to the input end of the third fusion unit. The output end of the third fusion unit is connected to the input end of the sixth feature extraction unit. The second output end of the sixth feature extraction unit is connected to the input end of the second convolutional unit. The output end of the second convolutional unit is connected to the input end of the fourth fusion unit. The input end of the fourth fusion unit is further connected to the input end of the first upsampling unit. The output end of the fourth fusion unit is connected to the input end of the seventh feature extraction unit.

5. The method according to claim 4, characterized in that, The structures of the first feature extraction unit, the second feature extraction unit, the third feature extraction unit, the fourth feature extraction unit, the fifth feature extraction unit, the sixth feature extraction unit, and the seventh feature extraction unit are the same, and they all serve as feature extraction units. The feature extraction unit includes: a first CBS layer, a split layer, a comprehensive bottleneck layer, a splicing layer, and a second CBS layer; Among them, the input end of the first CBS layer serves as the input end of the feature extraction unit; The output end of the first CBS layer is connected to the input end of the split layer. The third output end of the split layer is connected to the input end of the comprehensive bottleneck layer. The first output end, the second output end of the split layer, and the output end of the comprehensive bottleneck layer are all connected to the input end of the splicing layer. The output end of the splicing layer is connected to the input end of the second CBS layer; The output end of the second CBS layer serves as the output end of the feature extraction unit.

6. The method according to claim 1, characterized in that Determining the area ratio of the garbage dump in the feeding port in the garbage material bit map according to the marked garbage dump contour includes: Converting the garbage material bit map into a binary map; Extracting the segmentation mask corresponding to the garbage dump contour from the binary map; Extracting the segmentation contour from the segmentation mask and calculating the area of the garbage dump contour by using the segmentation contour; Obtaining the width and height of the garbage material bit map; Calculating the area when the bin is full according to the preset ratio, width, and height; Dividing the area of the garbage dump contour by the area when the bin is full to obtain the area ratio of the garbage dump in the feeding port.

7. A feeding monitoring device for a garbage feeding port, characterized in that, It includes: An acquisition module for acquiring each feeding image during the feeding process of the garbage feeding port; A model processing module for processing each of the feeding images by using a pre-trained contour recognition model to obtain each garbage material bit map marked with the garbage dump contour; An area ratio calculation module for determining, for each of the garbage material bit maps, the area ratio of the garbage dump in the feeding port in the garbage material bit map according to the marked garbage dump contour; A time node determination module, configured to obtain the time node corresponding to the waste material bitmap, and determine the time node when the feeding port corresponding to the time node is empty; A time interval calculation module, configured to calculate the time interval between the time node corresponding to the waste material bitmap and the time node when the feeding port is empty; A prediction module, configured to predict the time interval between any time node and the time node when the corresponding feeding port is empty according to the time interval corresponding to each waste material bitmap and the area ratio, so as to implement feeding monitoring; including: establishing a falling experience function according to the time interval corresponding to each waste material bitmap and the area ratio; for each waste material bitmap, determining each waste material bitmap corresponding to the previous preset number of frames of the time node corresponding to the waste material bitmap and each waste material bitmap corresponding to the subsequent preset number of frames as each adjacent material bitmap; determining the area ratio of each adjacent material bitmap; calculating the waste falling speed corresponding to the waste material bitmap based on the area ratio; establishing a time interval function according to the falling experience function and the waste falling speed corresponding to each waste material bitmap; using the time interval function to predict the time interval between any time node and the time node when the corresponding feeding port is empty.

8. A feeding monitoring device for a garbage feeding port, characterized in that, Comprising a memory and a processor; The memory is used for storing programs; The processor is configured to execute the program to implement each step of the feeding monitoring method for the waste feeding port according to any one of claims 1-6.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, each step of the feeding monitoring method for the waste feeding port according to any one of claims 1-6 is implemented.

Images