Feeding monitoring method, device and equipment for garbage feeding port and storage medium

Abstract

The invention discloses a feeding monitoring method and device for a garbage feeding port, equipment and a storage medium. The feeding monitoring method comprises the steps that all feeding images of the garbage feeding port are obtained; processing each feeding image by using a contour recognition model to obtain each garbage level map marked with a garbage heap contour; for each garbage level map, the area proportion of the garbage heap in the feeding port is determined; acquiring a time node corresponding to the garbage level map, and determining a time node when the corresponding feeding port is empty; calculating a time interval; according to the time interval and the area proportion corresponding to each garbage level map, the time interval between any time node and the time node when the corresponding feeding port is empty is predicted, and feeding monitoring is achieved. Compared with manual monitoring, the real feeding condition of the garbage feeding port can be determined more accurately, workers can be helped to reasonably arrange time and resources of garbage feeding or other treatment processes, the garbage feeding efficiency is improved, and time and labor are saved.

Description

Technical Field

[0001] The present application relates to the technical field of garbage feeding monitoring, and in particular to a method, device, equipment and storage medium for monitoring the feeding of a garbage feeding port. Background Art

[0002] In recent years, environmental protection has become a hot topic worldwide, and all sectors of society have paid great attention to it. With the advancement of science and technology and the pursuit of sustainable development, a series of innovative industrial environmental protection technologies have emerged, and higher requirements have been placed on the effect of garbage disposal. Garbage is generally thrown in from the garbage feeding port, so monitoring the garbage feeding port is very important, such as paying real-time attention to whether the current feeding is completed, judging when the feeding port is emptied, when the next round of garbage delivery should be started, etc., and these processes are related to the overall efficiency of garbage disposal.

[0003] However, the existing method of monitoring the garbage feeding port remains at manual monitoring, which requires staff to continuously monitor the feeding situation of the feeding port. This method is easily affected by human factors and may lead to misjudgment, seriously affecting the efficiency of garbage disposal. Summary of the invention

[0004] In view of this, the present application provides a method, device, equipment and storage medium for monitoring the feeding of garbage feeding ports, which is used to solve the problem that the existing method of monitoring garbage feeding ports remains based on manual monitoring, requiring staff to continuously monitor the feeding situation of the feeding ports. This method is easily affected by human factors and may lead to misjudgment, seriously affecting the efficiency of garbage disposal.

[0005] To achieve the above objectives, the proposed solution is as follows:

[0006] In a first aspect, a method for monitoring a garbage feeding port includes:

[0007] Acquire various feeding images of the garbage feeding port during the feeding process;

[0008] Using a pre-trained contour recognition model to process each of the feeding images, to obtain each garbage material location map marked with the contour of the garbage pile;

[0009] For each of the garbage material location maps, determine the area ratio of the garbage pile in the material feeding port in the garbage material location map according to the marked garbage pile contour;

[0010] Obtain the time node corresponding to the garbage material position map, and determine the time node when the feeding port corresponding to the time node is empty;

[0011] Calculate the time interval between the time node corresponding to the garbage material position map and the time node when the feeding port is empty;

[0012] According to the time interval corresponding to each of the garbage material position maps and the area ratio, the time interval between any time node and the time node when the corresponding feeding port is empty is predicted to realize feeding monitoring.

[0013] Preferably, the contour recognition model includes a backbone network module, a neck network module, and a head network module;

[0014] Among them, the input end of the backbone network module serves as the input end of the contour recognition model, the output end of the backbone network module is connected to the input end of the neck network module, the output end of the neck network module is connected to the input end of the head network module, and the output end of the head network module serves as the output end of the contour recognition model.

[0015] Preferably, the backbone network module includes a first downsampling unit, a second downsampling unit, a third downsampling unit, a fourth downsampling unit, a fifth downsampling unit, a first feature extraction unit, a second feature extraction unit, a third feature extraction unit, a fourth feature extraction unit and a spatial pyramid pooling unit;

[0016] Wherein, the input end of the first down-sampling unit is used as the input end of the backbone network module, the output end of the first down-sampling unit is connected to the input end of the second down-sampling unit, the output end of the second down-sampling unit is connected to the input end of the first feature extraction unit, the output end of the first feature extraction unit is connected to the input end of the third down-sampling unit, the output end of the third down-sampling unit is connected to the input end of the second feature extraction unit, the output end of the second feature extraction unit is connected to the input end of the fourth down-sampling unit, the output end of the fourth down-sampling unit is connected to the input end of the third feature extraction unit, the output end of the third feature extraction unit is connected to the input end of the fifth down-sampling unit, the output end of the fifth down-sampling unit is connected to the input end of the fourth feature extraction unit, and the output end of the fourth feature extraction unit is connected to the input end of the spatial pyramid pooling unit;

[0017] The output end of the second feature extraction unit, the output end of the third feature extraction unit and the output end of the spatial pyramid pooling unit serve as the first output end, the second output end and the third output end of the backbone network module respectively, and are all connected to the input end of the neck network module.

[0018] Preferably, the neck network module includes a first upsampling unit, a second upsampling unit, a first fusion unit, a second fusion unit, a third fusion unit, a fourth fusion unit, a fifth feature extraction unit, a sixth feature extraction unit, a seventh feature extraction unit, a first convolution unit, and a second convolution unit;

[0019] The input end of the second fusion unit, the input end of the first fusion unit and the input end of the first upsampling unit are respectively used as the first input end, the second input end and the third input end of the neck network module;

[0020] The first output end of the second fusion unit is used as the first output end of the neck network module, the first output end of the sixth feature extraction unit is used as the second output end of the neck network module, and the output end of the seventh feature extraction unit is used as the third output end of the neck network module;

[0021] The first input end, the second input end and the third input end of the neck network module are connected to the first output end, the second output end and the third output end of the backbone network module respectively;

[0022] The output end of the first upsampling unit and the second output end of the backbone network module are both connected to the second input end of the neck network module, the input end of the fifth feature extraction unit and the input end of the third fusion unit are both connected to the output end of the first fusion unit, the output end of the fifth feature extraction unit is connected to the input end of the second upsampling unit, the output end of the second upsampling unit and the first output end of the backbone network module are both connected to the first input end of the neck network module, the second output end of the second fusion unit is connected to the input end of the first convolution unit, the output end of the first convolution unit and the output end of the first fusion unit are both connected to the input end of the third fusion unit, the output end of the third fusion unit is connected to the input end of the sixth feature extraction unit, the second output end of the sixth feature extraction unit is connected to the input end of the second convolution unit, the output end of the second convolution unit is connected to the input end of the fourth fusion unit, the input end of the fourth fusion unit is also connected to the input end of the first upsampling unit, and the output end of the fourth fusion unit is connected to the input end of the seventh feature extraction unit.

[0023] Preferably, the first feature extraction unit, the second feature extraction unit, the third feature extraction unit, the fourth feature extraction unit, the fifth feature extraction unit, the sixth feature extraction unit, and the seventh feature extraction unit have the same structure and are all used as feature extraction units. The feature extraction units include: a first CBS layer, a splitting layer, a comprehensive bottleneck layer, a splicing layer, and a second CBS layer;

[0024] Wherein, the input end of the first CBS layer serves as the input end of the feature extraction unit;

[0025] The output end of the first CBS layer is connected to the input end of the split layer, the third output end of the split layer is connected to the input end of the comprehensive bottleneck layer, the first output end of the split layer, the second output end of the split layer and the output end of the comprehensive bottleneck layer are all connected to the input end of the splicing layer, and the output end of the splicing layer is connected to the input end of the second CBS layer;

[0026] The output end of the second CBS layer serves as the output end of the feature extraction unit.

[0027] Preferably, the determining, according to the marked garbage pile contour, the area ratio of the garbage pile in the feeding port in the garbage material level map comprises:

[0028] Convert the garbage material location map into a binary image;

[0029] Extracting a segmentation mask corresponding to the garbage dump outline from the binary image;

[0030] Extracting a segmentation contour from the segmentation mask, and calculating the area of ​​the garbage dump contour using the segmentation contour;

[0031] Obtaining the width and height of the garbage material map;

[0032] Calculate the area when the garbage bin is full according to the preset ratio, width and height;

[0033] The area of ​​the garbage pile outline is divided by the area of ​​the garbage bin when it is full to obtain the area ratio of the garbage pile in the feeding port.

[0034] Preferably, the method of predicting the time interval between any time node and the time node when the corresponding feeding port is empty according to the time interval corresponding to each of the garbage material level maps and the area ratio includes:

[0035] Establishing a falling empirical function according to the time interval corresponding to each of the garbage material level maps and the area ratio;

[0036] For each of the garbage material level maps, determining the garbage material level maps corresponding to a preset number of frames before and after a preset number of frames corresponding to a time node of the garbage material level map as adjacent material level maps;

[0037] Determining the area ratio of each of the adjacent material level maps;

[0038] Calculate the garbage falling speed corresponding to the garbage level map based on the area ratio;

[0039] Establishing a time interval function according to the falling empirical function and the garbage falling speed corresponding to each of the garbage material level maps;

[0040] The time interval function is used to predict the time interval between any time node and the time node when the corresponding feeding port is empty.

[0041] In a second aspect, a garbage feeding port feeding monitoring device comprises:

[0042] An acquisition module is used to acquire each feeding image of the garbage feeding port during the feeding process;

[0043] A model processing module, used to process each of the feeding images using a pre-trained contour recognition model to obtain garbage material location maps each marked with a garbage pile contour;

[0044] An area ratio calculation module is used to determine, for each garbage material location map, the area ratio of the garbage pile in the material feeding port in the garbage material location map according to the marked garbage pile contour;

[0045] A time node determination module is used to obtain the time node corresponding to the garbage material position map and determine the time node when the feeding port corresponding to the time node is empty;

[0046] A time interval calculation module, used to calculate the time interval between the time node corresponding to the garbage material position map and the time node when the feeding port is empty;

[0047] The prediction module is used to predict the time interval between any time node and the time node when the corresponding feeding port is empty according to the time interval corresponding to each of the garbage material position maps and the area ratio, so as to realize feeding monitoring.

[0048] In a third aspect, a garbage feeding port feeding monitoring device includes a memory and a processor;

[0049] The memory is used to store programs;

[0050] The processor is used to execute the program to implement each step of the garbage feeding port feeding monitoring method as described in any one of the first aspects.

[0051] In a fourth aspect, a storage medium stores a computer program thereon, and when the computer program is executed by a processor, the steps of the method for monitoring the feeding of a garbage feeding port as described in any one of the first aspects are implemented.

[0052] It can be seen from the above technical scheme that the present application obtains each feeding image of the garbage feeding port during the feeding process; uses a pre-trained contour recognition model to process each of the feeding images to obtain each garbage material level map marked with the contour of the garbage pile; for each of the garbage material level maps, determines the area ratio of the garbage pile in the feeding port in the garbage material level map according to the marked garbage pile contour; obtains the time node corresponding to the garbage material level map, and determines the time node when the feeding port corresponding to the time node is empty; calculates the time interval between the time node corresponding to the garbage material level map and the time node when the feeding port is empty; according to the time interval corresponding to each of the garbage material level maps and the area ratio, predicts the time interval between any time node and the time node when the corresponding feeding port is empty, so as to realize feeding monitoring. The present application first obtains various feeding images of the garbage feeding port during the feeding process. Compared with manual monitoring, the actual feeding situation of the garbage feeding port can be determined more accurately. The pre-trained contour recognition model can process the feeding images to obtain various garbage material position maps marked with garbage pile contours, reducing the subjectivity of manual judgment. Since the time interval information needs to be determined, it is crucial to determine the area ratio of the garbage pile contour in each garbage material position map in the feeding port. By comparing the time node of each garbage material position map with the time node when the corresponding feeding port is empty, the time interval can be accurately obtained, which can reduce the workload of manual recording and judgment, avoid time recording errors caused by human factors, and can predict the garbage pile feeding situation at any time node, thereby helping staff to reasonably arrange the time and resources for garbage feeding or other processing processes, improve garbage feeding efficiency, save time and effort, and reduce feeding or other processing delays caused by excessive garbage accumulation. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0054] Figure 1 An optional flow chart of a method for monitoring a garbage feeding port provided in an embodiment of the present application;

[0055] Figure 2 A schematic diagram of garbage material with a garbage pile outline marked thereon provided in an embodiment of the present application;

[0056] Figure 3 A schematic diagram of the structure of a contour recognition model provided in an embodiment of the present application;

[0057] Figure 4 A schematic diagram of the structure of a backbone network module provided in an embodiment of the present application;

[0058] Figure 5 A schematic diagram of the structure of a spatial pyramid pooling unit provided in an embodiment of the present application;

[0059] Figure 6 A schematic diagram of the structure of a neck network module provided in an embodiment of the present application;

[0060] Figure 7 A schematic diagram of the structure of a Segment unit provided in an embodiment of the present application;

[0061] Figure 8 A structural schematic diagram of a feature extraction unit provided in an embodiment of the present application;

[0062] Fig. 9 A schematic diagram of the structure of a garbage feeding port feeding monitoring device provided in an embodiment of the present application;

[0063] Fig.10 A schematic diagram of the structure of a garbage feeding port feeding monitoring device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0065] This application can be applied to any scenario that includes a garbage feeding port, such as a garbage power plant. By converting waste into electrical energy, the garbage power plant not only helps solve the increasingly serious garbage disposal problem, but also adds new vitality to the field of renewable energy.

[0066] The present invention can be used in many general or special computing device environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multi-processor devices, distributed computing environments including any of the above devices or equipment, etc.

[0067] The embodiment of the present invention provides a method for monitoring the feeding of garbage feeding ports. The method can be applied to various computer terminals or intelligent terminals. The execution subject can be a processor or server of the computer terminal or intelligent terminal. The method flow chart of the method is as follows: Figure 1 As shown, specifically including:

[0068] S1: Acquire each feeding image of the garbage feeding port during the feeding process.

[0069] Nowadays, image recognition technology has become a key factor in improvement in various fields. This application also applies this idea to monitor feeding. First, video data for monitoring the garbage feeding port is obtained. The monitoring video data of the garbage feeding port is the video data collected by the camera or monitor that monitors the garbage feeding port. The field of view of the camera or monitor should cover the garbage feeding port, and can clearly monitor the entire process of garbage feeding, including the moment of feeding, the falling process, the process of garbage completely entering the garbage bin from the feeding port, etc., to ensure the integrity of the feeding process, so as to ensure the accuracy of monitoring. Then, image segmentation can be performed from the monitoring video data to obtain each feeding image.

[0070] It is understandable that the monitoring video data contains multiple frames of images, each frame of image corresponds to a time node, or multiple frames of image correspond to a time node at the same time, and the length of a frame can be determined according to the duration of the monitoring video data. The monitoring video data may contain multiple rounds of feeding processes, or may only contain one round of feeding processes. Therefore, the time period for segmenting the monitoring video data can be determined according to the situation, for example, a feeding image is segmented every preset time period.

[0071] By acquiring surveillance video data and then segmenting the image, the integrity and certainty of the data can be ensured, avoiding the discontinuity and instability of manual monitoring.

[0072] S2: using a pre-trained contour recognition model to process each of the feeding images to obtain garbage material location maps each marked with a garbage pile contour.

[0073] The contour recognition model is trained with a dataset containing countless feeding sample images as the training sample set, and the real garbage dump contour of each feeding sample image as the sample label. Feeding images of monitoring garbage feeding ports can be collected from various industries or scenes. In one example, a sample set containing 1,300 feeding sample images was collected. In order to improve the training accuracy of the model, these feeding sample images can be preprocessed, and a variety of data enhancement methods can be adopted to expand the data volume, so that the model is more adaptable and can handle various feeding images, including flipping, rotation, scaling, translation, cropping, color change, mirror symmetry, adding noise, etc. After the training is completed, the training set and the test set are split in a ratio of 8:2. The training set is used to train the contour recognition model, and the test set is used to test the trained contour recognition model. If the test passes, the training is completed. If the test fails, the training continues to ensure the availability of the model.

[0074] Using a pre-trained contour recognition model, the contour of the garbage dump can be automatically identified, reducing the subjectivity and errors of manual judgment. The contour recognition model can quickly and accurately process a large number of feeding images, which can improve monitoring efficiency.

[0075] The garbage dump outline involved in this application is the outline of the garbage dump itself, so that the calculated area ratio is more accurate.

[0076] S3: For each of the garbage material location maps, determine the area ratio of the garbage pile in the material feeding port in the garbage material location map according to the marked garbage pile contour.

[0077] Since the outline of the garbage pile is marked in the garbage material level map, the area of ​​the feeding port in the garbage material level map can also be determined. Therefore, the area ratio of the garbage pile in the feeding port can be calculated. The accumulation of garbage can be quantified based on the area ratio, providing accurate data support for subsequent time interval predictions. This will be more objective than manual visual inspection and avoid a high misjudgment rate.

[0078] Specifically, considering that under normal circumstances, the feeding port is not only monitored at the feeding port, but also the outer area of ​​the garbage bin, the ground and other areas around the feeding port are monitored at the same time. Therefore, the feeding port does not occupy the entire garbage material location map. In the subsequent monitoring process, the focus is on the situation of garbage being put into the feeding port. Therefore, the area ratio is calculated according to the area of ​​the garbage pile and the area of ​​the feeding port, which will be more accurate.

[0079] S4: Obtain the time node corresponding to the garbage material position map, and determine the time node when the feeding port corresponding to the time node is empty.

[0080] Each garbage material level map corresponds to a time node. This time node is not the specific time corresponding to the garbage material level map (for example, accurate to the second), but an information indicating the time scale, in which one or more garbage material level maps correspond to a time node at the same time.

[0081] It should be noted that if the time node corresponding to the garbage material position map is the first time node, and the time node when the corresponding feeding port is empty is the second time node, then the second time node is equal to or later than the first time node. In other words, only the time node corresponding to the time when all the garbage in the garbage material position map enters the garbage bin below the feeding port, or the situation where there is no garbage at the feeding port after the garbage is fed, is regarded as the time node when the feeding port corresponding to the time node is empty as specified above.

[0082] In this way, the feeding situation can be analyzed by recording and comparing between time nodes.

[0083] S5: Calculate the time interval between the time node corresponding to the garbage material position map and the time node when the feeding port is empty.

[0084] By calculating the time interval between two time nodes, the speed and frequency of the garbage pile entering the feed port or falling can be quantified. Subsequent monitoring can provide data support for the management and scheduling of garbage feeding or other treatment processes. This method is more accurate and efficient than manual monitoring, and can detect abnormal situations in a timely manner and deal with them.

[0085] Specifically, the time interval can be measured with lower time dimensions such as minutes and seconds. For example, the time node corresponding to a garbage material position map is 14:23, and the time node when the corresponding feeding port is empty is 14:24, then the time interval is 1 minute / 60 seconds.

[0086] S6: According to the time interval corresponding to each of the garbage material position maps and the area ratio, the time interval between any time node and the time node when the corresponding feeding port is empty is predicted to realize feeding monitoring.

[0087] Since the garbage material position map involved in the above steps does not represent the image of the feeding port at every moment / every frame / every second during the feeding process, the determined time interval and area ratio can summarize and analyze the time interval between any time node and the time node when the corresponding feeding port is empty. The time node can be any dimension representing time, such as frame / second / moment, etc., so the situation of each time node of the current garbage feeding work can be determined, so as to achieve the purpose of early warning of garbage accumulation, and reduce the idle time between two rounds of feeding to improve efficiency, and help staff to reasonably arrange the time and resources for garbage feeding or other processing processes, such as preparing for the next round of feeding before the predicted time interval of the current round of feeding ends.

[0088] The present application first obtains various feeding images of the garbage feeding port during the feeding process. Compared with manual monitoring, the actual feeding situation of the garbage feeding port can be determined more accurately. The pre-trained contour recognition model can process the feeding images to obtain various garbage material position maps marked with garbage pile contours, reducing the subjectivity of manual judgment. Since the time interval information needs to be determined, it is crucial to determine the area ratio of the garbage pile contour in each garbage material position map in the feeding port. By comparing the time node of each garbage material position map with the time node when the corresponding feeding port is empty, the time interval can be accurately obtained, which can reduce the workload of manual recording and judgment, avoid time recording errors caused by human factors, and can predict the garbage pile feeding situation at any time node, thereby helping staff to reasonably arrange the time and resources for garbage feeding or other processing processes, improve garbage feeding efficiency, save time and effort, and reduce feeding delays or other processing delays caused by excessive garbage accumulation.

[0089] In one example, a garbage location map with garbage dump outlines is shown in Figure 1. Figure 2 As shown in the figure, the area ratio in this figure is 24%. In the process of the contour recognition model identifying the contour of the garbage dump, a garbage dump identification and confirmation process is also involved. For example, the smallest circumscribed rectangle of the suspected garbage dump is first identified, and the probability of the suspected garbage dump (rubbish heap) in the rectangle is obtained to be 0.95. If the probability is high, it is determined to be a garbage dump, and then the specific contour of the garbage dump is identified.

[0090] In the method provided by the embodiment of the present invention, for the contour recognition model, its structure is as follows Figure 3 As shown, it includes a backbone network module, a neck network module, and a head network module;

[0091] Among them, the input end of the backbone network module serves as the input end of the contour recognition model, the output end of the backbone network module is connected to the input end of the neck network module, the output end of the neck network module is connected to the input end of the head network module, and the output end of the head network module serves as the output end of the contour recognition model.

[0092] Specifically, the contour recognition model belongs to a deep learning model, and the backbone network module adopts the Backbone network; the neck network (Neck network) module can adopt a feature pyramid network with a structure of PAN-FPN (Path Aggregation Network-FeaturePyramid Network), and the head network module can adopt the Head network.

[0093] The structures of the backbone network module, neck network module, and head network module are described in detail below.

[0094] 1) Backbone network module.

[0095] like Figure 4 As shown, the backbone network module includes a first downsampling unit, a second downsampling unit, a third downsampling unit, a fourth downsampling unit, a fifth downsampling unit, a first feature extraction unit, a second feature extraction unit, a third feature extraction unit, a fourth feature extraction unit and a spatial pyramid pooling unit;

[0096] Wherein, the input end of the first down-sampling unit is used as the input end of the backbone network module, the output end of the first down-sampling unit is connected to the input end of the second down-sampling unit, the output end of the second down-sampling unit is connected to the input end of the first feature extraction unit, the output end of the first feature extraction unit is connected to the input end of the third down-sampling unit, the output end of the third down-sampling unit is connected to the input end of the second feature extraction unit, the output end of the second feature extraction unit is connected to the input end of the fourth down-sampling unit, the output end of the fourth down-sampling unit is connected to the input end of the third feature extraction unit, the output end of the third feature extraction unit is connected to the input end of the fifth down-sampling unit, the output end of the fifth down-sampling unit is connected to the input end of the fourth feature extraction unit, and the output end of the fourth feature extraction unit is connected to the input end of the spatial pyramid pooling unit;

[0097] The output end of the second feature extraction unit, the output end of the third feature extraction unit and the output end of the spatial pyramid pooling unit serve as the first output end, the second output end and the third output end of the backbone network module respectively, and are all connected to the input end of the neck network module.

[0098] Specifically, the first downsampling unit, the second downsampling unit, the third downsampling unit, the fourth downsampling unit, and the fifth downsampling unit have the same structure, which are all convolution groups (CBS) consisting of 1 Convolution layer, 1 BatchNorm layer, and 1 SILU layer. The convolution kernel k of these downsampling units is 3, the step size s is 2, and the padding p is 1. The backbone network module is mainly used for feature extraction. In an example, the resolution of the feed image input to the backbone network module is 640×640 and the number of channels is 3. It will first be downsampled by the first downsampling unit, and the resolution will be changed to 320×320 and the number of channels will be changed to 64. Then, it will pass through the second downsampling unit and the first feature extraction unit, and the resolution will be changed to 160×160 and the number of channels will be changed to 128. In this process, the size of the feed image does not change, and then it will pass through the three groups of downsampling units and feature extraction units in turn. That is, the feed image with a resolution of 160×160 and a number of channels of 128 will first pass through the first downsampling unit and the first feature extraction unit in turn. After the third downsampling unit and the second feature extraction unit, a feed image P4 with a resolution reduced to 80×80 and a number of channels of 256 is obtained. After the feed image P4 passes through the fourth downsampling unit and the third feature extraction unit in sequence, a feed image P6 with a resolution reduced to 40×40 and a number of channels of 512 is obtained. After the feed image P6 passes through the fifth downsampling unit, the fourth feature extraction unit and the spatial pyramid pooling unit in sequence, a feed image P9 with a resolution reduced to 20×20 and a number of channels of 1024 is obtained, thereby outputting three feed images of different scales from the first output end, the second output end and the third output end of the backbone network module respectively.

[0099] Spatial Pyramid Pooling Fast (SPPF) is an improved structure. Figure 5 As shown, there are two CBS1 layers, the convolution kernel k of the two CBS1 layers is 3, the step size s is 1, and the padding p is 0, as well as three MaxPool2d layers (layers for two-dimensional maximum pooling Max Pooling operations) and 1 Concat layer.

[0100] This hierarchical downsampling and feature extraction method can gradually extract abstract image information. Combined with multi-scale output, it can provide rich feature representation for the subsequent neck network module, making the contour recognition model more performant.

[0101] 2) Neck network module.

[0102] The structure of the neck network module is as follows Figure 6As shown, it includes a first upsampling unit, a second upsampling unit, a first fusion unit, a second fusion unit, a third fusion unit, a fourth fusion unit, a fifth feature extraction unit, a sixth feature extraction unit, a seventh feature extraction unit, a first convolution unit, and a second convolution unit;

[0103] The input end of the second fusion unit, the input end of the first fusion unit and the input end of the first upsampling unit are respectively used as the first input end, the second input end and the third input end of the neck network module;

[0104] The first output end of the second fusion unit is used as the first output end of the neck network module, the first output end of the sixth feature extraction unit is used as the second output end of the neck network module, and the output end of the seventh feature extraction unit is used as the third output end of the neck network module;

[0105] The first input end, the second input end and the third input end of the neck network module are connected to the first output end, the second output end and the third output end of the backbone network module respectively;

[0106] The output end of the first upsampling unit and the second output end of the backbone network module are both connected to the second input end of the neck network module, the input end of the fifth feature extraction unit and the input end of the third fusion unit are both connected to the output end of the first fusion unit, the output end of the fifth feature extraction unit is connected to the input end of the second upsampling unit, the output end of the second upsampling unit and the first output end of the backbone network module are both connected to the first input end of the neck network module, the second output end of the second fusion unit is connected to the input end of the first convolution unit, the output end of the first convolution unit and the output end of the first fusion unit are both connected to the input end of the third fusion unit, the output end of the third fusion unit is connected to the input end of the sixth feature extraction unit, the second output end of the sixth feature extraction unit is connected to the input end of the second convolution unit, the output end of the second convolution unit is connected to the input end of the fourth fusion unit, the input end of the fourth fusion unit is also connected to the input end of the first upsampling unit, and the output end of the fourth fusion unit is connected to the input end of the seventh feature extraction unit.

[0107] Specifically, the neck network module processes the three types of feed images output by the backbone network module. First, P9 is upsampled by 2 times by the first upsampling unit and output to the first fusion unit. It is fused with P6 in the first fusion unit to obtain image F1 (resolution is 40×40); F1 then enters the second upsampling unit after passing through the fifth feature extraction unit for upsampling by 2 times and is output to the second fusion unit. It is fused with P4 in the second fusion unit to obtain feature map T1 (resolution is 80×80). T1 will be output from the first output end of the neck network module; at the same time, T1 will be input to the first volume The convolution unit performs convolution processing and outputs it to the third fusion unit for fusion with F1 to obtain image F2 (resolution is 40×40); F2 will pass through the sixth feature extraction unit to obtain feature map T2 (resolution is 40×40), and T2 will be output from the second output end of the neck network module; at the same time, the feature map T2 is input to the second convolution module for convolution, and then fused with P9 in the fourth fusion unit to obtain F3 (resolution is 40×40); finally, F3 is processed by the seventh feature extraction unit to obtain feature map T3 (resolution is 40×40), which is then output from the third output end of the neck network module.

[0108] That is, after being processed by the neck network module, the feature map T1, feature map T2 and feature map T3 are finally output through its own first output end, second output end and third output end respectively, and these three feature maps will be input into the head network module for further processing.

[0109] A feature map of a single scale may not be able to fully express the complex features of the target, especially for small targets or occluded targets. It can be understood that images of different scales contain information at different levels. For example, P4 with a resolution of 80×80 belongs to a shallow feature map, which contains rich detail information, but weak semantic information, while P9 with a resolution of 20×20 does not have sufficient spatial details, but contains stronger semantic information. Therefore, the neck network module set in this application can enhance the detection ability of the contour recognition model for multi-scale targets by fusing multi-scale image information and utilizing rich detail information and deep semantic information, so that the contour recognition model can better capture the global and local features of the target.

[0110] Among them, the first convolution unit and the second convolution unit have the same structure, both of which are CBS structures, with the convolution kernel k being 3, the step size s being 2, and the padding p being 1.

[0111] The PAN-FPN feature pyramid network includes three parts: multi-scale feature extraction, bidirectional feature fusion, and enhanced feature representation. Therefore, after the backbone network module extracts feature maps of different scales, the top-down feature pyramid in the Feature Pyramid Network (FPN) structure in the neck network can be used to achieve preliminary fusion of multi-scale features. Then, based on FPN, the bottom-up path of the Path Aggregation Network (PAN) is introduced to pass the underlying features to the high-level layers layer by layer to further enrich the multi-scale features. Through lateral connections, features of different scales are fused to ensure that each layer of features contains rich contextual information. Finally, through the fusion of bidirectional paths, the fused feature maps will contain richer contextual information and semantic information, which can enhance the detection ability of the contour recognition model for targets of different scales.

[0112] 3) Head network module.

[0113] The head network module can adopt the structure of the YOLACT (You Only Look At CoefficienTs) model, which can predict the prototype mask of the image and the mask coefficients of each Bbox instance in parallel. In the head network module, T1, T2 and T3 are upsampled to obtain a feature map of size 160×160 as a native mask for network prediction. The prediction head is used to generate the position, category and mask coefficient of the prediction box, and then two targets are screened out through NMS (Non-Maximum Suppression). The predicted masks of the two targets are generated with the help of the native mask and the mask coefficient, and then the predicted masks of the two targets are cropped according to the prediction box, and then the threshold is set and binarized to obtain the final result.

[0114] This module contains 3 Segment units and 1 Proto (Prototype Unit) unit. The 3 Segment units process the three feature maps T1, T2 and T3 respectively, and T1 is processed by the Proto unit. The structure of the Segment unit is as follows: Figure 7 As shown in the figure, it includes three branches, each of which contains 2 CBS2 layers, 1 Conv2d layer and 1 Box.Loss layer (Bounding Box Loss Layer). The convolution kernel k of the CBS2 layer is 3, the step size s is 1, and the padding p is 1. The convolution kernel k of the Conv2d layer is 3, the step size s is 1, and the padding p is 0.

[0115] Optionally, the first feature extraction unit, the second feature extraction unit, the third feature extraction unit, the fourth feature extraction unit, the fifth feature extraction unit, the sixth feature extraction unit and the seventh feature extraction unit have the same structure. For the convenience of explanation, these seven feature extraction units are all referred to as feature extraction units.

[0116] The feature extraction unit includes: the first CBS layer, the splitting layer, the comprehensive bottleneck layer, the splicing layer and the second CBS layer;

[0117] Among them, the input end of the first CBS layer is used as the input end of the feature extraction unit;

[0118] The output end of the first CBS layer is connected to the input end of the split layer, the third output end of the split layer is connected to the input end of the comprehensive bottleneck layer, the first output end of the split layer, the second output end of the split layer and the output end of the comprehensive bottleneck layer are all connected to the input end of the splicing layer, and the output end of the splicing layer is connected to the input end of the second CBS layer;

[0119] The output end of the second CBS layer serves as the output end of the feature extraction unit.

[0120] Specifically, the feature extraction unit C2f_Dual in the present application integrates the dual convolution mechanism into the existing C2f module (the C2f module is a modular design for convolutional neural networks), and uses the principles of heterogeneous convolution and group convolution to make lightweight improvements to the core structure Bottleneck of the existing C2f module, which can reduce the computational complexity of the model without losing feature information. The structure of the feature extraction unit is as follows: Figure 8 shown.

[0121] The following takes the feature map T1 as an example. After T1 is input into the feature extraction unit, it first passes through the first CBS layer, which includes the conv2d layer (the convolution kernel size is 1×1), the BatchNorm layer (represented by BN in the formula), and the SiLU activation function. That is, after T1 passes through the conv2d layer, the feature map is obtained. , obtained through the BatchNorm layer , and then through the SiLU activation function, the final calculation result is obtained , Represents T1, the formula is as follows:

[0122] ;

[0123] ;

[0124] ;

[0125] Then As the output of the first CBS layer, it is input to the splitting layer, and the torch.chunk function is introduced to split it to obtain , Two parts, they contrast The number of channels is halved, i.e. ( for number of channels).

[0126] ;

[0127] , These two parts are the two outputs of the split layer, and there is also a third output, which is also , and the comprehensive bottleneck layer needs to process the third output. The comprehensive bottleneck layer contains n improved bottleneck blocks. Each improved bottleneck block will output a channel number of The feature map branch That is to say, it needs to go through n improved bottleneck blocks in sequence, and finally the comprehensive bottleneck layer will output n feature map branches with half the number of channels of the previous feature map. , then these n outputs and , Input the concatenation layer together for concatenation, and get :

[0128] ;

[0129] Number of output channels for:

[0130] ;

[0131] Finally, it passes through the second CBS layer (the structure of the first CBS layer is the same as that of the second CBS layer, the convolution kernel k is 3, the step size s is 2, and the padding p is 1; among them, the k of the conv2d layer is 3, the step size s is 1, and the padding p is 0), that is, it passes through the conv2d layer in sequence to obtain , obtained through the BatchNorm layer , and then through the SiLU activation function, the final output graph is obtained , the formula is as follows:

[0132] ;

[0133] ;

[0134] ;

[0135] This is the garbage material map with the garbage pile outline marked on it, which is the final output of the contour recognition model.

[0136] Furthermore, the improved bottleneck block is a residual module consisting of a Dualconv layer and a third CBS layer, which includes two branches. For example, it will As the input and output of a branch, no processing is done. As input to another branch, in which After passing through a third CBS layer (the convolution kernel k of the third CBS layer is 1, the step size s is 1, and the padding p is 0) and a Dualconv layer, the output , and then merge and concatenate the outputs of these two branches to get:

[0137] .

[0138] Setting up two branches can be retained The original features of the image are preserved to avoid information loss and accelerate the training process of the model. The Dualconv layer includes two parallel conv2d layers, that is, the feature maps entering the Dualconv layer will be processed by two conv2d layers at the same time, and then the outputs of the two conv2d layers will be fused. This dual convolution combines the advantages of group convolution and heterogeneous convolution, and heterogeneous convolution combines the advantages of 3×3 convolution kernel and 1×1 convolution kernel. The 3×3 convolution kernel can capture more spatial information when extracting features, while the 1×1 convolution kernel can interact and integrate information between feature channels without increasing too many parameters and computational complexity. The Dualconv layer uses group convolution to divide the feature map into multiple groups, and the convolution of each group only processes part of the corresponding input feature map, which greatly reduces the complexity of the model.

[0139] The structures of these two conv2d layers are different. One conv2d layer has a convolution kernel k size of 3, a stride s of 1, a padding p of 1, and a group number g of 4, while the other conv2d layer has a convolution kernel k size of 1, a stride s of 1, and a padding p of 0.

[0140] In one instance, the output feature map size is , is the width and height of the output feature map, then in the standard convolution, the input feature map passes through the standard convolution layer The size is The convolution filter is filtered, where is the convolution kernel size, is the number of convolution kernel channels, so the standard convolution layer Executed The total number of floating point operations is:

[0141] .

[0142] In the Dualconv layer of this application, the number of convolution filter groups used is To control the convolution filter The ratio of convolution kernels, for a given , in all channels, the size is ( ) is the ratio of the combined synchronized convolution kernels to , while the rest The ratio of the convolution kernel is ( ), therefore, in In the double convolution layer structure composed of convolution filter groups, the combined convolution kernel of for:

[0143] ;

[0144] the remaining Convolution kernel for:

[0145] ;

[0146] but The total is:

[0147] ;

[0148] Then compare the computational cost of the double convolutional layer provided by this application with that of the standard convolutional layer, and the computational reduction ratio is :

[0149] .

[0150] Therefore, the dual convolutional layer provided in this application has improved computational efficiency compared to the standard convolutional layer and significantly reduced computational cost.

[0151] Through the improvement of some of the above modules or units, the present application finally trains a lightweight contour recognition model, which can reduce the complexity and number of parameters of the model, significantly reduce the demand for computing resources while maintaining high accuracy, and can be adapted to various fields, especially deployment in actual industrial environments.

[0152] Next, the process of determining the area ratio of the garbage pile in the feeding port in the garbage material location map according to the marked garbage pile outline in the present application is described in detail.

[0153] Convert the garbage material location map into a binary image;

[0154] Extracting a segmentation mask corresponding to the garbage dump outline from the binary image;

[0155] Extracting a segmentation contour from the segmentation mask, and calculating the area of ​​the garbage dump contour using the segmentation contour;

[0156] Obtaining the width and height of the garbage material map;

[0157] Calculate the area when the garbage bin is full according to the preset ratio, width and height;

[0158] The area of ​​the garbage pile outline is divided by the area of ​​the garbage bin when it is full to obtain the area ratio of the garbage pile in the feeding port.

[0159] Specifically, the mask is usually a binary image, where the garbage dump outline and its internal area are white, with pixel values ​​of 255, and the background is white, with pixel values ​​of 0. The function finds the segmentation contour of the garbage pile in the segmentation mask and calculates the area of ​​the garbage pile contour. The area of ​​the garbage pile contour actually refers to the area of ​​the garbage pile. The calculation formula is:

[0160] ;

[0161] The preset ratio can be determined according to the on-site feeding situation. This embodiment does not limit this. In one example, the area occupied by the feeding port in the entire garbage material level map is approximately half of the entire garbage material level map. Therefore, the preset ratio is set to 1 / 2, and the area of ​​the garbage pile outline is recorded as , the width of the garbage bitmap is recorded as , Gao Jizuo , from which we can get the area ratio of the garbage pile in the feeding port The calculation formula is:

[0162] .

[0163] The following embodiment specifically explains the process of predicting the time interval between any time node and the time node when the corresponding feeding port is empty based on the time interval corresponding to each garbage material level map and the area ratio in the present application.

[0164] Establishing a falling empirical function according to the time interval corresponding to each of the garbage material level maps and the area ratio;

[0165] For each of the garbage material level maps, determining the garbage material level maps corresponding to a preset number of frames before and after a preset number of frames corresponding to a time node of the garbage material level map as adjacent material level maps;

[0166] Determining the area ratio of each of the adjacent material level maps;

[0167] Calculate the garbage falling speed corresponding to the garbage level map based on the area ratio;

[0168] Establishing a time interval function according to the falling empirical function and the garbage falling speed corresponding to each of the garbage material level maps;

[0169] The time interval function is used to predict the time interval between any time node and the time node when the corresponding feeding port is empty.

[0170] Specifically, the time node is recorded as , the time interval is recorded as , time node The corresponding area ratio is ,use and The law of garbage level drop is obtained by empirical function :

[0171] ;

[0172] As the area ratio, select the garbage material map with a preset number of frames earlier than the time node corresponding to the garbage material map, such as the first 1 to 3 frames of the garbage material map, and select the garbage material map with a preset number of frames later than the time node corresponding to the garbage material map, all as adjacent material maps, so that the garbage falling speed corresponding to the garbage material map can be calculated. .

[0173] The falling experience function The speed of garbage falling By performing weighted analysis, the current time node can be predicted more accurately. The time interval between the time nodes when the feeding port is empty ,in is the viscosity coefficient, The time interval function is established as the garbage falling rate coefficient at the feeding port:

[0174] .

[0175] Therefore, the time interval function can be used to predict the time interval between any time node and the time node when the corresponding feeding port is empty. Then, by adding the time length corresponding to the time interval to the time node, it can be determined when this round of garbage will be fed, so as to facilitate the preparation for the next round of feeding and improve the overall feeding efficiency.

[0176] and Figure 1 Corresponding to the method described above, the embodiment of the present invention also provides a garbage feeding port feeding monitoring device for monitoring Figure 1 In the specific implementation of the method, the garbage feeding port feeding monitoring device provided by the embodiment of the present invention can be combined with a computer terminal or various mobile devices. Fig. 9 , introduce the feeding monitoring device of the garbage feeding port, such as Fig. 9 As shown, the device may include:

[0177] The acquisition module 10 is used to acquire each feeding image of the garbage feeding port during the feeding process;

[0178] The model processing module 20 is used to process each of the feeding images using a pre-trained contour recognition model to obtain each garbage material position map marked with the contour of the garbage pile;

[0179] The area ratio calculation module 30 is used to determine the area ratio of the garbage pile in the feeding port in each garbage material level map according to the marked garbage pile contour;

[0180] A time node determination module 40 is used to obtain the time node corresponding to the garbage material position map, and determine the time node when the feeding port corresponding to the time node is empty;

[0181] The time interval calculation module 50 is used to calculate the time interval between the time node corresponding to the garbage material position map and the time node when the feeding port is empty;

[0182] The prediction module 60 is used to predict the time interval between any time node and the time node when the corresponding feeding port is empty according to the time interval corresponding to each of the garbage material position maps and the area ratio, so as to realize feeding monitoring.

[0183] The present application first obtains various feeding images of the garbage feeding port during the feeding process. Compared with manual monitoring, the actual feeding situation of the garbage feeding port can be determined more accurately. The pre-trained contour recognition model can process the feeding images to obtain various garbage material position maps marked with garbage pile contours, reducing the subjectivity of manual judgment. Since the time interval information needs to be determined, it is crucial to determine the area ratio of the garbage pile contour in each garbage material position map in the feeding port. By comparing the time node of each garbage material position map with the time node when the corresponding feeding port is empty, the time interval can be accurately obtained, which can reduce the workload of manual recording and judgment, avoid time recording errors caused by human factors, and can predict the garbage pile feeding situation at any time node, thereby helping staff to reasonably arrange the time and resources for garbage feeding or other processing processes, improve garbage feeding efficiency, save time and effort, and reduce feeding or other processing delays caused by excessive garbage accumulation.

[0184] Furthermore, the embodiment of the present application provides a garbage feeding port feeding monitoring device. Optionally, Fig.10 The hardware structure diagram of the garbage feeding port monitoring device is shown. Fig.10 The hardware structure of the garbage feeding port monitoring device may include: at least one processor 01, at least one communication interface 02, at least one memory 03 and at least one communication bus 04.

[0185] In the embodiment of the present application, the number of the processor 01 , the communication interface 02 , the memory 03 , and the communication bus 04 is at least one, and the processor 01 , the communication interface 02 , and the memory 03 communicate with each other through the communication bus 04 .

[0186] The processor 01 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0187] The memory 03 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), etc., such as at least one disk memory.

[0188] The memory stores a program, and the processor can call the program stored in the memory, and the program is used to execute the following garbage feeding port feeding monitoring method, including:

[0189] Acquire various feeding images of the garbage feeding port during the feeding process;

[0190] Using a pre-trained contour recognition model to process each of the feeding images, to obtain each garbage material location map marked with the contour of the garbage pile;

[0191] For each of the garbage material location maps, determine the area ratio of the garbage pile in the material feeding port in the garbage material location map according to the marked garbage pile contour;

[0192] Obtain the time node corresponding to the garbage material position map, and determine the time node when the feeding port corresponding to the time node is empty;

[0193] Calculate the time interval between the time node corresponding to the garbage material position map and the time node when the feeding port is empty;

[0194] According to the time interval corresponding to each of the garbage material position maps and the area ratio, the time interval between any time node and the time node when the corresponding feeding port is empty is predicted to realize feeding monitoring.

[0195] Optionally, the detailed functions and extended functions of the program may refer to the description of the garbage feeding port feeding monitoring method in the method embodiment.

[0196] The embodiment of the present application further provides a storage medium, which can store a program suitable for execution by a processor, and when the program is running, controls the device where the storage medium is located to execute the following garbage feeding port feeding monitoring method, including:

[0197] Acquire various feeding images of the garbage feeding port during the feeding process;

[0198] Using a pre-trained contour recognition model to process each of the feeding images, to obtain each garbage material location map marked with the contour of the garbage pile;

[0199] For each of the garbage material location maps, determine the area ratio of the garbage pile in the material feeding port in the garbage material location map according to the marked garbage pile contour;

[0200] Obtain the time node corresponding to the garbage material position map, and determine the time node when the feeding port corresponding to the time node is empty;

[0201] Calculate the time interval between the time node corresponding to the garbage material position map and the time node when the feeding port is empty;

[0202] According to the time interval corresponding to each of the garbage material position maps and the area ratio, the time interval between any time node and the time node when the corresponding feeding port is empty is predicted to realize feeding monitoring.

[0203] Specifically, the storage medium may be a computer-readable storage medium, and the computer-readable storage medium may be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk or a ROM.

[0204] Optionally, the detailed functions and extended functions of the program may refer to the description of the garbage feeding port feeding monitoring method in the method embodiment.

[0205] In addition, the functional modules in the various embodiments of the present disclosure can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. If the function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a live broadcast device, or a network device, etc.) to perform all or part of the steps of the methods of the various embodiments of the present disclosure.

[0206] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0207] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0208] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for monitoring the feeding of garbage feeding ports, characterized in that: include: Acquire various feeding images of the garbage feeding port during the feeding process; Using a pre-trained contour recognition model to process each of the feeding images, to obtain garbage material location maps each marked with a garbage pile contour; For each of the garbage material location maps, determine the area ratio of the garbage pile in the material feeding port in the garbage material location map according to the marked garbage pile contour; Obtain the time node corresponding to the garbage material position map, and determine the time node when the feeding port corresponding to the time node is empty; Calculate the time interval between the time node corresponding to the garbage material position map and the time node when the feeding port is empty; According to the time interval corresponding to each of the garbage material position maps and the area ratio, the time interval between any time node and the time node when the corresponding feeding port is empty is predicted to realize feeding monitoring.

2. The method according to claim 1, characterized in that The contour recognition model includes a backbone network module, a neck network module, and a head network module; Among them, the input end of the backbone network module serves as the input end of the contour recognition model, the output end of the backbone network module is connected to the input end of the neck network module, the output end of the neck network module is connected to the input end of the head network module, and the output end of the head network module serves as the output end of the contour recognition model.

3. The method according to claim 2, characterized in that The backbone network module includes a first downsampling unit, a second downsampling unit, a third downsampling unit, a fourth downsampling unit, a fifth downsampling unit, a first feature extraction unit, a second feature extraction unit, a third feature extraction unit, a fourth feature extraction unit and a spatial pyramid pooling unit; Wherein, the input end of the first down-sampling unit is used as the input end of the backbone network module, the output end of the first down-sampling unit is connected to the input end of the second down-sampling unit, the output end of the second down-sampling unit is connected to the input end of the first feature extraction unit, the output end of the first feature extraction unit is connected to the input end of the third down-sampling unit, the output end of the third down-sampling unit is connected to the input end of the second feature extraction unit, the output end of the second feature extraction unit is connected to the input end of the fourth down-sampling unit, the output end of the fourth down-sampling unit is connected to the input end of the third feature extraction unit, the output end of the third feature extraction unit is connected to the input end of the fifth down-sampling unit, the output end of the fifth down-sampling unit is connected to the input end of the fourth feature extraction unit, and the output end of the fourth feature extraction unit is connected to the input end of the spatial pyramid pooling unit; The output end of the second feature extraction unit, the output end of the third feature extraction unit and the output end of the spatial pyramid pooling unit serve as the first output end, the second output end and the third output end of the backbone network module respectively, and are all connected to the input end of the neck network module.

4. The method according to claim 3, characterized in that The neck network module includes a first upsampling unit, a second upsampling unit, a first fusion unit, a second fusion unit, a third fusion unit, a fourth fusion unit, a fifth feature extraction unit, a sixth feature extraction unit, a seventh feature extraction unit, a first convolution unit, and a second convolution unit; The input end of the second fusion unit, the input end of the first fusion unit and the input end of the first upsampling unit are respectively used as the first input end, the second input end and the third input end of the neck network module; The first output end of the second fusion unit is used as the first output end of the neck network module, the first output end of the sixth feature extraction unit is used as the second output end of the neck network module, and the output end of the seventh feature extraction unit is used as the third output end of the neck network module; The first input end, the second input end and the third input end of the neck network module are connected to the first output end, the second output end and the third output end of the backbone network module respectively; The output end of the first upsampling unit and the second output end of the backbone network module are both connected to the second input end of the neck network module, the input end of the fifth feature extraction unit and the input end of the third fusion unit are both connected to the output end of the first fusion unit, the output end of the fifth feature extraction unit is connected to the input end of the second upsampling unit, the output end of the second upsampling unit and the first output end of the backbone network module are both connected to the first input end of the neck network module, the second output end of the second fusion unit is connected to the input end of the first convolution unit, the output end of the first convolution unit and the output end of the first fusion unit are both connected to the input end of the third fusion unit, the output end of the third fusion unit is connected to the input end of the sixth feature extraction unit, the second output end of the sixth feature extraction unit is connected to the input end of the second convolution unit, the output end of the second convolution unit is connected to the input end of the fourth fusion unit, the input end of the fourth fusion unit is also connected to the input end of the first upsampling unit, and the output end of the fourth fusion unit is connected to the input end of the seventh feature extraction unit.

5. The method according to claim 4, characterized in that The first feature extraction unit, the second feature extraction unit, the third feature extraction unit, the fourth feature extraction unit, the fifth feature extraction unit, the sixth feature extraction unit, and the seventh feature extraction unit have the same structure and are all used as feature extraction units. The feature extraction unit includes: a first CBS layer, a splitting layer, a comprehensive bottleneck layer, a splicing layer, and a second CBS layer; Wherein, the input end of the first CBS layer serves as the input end of the feature extraction unit; The output end of the first CBS layer is connected to the input end of the split layer, the third output end of the split layer is connected to the input end of the comprehensive bottleneck layer, the first output end of the split layer, the second output end of the split layer and the output end of the comprehensive bottleneck layer are all connected to the input end of the splicing layer, and the output end of the splicing layer is connected to the input end of the second CBS layer; The output end of the second CBS layer serves as the output end of the feature extraction unit.

6. The method according to claim 1, characterized in that Determining the area ratio of the garbage pile in the feeding port in the garbage material location map according to the marked garbage pile contour includes: Convert the garbage material location map into a binary image; Extracting a segmentation mask corresponding to the garbage dump outline from the binary image; Extracting a segmentation contour from the segmentation mask, and calculating the area of ​​the garbage dump contour using the segmentation contour; Obtaining the width and height of the garbage material map; Calculate the area when the garbage bin is full according to the preset ratio, width and height; The area of ​​the garbage pile outline is divided by the area of ​​the garbage bin when it is full to obtain the area ratio of the garbage pile in the feeding port.

7. The method according to any one of claims 1 to 6, characterized in that: The predicting, based on the time interval corresponding to each of the garbage material position maps and the area ratio, of the time interval between any time node and the time node when the corresponding feeding port is empty comprises: Establishing a falling empirical function according to the time interval corresponding to each of the garbage material level maps and the area ratio; For each of the garbage material level maps, determining the garbage material level maps corresponding to a preset number of frames before and after a preset number of frames corresponding to a time node of the garbage material level map as adjacent material level maps; Determining the area ratio of each of the adjacent material level maps; Calculate the garbage falling speed corresponding to the garbage level map based on the area ratio; Establishing a time interval function according to the falling empirical function and the garbage falling speed corresponding to each of the garbage material level maps; The time interval function is used to predict the time interval between any time node and the time node when the corresponding feeding port is empty.

8. A garbage feeding port feeding monitoring device, characterized in that: include: An acquisition module is used to acquire each feeding image of the garbage feeding port during the feeding process; A model processing module, used to process each of the feeding images using a pre-trained contour recognition model to obtain garbage material location maps each marked with a garbage pile contour; An area ratio calculation module is used to determine, for each garbage material location map, the area ratio of the garbage pile in the material feeding port in the garbage material location map according to the marked garbage pile contour; A time node determination module is used to obtain the time node corresponding to the garbage material position map and determine the time node when the feeding port corresponding to the time node is empty; A time interval calculation module, used to calculate the time interval between the time node corresponding to the garbage material position map and the time node when the feeding port is empty; The prediction module is used to predict the time interval between any time node and the time node when the corresponding feeding port is empty according to the time interval corresponding to each of the garbage material position maps and the area ratio, so as to realize feeding monitoring.

9. A garbage feeding port feeding monitoring device, characterized in that: including memory and processor; The memory is used to store programs; The processor is used to execute the program to implement the various steps of the garbage feeding port feeding monitoring method as described in any one of claims 1-7.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the method for monitoring the feeding of a garbage feeding port as described in any one of claims 1 to 7 is implemented.

Images