Garbage can overflow state detection method, device and equipment and storage medium

Abstract

The application relates to the technical field of image processing, and provides a garbage can overflow state detection method and device, equipment and a storage medium. The method comprises the following steps: acquiring a monitoring image of a garbage can, extracting image information and depth information of the monitoring image; based on the image information and the depth information, calculating image information difference and depth information difference between two adjacent images in the monitoring image; extracting multi-level features of the image information difference and the depth information difference and fusing the multi-level features to obtain fused features; and detecting the overflow state of the garbage can according to the fused features. The garbage can overflow state detection method provided in the application can effectively inhibit the influence of environmental changes and other factors on the detection effect by fusing the difference values of the depth information and the image information, and improve the detection accuracy of the overflow state of the garbage can in the environment of rain, snow and heavy fog.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a garbage can overflow state detection method and device, equipment and a storage medium. BACKGROUND

[0002] At present, the detection of the garbage can overflow state is mainly realized through an image detection method. The detection image mainly comes from a video monitoring picture of the garbage can, and the information source is single. In a low-visibility environment such as rain, snow and fog, the image information collected by monitoring is affected, thereby affecting the detection effect of the garbage can overflow state and leading to low detection accuracy. SUMMARY

[0003] The embodiments of the present application provide a garbage can overflow state detection method, device, equipment and storage medium to solve the technical problem of low detection accuracy of the existing detection method.

[0004] In a first aspect, the embodiments of the present application provide a garbage can overflow state detection method, comprising:

[0005] obtaining a monitoring image of a garbage can, extracting image information and depth information of the monitoring image;

[0006] calculating image information difference and depth information difference between adjacent two frames of images in the monitoring image based on the image information and the depth information;

[0007] extracting multi-level features of the image information difference and the depth information difference and fusing to obtain fused features;

[0008] detecting the overflow state of the garbage can according to the fused features.

[0009] In one embodiment, the monitoring image includes an RGB image and a depth image, and the step of calculating the image information difference and the depth information difference between adjacent two frames of images in the monitoring image based on the image information and the depth information includes:

[0010] calculating the difference value between adjacent two frames of RGB images in the monitoring image based on the image information to obtain the image information difference of the monitoring image;

[0011] aligning the depth image and the RGB image through interpolation processing, determining the corresponding relationship between each pixel point in the RGB image and the depth information, and calculating the difference value between adjacent two frames of depth images in the monitoring image based on the corresponding relationship to obtain the depth information difference of the monitoring image.

[0012] In an embodiment, the step of extracting and fusing the multi-level features of the image information difference and the depth information difference comprises:

[0013] The image information difference and the depth information difference are input into a preset backbone network, and the multi-level features of the image information difference and the depth information difference are extracted by using the backbone network, wherein the backbone network is configured with a convolution layer in which a dilated convolution and a deformable convolution are parallel, and does not contain a full connection layer and a pooling layer.

[0014] The multi-level features are input into a lateral connection structure of the backbone network and a preset feature pyramid network, and the multi-level features of the image information difference and the depth information difference are fused to obtain fused features, wherein the lateral connection structure contains a parallel spatial attention mechanism and a channel attention mechanism.

[0015] In an embodiment, the step of inputting the multi-level features into the lateral connection structure of the backbone network and the preset feature pyramid network, and fusing the multi-level features of the image information difference and the depth information difference comprises:

[0016] Each of the multi-level features is input into the channel attention mechanism in the lateral connection structure to perform a channel attention operation to obtain a first attention image.

[0017] The first attention image is input into the spatial attention mechanism in the lateral connection structure to perform a spatial attention operation to obtain a second attention image.

[0018] Convolution and up-sampling operations are performed on the second attention image to obtain high-level features corresponding to each of the multi-level features, and the high-level features contain RGB features and depth features.

[0019] The RGB features and the depth features in the high-level features are subjected to splicing processing to fuse the multi-level features of the image information difference and the depth information difference.

[0020] In an embodiment, the step of inputting each of the multi-level features into the channel attention mechanism in the lateral connection structure to perform a channel attention operation to obtain a first attention image comprises:

[0021] Each of the multi-level features is input into the channel attention mechanism in the lateral connection structure, and global maximum pooling and mean pooling operations are performed on the multi-level features to obtain a first feature image.

[0022] performing a convolution operation on the first feature image to obtain a second feature image, and obtaining a first attention image according to a product of the second feature image and the hierarchical features.

[0023] In one embodiment, the step of inputting the first attention image into the spatial attention mechanism in the lateral connection structure to perform a spatial attention operation to obtain a second attention image comprises:

[0024] inputting the first attention image into the spatial attention mechanism in the lateral connection structure, performing a mean pooling operation on the first attention image to obtain a third feature image;

[0025] performing a global max pooling and a convolution operation on the third feature image to obtain a fourth feature image, and obtaining a second attention image according to a product of the fourth feature image and the first attention image.

[0026] In one embodiment, the step of detecting the overflow state of the garbage can according to the fusion feature further comprises:

[0027] displaying a detection result of the overflow state of the garbage can;

[0028] outputting a warning prompt information according to the detection result when the garbage can is in the overflow state.

[0029] In a second aspect, the embodiments of the present application provide a garbage can overflow state detection device, comprising:

[0030] an image acquisition module configured to acquire a monitoring image of a garbage can, and extract image information and depth information of the monitoring image;

[0031] a difference calculation module configured to calculate image information difference and depth information difference between two adjacent frames of images in the monitoring image based on the image information and the depth information;

[0032] a feature fusion module configured to extract and fuse multi-level features of the image information difference and the depth information difference to obtain a fusion feature;

[0033] a state detection module configured to detect an overflow state of the garbage can according to the fusion feature.

[0034] In a third aspect, the embodiments of the present application provide an electronic device, comprising a processor and a memory storing a computer program, wherein the processor implements the steps of the garbage can overflow state detection method of the first aspect when executing the program.

[0035] In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the steps of the garbage can overflow state detection method in the first aspect.

[0036] The garbage can overflow state detection method, device, equipment and computer readable storage medium provided by the embodiment of the present application can effectively inhibit the influence of environmental changes and other factors on the detection effect by fusing the difference value of the depth information and the image information, and improve the detection accuracy of the garbage can overflow state in the rain, snow, fog and other environments. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the present application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0038] Figure 1 is one of the flowcharts of the garbage can overflow state detection method provided by the embodiment of the present application;

[0039] Figure 2 is one of the image processing flowcharts of the garbage can overflow state detection method provided by the embodiment of the present application;

[0040] Figure 3 is one of the structural diagrams of the garbage can overflow state detection device provided by the embodiment of the present application;

[0041] Figure 4 is the structural diagram of the electronic equipment provided by the embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the present application more clear, the technical solutions in the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0043] Figure 1 is one of the flowcharts of the garbage can overflow state detection method provided by the embodiment of the present application. Referring to Figure 1 The garbage can overflow state detection method provided by the embodiment of the present application can include:

[0044] Step 100, acquiring a monitoring image of a garbage can, extracting image information and depth information of the monitoring image;

[0045] It should be noted that, for the case that the monitoring video image of the garbage can collected in the environment with low visibility such as rain, snow, heavy fog and night is not clear, which affects the detection accuracy of the garbage can overflow state, the embodiment of the present application provides a garbage can overflow state detection method, which can improve the detection accuracy of the garbage can overflow state in the environment such as rain, snow, heavy fog and night. Specifically, first, a monitoring image of a garbage can is acquired, which is an image containing the garbage can, which can be a video monitoring image of the garbage can. The acquired monitoring image includes an RGB image and a depth image, and the acquired monitoring image is input into a pre-trained detection model to perform filtering, noise reduction, cropping and other preprocessing operations to obtain corresponding RGB images and depth images, and the image information and depth information of the garbage can are extracted therefrom.

[0046] Step 200, based on the image information and the depth information, calculating the image information difference and the depth information difference between adjacent two frames of images in the monitoring image;

[0047] The acquired monitoring image includes an RGB image and a depth image, and according to the extracted image information and depth information, the information difference value between adjacent two frames of images in the acquired monitoring image is calculated to obtain the image information difference and the depth information difference of the monitoring image.

[0048] Further, step 200 can further include:

[0049] Step 201, based on the image information, calculating the difference value between adjacent two frames of RGB images in the monitoring image to obtain the image information difference of the monitoring image;

[0050] Step 202, aligning the depth image and the RGB image through interpolation processing, determining the corresponding relationship between each pixel point in the RGB image and the depth information, and calculating the difference value between adjacent two frames of depth images in the monitoring image based on the corresponding relationship to obtain the depth information difference of the monitoring image.

[0051] In the calculation of the image information difference and the depth information difference of the monitoring image, specifically, the difference value between the RGB images of two adjacent frames is calculated according to the extracted image information, to obtain the image information difference, and the difference value between the depth images of two adjacent frames is calculated according to the extracted depth information, to obtain the depth information difference. Further, before the calculation of the depth information difference, the RGB image and the depth image are aligned through interpolation, alignment and other processing to determine the correspondence between each pixel point in the RGB image and the depth information of the depth image, thereby determining the depth information of each pixel point in the RGB image, and based on the depth information of each pixel point, the difference value between the depth images of two adjacent frames is calculated to obtain the depth information difference of the monitoring image.

[0052] Step 300, extracting multi-level features of the image information difference and the depth information difference and fusing to obtain fused features;

[0053] Based on the calculated image information difference and depth information difference of the monitoring image, a multi-level feature image is constructed for monitoring the overfilling state of the garbage can. In the construction of the multi-level feature image, the calculated image information difference and depth information difference are fused and input into the information fusion network and the detection network of the detection model, the multi-level features of the image information difference and the depth information difference are extracted and fused to obtain the fused features, and the detection is performed according to the fused features. The pre-trained detection model can be any deep learning neural network, such as multilayer perceptron, VGG (Visual Geometry Group Network, VGG model), Resnet (Residual Network, residual network), etc., which is not limited herein.

[0054] Further, step 300 can further include:

[0055] Step 301, inputting the image information difference and the depth information difference into a preset backbone network, and extracting multi-level features of the image information difference and the depth information difference by using the backbone network, wherein the backbone network is configured with a convolutional layer in which a dilated convolution and a deformable convolution are parallel, and does not contain a full connection layer and a pooling layer;

[0056] Step 302, inputting the multi-level features into a lateral connection structure of the backbone network and a preset feature pyramid network, fusing the multi-level features of the image information difference and the depth information difference to obtain fused features, wherein the lateral connection structure contains a parallel spatial attention mechanism and a channel attention mechanism.

[0057] In constructing the multi-level feature image, the difference values of the adjacent two frames of depth images and the adjacent two frames of RGB images are fused and input into a preset backbone network, taking the Resnet50 network structure as the backbone network as an example, the backbone network is a Resnet50 network which is configured with a convolutional layer in parallel with a dilated convolution and a deformable convolution, and the full connection layer and the pooling layer are removed, the features of the difference of the adjacent two frames of depth images and the adjacent two frames of RGB images of each layer are extracted by using the backbone network, to obtain multi-level features. By using the parallel structure of dilated convolution and deformable convolution, the high-level features of image information and depth information can be effectively extracted, and the detection effect of small size in the image is enhanced.

[0058] The features of each level in the extracted multi-level features are input into the lateral connection structure of the backbone network and the preset feature pyramid network (FPN), and the multi-level features of the difference of image information and the difference of depth information are fused by using the attention mechanism to obtain fused features. Among them, the lateral connection structure of the feature pyramid and the backbone network is configured with parallel channel attention mechanism and spatial attention mechanism. The parallel structure of spatial attention and channel attention mechanism suppresses the noise interference of low-quality features. By fusing the high-level features of depth information and image information, the detection effect of the full-overflow state of the garbage can is enhanced.

[0059] Step 400, detecting the full-overflow state of the garbage can according to the fused features.

[0060] According to the fused features, the full-overflow state of the garbage can can be detected, which can improve the detection effect, especially in rainy, snowy, foggy and other weather conditions. Specifically, by fusing the difference values of depth information and RGB image information, the influence of environmental changes and other factors on the detection effect can be effectively suppressed, and the detection accuracy of the full-overflow state of the garbage can in rainy, snowy, foggy and other environments can be improved. That is, by fusing the difference values of depth information and RGB information, the influence of environmental changes and other factors on the detection effect can be effectively suppressed, and the detection accuracy of the full-overflow state of the garbage can in rainy, snowy, foggy and other environments can be improved.

[0061] Reference Figure 2 , Figure 2 The image processing flowchart of the garbage can full-overflow state detection method provided by the embodiment of the application is shown in Figure 2In the specific implementation, the collected monitoring images include an RGB image and a depth image, wherein the RGB image can be collected by a visual receiver, and the depth image can be collected by a depth information receiver; the collected images are input into a pre-trained detection model; after pre-processing the collected RGB image and depth image, the collected RGB image and depth image are subjected to feature extraction and fusion through an information fusion network and a detection network to obtain fusion features, and the overfill state of the garbage can is detected according to the fusion features. The detection model includes a backbone network, an information fusion network and a detection network required for pre-processing, wherein the backbone network is, for example, a Resnet50 network, the information fusion network is, for example, an improved feature pyramid network (FPN), and the detection network is, for example, a FCN convolutional network. In the backbone network, multi-level features of the difference between adjacent two frames of image information and depth information are extracted; in the information fusion network, high-level features of RGB information and depth information of the same level structure in the feature pyramid network (FPN) are subjected to feature fusion to obtain multi-level fusion features; and in the detection network, a FCN network is added behind the feature fusion structure of each level of the information fusion network to detect the overfill state of the garbage can.

[0062] Further, with reference to Figure 2 , after step 400, the method can further include:

[0063] Step 501: displaying a detection result of the overfill state of the garbage can.

[0064] Step 502: outputting a warning prompt information according to the detection result if the garbage can is in the overfill state.

[0065] In Figure 2 , the detection of the overfill state of the garbage can further includes publishing of the detection result and the warning prompt information; the detection result output by the detection network is displayed, that is, the detection result of the overfill state of the garbage can is displayed, which is, for example, whether the garbage can is currently in the overfill state or the overfill degree of the garbage can, which reflects the amount of garbage currently contained in the garbage can. According to the detection result, if the garbage can is in the overfill state or the overfill degree of the garbage can exceeds a preset threshold, the warning prompt information is output.

[0066] In the embodiment, by fusing the difference between the depth information and the image information, the influence of environmental changes and other factors on the detection effect can be effectively inhibited, and the detection accuracy of the overfill state of the garbage can in the rain, snow, fog and other environments can be improved.

[0067] Further, through the convolutional layers in parallel by the hollow convolution and the deformable convolution, the high-level features of the image information and the depth information can be effectively extracted, and through the structure in parallel by the spatial attention mechanism and the channel attention mechanism, the high-level features of the image information difference and the depth information difference are fused, the detection effect on small size in the image is enhanced, the noise interference of low quality features is inhibited, and the detection accuracy of the garbage can overflow state is further improved.

[0068] In one embodiment, step 302 can further include:

[0069] Step 3021, input each level feature in the multi-level feature into the channel attention mechanism in the lateral connection structure to perform channel attention operation, and obtain a first attention image;

[0070] Step 3022, input the first attention image into the spatial attention mechanism in the lateral connection structure to perform spatial attention operation, and obtain a second attention image;

[0071] Step 3023, perform convolution and up-sampling operation on the second attention image to obtain high-level features corresponding to each level feature in the multi-level feature, and the high-level features include RGB features and depth features;

[0072] Step 3024, perform splicing processing on the RGB features and the depth features in the high-level features to fuse the multi-level features of the image information difference and the depth information difference.

[0073] In the fusion of the high-level features of the image information difference and the depth information difference, the multi-level features of the image information difference and the depth information difference are input into the channel attention mechanism in the lateral connection structure to perform channel attention operation on the multi-level features, and a corresponding first attention image is obtained.

[0074] Further, in step 3021, the step of performing channel attention operation on the multi-level features can include:

[0075] Step 3121, input each level feature in the multi-level feature into the channel attention mechanism in the lateral connection structure to perform global maximum pooling and mean pooling operation on the each level feature, and obtain a first feature image;

[0076] Step 3221, perform convolution operation on the first feature image to obtain a second feature image, and according to the product of the second feature image and the each level feature, obtain a first attention image.

[0077] As shown in formula 1, in the lateral connection structure, first, the features obtained by each level in the backbone network Resnet50 are input into the channel attention for operation:

[0078] Mc(F′ i )=Conv(Pave(F i )+Pmax(F i ))*F i (1)

[0079] wherein, F i represents the feature map output by the i-th level of the backbone network, F′ i represents the feature map output by the corresponding level in the first attention image, Conv(·) represents the convolution operation, Pmax(·) represents the global maximum pooling operation, Pave(·) represents the average pooling operation, * represents the multiplication of corresponding elements in the feature map, and + represents the addition of corresponding elements in the feature map.

[0080] When performing the channel attention operation on the image information difference and the depth information difference, first, the pooling operation is performed on each level feature F i in the extracted multi-level feature to obtain the corresponding first feature image, wherein the pooling operation includes the global maximum pooling and the average pooling operation. The features obtained by different pooling operations are added to obtain the corresponding first feature image. The convolution operation is performed on the first feature image feature obtained by the pooling operation to obtain the second feature image, and the second feature image is multiplied by the original feature before the pooling operation. According to the product of the second feature image and the original level feature, the first attention image is obtained.

[0081] The spatial attention operation is performed on the first attention image obtained by the channel attention operation to obtain the corresponding second attention image. In step 3022, the spatial attention is performed on the first attention image to obtain the second attention image, and the steps include:

[0082] In step 3122, the first attention image is input into the spatial attention mechanism in the lateral connection structure, and the average pooling operation is performed on the first attention image to obtain the third feature image.

[0083] In step 3222, the global maximum pooling and the convolution operation are performed on the third feature image to obtain the fourth feature image, and the second attention image is obtained according to the product of the fourth feature image and the first attention image.

[0084] When the spatial attention operation is performed on the first attention image, first, the average pooling operation is performed on the feature map of each level in the first attention image to obtain the corresponding third feature image. The global maximum pooling operation and the convolution operation are sequentially performed on the third feature image to obtain the corresponding fourth feature image. The fourth feature image is multiplied by the feature map in the first attention image, and the second attention image is obtained according to the product of the fourth feature image and the feature map in the first attention image. As shown in the following formula 2:

[0085] Ms(F″ i )=Conv(Pmax(Pave(F′ i )))*F′ i (2)

[0086] Among them, F′ i F″ represents the feature map corresponding to the i-th layer of the backbone network in the first attention image. i This represents the feature map corresponding to the i-th layer of the backbone network in the second attention image.

[0087] After obtaining the second attention image, the Feature Pyramid Network (FPN) is used to perform convolution and upsampling operations on the second attention image to obtain high-level features corresponding to each level of the multi-level features of image information difference and depth information difference. These high-level features include RGB features and depth features. The RGB features and depth features in the high-level features are then concatenated to obtain multi-level fused features. The process of performing convolution and upsampling operations on the second attention image is shown in Equation 3 below:

[0088] add(P′ i =Conv(Ms(F″) i ))+Up(P′ i+1 (3)

[0089] Among them, P′ i Let represent the feature map output by the i-th layer of the FPN network, and Up(·) denotes the upsampling operation. Using the FPN network allows for better fusion of high-level features from different layers of the backbone network, including RGB and depth information, to construct multi-scale feature maps for detecting the overflow state of trash cans at different scales. This reduces interference caused by semantic differences in high-level features from different layers of RGB and depth information, thus improving the accuracy of trash can overflow detection.

[0090] Furthermore, the high-level features obtained after convolution and upsampling operations using the Feature Pyramid Network (FPN) include RGB features and depth features. Let's assume that the RGB features and depth features in the high-level features obtained at the i-th layer of the FPN network are represented as follows: and The fused features obtained through splicing can then be represented as:

[0091]

[0092] Among them, c iThe fusion feature represents the i-th level of the FPN network, WxH represents the size of the feature map, C1 and C2 respectively represent the channel number of the feature map of the high-level feature obtained by the i-th layer of the FPN network, Concat(·) represents a splicing operation, and C1+C2 represents the addition of feature channels.

[0093] When detecting the overflow state of the garbage can according to the fusion feature, the fusion feature is input into a detection network, and one embodiment of the configuration of the detection network is as follows:

[0094] Two FCN (Fully Convolutional Networks for Semantic Segmentation) networks are configured after the fusion feature of each level, the first FCN network is composed of four 3x3xC convolutional layers, and the activation function is ReLU (Linear rectification function). Wherein, C represents the channel number of the input feature map, which can be set by the user, and one 3x3x9 output layer is configured after the convolutional layer, and the activation function is sigmoid, which is used to predict the overflow state of each garbage can.

[0095] The second FCN network is composed of four 3x3xC convolutional layers, and the activation function is ReLU. (C also represents the channel number of the input feature map set by the user) One 4x9 output layer is configured after the convolutional layer, which is used to predict the position information of the garbage can. Combining the detection results of the position information and the overflow state of the garbage can, when the garbage can is in the overflow state or the overflow degree of the garbage can exceeds the preset threshold, an early warning prompt information is output.

[0096] In this embodiment, the feature pyramid can better fuse the high-level features of image information and depth information, so as to more effectively detect the overflow state of the garbage can and improve the detection accuracy.

[0097] The garbage can overflow state detection device provided by the embodiments of the present application will be described below. The garbage can overflow state detection device described below can be correspondingly referred to the garbage can overflow state detection method described above.

[0098] Referring to Figure 3 The garbage can overflow state detection device provided by the embodiments of the present application comprises:

[0099] The image acquisition module 10 is configured to acquire a monitoring image of a garbage can, and extract image information and depth information of the monitoring image.

[0100] The difference calculation module 20 is configured to calculate the image information difference and the depth information difference between two adjacent frames of images in the monitoring image based on the image information and the depth information.

[0101] a feature fusion module 30 configured to extract and fuse multi-level features of the image information difference and the depth information difference to obtain fused features;

[0102] a state detection module 40 configured to detect an overflow state of the garbage can according to the fused features.

[0103] In an embodiment, the monitoring image comprises an RGB image and a depth image, and the difference calculation module 20 is further configured to:

[0104] calculate a difference between two adjacent frames of the RGB image in the monitoring image based on the image information to obtain an image information difference of the monitoring image;

[0105] align the depth image and the RGB image through interpolation processing, determine a corresponding relationship between each pixel point in the RGB image and the depth information, and calculate a difference between two adjacent frames of the depth image in the monitoring image based on the corresponding relationship to obtain a depth information difference of the monitoring image.

[0106] In an embodiment, the feature fusion module 30 is further configured to:

[0107] input the image information difference and the depth information difference into a preset backbone network, and extract multi-level features of the image information difference and the depth information difference by using the backbone network, wherein the backbone network is configured with a convolution layer in which a dilated convolution and a deformable convolution are parallel, and does not contain a full connection layer and a pooling layer;

[0108] input the multi-level features into a lateral connection structure of the backbone network and a preset feature pyramid network, fuse the multi-level features of the image information difference and the depth information difference to obtain fused features, wherein the lateral connection structure contains a parallel spatial attention mechanism and a channel attention mechanism.

[0109] In an embodiment, the feature fusion module 30 is further configured to:

[0110] input each level feature in the multi-level features into a channel attention mechanism in the lateral connection structure to perform a channel attention operation to obtain a first attention image;

[0111] input the first attention image into a spatial attention mechanism in the lateral connection structure to perform a spatial attention operation to obtain a second attention image;

[0112] perform convolution and up-sampling operations on the second attention image to obtain high-level features corresponding to each level feature in the multi-level features, and the high-level features contain RGB features and depth features.

[0113] The RGB features and depth features in the high-level features are concatenated to fuse the multi-level features of the image information difference and the depth information difference.

[0114] In one embodiment, the feature fusion module 30 is further configured to:

[0115] Each level of the multi-level features is input into the channel attention mechanism in the lateral connection structure, and global max pooling and mean pooling operations are performed on each level of features to obtain the first feature image;

[0116] A convolution operation is performed on the first feature image to obtain a second feature image, and a first attention image is obtained based on the product of the second feature image and the features of each level.

[0117] In one embodiment, the feature fusion module 30 is further configured to:

[0118] The first attention image is input into the spatial attention mechanism in the lateral connection structure, and mean pooling is performed on the first attention image to obtain the third feature image;

[0119] Global max pooling and convolution operations are performed on the third feature image to obtain the fourth feature image. The second attention image is obtained by multiplying the fourth feature image and the first attention image.

[0120] In one embodiment, the trash can overflow detection device further includes an early warning module for:

[0121] The display shows the detection results of the overflow status of the trash can;

[0122] If the trash can is overflowing based on the detection results, an early warning message will be output.

[0123] Figure 4 Example: A schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 can call a computer program in the memory 430 to execute the steps of the trash can overflow detection method, such as including:

[0124] Acquire monitoring images of the trash can, and extract image information and depth information from the monitoring images;

[0125] based on the image information and the depth information, calculating image information difference and depth information difference between two adjacent frames of images in the monitoring images;

[0126] extracting multi-level features of the image information difference and the depth information difference and fusing to obtain fused features;

[0127] detecting the overflow state of the garbage can according to the fused features.

[0128] In addition, the logic instructions in the memory 430 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0129] On the other hand, the embodiments of the present application also provide a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program is executed by a processor, so that the computer can execute the steps of the garbage can overflow state detection method provided by the above-mentioned embodiments, for example, including:

[0130] obtaining monitoring images of a garbage can, and extracting image information and depth information of the monitoring images;

[0131] based on the image information and the depth information, calculating image information difference and depth information difference between two adjacent frames of images in the monitoring images;

[0132] extracting multi-level features of the image information difference and the depth information difference and fusing to obtain fused features;

[0133] detecting the overflow state of the garbage can according to the fused features.

[0134] On the other hand, the embodiments of the present application also provide a processor readable storage medium, which stores a computer program, and the computer program is used to make the processor execute the steps of the method provided by the above-mentioned embodiments, for example, including:

[0135] obtain a monitoring image of a garbage can, extract image information and depth information of the monitoring image;

[0136] based on the image information and the depth information, calculate image information difference and depth information difference between two adjacent frames of images in the monitoring image;

[0137] extract multi-level features of the image information difference and the depth information difference and fuse them to obtain fused features;

[0138] detect the overflow state of the garbage can according to the fused features.

[0139] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to a magnetic memory (such as a floppy disk, a hard disk, a magnetic tape, a magneto-optical disk (MO), etc.), an optical memory (such as a CD, a DVD, a BD, a HVD, etc.), and a semiconductor memory (such as a ROM, an EPROM, an EEPROM, a non-volatile memory (NAND FLASH), a solid state disk (SSD)), etc.

[0140] The device embodiments described above are only schematic, wherein the units illustrated as separate components can or can not be physically separated, and the components illustrated as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments. Those skilled in the art can understand and implement without creative labor.

[0141] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus necessary universal hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software products, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and include a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0142] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the same; although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of detecting an overfill condition of a trash receptacle, the method comprising: The method comprises the following steps: obtaining a monitoring image of a garbage can, extracting image information and depth information of the monitoring image; based on the image information and the depth information, calculating the image information difference and the depth information difference between two adjacent frames of images in the monitoring image; extracting multi-level features of the image information difference and the depth information difference and fusing them to obtain fused features; detecting the overflow state of the garbage can according to the fused features; the monitoring image includes RGB images and depth images, and the step of calculating the image information difference and the depth information difference between two adjacent frames of images in the monitoring image based on the image information and the depth information comprises: based on the image information, calculating the difference value between two adjacent frames of RGB images in the monitoring image to obtain the image information difference of the monitoring image; align the depth image and the RGB image through interpolation processing, determine the corresponding relationship between each pixel point in the RGB image and the depth information, and calculate the difference value between two adjacent frames of depth images in the monitoring image based on the corresponding relationship to obtain the depth information difference of the monitoring image.

2. The trash can overflow state detection method of claim 1, wherein, The step of extracting multi-level features of the image information difference and the depth information difference and fusing them to obtain fused features comprises: input the image information difference and the depth information difference into a preset backbone network, and extract multi-level features of the image information difference and the depth information difference using the backbone network, wherein the backbone network is configured with a convolution layer in which a hollow convolution and a deformable convolution are parallel, and does not contain a full connection layer and a pooling layer; input the multi-level features into a lateral connection structure of the backbone network and a preset feature pyramid network, fuse the multi-level features of the image information difference and the depth information difference to obtain fused features, wherein the lateral connection structure contains a parallel spatial attention mechanism and a channel attention mechanism.

3. The trash can overflow state detection method of claim 2, wherein, The step of inputting the multi-level features into the lateral connection structure of the backbone network and the preset feature pyramid network to fuse the multi-level features of the image information difference and the depth information difference comprises: input each level feature in the multi-level features into the channel attention mechanism in the lateral connection structure to perform channel attention operation to obtain a first attention image; input the first attention image into the spatial attention mechanism in the lateral connection structure to perform spatial attention operation to obtain a second attention image; perform convolution and up-sampling operation on the second attention image to obtain high-level features corresponding to each level feature in the multi-level features, and the high-level features contain RGB features and depth features; perform splicing processing on the RGB features and the depth features in the high-level features to fuse the multi-level features of the image information difference and the depth information difference.

4. The trash can overflow state detection method of claim 3, wherein, The step of inputting each level feature in the multi-level features into the channel attention mechanism in the lateral connection structure to perform channel attention operation to obtain a first attention image comprises: The hierarchical features in the multi-level features are input into a channel attention mechanism in the lateral connection structure, global max pooling and mean pooling operations are performed on the hierarchical features to obtain first feature images; Convolution operations are performed on the first feature images to obtain second feature images, and first attention images are obtained according to the product of the second feature images and the hierarchical features.

5. The trash can overflow state detection method of claim 3, wherein, The step of inputting the first attention images into a spatial attention mechanism in the lateral connection structure to perform spatial attention operations to obtain second attention images comprises: The first attention images are input into a spatial attention mechanism in the lateral connection structure, and mean pooling operations are performed on the first attention images to obtain third feature images; Global max pooling and convolution operations are performed on the third feature images to obtain fourth feature images, and second attention images are obtained according to the product of the fourth feature images and the first attention images.

6. The trash can overflow state detection method of claim 1, wherein, After the step of detecting the overflow state of the garbage can according to the fusion features, the method further comprises: displaying the detection result of the overflow state of the garbage can; outputting a warning prompt information when the garbage can is in the overflow state according to the detection result.

7. A garbage can overflow state detecting apparatus characterized by comprising: Comprise: An image acquisition module is configured to acquire a monitoring image of a garbage can, and extract image information and depth information of the monitoring image; A difference calculation module is configured to calculate image information difference and depth information difference between adjacent two frames of images in the monitoring image based on the image information and the depth information; A feature fusion module is configured to extract multi-level features of the image information difference and the depth information difference and fuse the multi-level features to obtain fusion features; A state detection module is configured to detect an overflow state of the garbage can according to the fusion features. The monitoring image comprises an RGB image and a depth image, and the difference calculation module is further configured to calculate a difference value between adjacent two frames of RGB images in the monitoring image based on the image information to obtain the image information difference of the monitoring image; align the depth image and the RGB image through interpolation processing, determine the corresponding relationship between each pixel point in the RGB image and the depth information, and calculate a difference value between adjacent two frames of depth images in the monitoring image based on the corresponding relationship to obtain the depth information difference of the monitoring image.

8. An electronic device comprising a processor and a memory having a computer program stored therein, characterized in that, The processor executes the computer program to realize the steps of the garbage can overflow state detection method in any one of claims 1 to 6. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the garbage can overflow state detection method in any one of claims 1 to 6.

Images