A method, apparatus, electronic device, and storage medium for recognizing images of backfill soil.

Abstract

This application discloses a method, apparatus, electronic device, and storage medium for recognizing images of landfill soil, relating to the field of image recognition technology. The method includes: acquiring an image of a target region to be detected; inputting the image to be detected into a feature extraction network to obtain global and local feature maps of the image, wherein the feature extraction network is an improved version based on a self-attention mechanism; fusing the global and local feature maps to obtain a fused feature map; and inputting the fused feature map into a landfill soil detection model to obtain the detection result of the image to be detected. The technical solution provided by this application can solve the bottleneck problem of scarce dataset samples and improve the recognition and detection effect.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a method, apparatus, electronic device and storage medium for recognizing images of landfill soil. Background Technology

[0002] In applications involving the early warning and detection of illegal landfill, traditional Convolutional Neural Networks (CNNs) and self-attention mechanisms are commonly used for identification and detection. CNN convolutional layers are not very effective at modeling long-term dependencies. Even with increasing convolutional pooling layers to expand the receptive field, this results in a large number of layers being stacked. Therefore, traditional CNN convolution suffers from a receptive field limitation, leading to mediocre detection performance. While self-attention mechanisms offer good performance in detecting long-term dependencies and can achieve a decent receptive field even at shallow layers, they require a large dataset to achieve good detection results. However, the scarcity of labeled illegal landfill data is a bottleneck. Therefore, designing a method for identifying landfill images that not only addresses the bottleneck of scarce datasets but also improves detection performance has become a pressing issue. Summary of the Invention

[0003] This application provides a method, apparatus, electronic device, and storage medium for recognizing images of landfill soil, which can solve the bottleneck problem of scarce dataset samples and improve the recognition and detection effect.

[0004] In a first aspect, this application provides a method for recognizing images of landfill soil, the method comprising:

[0005] Acquire the image of the target region to be detected;

[0006] The image to be detected is input into a feature extraction network to obtain the global feature map and local feature map of the image to be detected. The feature extraction network is an improvement based on the self-attention mechanism.

[0007] The global feature map and the local feature map are fused to obtain a fused feature map;

[0008] The fused feature map is input into the fill soil detection model to obtain the detection result of the image to be detected.

[0009] This application provides a method for recognizing images of landfill soil. To avoid the receptive field problem of traditional CNN convolutions and address the bottleneck of scarce landfill soil datasets, this application incorporates constraint mechanisms such as the weight vector parameters and membership matrix parameters to be learned into a self-attention mechanism to obtain a feature extraction network. This network extracts global and local feature maps, which are then fused to obtain a fused feature map. Finally, a landfill soil detection model identifies the fused feature map to obtain the detection result of the image to be detected. This application not only solves the bottleneck problem of scarce dataset samples but also improves the recognition and detection performance.

[0010] Furthermore, the self-attention mechanism includes a matrix operation module and a variant operation module. The matrix operation module includes a query matrix, a keyword matrix, and a value matrix. The feature extraction network is obtained by improving the self-attention mechanism as follows: In the matrix operation module, a first weight vector parameter and a first membership matrix parameter are set to be learned, so that the query matrix and the keyword matrix can be autonomously learned through the first weight vector parameter, and the autonomous learning ability of the query matrix and the keyword matrix is ​​suppressed through the first membership matrix parameter, thereby obtaining an improved matrix operation module. In the variant operation module, a second weight vector parameter and a second membership matrix parameter are set to be learned, so that the output result of the improved matrix operation module can be autonomously learned through the second weight vector parameter, and the autonomous learning ability of the output result of the improved matrix operation module and the value matrix is ​​suppressed through the second membership matrix parameter, thereby obtaining an improved variant operation module.

[0011] Furthermore, the step of inputting the image to be detected into the feature extraction network to obtain the global feature map of the image to be detected includes: determining the target query matrix, target keyword matrix, and target value matrix corresponding to the image to be detected based on the preset first weight matrix, second weight matrix, and third weight matrix; processing the target query matrix and the target keyword matrix through the improved matrix operation module to obtain initial global features; and processing the initial global features and the target value matrix through the improved variant operation module to obtain the global feature map.

[0012] Furthermore, the step of processing the target query matrix and the target keyword matrix using the improved matrix operation module to obtain the initial global features includes: calculating the similarity matrix between the target query matrix and the target keyword matrix; performing matrix multiplication of the target query matrix and the first weight vector parameter to obtain the weight query matrix, performing matrix multiplication of the target keyword matrix and the first weight vector parameter to obtain the weight keyword matrix; performing matrix multiplication of the weight query matrix and the first membership matrix parameter to obtain the membership query matrix, performing matrix multiplication of the weight keyword matrix and the first membership matrix parameter to obtain the membership keyword matrix; and adding the similarity matrix, the weight query matrix, the weight keyword matrix, the membership query matrix, and the membership keyword matrix to obtain the initial global features.

[0013] Furthermore, the step of processing the initial global features and the target value matrix to obtain a global feature map through the improved variant operation module includes: normalizing the initial global features using the Softmax function to obtain an initial feature tensor; performing matrix multiplication on the initial feature tensor, the second weight vector parameter, and the second membership matrix parameter to obtain a first intermediate matrix; performing matrix multiplication on the target value matrix and the second membership matrix parameter to obtain a second intermediate matrix; and adding the first intermediate matrix and the second intermediate matrix to obtain the global feature map.

[0014] Furthermore, calculating the similarity matrix between the target query matrix and the target keyword matrix includes: performing dot products on the elements of each row in the target query matrix with the elements of all rows in the target keyword matrix to obtain a row similarity matrix; performing dot products on the elements of each column in the target query matrix with the elements of all columns in the target keyword matrix to obtain a column similarity matrix; and adding the row similarity matrix and the column similarity matrix to obtain the similarity matrix.

[0015] Furthermore, the step of fusing the global feature map and the local feature map to obtain a fused feature map includes: encoding the global feature map through a first encoding network to obtain a global feature vector, the first encoding network including three attention layers, a shotcut & normal layer, and a feed forward layer; encoding the local feature map through a second encoding network to obtain a local feature vector, the second encoding network including nine attention layers, a shotcut & normal layer, and a feed forward layer; and fusing the global feature vector and the local feature vector through a concat function to obtain the fused feature map.

[0016] Secondly, this application provides a device for recognizing images of landfill soil, the device comprising:

[0017] The image acquisition module is used to acquire the image to be detected in the target area;

[0018] The feature extraction module is used to input the image to be detected into the feature extraction network to obtain the global feature map and local feature map of the image to be detected, respectively. The feature extraction network is an improvement based on the self-attention mechanism.

[0019] The feature fusion module is used to fuse the global feature map and the local feature map to obtain a fused feature map;

[0020] The image recognition module is used to input the fused feature map into the fill soil detection model to obtain the detection result of the image to be detected.

[0021] Thirdly, this application provides an electronic device comprising:

[0022] At least one processor; and

[0023] A memory communicatively connected to the at least one processor; wherein,

[0024] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the method for recognizing landfill images according to any embodiment of this application.

[0025] Fourthly, this application provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the method for recognizing landfill images as described in any embodiment of this application.

[0026] It should be noted that the aforementioned computer instructions may be stored, in whole or in part, on a computer-readable storage medium. This computer-readable storage medium may be packaged together with the processor of the soil filling image recognition device, or it may be packaged separately from the processor of the soil filling image recognition device; this application does not impose any limitations on this.

[0027] The descriptions of the second, third, and fourth aspects in this application can be referenced to the detailed description of the first aspect; and the beneficial effects described in the second, third, and fourth aspects can be referenced to the analysis of the beneficial effects of the first aspect, which will not be repeated here.

[0028] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description.

[0029] It is understood that before using the technical solutions disclosed in the various embodiments of this application, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this application in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 A schematic diagram of the first process of a method for recognizing images of landfill soil provided in an embodiment of this application;

[0032] Figure 2 A schematic diagram of the second process of a method for recognizing images of landfill provided in an embodiment of this application;

[0033] Figure 3 A schematic diagram of the structure of a soil filling image recognition device provided in an embodiment of this application;

[0034] Figure 4 This is a block diagram of an electronic device used to implement a method for recognizing images of landfill soil according to an embodiment of this application. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0036] It should be noted that the terms "first," "second," "target," and "original," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein. Furthermore, the terms "comprising," "having," and any variations thereof are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] Figure 1 This is a schematic diagram of the first process of a method for recognizing landfill images provided in this application embodiment. This embodiment is applicable to the situation of recognizing landfill images in the application scenario of monitoring and early warning detection of illegal landfill. The landfill image recognition method provided in this embodiment can be executed by the landfill image recognition device provided in this application embodiment. This device can be implemented by software and / or hardware and integrated into the electronic device executing the method. Preferably, the electronic device can be the algorithm server of an intelligent analysis center.

[0038] See Figure 1 The method in this embodiment includes, but is not limited to, the following steps:

[0039] S110. Obtain the image to be detected in the target area.

[0040] The target area refers to the area monitored by the data acquisition equipment, which can be a lidar sensor, millimeter-wave radar sensor, ultrasonic radar sensor, or vision sensor, preferably a camera. The image to be detected refers to the frame image in the video stream captured by the monitored target area, which is an image of the landfill soil and construction waste in the target area.

[0041] In this embodiment, a data acquisition device monitors the landfill and construction waste in the target area. The data acquisition device pushes the captured video stream to the algorithm server in real time. The algorithm server receives the video stream of the target area pushed by the data acquisition device and obtains the image to be detected from the video stream.

[0042] Ideally, after obtaining the image to be detected, data augmentation processing can be performed on it. For example, the Mosaic algorithm can be used to randomly scale, crop, and arrange the images multiple times to perform data augmentation; alternatively, the MixUp algorithm can be used to fill the edges of the image to enhance its data.

[0043] S120. Input the image to be detected into the feature extraction network to obtain the global feature map and local feature map of the image to be detected.

[0044] The feature extraction network in this step is an improvement based on the self-attention mechanism, which includes a matrix operation module and a variant operation module. The matrix operation module includes a query matrix, a keyword matrix, and a value matrix.

[0045] Specifically, the process of improving the self-attention mechanism to obtain the feature extraction network is as follows: In the matrix operation module, a first weight vector parameter and a first membership matrix parameter to be learned are set to enable autonomous learning of the query matrix and keyword matrix through the first weight vector parameter, and the autonomous learning ability of the query matrix and keyword matrix is ​​suppressed through the first membership matrix parameter, thus obtaining an improved matrix operation module; In the variant operation module, a second weight vector parameter and a second membership matrix parameter to be learned are set to enable autonomous learning of the output result of the improved matrix operation module through the second weight vector parameter, and the autonomous learning ability of the output result and value matrix of the improved matrix operation module is suppressed through the second membership matrix parameter, thus obtaining an improved variant operation module.

[0046] The weight vector parameters and membership matrix parameters to be learned serve as constraint mechanisms. Adding additional constraint mechanisms to the existing self-attention mechanism can enhance its recognition and detection performance. The significance of setting weight vector parameters lies in the fact that, since the query matrix and keyword matrix inherently possess different weight information, adding learnable weight vectors for encoding matrix operations allows the feature extraction network to autonomously learn the parameters, thereby improving the query and keyword matrices. Furthermore, the feature extraction network itself must also autonomously learn the weights of different query and keyword matrices. The significance of setting membership matrix parameters lies in controlling and suppressing the learning ability of the feature extraction network and preventing overfitting by adding learnable membership matrices for matrix operations.

[0047] In this embodiment, to further improve the running speed of the matrix operation module, the image to be detected is segmented according to a segmentation method before being input into the BACKBONE. However, segmentation alone limits the acquisition of information related to the relationships between segments, making it difficult to obtain global information. Therefore, a global branch and a local branch are set in the feature extraction network, with both global and local branches propagating into the BACKBONE simultaneously. The global branch is used to extract global feature maps, and its network can be designed as a simple BACKBONE; the local branch is used to extract local feature maps, and its network can be designed as a complex BACKBONE for computation.

[0048] Global feature maps are extracted through the global branch of the feature extraction network, while local feature maps are extracted through the local branch. The difference between global and local feature map extraction lies in the fact that, before extracting local feature maps, the image to be detected needs to be segmented into a predetermined number of blocks, such as 9x9 tokens. The BACKBONE network receives the embedded sequences of these 81 locally segmented tokens as input. To process local features in a 2D image, the image is converted into a series of flattened patches. Assuming the input image resolution is W and H, and the number of channels is 3, then the resolution of each block is W / 9 and H / 9. The subsequent feature extraction method within each block is the same as that for global feature map extraction.

[0049] S130. Perform feature fusion on the global feature map and the local feature map to obtain a fused feature map.

[0050] In this embodiment, a global feature vector is obtained by encoding the global feature map through a first encoding network, which includes three attention layers, a shotcut & normalization layer, and a feed forward layer; a local feature vector is obtained by encoding the local feature map through a second encoding network, which includes nine attention layers, a shotcut & normalization layer, and a feed forward layer; and a fused feature map is obtained by fusing the global feature vector and the local feature vector through a concat function.

[0051] S140. Input the fused feature map into the fill soil detection model to obtain the detection result of the image to be detected.

[0052] In this embodiment of the application, the landfill detection model can use an existing image detection model to identify the fused feature map, and determine whether there is landfill or construction waste in the image to be tested based on the output of the landfill detection model, i.e., the detection result.

[0053] Since landfill soil and construction waste are similar to hillside mounds and ordinary garbage, the landfill soil image recognition method of this application can be improved by acquiring these interfering negative samples for negative sample suppression, and by preprocessing and enhancing the negative samples to improve the model accuracy.

[0054] The technical solution provided in this embodiment involves acquiring a target region image to be detected; inputting the image to be detected into a feature extraction network to obtain global and local feature maps of the image; fusing the global and local feature maps to obtain a fused feature map; and inputting the fused feature map into a landfill detection model to obtain the detection result of the image to be detected. To avoid the receptive field problem of traditional CNN convolutions and to address the bottleneck of scarce landfill dataset samples in landfill detection, this application incorporates constraint mechanisms such as the weight vector parameters and membership matrix parameters to be learned into the self-attention mechanism to obtain a feature extraction network. The feature extraction network extracts global and local feature maps respectively, then fuses them to obtain a fused feature map, which is then used by the landfill detection model to identify the image to be detected. This application not only solves the bottleneck problem of scarce dataset samples but also improves the recognition and detection effect.

[0055] The method for recognizing landfill images provided in the embodiments of this application is further described below. Figure 2 This is a second flowchart illustrating a method for recognizing landfill images provided in an embodiment of this application. This embodiment optimizes the above-described embodiment, specifically by providing a detailed explanation of the global feature map extraction process.

[0056] See Figure 2 The method in this embodiment includes, but is not limited to, the following steps:

[0057] S210. Determine the target query matrix, target keyword matrix, and target value matrix corresponding to the image to be detected based on the preset first weight matrix, second weight matrix, and third weight matrix.

[0058] In this embodiment of the application, the first weight matrix, the second weight matrix, and the third weight matrix can be determined by random initialization at the beginning of the autonomous learning stage, and then the first weight matrix, the second weight matrix, and the third weight matrix are continuously updated according to the autonomous learning.

[0059] Let X be the pixel matrix of the image to be detected, WQ be the first weight matrix, WK be the second weight matrix, WV be the third weight matrix, Q be the target query matrix, K be the target keyword matrix, and V be the target value matrix. The calculation formula is: X*WQ=Q, X*WK=K, X*WV=V.

[0060] S220. Calculate the similarity matrix between the target query matrix and the target keyword matrix.

[0061] In this embodiment, a row similarity matrix is ​​obtained by performing dot products on the elements of each row in the target query matrix and the elements of all rows in the target keyword matrix; a column similarity matrix is ​​obtained by performing dot products on the elements of each column in the target query matrix and the elements of all columns in the target keyword matrix; and the row similarity matrix and the column similarity matrix are added together to obtain a similarity matrix.

[0062] Let the similarity matrix be denoted as QA, the row similarity matrix as RowSelfAtten(Q*K), and the column similarity matrix as ColSelfAtten(Q*K). The formula for calculating the similarity matrix is: QA=RowSelfAtten(Q*K)+ColSelfAtten(Q*K), where Q is the target query matrix and K is the target keyword matrix.

[0063] In machine vision, the application of attention mechanisms faces significant computational challenges. Since images are two-dimensional (rows and columns), the computational cost of self-attention is quadratic (second order) (as the self-attention mechanism requires each pixel in a row and column to be associated with a weight). This leads to excessive computation. To mitigate this, the computational cost is reduced by transforming the calculation of row similarity matrix * column similarity matrix into the calculation of row similarity matrix + column similarity matrix, i.e., by calculating QA. It's important to note that considering only the row or column similarity matrices, even with multiple layers, fails to fuse the global information. Therefore, QA is obtained by combining QA = RowSelfAtten(Q*K) + ColSelfAtten(Q*K), and this QA is used to better fuse global information.

[0064] S230. Multiply the target query matrix and the first weight vector parameter to obtain the weight query matrix, and multiply the target keyword matrix and the first weight vector parameter to obtain the weight keyword matrix.

[0065] In this embodiment, since the target query matrix and the target keyword matrix have different weight information, the feature extraction network learns the parameters autonomously by adding a learnable first weight vector parameter to the encoding matrix operation, thereby improving the target query matrix and the target keyword matrix. In addition, the feature extraction network itself also learns the weights of different target query matrices and target keyword matrices.

[0066] Let the first power vector parameters be denoted as PQ and PK, the power query matrix as TQ, and the power key matrix as TK. The specific calculation formulas are: Q*PQ=TQ, K*PK=TK.

[0067] S240. Multiply the power query matrix and the first membership matrix parameters to obtain the membership query matrix, and multiply the power keyword matrix and the first membership matrix parameters to obtain the membership keyword matrix.

[0068] In this embodiment, matrix operations are performed by adding a learnable membership matrix to control and suppress the learning ability of the feature extraction network and prevent overfitting.

[0069] Let the parameters of the first membership matrix be MQ and MK, the membership query matrix be EQ, and the membership keyword matrix be EK. The specific calculation formulas are: TQ*MQ=EQ, TK*MK=EK.

[0070] S250. Add the similarity matrix, power query matrix, power keyword matrix, membership query matrix, and membership keyword matrix together to obtain the initial global features.

[0071] In this embodiment, the initial global features are obtained by adding a membership matrix and a power vector encoding to the self-attention mechanism to obtain the vector representation of the corresponding position. The initial global features are denoted as STEP1RESULT, and the specific calculation formula is: STEP1RESULT=QA+TQ+TK+EQ+EK, where QA is the similarity matrix, TQ is the power query matrix, TK is the power keyword matrix, EQ is the membership query matrix, and EK is the membership keyword matrix.

[0072] S260. The initial feature tensor is obtained by normalizing the initial global features using the Softmax function.

[0073] In this embodiment of the application, the Softmax function is a normalization function, but other normalization functions may also be used in this step.

[0074] S270. Multiply the initial feature tensor, the second weight vector parameter, and the second membership matrix parameter to obtain the first intermediate matrix.

[0075] In this embodiment of the application, the initial feature tensor is denoted as SOFTRESULT, the second weight vector parameter is denoted as MVA, the second membership matrix parameter is denoted as MVB, and the first intermediate matrix is ​​denoted as STEP2RESULT1. The specific calculation formula is: STEP2RESULT1=SOFTRESULT*MVA*MVB.

[0076] S280. Multiply the parameters of the target value matrix and the second membership matrix to obtain the second intermediate matrix.

[0077] In this embodiment, the second intermediate matrix is ​​denoted as STEP2RESULT2, and the specific calculation formula is: STEP2RESULT2=V*MVB.

[0078] S290. Add the first intermediate matrix and the second intermediate matrix to obtain the global feature map.

[0079] In this embodiment of the application, the global feature map is denoted as STEP3RESULT, and the specific calculation formula is: STEP3RESULT = STEP2RESULT1 + STEP2RESULT2.

[0080] The technical solution provided in this embodiment determines the target query matrix, target keyword matrix, and target value matrix corresponding to the image to be detected based on the preset first weight matrix, second weight matrix, and third weight matrix; calculates the similarity matrix between the target query matrix and the target keyword matrix; performs matrix multiplication of the target query matrix and the first weight vector parameter to obtain the power query matrix, and performs matrix multiplication of the target keyword matrix and the first weight vector parameter to obtain the power keyword matrix; performs matrix multiplication of the power query matrix and the first membership matrix parameter to obtain the membership query matrix, and performs matrix multiplication of the power keyword matrix and the first membership matrix parameter to obtain the membership keyword matrix; adds the similarity matrix, power query matrix, power keyword matrix, membership query matrix, and membership keyword matrix to obtain the initial global features; normalizes the initial global features using the Softmax function to obtain the initial feature tensor; performs matrix multiplication of the initial feature tensor, the second weight vector parameter, and the second membership matrix parameter to obtain the first intermediate matrix; performs matrix multiplication of the target value matrix and the second membership matrix parameter to obtain the second intermediate matrix; and adds the first intermediate matrix and the second intermediate matrix to obtain the global feature map. To avoid the receptive field problem of traditional CNN convolutions and address the bottleneck of scarce samples in landfill datasets in landfill detection, this application incorporates constraints such as the weight vector parameters and membership matrix parameters to be learned into a self-attention mechanism to obtain a feature extraction network. This network extracts global and local feature maps separately, which are then fused to obtain a fused feature map. Finally, a landfill detection model uses this fused feature map to identify the detected image. This application not only solves the bottleneck problem of scarce dataset samples but also improves the recognition and detection performance.

[0081] Figure 3 This is a schematic diagram of the structure of a device for recognizing images of landfill soil provided in an embodiment of this application, as shown below. Figure 3 As shown, the device 300 may include:

[0082] Image acquisition module 310 is used to acquire the image to be detected in the target area;

[0083] The feature extraction module 320 is used to input the image to be detected into the feature extraction network to obtain the global feature map and local feature map of the image to be detected, respectively. The feature extraction network is obtained by improving the self-attention mechanism.

[0084] Feature fusion module 330 is used to fuse the global feature map and the local feature map to obtain a fused feature map;

[0085] The image recognition module 340 is used to input the fused feature map into the fill soil detection model to obtain the detection result of the image to be detected.

[0086] Optionally, the self-attention mechanism includes a matrix operation module and a variant operation module, wherein the matrix operation module includes a query matrix, a keyword matrix, and a value matrix;

[0087] Furthermore, the aforementioned image recognition device for landfill soil may also include: an improved feature extraction network module;

[0088] The improved feature extraction network module is used to set a first weight vector parameter and a first membership matrix parameter to be learned in the matrix operation module, so that the query matrix and the keyword matrix can be autonomously learned through the first weight vector parameter, and the autonomous learning ability of the query matrix and the keyword matrix can be suppressed through the first membership matrix parameter, thereby obtaining an improved matrix operation module; in the variant operation module, a second weight vector parameter and a second membership matrix parameter to be learned are set, so that the output result of the improved matrix operation module can be autonomously learned through the second weight vector parameter, and the autonomous learning ability of the output result of the improved matrix operation module and the value matrix can be suppressed through the second membership matrix parameter, thereby obtaining an improved variant operation module.

[0089] Furthermore, the aforementioned feature extraction module 320 can be specifically used to: determine the target query matrix, target keyword matrix, and target value matrix corresponding to the image to be detected based on the preset first weight matrix, second weight matrix, and third weight matrix; process the target query matrix and the target keyword matrix through the improved matrix operation module to obtain initial global features; and process the initial global features and the target value matrix through the improved variant operation module to obtain a global feature map.

[0090] Furthermore, the feature extraction module 320 described above may include a first extraction unit and a second extraction unit;

[0091] The first extraction unit is used to calculate the similarity matrix between the target query matrix and the target keyword matrix; multiply the target query matrix and the first weighted vector parameter to obtain the weighted query matrix, multiply the target keyword matrix and the first weighted vector parameter to obtain the weighted keyword matrix; multiply the weighted query matrix and the first membership matrix parameter to obtain the membership query matrix, multiply the weighted keyword matrix and the first membership matrix parameter to obtain the membership keyword matrix; and add the similarity matrix, the weighted query matrix, the weighted keyword matrix, the membership query matrix, and the membership keyword matrix to obtain the initial global feature.

[0092] The second extraction unit is used to normalize the initial global features using the Softmax function to obtain an initial feature tensor; multiply the initial feature tensor, the second weight vector parameter, and the second membership matrix parameter to obtain a first intermediate matrix; multiply the target value matrix and the second membership matrix parameter to obtain a second intermediate matrix; and add the first intermediate matrix and the second intermediate matrix to obtain the global feature map.

[0093] Furthermore, the first extraction unit described above can be specifically used to: obtain a row similarity matrix by performing a dot product between the elements of each row in the target query matrix and the elements of all rows in the target keyword matrix; obtain a column similarity matrix by performing a dot product between the elements of each column in the target query matrix and the elements of all columns in the target keyword matrix; and add the row similarity matrix and the column similarity matrix to obtain the similarity matrix.

[0094] Furthermore, the aforementioned feature fusion module 330 can be specifically used to: encode the global feature map through a first encoding network to obtain a global feature vector, the first encoding network including three attention layers, a shotcut & normal layer, and a feed forward layer; encode the local feature map through a second encoding network to obtain a local feature vector, the second encoding network including nine attention layers, a shotcut & normal layer, and a feed forward layer; and fuse the global feature vector and the local feature vector through a concat function to obtain the fused feature map.

[0095] The soil filling image recognition device provided in this embodiment can be applied to the soil filling image recognition method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0096] Figure 4This is a block diagram of an electronic device used to implement a method for recognizing images of landfill soil according to embodiments of this application. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present application described and / or claimed herein.

[0097] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0098] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0099] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as methods for recognizing landfill images.

[0100] In some embodiments, the method for recognizing landfill images can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the landfill image recognition method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the landfill image recognition method by any other suitable means (e.g., by means of firmware).

[0101] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0102] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0103] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0104] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0105] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0106] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0107] Note that the above are merely preferred embodiments and technical principles applied in this application. Those skilled in the art will understand that this application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this application. For example, those skilled in the art can use the various forms of processes shown above to reorder, add, or delete steps; the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solution of this application can be achieved, and no limitations are imposed herein.

[0108] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for recognizing images of landfill soil, characterized in that, The method includes: Acquire the image of the target region to be detected; The image to be detected is input into a feature extraction network to obtain the global feature map and local feature map of the image to be detected. The feature extraction network is an improvement based on the self-attention mechanism. The global feature map and the local feature map are fused to obtain a fused feature map; The fused feature map is input into the fill soil detection model to obtain the detection result of the image to be detected; The self-attention mechanism includes a matrix operation module and a variant operation module. The matrix operation module includes a query matrix, a keyword matrix, and a value matrix. The feature extraction network is obtained by improving the self-attention mechanism in the following way: In the matrix operation module, a first weight vector parameter to be learned and a first membership matrix parameter to be learned are set so that the query matrix and the keyword matrix can be autonomously learned through the first weight vector parameter, and the autonomous learning ability of the query matrix and the keyword matrix can be suppressed through the first membership matrix parameter, thereby obtaining an improved matrix operation module; In the variant operation module, a second weight vector parameter and a second membership matrix parameter to be learned are set so that the output result of the improved matrix operation module can be autonomously learned through the second weight vector parameter, and the autonomous learning ability of the output result of the improved matrix operation module and the value matrix can be suppressed through the second membership matrix parameter, thereby obtaining the improved variant operation module.

2. The method for recognizing images of landfill soil according to claim 1, characterized in that, The step of inputting the image to be detected into a feature extraction network to obtain the global feature map of the image to be detected includes: The target query matrix, target keyword matrix, and target value matrix corresponding to the image to be detected are determined based on the preset first weight matrix, second weight matrix, and third weight matrix. The improved matrix operation module performs data processing on the target query matrix and the target keyword matrix to obtain initial global features; The improved variant operation module processes the initial global features and the target value matrix to obtain a global feature map.

3. The method for recognizing images of landfill soil according to claim 2, characterized in that, The process of obtaining initial global features by processing the target query matrix and the target keyword matrix using the improved matrix operation module includes: Calculate the similarity matrix between the target query matrix and the target keyword matrix; The power query matrix is ​​obtained by performing matrix multiplication on the target query matrix and the first power vector parameter, and the power keyword matrix is ​​obtained by performing matrix multiplication on the target keyword matrix and the first power vector parameter. The membership query matrix is ​​obtained by performing matrix multiplication between the power query matrix and the first membership matrix parameters, and the membership keyword matrix is ​​obtained by performing matrix multiplication between the power keyword matrix and the first membership matrix parameters. The initial global feature is obtained by adding the similarity matrix, the power query matrix, the power keyword matrix, the membership query matrix, and the membership keyword matrix.

4. The method for recognizing images of landfill soil according to claim 2, characterized in that, The step of obtaining a global feature map by processing the initial global features and the target value matrix through the improved variant operation module includes: The initial feature tensor is obtained by normalizing the initial global features using the Softmax function; The first intermediate matrix is ​​obtained by performing matrix multiplication on the initial feature tensor, the second weight vector parameter, and the second membership matrix parameter. The second intermediate matrix is ​​obtained by performing matrix multiplication on the target value matrix and the parameters of the second membership matrix; The global feature map is obtained by adding the first intermediate matrix and the second intermediate matrix.

5. The method for recognizing images of landfill soil according to claim 3, characterized in that, The calculation of the similarity matrix between the target query matrix and the target keyword matrix includes: The row similarity matrix is ​​obtained by performing dot products on the elements of each row in the target query matrix and the elements of all rows in the target keyword matrix. The column similarity matrix is ​​obtained by performing dot products on the elements of each column in the target query matrix and the elements of all columns in the target keyword matrix. The similarity matrix is ​​obtained by adding the row similarity matrix and the column similarity matrix.

6. The method for recognizing images of landfill soil according to claim 1, characterized in that, The step of fusing the global feature map and the local feature map to obtain a fused feature map includes: The global feature map is encoded by a first encoding network to obtain a global feature vector. The first encoding network includes three attention layers, a shotcut & normal layer, and a feed forward layer. The local feature map is encoded by a second encoding network to obtain a local feature vector. The second encoding network includes 9 attention layers, shotcut & normal layers and feed forward layers. The global feature vector and the local feature vector are fused using the concat function to obtain the fused feature map.

7. A device for recognizing images of landfill soil, characterized in that, The device includes: The image acquisition module is used to acquire the image to be detected in the target area; The feature extraction module is used to input the image to be detected into the feature extraction network to obtain the global feature map and local feature map of the image to be detected, respectively. The feature extraction network is an improvement based on the self-attention mechanism. The feature fusion module is used to fuse the global feature map and the local feature map to obtain a fused feature map; An image recognition module is used to input the fused feature map into the fill soil detection model to obtain the detection result of the image to be detected; The self-attention mechanism includes a matrix operation module and a variant operation module, wherein the matrix operation module includes a query matrix, a keyword matrix, and a value matrix; The device for recognizing the image of the fill soil further includes: an improved feature extraction network module; The improved feature extraction network module is used to set a first weight vector parameter and a first membership matrix parameter to be learned in the matrix operation module, so that the query matrix and the keyword matrix can be autonomously learned through the first weight vector parameter, and the autonomous learning ability of the query matrix and the keyword matrix can be suppressed through the first membership matrix parameter, thereby obtaining an improved matrix operation module; in the variant operation module, a second weight vector parameter and a second membership matrix parameter to be learned are set, so that the output result of the improved matrix operation module can be autonomously learned through the second weight vector parameter, and the autonomous learning ability of the output result of the improved matrix operation module and the value matrix can be suppressed through the second membership matrix parameter, thereby obtaining an improved variant operation module.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the method for recognizing the image of the fill soil as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method for recognizing images of fill soil as described in any one of claims 1 to 6.

Images