Spam detection method and system, electronic device and storage medium

Abstract

The embodiment of the application provides a garbage detection method, system, electronic equipment and storage medium, and belongs to the technical field of artificial intelligence. The method preliminarily identifies garbage in a to-be-detected image through a target detection model, obtains a plurality of garbage detection features and corresponding garbage local images, then inputs the plurality of garbage local images and the to-be-detected image into a visual encoder to obtain a multi-scale image feature of each garbage local image, obtains a visual feature according to the multi-scale image feature of the garbage local image and the garbage detection feature, further inputs the visual feature of the garbage local image and a garbage recognition text instruction set into a visual reasoning large language model to perform associated understanding and reasoning on the garbage local image according to the actual scene and environment and the garbage, and realize garbage detection. Without the need of a large number of small models for training, the garbage detection method of the application has high applicability and high garbage recognition accuracy.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a waste detection method, system, electronic device, and storage medium. Background Technology

[0002] Video surveillance-based waste detection technology is widely used in various business scenarios, such as urban order detection, illegal waste monitoring, smart trash can management, and waste disposal facility layout optimization. However, due to the diversity of urban environments, such as streets and alleyways, models often struggle to accurately understand the potential locations of waste in different settings. Furthermore, the diversity of waste types adds to the challenge, as some waste may resemble common objects, such as paper waste and advertising materials, leading to frequent false alarms.

[0003] Currently, waste detection methods employ neural network-based image processing and recognition target detection models, such as convolutional network detection. However, these methods have limitations in generalizing to different scenarios and waste types, only detecting waste in specific scenarios and of specific categories. For other scenarios or waste types, the models exhibit poor robustness and low detection accuracy. To adapt to a wide range of waste detection scenarios and waste types, a large number of waste detection samples of different types and scenarios are needed to construct a training set for training and optimizing the target detection model. This results in long training times, lengthy deployment processes, and heavy computational burdens. Summary of the Invention

[0004] The main objective of this application is to propose a waste detection method, system, electronic device, and storage medium, aiming to improve the robustness of the waste identification model and the accuracy of waste identification in different business scenarios.

[0005] To achieve the above objectives, one aspect of this application proposes a waste detection method, comprising the following steps:

[0006] Acquire the image to be detected;

[0007] The garbage in the image to be detected is identified by the target detection model, and multiple garbage detection features and corresponding local garbage images are obtained.

[0008] Multiple local images of the waste and the image to be detected are input into a visual encoder to extract multi-scale features from each local image of the waste, thereby obtaining multi-scale image features for each local image of the waste.

[0009] Visual features are obtained based on the multi-scale image features and garbage detection features of the local garbage image;

[0010] The visual features of the local garbage image and the garbage recognition text instruction set are input into the visual reasoning big language model to obtain the detection results of the local garbage image.

[0011] In some embodiments, identifying trash in the image to be detected using a target detection model to obtain multiple trash detection features and corresponding local trash images includes the following steps:

[0012] The image to be detected is input into the target detection model for garbage identification, resulting in multiple target detection boxes and garbage detection features of the image block enclosed by each target detection box;

[0013] Multiple target detection boxes are expanded according to the preset expansion pixel specifications to obtain multiple expanded detection boxes;

[0014] The image to be detected is cropped according to the expanded detection frame to obtain a local image of the garbage.

[0015] In some embodiments, the step of inputting the image to be detected into a target detection model for garbage identification to obtain multiple target detection boxes and garbage detection features of the image block enclosed by each target detection box includes the following steps:

[0016] The image to be detected is input into the target detection model for garbage identification, and multiple initial identification results are obtained. The initial identification results include the detection score of the identified candidate objects as garbage and the corresponding initial detection box.

[0017] Select the initial recognition result with a detection score greater than the score threshold from multiple initial recognition results, and use the initial detection box corresponding to the selected initial recognition result as the target detection box.

[0018] In some embodiments, the step of inputting multiple local images of waste and the image to be detected into a visual encoder to extract multi-scale features from each local image of waste and obtain the multi-scale image features of each local image of waste includes the following steps:

[0019] The local images of the garbage are mapped to multiple feature extraction layers of different dimensions to obtain local garbage features of multiple dimensions.

[0020] The local image of the garbage is mapped to a global feature extraction layer based on the image to be detected to obtain the global features of the garbage;

[0021] The feature set formed by the global garbage feature and multiple local garbage features is input into a multi-layer cross-attention interactive coding layer to obtain the image feature corresponding to each element of the feature set;

[0022] Image features corresponding to the global features of the garbage are selected as the multi-scale image features of the local garbage image.

[0023] In some embodiments, obtaining visual features based on the multi-scale image features and garbage detection features of the local garbage image includes the following steps:

[0024] The detection features are input into the detection adapter for first-dimensional adaptation processing to obtain first visual information;

[0025] The multi-scale image features are input into the image adapter for second-dimensional adaptation processing to obtain second visual information;

[0026] The first visual information and the second visual information are input into a fully connected layer for information fusion to obtain visual features.

[0027] In some embodiments, the visual reasoning large language model includes multiple sequentially connected intermediate language model layers, with a visual-language-semantic information fusion layer connecting two intermediate language model layers.

[0028] The intermediate layer of the language model is used to encode language reasoning information based on the garbage identification text instruction set to obtain semantic features;

[0029] The visual-language-semantic information fusion layer is used to fuse the semantic features and visual features output from the intermediate layer of the previous language model and perform image reasoning based on the garbage identification text instruction set to obtain language reasoning information.

[0030] In some embodiments, inputting the visual features of the localized garbage image and the garbage recognition text instruction set into a visual reasoning large language model to obtain the detection result of the localized garbage image includes the following steps:

[0031] The garbage identification text instruction set is input into the first language model intermediate layer, and the semantic features output by the first language model intermediate layer and the visual features are input into the first visual language semantic information fusion layer to obtain the corresponding language reasoning information.

[0032] The language reasoning information output by the first visual-language-semantic information fusion layer is input into the second language model intermediate layer, and the semantic features output by the second language model intermediate layer are input into the corresponding language reasoning information of the second visual-language-semantic information fusion layer. This process is repeated until the last language model intermediate layer outputs semantic features. The semantic features output by the last language model intermediate layer are used as the detection result of the garbage local image. The detection result includes the probability that garbage exists in the garbage local image.

[0033] To achieve the above objectives, another aspect of this application proposes a waste detection system, comprising:

[0034] The first module is used to acquire the image to be detected;

[0035] The second module is used to identify garbage in the image to be detected through a target detection model, and obtain multiple garbage detection features and corresponding local garbage images.

[0036] The third module is used to input multiple local images of the garbage and the image to be detected into the visual encoder to extract multi-scale features from each local image of the garbage and obtain multi-scale image features of each local image of the garbage.

[0037] The fourth module is used to obtain visual features based on the multi-scale image features and garbage detection features of the local garbage image;

[0038] The fifth module is used to input the visual features of the local garbage image and the garbage recognition text instruction set into the visual reasoning big language model to obtain the detection results of the local garbage image.

[0039] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory. When the program is executed by the processor, it implements the garbage detection method described above.

[0040] To achieve the above objectives, another aspect of the embodiments of this application proposes a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs that can be executed by one or more processors to implement the garbage detection method described above.

[0041] The proposed waste detection method, system, electronic device, and storage medium initially identify waste in the image to be detected through a target detection model, obtaining multiple waste detection features and corresponding local waste images. These local waste images and the image to be detected are then input into a visual encoder to obtain multi-scale image features for each local waste image. Visual features are obtained based on the multi-scale image features and waste detection features of the local waste images. Furthermore, the visual features of the local waste images and the waste recognition text instruction set are input into a visual reasoning language model to perform association reasoning between the local waste images and the actual scene and environment, thus achieving waste detection. Without requiring extensive training of small models, the waste detection method of this application has high applicability and high accuracy in waste recognition. Attached Figure Description

[0042] Figure 1This is a flowchart of the waste detection method provided in the embodiments of this application;

[0043] Figure 2 This is a schematic diagram illustrating the training and deployment process of the garbage detection algorithm provided in this application embodiment;

[0044] Figure 3 yes Figure 1 The flowchart of step S102 in the document;

[0045] Figure 4 This is a schematic diagram of the target detection model processing procedure provided in the embodiments of this application;

[0046] Figure 5 yes Figure 3 The flowchart of step S201 in the text;

[0047] Figure 6 yes Figure 1 The flowchart of step S103 in the process;

[0048] Figure 7 This is a schematic diagram of the multi-scale image feature extraction process provided in the embodiments of this application;

[0049] Figure 8 yes Figure 1 The flowchart of step S104 in the process;

[0050] Figure 9 This is a schematic diagram of the visual feature fusion and visual reasoning large language model processing provided in the embodiments of this application;

[0051] Figure 10 yes Figure 1 The flowchart of step S105 in the process;

[0052] Figure 11 This is a schematic diagram of a distributed system consisting of a central terminal and edge terminals provided in an embodiment of this application;

[0053] Figure 12 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0057] First, let's analyze some of the terms used in this application:

[0058] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0059] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0060] A Visual Reasoning Language Model (VLM) is an artificial intelligence model that transforms visual information (such as images and videos) into linguistic information and uses natural language processing (NLP) techniques for reasoning and decision-making. A VLM typically consists of two parts: a visual encoder and a language model. The visual encoder converts visual information into linguistic information, while the language model uses NLP techniques for reasoning and decision-making. VLM has a wide range of applications, including intelligent customer service, intelligent question answering, smart homes, and smart healthcare. VLM can use high-level semantic representations of input text to complete defined tasks. Theoretically, as long as the input image is encoded with the same semantic representation, VLM can complete other corresponding visual tasks without training. This technique uses natural language data to train the model, then uses visual input to obtain semantic representations to complete reasoning tasks, thus enabling the application of knowledge learned in one modality to another.

[0061] Object detection models are image feature extraction and classification models built upon convolutional neural networks (CNNs). CNNs are a type of feedforward neural network that incorporates convolutional computations and has a deep structure. They can undergo supervised learning using labeled training data to perform tasks such as visual image recognition and object detection. Compared to large language models for visual reasoning, object detection models are relatively small, and their object detection performance is limited by the influence of the training samples, resulting in poor generalization.

[0062] This application provides a waste detection method, system, electronic device, and storage medium, aiming to improve the robustness of waste identification models and the accuracy of waste identification in different business scenarios.

[0063] The waste detection method, system, electronic device, and storage medium provided in this application are specifically described through the following embodiments. First, the waste detection method in this application is described.

[0064] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0065] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0066] The spam detection method provided in this application relates to the field of artificial intelligence technology. The spam detection method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the spam detection method, but is not limited to the above forms.

[0067] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0068] Figure 1 This is an optional flowchart of the garbage detection method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S105.

[0069] Step S101: Obtain the image to be detected;

[0070] Step S102: Identify the garbage in the image to be detected using the target detection model to obtain multiple garbage detection features and corresponding local garbage images;

[0071] Step S103: Input multiple local images of garbage and the image to be detected into the visual encoder to extract multi-scale features from each local image of garbage, and obtain the multi-scale image features of each local image of garbage.

[0072] Step S104: Obtain visual features based on the multi-scale image features and garbage detection features of the local garbage image;

[0073] Step S105: Input the visual features of the local garbage image and the garbage recognition text instruction set into the visual reasoning big language model to obtain the detection result of the local garbage image.

[0074] Steps S101 to S105 of this application embodiment involve using a target detection model to initially identify garbage in the image to be detected, obtaining multiple garbage detection features and corresponding local garbage images. These local garbage images and the image to be detected are then input into a visual encoder to obtain multi-scale image features for each local garbage image. Visual features are obtained based on the multi-scale image features and garbage detection features of the local garbage images. Furthermore, the visual features of the local garbage images and the garbage recognition text instruction set are input into a visual reasoning large language model to perform association reasoning between the local garbage images and the actual scene and environment, thereby achieving garbage detection. Without requiring extensive training of small models, the garbage detection method of this application has high applicability and high garbage recognition accuracy.

[0075] In step S101 of some embodiments, the image to be detected can be acquired by cameras installed on streets, alleys, etc. The camera captures the monitoring image of the monitoring location, and then each frame of the monitoring image is used as the image to be detected for garbage detection, thereby enabling the detection of business scenarios such as whether there is garbage on the ground or whether the garbage can is overflowing.

[0076] In step S102 of some embodiments, the object detection model is an image recognition model built based on a convolutional neural network. For example, the object detection model can be a YOLO model, a deep learning algorithm used for image recognition in computer vision technology. The YOLO model transforms the object detection problem into a regression problem, that is, given an input image, it directly regresses the bounding box of the target and its classification category at multiple locations in the image. YOLO models include, but are not limited to, YOLOv3, YOLOv4, and YOLOv5 (different versions of YOLO). It is understood that before using the object detection model to recognize the image to be detected, a small batch of garbage image samples is needed to train the object detection model, enabling it to identify the object regions in the image to be detected that are marked as garbage by the detection boxes, and to crop the detected local regions to generate local garbage images. Each detection box corresponds to a garbage detection feature, which is a feature extracted by the object detection model from the local image during image processing. In this embodiment, a small batch of data samples is used to train the target detection model, which can reduce the training amount for the small model. The target detection model can perform preliminary identification of garbage in the image to be detected and extract the local area of ​​suspected garbage. In order to further improve the accuracy of garbage detection, a visual reasoning big language model is subsequently used to perform secondary screening of the garbage identification results of the target detection model.

[0077] In step S103 of some embodiments, the visual encoder can encode and compress digital image data based on a certain algorithm. In the visual encoder, multi-scale feature extraction technology is used on the image data to effectively extract and represent multiple scale features of the image, thereby achieving better encoding results and visual quality. Multi-scale feature extraction techniques include wavelet transform, pyramids, and multi-resolution analysis. For example, the visual encoder can use a pyramid model to extract multi-scale features of the image data. The pyramid model is a hierarchical structure where the image starts from the bottom layer and gradually extracts more abstract and global features from the top layer. Through the pyramid, the image can be decomposed into sub-images of multiple scales, each sub-image representing information at a certain scale of the image. During the multi-scale feature extraction process of the local garbage image itself, the visual encoder simultaneously integrates features from the image to be detected (i.e., the global image), so that the multi-scale image features of the local garbage image not only include local detail features but also global environmental features, which can improve the accuracy of subsequent garbage detection.

[0078] In step S104 of some embodiments, garbage detection features of the local garbage image are obtained through a target detection model, and multi-scale image features of the local garbage image are obtained through a visual encoder. The garbage detection features and multi-scale image features are then fused to obtain the visual features of the local garbage image. The fusion method of garbage detection features and multi-scale image features can be direct concatenation or feature fusion using a weighted algorithm.

[0079] In step S105 of some embodiments, the visual reasoning large language model can infer the visual features of a local garbage image based on the guidance of the garbage identification text instruction set, thereby obtaining a reasoning result on whether the local garbage image contains garbage. The visual reasoning large language model includes multiple sequentially connected intermediate language model layers, with a visual-language-semantic information fusion layer connecting two intermediate language model layers. The intermediate language model layers are used to encode language reasoning information based on the garbage identification text instruction set to obtain the semantic features of the text. The visual-language-semantic information fusion layer is used to fuse the semantic features and visual features output by the previous intermediate language model layer and perform image reasoning based on the garbage identification text instruction set to obtain language reasoning information. The visual reasoning large language model alternately encodes language reasoning information and infers visual features, finally obtaining the detection result of the local garbage image.

[0080] It should be noted that the visual reasoning large language model also needs to be trained before applying it for secondary garbage detection. Specifically, refer to... Figure 2 First, garbage is described and defined in natural language and preprocessed into an instruction set centered on garbage recognition. Then, an object detection model trained on a small batch of garbage-related data is used to identify object regions marked as garbage in the image to be detected, and the detected local regions are cropped to generate local garbage images. Next, the local garbage images are fed into a visual encoder to extract multi-scale features and further generate visual features. These visual features are fused with the feature distributions of the self-attention layers of the visual reasoning large language model, combined with actual labels to generate a large language model capable of visual reasoning. After obtaining the large language model for visual reasoning through offline training, it is deployed. Based on the predefined garbage recognition instruction set, the large language model guides the model to infer the probability of garbage presence in the local garbage images of the image to be detected, thus assisting the smaller model in secondary filtering.

[0081] In this embodiment, natural language can be used to describe and define garbage to construct a garbage identification instruction set. The specific construction process is as follows:

[0082] First, an image description generation model is used to construct a natural language description (caption) for an image containing a garbage scene, in order to describe the entire scene containing garbage;

[0083] Second, based on the business scenario and the image construction question, such as "Is there garbage in the image?";

[0084] Third, common-sense text is constructed based on the specialized terms of the business scenario to define the waste identification scenario (Lecture);

[0085] Fourth, based on the concept of thought chain, an explanatory text (Solution) is constructed to explain why there is garbage in this place and what kind of garbage is there, prompting the model to think step by step, thereby enhancing the multimodal reasoning and comprehension ability of the multimodal language model;

[0086] Fifth, construct a multimodal language dialogue instruction set for model training. For example, the instruction set could be: "Question: Question? Image description: Caption"; positive answer: "Yes, because: Lecture, therefore: Solution"; negative answer: "No, because: Lecture, therefore: Solution".

[0087] To guide the visual reasoning large language model in outputting specified multimodal understandings, during model training, a set of garbage identification instructions defined in the first step is given and input into the large language model in a "question, answer" format. Visual feature information corresponding to the question and answer is fused in the intermediate layers of each layer to train the model to fit the answers. The formula for calculating the model loss during the training process of the visual reasoning large language model is:

[0088]

[0089] Where w is the word predicted by the model, w j |w ＜j The representation uses all words before position j as input to predict the word at the current position, where answer refers to the correct answer, and J is the maximum length of the predicted word.

[0090] Please see Figure 3 In some embodiments, step S102, which involves identifying trash in the image to be detected using a target detection model to obtain multiple trash detection features and corresponding local trash images, may include, but is not limited to, the following steps:

[0091] Step S201: Input the image to be detected into the target detection model for garbage identification to obtain multiple target detection boxes and the garbage detection features of the image block surrounded by each target detection box;

[0092] Step S202: Expand multiple target detection boxes according to preset expansion pixel specifications to obtain multiple expanded detection boxes;

[0093] Step S203: The image to be detected is cropped according to the expanded detection frame to obtain a local image of the garbage.

[0094] In this embodiment, please refer to Figure 4 The image to be detected is input into the target detection model (i.e., the detection mini-model) to identify trash. The detection mini-model outputs multiple target detection boxes for trash objects. The target detection boxes include the detection box position (e.g., the horizontal and vertical coordinates of the top left corner) and the detection box size (e.g., the length and width of the detection box). After obtaining the target detection boxes, each detection box is expanded outward by a preset outward expansion pixel specification in each direction, for example, by 10 pixels in each direction, to obtain the expanded detection box (Box) for each object. t Preserve the features f of the image block enclosed by each detection box. t , forming a set n represents the number of detection boxes output by the object detection model. Then, the image to be detected is cropped based on the expanded detection boxes to obtain a local garbage image. Specifically, when expanding the detection boxes, for detection boxes less than 10 pixels in a given direction, the expansion is made to the maximum pixel size in that direction.

[0095] Please see Figure 5 In some embodiments, step S201, which involves inputting the image to be detected into a target detection model for garbage identification to obtain multiple target detection boxes and the garbage detection features of the image block enclosed by each target detection box, may include, but is not limited to, the following steps:

[0096] Step S301: Input the image to be detected into the target detection model for garbage identification and obtain multiple initial identification results. The initial identification results include the detection scores of the identified candidate objects as garbage and the corresponding initial detection boxes.

[0097] Step S302: Select an initial recognition result with a detection score greater than a score threshold from multiple initial recognition results, and use the initial detection box corresponding to the selected initial recognition result as the target detection box.

[0098] In this embodiment, when the object detection model extracts objects from the image to be detected, it may extract multiple candidate objects. Each object corresponds to an initial recognition result, which includes a detection score indicating whether the candidate object is garbage and its corresponding initial detection box. For initial recognition results with a detection score greater than a score threshold, the object in its initial detection box is considered likely to be garbage, and therefore, the initial detection box is used as the target detection box for subsequent secondary detection processing. For initial recognition results with a detection score less than or equal to the score threshold, the object in its initial detection box is considered not to be garbage, and therefore, this part of the initial detection box is filtered out to reduce the amount of computer computation. It should be noted that, to improve the robustness of the object detection model, the score threshold N can be limited to a small numerical range, for example, 0.3. <N<0.6。

[0099] Please see Figure 6 In some embodiments, step S103, which involves inputting multiple local images of waste and the image to be detected into a visual encoder to extract multi-scale features from each local image of waste, and obtaining the multi-scale image features of each local image of waste, may include, but is not limited to, the following steps:

[0100] Step S401: Map the local garbage image to multiple feature extraction layers of different dimensions to obtain local garbage features of multiple dimensions.

[0101] Step S402: Map the local garbage image to a global feature extraction layer based on the image to be detected to obtain the global garbage features;

[0102] Step S403: Input the feature set formed by the global garbage features and multiple local garbage features into the multi-layer cross-attention interactive coding layer to obtain the image features corresponding to each element in the feature set;

[0103] Step S404: Select the image features corresponding to the global garbage features as the multi-scale image features of the local garbage image.

[0104] In this embodiment, please refer to Figure 7 Multiple local images of garbage and the image to be detected are used as the input image set for the visual encoder. The input image set is represented as {I}. G ,I l1 ,…,I ln}, where I G It is the global image (i.e., the image to be detected), I lThese are local images (i.e., junk local images). In the visual encoder, each input image undergoes multi-scale transformations, for example, mapping the original image to three different dimensions: 64*64, 256*256, and 512*512. Image feature encoding is performed on images of different dimensions to obtain multiple junk local features of different dimensions. Specifically, the encoding process for each dimension involves dividing each image into 49 local regions on average. A convolutional neural network is used to map each local region into a 512-dimensional feature, and a learnable 512-dimensional feature is added as a global feature. A 50*512 feature set is constructed and fed into a 4-layer Transformer encoder for cross-attention interactive encoding. The learnable 512-dimensional feature is then output as the encoded feature of that dimension of the image (i.e., junk local features). The encoded features of the three-dimensional images are {f}. 64 ,f 256 ,f 512}

[0105] The encoded features of different dimensions are constructed into a (3+1)*512 dimensional feature set, which is then fed into a 4-layer Transformer encoder for cross-attention encoding. Here, 3 represents the three-dimensional features, and 1 represents additional learnable features used to learn the features of the global image. After encoding, the image features {f} of the garbage local image are obtained. s ,f e1 ,…,f ek Then, the (k+1)*512 image feature groups are fed into a 4-layer Transformer for encoding, and the output feature f of the added learnable feature channels is also selected. s Multi-scale image features f of local garbage images m .

[0106] Please see Figure 8 In some embodiments, step S104, which involves obtaining visual features based on multi-scale image features of the local garbage image and garbage detection features, may include, but is not limited to, the following steps:

[0107] Step S501: Input the detection features into the detection adapter for first-dimensional adaptation processing to obtain first visual information;

[0108] Step S502: Input the multi-scale image features into the image adapter for second-dimensional adaptation processing to obtain second visual information;

[0109] Step S503: Input the first visual information and the second visual information into the fully connected layer for information fusion to obtain visual features.

[0110] In this embodiment, refer to Figure 9 The detection features of the local image cropped by each expanded detection box and the multi-scale image features corresponding to this local image. Visual information is input into the large language model for visual reasoning. Before inputting detection features and multi-scale image features into the large language model, they need to be processed and fused to adapt to the input requirements of the large language model. Specifically, detection features are input into a detection adapter for first-dimensional adaptation to obtain first visual information, and multi-scale image features are input into an image adapter for second-dimensional adaptation to obtain second visual information. Both the detection adapter and the image adapter use fully connected layer networks based on residual connections to transform dimensions. Then, the first and second visual information are concatenated and fused to construct features with longer dimensions, and a unified fully connected layer is used for mapping to reduce the dimensionality, resulting in visual features. The process of obtaining visual features can be represented as follows:

[0111] f vl =M3([(M1(f m M2(f) t ))]);

[0112] M1 and M2 are the fully connected layer networks constructed for detection feature adaptation and multi-scale image feature adaptation, respectively, and M3 is the fully connected layer network used for multi-source visual information fusion.

[0113] Please see Figure 10 In some embodiments, step S105, which involves inputting the visual features of the local garbage image and the garbage recognition text instruction set into the visual reasoning large language model to obtain the detection result of the local garbage image, may include, but is not limited to, the following steps:

[0114] Step S601: Input the garbage identification text instruction set into the first language model intermediate layer, and input the semantic features and visual features output by the first language model intermediate layer into the first visual language semantic information fusion layer to obtain the corresponding language reasoning information.

[0115] Step S602: Input the language reasoning information output by the first visual-language-semantic information fusion layer into the second language model intermediate layer, and input the semantic features output by the second language model intermediate layer into the corresponding language reasoning information of the second visual-language-semantic information fusion layer, and so on, until the last language model intermediate layer outputs semantic features, and use the semantic features output by the last language model intermediate layer as the detection result of the garbage local image. The detection result includes the probability that garbage exists in the garbage local image.

[0116] In this embodiment, please continue to refer to Figure 9In a large language model for visual reasoning, a visual-language-semantic information fusion layer is added between each intermediate layer of the language model. This fusion layer connects the obtained visual features with the intermediate encoded language information to obtain f. mu =[f vl ;f text ], f text These are the language features encoded by the intermediate layers of the language model. The intermediate layers of the language model can use self-attention networks for encoding. The visual-language-semantic information fusion layer, based on the dimensional relationships of the input connectivity features, transforms the original f... text Output information extraction As the next stage of language information encoding Through 32 layers of mapping and fusion, the large language model learns multimodal information. During the model's visual reasoning process, the questions in the defined spam identification instruction set are input into the large language model, and the visual features corresponding to the questions are fused in the middle of each layer to prompt the model to give the probability of the identification answer: "yes / no". This is used to output the spam identification of the large language model and help the small model to perform secondary filtering of false positive results.

[0117] According to some specific embodiments of this application, please refer to Figure 11 The following example illustrates the application of the garbage detection method of this application to a distributed system consisting of a central terminal and edge terminals:

[0118] At the edge, small models are deployed on cameras and for garbage detection. The cameras acquire images to be detected, and the small models perform detection and recognition on the images, outputting initial detection boxes and corresponding detection scores. The small models then filter out target detection boxes based on the detection scores. The target detection boxes are expanded to obtain expanded detection boxes. The local images and detection features corresponding to the expanded detection boxes, as well as the global image, are sent to the cloud for secondary filtering using a large visual reasoning language model in the cloud.

[0119] In the cloud, visual features are obtained through multi-scale extraction and feature fusion based on the detection features of local, global, and local images transmitted from the edge. These visual features serve as visual information cues for the visual reasoning language model, inputting them into each layer of the language model. Simultaneously, a pre-defined set of spam identification instructions is input to the language model's input, prompting the model to identify the probability P of whether or not the input image contains spam. For local images where P > M (where M is the spam identification threshold of the language model), the image is determined to be spam, an alarm is returned to the edge, and the image and timestamp are saved as evidence.

[0120] In the above embodiments, the computation relies on the powerful understanding ability of the large language model. The small edge model and the large visual understanding model only need to be trained with a small batch of data to prompt the model to identify the type of garbage and determine whether garbage exists by combining the spatial relationships of the monitoring scene. The garbage detection accuracy and applicability are high.

[0121] This application also provides a waste detection system, including:

[0122] The first module is used to acquire the image to be detected;

[0123] The second module is used to identify garbage in the image to be detected through the target detection model, and obtain multiple garbage detection features and corresponding local garbage images;

[0124] The third module is used to input multiple local images of garbage and the image to be detected into the visual encoder to extract multi-scale features from each local image of garbage, thereby obtaining the multi-scale image features of each local image of garbage.

[0125] The fourth module is used to obtain visual features based on the multi-scale image features of local garbage images and garbage detection features;

[0126] The fifth module is used to input the visual features of the local garbage image and the garbage recognition text instruction set into the visual reasoning big language model to obtain the detection results of the local garbage image.

[0127] It is understood that the content of the above-described waste detection method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above-described waste detection method embodiments, and the beneficial effects achieved are also the same as those achieved in the above-described waste detection method embodiments.

[0128] This application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned garbage detection method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0129] Please see Figure 12 , Figure 12 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0130] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0131] The memory 902 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the garbage detection method of the embodiments of this application.

[0132] The input / output interface 903 is used to implement information input and output;

[0133] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0134] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0135] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0136] This application also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, which can be executed by one or more processors to implement the above-described garbage detection method.

[0137] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0138] The waste detection method, system, electronic device, and storage medium provided in this application preliminarily identify waste in the image to be detected through a target detection model, obtaining multiple waste detection features and corresponding local waste images. Then, the multiple local waste images and the image to be detected are input into a visual encoder to obtain multi-scale image features of each local waste image. Based on the multi-scale image features and waste detection features of the local waste images, visual features are obtained. Furthermore, the visual features of the local waste images and the waste recognition text instruction set are input into a visual reasoning large language model to perform association reasoning between the local waste images and waste in real-world scenarios and environments, thereby achieving waste detection. Without requiring extensive training of small models, the waste detection method of this application has high applicability and high accuracy in waste recognition.

[0139] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0140] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0141] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0142] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0143] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0144] It should be understood that in this application, "at least one" means one or more, and "more" means two or more.

[0145] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0146] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0147] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0148] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0149] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A waste detection method, characterized in that, Includes the following steps: Acquire the image to be detected; The garbage in the image to be detected is identified by the target detection model, and multiple garbage detection features and corresponding local garbage images are obtained. Multiple local images of the waste and the image to be detected are input into a visual encoder to extract multi-scale features from each local image of the waste, thereby obtaining multi-scale image features for each local image of the waste; while the visual encoder is extracting multi-scale features from the local images of the waste itself, it also fuses the features of the image to be detected. Visual features are obtained based on the multi-scale image features and garbage detection features of the local garbage image; The visual features of the local garbage image and the garbage recognition text instruction set are input into the visual reasoning big language model to obtain the detection result of the local garbage image; The visual reasoning large language model includes multiple sequentially connected intermediate language model layers, with a visual-language-semantic information fusion layer connecting two intermediate language model layers. The intermediate language model layers are used to encode language reasoning information based on the garbage identification text instruction set to obtain the semantic features of the text. The visual-language-semantic information fusion layer is used to fuse the semantic features and visual features output by the previous intermediate language model layer and perform image reasoning based on the garbage identification text instruction set to obtain language reasoning information. The visual reasoning large language model alternately encodes language reasoning information and reasons about visual features to obtain the detection results of local garbage images.

2. The waste detection method according to claim 1, characterized in that, The step of identifying trash in the image to be detected using a target detection model to obtain multiple trash detection features and corresponding local images of trash includes the following steps: The image to be detected is input into the target detection model for garbage identification, resulting in multiple target detection boxes and garbage detection features of the image block enclosed by each target detection box; Multiple target detection boxes are expanded according to the preset expansion pixel specifications to obtain multiple expanded detection boxes; The image to be detected is cropped according to the expanded detection frame to obtain a local image of the garbage.

3. The waste detection method according to claim 2, characterized in that, The process of inputting the image to be detected into a target detection model for garbage identification, and obtaining multiple target detection boxes and garbage detection features of the image block enclosed by each target detection box, includes the following steps: The image to be detected is input into the target detection model for garbage identification, and multiple initial identification results are obtained. The initial identification results include the detection score of the identified candidate objects as garbage and the corresponding initial detection box. Select the initial recognition result with a detection score greater than the score threshold from multiple initial recognition results, and use the initial detection box corresponding to the selected initial recognition result as the target detection box.

4. The waste detection method according to claim 1, characterized in that, The step of inputting multiple local images of the waste and the image to be detected into a visual encoder to extract multi-scale features from each local image of the waste, thereby obtaining the multi-scale image features of each local image of the waste, includes the following steps: The local images of the garbage are mapped to multiple feature extraction layers of different dimensions to obtain local garbage features of multiple dimensions. The local image of the garbage is mapped to a global feature extraction layer based on the image to be detected to obtain the global features of the garbage; The feature set formed by the global garbage feature and multiple local garbage features is input into a multi-layer cross-attention interactive coding layer to obtain the image feature corresponding to each element of the feature set; Image features corresponding to the global features of the garbage are selected as the multi-scale image features of the local garbage image.

5. The waste detection method according to claim 1, characterized in that, The step of obtaining visual features based on the multi-scale image features and garbage detection features of the local garbage image includes the following steps: The detection features are input into the detection adapter for first-dimensional adaptation processing to obtain first visual information; The multi-scale image features are input into the image adapter for second-dimensional adaptation processing to obtain second visual information; The first visual information and the second visual information are input into a fully connected layer for information fusion to obtain visual features.

6. The waste detection method according to claim 1, characterized in that, The process of inputting the visual features of the localized garbage image and the garbage recognition text instruction set into the visual reasoning large language model to obtain the detection result of the localized garbage image includes the following steps: The garbage identification text instruction set is input into the first language model intermediate layer, and the semantic features output by the first language model intermediate layer and the visual features are input into the first visual language semantic information fusion layer to obtain the corresponding language reasoning information. The language reasoning information output by the first visual-language-semantic information fusion layer is input into the second language model intermediate layer, and the semantic features output by the second language model intermediate layer are input into the corresponding language reasoning information of the second visual-language-semantic information fusion layer. This process is repeated until the last language model intermediate layer outputs semantic features. The semantic features output by the last language model intermediate layer are used as the detection result of the garbage local image. The detection result includes the probability that garbage exists in the garbage local image.

7. A waste detection system, characterized in that, include: The first module is used to acquire the image to be detected; The second module is used to identify garbage in the image to be detected through a target detection model, and obtain multiple garbage detection features and corresponding local garbage images. The third module is used to input multiple local images of the waste and the image to be detected into the visual encoder to perform multi-scale feature extraction on each local image of the waste, thereby obtaining multi-scale image features of each local image of the waste; the visual encoder simultaneously fuses the features of the image to be detected during the multi-scale feature extraction process of the local images of the waste itself. The fourth module is used to obtain visual features based on the multi-scale image features and garbage detection features of the local garbage image; The fifth module is used to input the visual features of the local garbage image and the garbage recognition text instruction set into the visual reasoning big language model to obtain the detection result of the local garbage image; The visual reasoning large language model includes multiple sequentially connected intermediate language model layers, with a visual-language-semantic information fusion layer connecting two intermediate language model layers. The intermediate language model layers are used to encode language reasoning information based on the garbage identification text instruction set to obtain the semantic features of the text. The visual-language-semantic information fusion layer is used to fuse the semantic features and visual features output by the previous intermediate language model layer and perform image reasoning based on the garbage identification text instruction set to obtain language reasoning information. The visual reasoning large language model alternately encodes language reasoning information and reasons about visual features to obtain the detection results of local garbage images.

8. An electronic device, characterized in that, The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for establishing communication between the processor and the memory. When the program is executed by the processor, it implements the steps of the garbage detection method as described in any one of claims 1 to 6.

9. A storage medium, said storage medium being a computer-readable storage medium for computer-readable storage, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the garbage detection method according to any one of claims 1 to 6.

Images