A method, device, computer equipment and storage medium for identifying garbage types

Through the X-ray-based garbage target recognition method, the feature extraction and segmentation network of X-ray images is used to solve the problem that traditional methods cannot identify blocked garbage, and high-accuracy garbage recognition is achieved, especially when the garbage bag is not opened. In the case of

CN114742989BActive Publication Date: 2024-11-08THE CHINESE UNIV OF HONG KONG (SHENZHEN)
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202210316855.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-28
Publication Date
2024-11-08
Estimated Expiration
2042-03-28

AI Technical Summary

Technical Problem

Traditional garbage type identification methods cannot effectively identify blocked garbage items, resulting in low recognition accuracy.

Method used

Using the X-ray-based garbage target recognition method, the combination of original X-ray images, feature extraction, target recognition and segmentation networks can realize the identification of occlusion garbage. The decomposition strategy from easy to difficult is adopted, and the masking of the simple identification box is used. The identification box is difficult to segment code information.

Benefits of technology

Efficiently identify garbage without disassembling the garbage bag, improve the recognition accuracy of X-ray images, and solve the problems of missed and missed detection caused by overlapping occlusion between targets.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114742989B_ABST
    Figure CN114742989B_ABST
Patent Text Reader

Abstract

The embodiments of the present application belong to the field of computer vision processing technology, and relate to a garbage target recognition method, device, computer equipment and storage medium based on X-ray. Based on a decomposition strategy from easy to difficult, the present application can efficiently extract edge information in X-ray images, and effectively solve the problems of missed detection and false detection caused by overlapping occlusion between various targets in X-ray garbage images, and can realize the recognition of various types of garbage in the bag without opening the garbage bag, and improve the recognition accuracy of X-ray images.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer vision processing technology, and in particular to an X-ray-based garbage target recognition method, device, computer equipment and storage medium. Background Art

[0002] In the context of complex global economic situation / economic globalization, how to inspect packaging waste is a very important part of the waste disposal process. By identifying the type and location of waste in the garbage bag, more effective information can be provided for the subsequent waste disposal process.

[0003] In the existing garbage type identification method, the existing general target recognition technology is usually used to perform type recognition operations on different garbage, so as to achieve the purpose of garbage classification.

[0004] However, the applicant found that traditional garbage type identification methods are generally not intelligent. When classifying the garbage to be classified using traditional garbage type identification methods, it is necessary to disassemble the garbage bags to be identified with complete packaging and display the items in the garbage bags flatly so that the RGB camera can identify them. It can be seen that traditional garbage type identification methods cannot effectively identify obscured garbage items. Summary of the invention

[0005] The purpose of the embodiments of the present application is to propose an X-ray-based garbage target identification method, device, computer equipment and storage medium to solve the problem that traditional garbage type identification methods cannot effectively identify obscured garbage items.

[0006] In order to solve the above technical problems, the embodiment of the present application provides a garbage target identification method based on X-ray, which adopts the following technical solution:

[0007] Obtaining original X-ray images of the garbage to be identified;

[0008] Performing a feature extraction operation on the original X-ray image to obtain original feature information;

[0009] Performing a target recognition operation on the garbage to be recognized in the original X-ray image according to the original feature information to obtain a target recognition frame, wherein the target recognition frame includes a simple recognition frame and a difficult recognition frame;

[0010] Performing a segmentation operation on the to-be-identified garbage in the difficult recognition frame according to the segmentation network and the mask information of the simple recognition frame to obtain a segmentation recognition frame;

[0011] The simple recognition frame and the segmentation recognition frame are the target recognition results of the garbage to be recognized.

[0012] In order to solve the above technical problems, the embodiment of the present application also provides a garbage target identification device based on X-ray, which adopts the following technical solution:

[0013] An image acquisition module, used to acquire the original X-ray image of the garbage to be identified;

[0014] A feature extraction module, used to perform a feature extraction operation on the original X-ray image to obtain original feature information;

[0015] A target recognition module, used for performing a target recognition operation on the garbage to be recognized in the original X-ray image according to the original feature information to obtain a target recognition frame, wherein the target recognition frame includes a simple recognition frame and a difficult recognition frame;

[0016] A frame segmentation module, used for performing a segmentation operation on the to-be-identified garbage of the difficult recognition frame according to the segmentation network and the mask information of the simple recognition frame to obtain a segmentation recognition frame;

[0017] A result confirmation module is used for the target recognition result that the simple recognition frame and the segmented recognition frame are the garbage to be recognized.

[0018] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0019] It includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned X-ray-based garbage target identification method when executing the computer-readable instructions.

[0020] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0021] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the X-ray-based garbage target identification method as described above are implemented.

[0022] The present application provides a garbage target recognition method based on X-ray, comprising: obtaining an original X-ray image of the garbage to be recognized; performing a feature extraction operation on the original X-ray image to obtain original feature information; performing a target recognition operation on the garbage to be recognized in the original X-ray image according to the original feature information to obtain a target recognition frame, wherein the target recognition frame includes a simple recognition frame and a difficult recognition frame; performing a segmentation operation on the garbage to be recognized in the difficult recognition frame according to the segmentation network and the mask information of the simple recognition frame to obtain a segmentation recognition frame; the simple recognition frame and the segmentation recognition frame are the target recognition results of the garbage to be recognized. Compared with the prior art, the present application is based on a decomposition strategy from easy to difficult, can efficiently extract edge information in X-ray images, and effectively solve the problems of missed detection and false detection caused by overlapping occlusion between various targets in X-ray garbage images, can realize the recognition of various types of garbage in the bag without opening the garbage bag, and improve the recognition accuracy of X-ray images. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0025] Figure 2 This is a flowchart of the implementation of the X-ray based garbage target identification method provided in the first embodiment of the present application;

[0026] Figure 3 yes Figure 2 A flowchart of a specific implementation of step S202;

[0027] Figure 4 yes Figure 2 A flowchart of a specific implementation of step S203;

[0028] Figure 5 This is a flowchart of a specific implementation method of distinguishing identification frames provided in Example 1 of the present application;

[0029] Figure 6 is a structural schematic diagram of a garbage target identification device based on X-rays provided in Example 1 of the present application;

[0030] Figure 7 yes Figure 6 A structural diagram of a specific implementation of the feature extraction module 220;

[0031] Figure 8 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0033] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0034] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0035] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, 103, network 104 and server 105. Network 104 is used to provide a medium for communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0036] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0037] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0038] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0039] It should be noted that the X-ray based garbage target identification method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the X-ray based garbage target identification device is generally set in the server / terminal device.

[0040] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0041] Embodiment 1

[0042] Continue to refer Figure 2 , shows an implementation flow chart of the X-ray-based garbage target identification method provided in Example 1 of the present application. For the sake of convenience of explanation, only the parts related to the present application are shown.

[0043] The above-mentioned X-ray-based garbage target identification method comprises the following steps:

[0044] Step S201: Obtaining the original X-ray image of the garbage to be identified.

[0045] In the embodiment of the present application, the model used in the algorithm needs to be trained with high-quality X-ray garbage images. Given that there is currently no high-quality X-ray garbage image dataset, the embodiment of the present disclosure contributes the first garbage detection X-ray image dataset in the field of computer vision, and applies it to model training and new garbage detection. The dataset contributed by the embodiment of the present disclosure contains a total of 5038 images. Based on the size of general packaging garbage images in X-ray machines, the resolution of a single image in the dataset is uniformly set to 450 × 450. According to the commonly used standards for garbage classification in my country, the dataset of the embodiment of the present disclosure divides the annotation categories into four categories: recyclable, hazardous, kitchen waste and other garbage; at the same time, the annotations of the dataset contributed by this example are further subdivided into twelve categories, including plastic bottles, cans, cartons, glass bottles, long sticks, spoons, hot packs, desiccants, lunch boxes, batteries, light bulbs and food residues, according to the frequency of occurrence of garbage categories and the value of recycling and processing; the dataset contains a total of 30,845 valid annotations, and each annotation includes two categories of annotations: target edge box and mask, which is convenient for the later use of the method proposed in the embodiment of the present disclosure to perform deep learning model training such as target recognition and instance segmentation.

[0046] In the embodiment of the present application, in terms of inputting the X-ray image of the packaged garbage, the complete packaged garbage is mainly passed through the X-ray machine, and the X-ray image of the garbage taken by the X-ray machine is obtained through an acquisition card connected to a high-performance computer, and the obtained input image is used for processing in subsequent steps. The number and image size of the input images can be determined according to the needs of the actual application scenario, and the embodiment of the present disclosure does not specifically limit its specific value.

[0047] In the embodiment of the present application, the X-ray garbage image dataset refers to a set of X-ray images collected by the inventors of the present disclosure and derived from household packaging waste.

[0048] In the embodiment of the present application, the X-ray image is an X-ray image of packaging waste taken by an X-ray machine and collected by an acquisition card connected to an external high-performance computer. The format of the collected image is determined, and there is no specific requirement for the size of the image.

[0049] Step S202: Perform feature extraction operation on the original X-ray image to obtain original feature information.

[0050] In the embodiment of the present application, the image features of the X-ray image of the packaging waste collected in step S201 are extracted to obtain the image features and contour features of the X-ray image.

[0051] Step S203: performing a target recognition operation on the garbage to be recognized in the original X-ray image according to the original feature information to obtain a target recognition frame, wherein the target recognition frame includes a simple recognition frame and a difficult recognition frame.

[0052] In an embodiment of the present application, the target recognition operation may be to perform a fusion operation on the image feature information and the thermal feature information according to the feature pyramid network to obtain feature fusion information; and then identify the target object in the feature fusion information according to the full convolution single-stage target detection model to obtain a target recognition frame.

[0053] Step S204: performing a segmentation operation on the difficult recognition frame according to the segmentation network and the mask information of the simple recognition frame to obtain a segmentation recognition frame.

[0054] In an embodiment of the present application, in X-ray image instance segmentation, a "difficult" target is usually an object that overlaps with multiple other targets in a target cluttered area. The disclosed embodiment is inspired by the instance segmentation method performed by humans, that is, first identifying "easy" (i.e., easy to distinguish) instances before segmenting "difficult" instances. The disclosed embodiment proposes a new easy-to-difficult decomposition strategy to simulate this process. The method first splits the obtained target bounding box set into a simple set and a difficult set, and then uses the mask extracted from the simple instance to help predict the target instance mask that is highly occluded in the difficult set.

[0055] In the embodiment of the present application, after obtaining the target recognition frame, a set of bounding boxes B can be obtained = {B 1 ,B 2 ,…,B N}∈R N×5 , where N is the number of bounding boxes after non-maximum suppression, and B i =[c x ,c y ,w,h,s] is the target box detected from the full convolution single-stage target detection head, where (c x ,c y ) represents the center coordinate of the detection box, w and h represent the width and height of the detection box respectively, and s is the prediction confidence given by the target detection head.

[0056] In the embodiment of the present application, based on the prediction confidence score of each target recognition frame, the obtained target detection frame set B is divided into N e The high confidence set B of boxes easy With N h The low confidence detection box set B of boxes hard After obtaining the simple and difficult detection box sets respectively, the disclosed embodiment plans to use the simple set to generate useful information to guide the generation of the difficult set target mask.

[0057] In the segmentation operation of the embodiment of the present application, performing instance segmentation on the target within the target detection bounding box with high confidence usually obtains better results. Therefore, the embodiment of the present disclosure first predicts the high confidence set Beasy The object instance masks in all boxes Use a segmentation head G s , where Z e is the updated feature of the graph convolutional network, is the i-th th The features of the region of interest of the box, A is the self-attention mechanism diagram:

[0058]

[0059]

[0060] A=softmax(F(X,X)),

[0061]

[0062] W is the weight of the graph network, A is the adjacency matrix of the graph, X e i is the feature of the easy part of the input to the attention graph network. θ(x) and φ(x) represent the output of the two fully connected layers (representing query and key in the attention mechanism) M e i is the mask of the simple set, Z e i is the feature of the simple set input graph after the attention network. σ is the activation function, and softmax is the softmax activation function.

[0063] In the embodiment of the present application, the estimated mask information obtained from the simple set is used to enhance the self-attention mechanism map to predict the mask of the difficult set. Specifically, the embodiment of the present disclosure first converts the mask M of the simple region obtained according to the coordinates of the region of interest e Convert back to the input image space and merge them into a single mask P through the element-based maximum operation in the sparse matrix. The mask P here is a soft mask, and each value contained represents the probability that the corresponding pixel position contains the target instance. Since the mask provides effective and obvious information for the location of simple instances, the embodiment of the present disclosure proposes a segmentation method based on a mask-guided attention mechanism to guide difficult instances.

[0064] In the embodiment of the present application, for the difficult set J th The proposed segmentation operation from easy to difficult performs region of interest alignment on mask P to obtain the region of interest features corresponding to the difficult set. The pixel-aligned mask P on which segmentation is performed j The mask-guided attention mechanism used here is A gIt is defined as follows, where ⊙ represents the Hadamard product, A s Then it is a similar probability matrix, whose elements represent the probability that two nodes contain objects:

[0065]

[0066] A s =1+P j (P j ) T ,

[0067] Where ⊙ represents the Hadamard product, A s is a similar probability matrix whose elements represent the probability that two nodes contain objects

[0068] Finally, the embodiment of the present disclosure can calculate the difficulty set by the following formula: The mask:

[0069]

[0070]

[0071] Among them, M h j Denotes the mask detection result of the difficult set, Z h j Features obtained after inputting graph attention for difficult features.

[0072] Step S205: The simple recognition frame and the segmented recognition frame are the target recognition results of the garbage to be recognized.

[0073] In an embodiment of the present application, guided by the mask attention mechanism, the difficult set can efficiently utilize the similarity matrix constructed by the mask results of the simple set to enhance the attention map, thereby improving the prediction accuracy of the overall mask of the difficult set target in instance segmentation.

[0074] In an embodiment of the present application, a garbage target recognition method based on X-ray is provided, including: obtaining an original X-ray image of the garbage to be recognized; performing a feature extraction operation on the original X-ray image to obtain original feature information; performing a target recognition operation on the garbage to be recognized in the original X-ray image according to the original feature information to obtain a target recognition frame, wherein the target recognition frame includes a simple recognition frame and a difficult recognition frame; performing a segmentation operation on the garbage to be recognized in the difficult recognition frame according to the segmentation network and the mask information of the simple recognition frame to obtain a segmentation recognition frame; the simple recognition frame and the segmentation recognition frame are the target recognition results of the garbage to be recognized. Compared with the prior art, the present application is based on a decomposition strategy from easy to difficult, can efficiently extract edge information in X-ray images, and effectively solve the problems of missed detection and false detection caused by overlapping occlusion between various targets in X-ray garbage images, can realize the recognition of various types of garbage in the bag without opening the garbage bag, and improve the recognition accuracy of X-ray images.

[0075] Continue reading Figure 3 , showing Figure 2 A flowchart of a specific implementation of step S202 is shown in FIG. 1 . For ease of explanation, only the part related to the present application is shown.

[0076] In some optional implementations of this embodiment, the original feature information includes image feature information and thermal feature information, and the above step S202 specifically includes:

[0077] Step S301: performing an image feature extraction operation on the original X-ray image according to the residual neural network to obtain image feature information.

[0078] In an embodiment of the present application, in the feature extraction stage, a residual neural network is selected as the backbone network to perform image feature extraction on the X-ray garbage image to obtain multiple image feature maps of different scales; the algorithm of the disclosed embodiment uses a residual 101 neural network (ResNet101) in the implementation to extract feature information of different scales of the original X-ray image, and inputs the information of different scales into the feature pyramid network for subsequent processing steps.

[0079] Step S302: performing a thermal feature extraction operation on the original X-ray image according to the global structure guidance model to obtain thermal feature information.

[0080] In an embodiment of the present application, after acquiring image features using a residual neural network, a global structure guidance model is proposed in the disclosed embodiment for features such as penetration and occlusion in X-ray images. The model design is universal and can be easily inserted into existing top-down deep learning methods to extract global contour information.

[0081] Specifically, the global structure guides the model to first upsample the low-resolution feature map to a high-resolution feature map, and then The embodiment of the present disclosure is also inspired by the human body posture estimation method. The embodiment of the present disclosure marks the correct mask J i It is represented as a Gaussian distribution heat map centered on the contour point with variance σ. Since the global contour map contains overlapping information between different instances, the embodiment of the present disclosure proposes that the global structure guidance model is integrated into the network to extract the semantic features of the whole image to better segment the overlapping objects.

[0082] Continue reading Figure 4 , showing Figure 2 A flowchart of a specific implementation of step S203 is shown in FIG. 1 . For ease of explanation, only the part related to the present application is shown.

[0083] In some optional implementations of this embodiment, the above step S203 specifically includes:

[0084] Step S401: performing a fusion operation on the image feature information and the thermal feature information according to the feature pyramid network to obtain feature fusion information.

[0085] Step S402: Identify the target object in the feature fusion information according to the full convolution single-stage target detection model to obtain a target recognition frame.

[0086] In an embodiment of the present application, the simple recognition frame and the difficult recognition frame obtained in step S203 are input into the feature pyramid network, and the multi-layer multi-scale images are fused through the feature pyramid network, and the fused results are input into the full convolution single-stage target detection head. Full convolution single-stage target detection is a single-stage, fully convolutional target detection algorithm that predicts objects in a per-pixel manner. Based on this anchor-free frame approach, the algorithm can make more use of foreground sample training while avoiding the influence of anchor-related parameters, and can obtain a more accurate target recognition frame, which is convenient for the next step of instance segmentation image processing.

[0087] Continue reading Figure 5 , shows a flowchart of a specific implementation method of distinguishing identification frames provided in Example 1 of the present application. For the sake of ease of explanation, only the parts related to the present application are shown.

[0088] In some optional implementations of this embodiment, the target recognition frame also includes prediction confidence, and after the above step S402, specifically including:

[0089] Step S501: taking the target recognition box whose prediction confidence is greater than or equal to the confidence threshold as a simple recognition box.

[0090] Step S502: The target recognition frame whose confidence is less than the confidence threshold is regarded as a difficult recognition frame.

[0091] In the embodiment of the present application, after obtaining the target recognition frame, a set of bounding boxes B can be obtained = {B 1 , B 2 , …, B N}∈R N×5 , where N is the number of bounding boxes after non-maximum suppression, and B i =[c x , c v , w, h, s] is the target box detected from the full convolution single-stage target detection head, where (c x , c y ) represents the center coordinate of the detection box, w and h represent the width and height of the detection box respectively, and s is the prediction confidence given by the target detection head.

[0092] In the embodiment of the present application, based on the prediction confidence score of each target recognition frame, the obtained target detection frame set B is divided into N e The high confidence set B of boxes easy With N h The low confidence detection box set B of boxes hard After obtaining the simple and difficult detection box sets respectively, the disclosed embodiment plans to use the simple set to generate useful information to guide the generation of the difficult set target mask.

[0093] In some optional implementations of this embodiment, before step S302, the following is further included:

[0094] Calculate the detection loss function of the fully convolutional single-stage target detection model, and perform tuning operations on the fully convolutional single-stage target detection model according to the detection loss function, where the detection loss function is expressed as:

[0095] L detect =L regression +L center +L classes

[0096] Among them, L regression represents the regression loss; L center represents the detector center point regression loss; L classes represents the detector classification loss.

[0097] In the embodiment of the present application, target detection is essentially performed based on an anchor-free method, which is more efficient, wherein full convolution single-stage target detection is selected as the target detector.

[0098] In some optional implementations of this embodiment, before step S104, the following steps are further included:

[0099] Calculate the segmentation loss function of the segmentation network and perform tuning operations on the segmentation network according to the segmentation loss function, where the segmentation loss function is expressed as:

[0100] L mask =λ 1 L e +λ 2 L h +λ 3 L heatmap

[0101] L e =L Occulder (M e )+L Occuldee (M e )

[0102] L h =L Occulder (M h )+L Occuldee (M h )

[0103]

[0104] Among them, L Occulder represents the mask loss function of the occluder; L Occuldee represents the occluded mask loss function; L heatmap represents the global edge loss function; J represents the edge heat map; L MSE represents the mean square error.

[0105] In the embodiment of the present application, among the variables of the above loss function, L e With L h Respectively represent the segmentation loss of the simple detection box and the difficult detection box set, L h The prediction of the global contour heat map is supervised, and L Occulder With L Occuldee Here, they represent the loss values ​​of the target detection box and mask in each region of interest. 1 ,λ 2 With λ 3 are the hyperparameters of the balanced loss function. According to the experience of the training set, the embodiments of the present disclosure adjust them to {0.5, 1.0, 0.5} respectively.

[0106] In an embodiment of the present application, the complete instance segmentation framework can be trained in an end-to-end manner by a multi-task loss function L defined as:

[0107] L=λLdetect +L mask ·

[0108] Generally, as the loss weight, λ=1.

[0109] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the computer-readable instructions are executed, they can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0110] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0111] Embodiment 2

[0112] Further references Figure 6 , as a response to the above Figure 2 The present application provides an embodiment of a garbage target identification device based on X-ray, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0113] like Figure 6 As shown, the X-ray based garbage target identification device 200 of this embodiment includes: an image acquisition module 210, a feature extraction module 220, a target identification module 230, a frame segmentation module 240 and a result confirmation module 250. Among them:

[0114] An image acquisition module 210 is used to acquire an original X-ray image of the garbage to be identified;

[0115] The feature extraction module 220 is used to perform feature extraction operations on the original X-ray image to obtain original feature information;

[0116] The target recognition module 230 is used to perform a target recognition operation on the garbage to be recognized in the original X-ray image according to the original feature information to obtain a target recognition frame, wherein the target recognition frame includes a simple recognition frame and a difficult recognition frame;

[0117] A frame segmentation module 240 is used to segment the garbage to be identified in the difficult identification frame according to the segmentation network and the mask information of the simple identification frame to obtain a segmented identification frame;

[0118] The result confirmation module 250 is used for simply identifying the frame and segmenting the frame as the target identification result of the garbage to be identified.

[0119] In the embodiment of the present application, the model used in the algorithm needs to be trained with high-quality X-ray garbage images. Given that there is currently no high-quality X-ray garbage image dataset, the embodiment of the present disclosure contributes the first garbage detection X-ray image dataset in the field of computer vision, and applies it to model training and new garbage detection. The dataset contributed by the embodiment of the present disclosure contains a total of 5038 images. Based on the size of general packaging garbage images in X-ray machines, the resolution of a single image in the dataset is uniformly set to 450 × 450. According to the commonly used standards for garbage classification in my country, the dataset of the embodiment of the present disclosure divides the annotation categories into four categories: recyclable, hazardous, kitchen waste and other garbage; at the same time, the annotations of the dataset contributed by this example are further subdivided into twelve categories, including plastic bottles, cans, cartons, glass bottles, long sticks, spoons, hot packs, desiccants, lunch boxes, batteries, light bulbs and food residues, according to the frequency of occurrence of garbage categories and the value of recycling and processing; the dataset contains a total of 30,845 valid annotations, and each annotation includes two categories of annotations: target edge box and mask, which is convenient for the later use of the method proposed in the embodiment of the present disclosure to perform deep learning model training such as target recognition and instance segmentation.

[0120] In the embodiment of the present application, in terms of inputting the X-ray image of the packaged garbage, the complete packaged garbage is mainly passed through the X-ray machine, and the X-ray image of the garbage taken by the X-ray machine is obtained through an acquisition card connected to a high-performance computer, and the obtained input image is used for processing in subsequent steps. The number and image size of the input images can be determined according to the needs of the actual application scenario, and the embodiment of the present disclosure does not specifically limit its specific value.

[0121] In the embodiment of the present application, the X-ray garbage image dataset refers to a set of X-ray images collected by the inventors of the present disclosure and derived from household packaging waste.

[0122] In the embodiment of the present application, the X-ray image is an X-ray image of packaging waste taken by an X-ray machine and collected by an acquisition card connected to an external high-performance computer. The format of the collected image is determined, and there is no specific requirement for the size of the image.

[0123] In an embodiment of the present application, image features of the collected X-ray image of packaging waste are extracted to obtain image features and contour features of the X-ray image.

[0124] In an embodiment of the present application, the target recognition operation may be to perform a fusion operation on the image feature information and the thermal feature information according to the feature pyramid network to obtain feature fusion information; and then identify the target object in the feature fusion information according to the full convolution single-stage target detection model to obtain a target recognition frame.

[0125] In an embodiment of the present application, in X-ray image instance segmentation, a "difficult" target is usually an object that overlaps with multiple other targets in a target cluttered area. The disclosed embodiment is inspired by the instance segmentation method performed by humans, that is, first identifying "easy" (i.e., easy to distinguish) instances before segmenting "difficult" instances. The disclosed embodiment proposes a new easy-to-difficult decomposition strategy to simulate this process. The method first splits the obtained target bounding box set into a simple set and a difficult set, and then uses the mask extracted from the simple instance to help predict the target instance mask that is highly occluded in the difficult set.

[0126] In the embodiment of the present application, after obtaining the target recognition frame, a set of bounding boxes B can be obtained = {B 1 , B 2 , …, B N}∈R N×5 , where N is the number of bounding boxes after non-maximum suppression, and B i =[c x , c y , w, h, s] is the target box detected from the full convolution single-stage target detection head, where (c x , c y ) represents the center coordinate of the detection box, w and h represent the width and height of the detection box respectively, and s is the prediction confidence given by the target detection head.

[0127] In the embodiment of the present application, based on the prediction confidence score of each target recognition frame, the obtained target detection frame set B is divided into N e The high confidence set B of boxes easy With N h The low confidence detection box set B of boxes hard After obtaining the simple and difficult detection box sets respectively, the disclosed embodiment plans to use the simple set to generate useful information to guide the generation of the difficult set target mask.

[0128] In the segmentation operation of the embodiment of the present application, performing instance segmentation on the target within the target detection bounding box with high confidence usually obtains better results. Therefore, the embodiment of the present disclosure first predicts the high confidence set B easy The object instance masks in all boxes Use a segmentation head G s , where Z e is the updated feature of the graph convolutional network, is the i-th th The features of the region of interest of the box, A is the self-attention mechanism diagram:

[0129]

[0130]

[0131] A=softmax(F(X,X))I

[0132]

[0133] W is the weight of the graph network, A is the adjacency matrix of the graph, X e i is the feature of the easy part of the input to the attention graph network. θ(x) and φ(x) represent the output of the two fully connected layers (representing query and key in the attention mechanism) M e i is the mask of the simple set, Z e i is the feature of the simple set input graph after the attention network. σ is the activation function, and softmax is the softmax activation function.

[0134] In the embodiment of the present application, the estimated mask information obtained from the simple set is used to enhance the self-attention mechanism map to predict the mask of the difficult set. Specifically, the embodiment of the present disclosure first converts the mask M of the simple region obtained according to the coordinates of the region of interest e Convert back to the input image space and merge them into a single mask P through the element-based maximum operation in the sparse matrix. The mask P here is a soft mask, and each value contained represents the probability that the corresponding pixel position contains the target instance. Since the mask provides effective and obvious information for the location of simple instances, the embodiment of the present disclosure proposes a segmentation method based on a mask-guided attention mechanism to guide difficult instances.

[0135] In the embodiment of the present application, for the difficult set J th The proposed segmentation operation from easy to difficult performs region of interest alignment on mask P to obtain the region of interest features corresponding to the difficult set. The pixel-aligned mask P on which segmentation is performed j The mask-guided attention mechanism used here is A g It is defined as follows, where ⊙ represents the Hadamard product, A s Then it is a similar probability matrix, whose elements represent the probability that two nodes contain objects:

[0136]

[0137] A s =1+P j (P j ) T ,

[0138] Where ⊙ represents the Hadamard product, A s is a similar probability matrix whose elements represent the probability that two nodes contain objects

[0139] Finally, the embodiment of the present disclosure can calculate the difficulty set by the following formula: The mask:

[0140]

[0141]

[0142] Among them, M h j Denotes the mask detection result of the difficult set, Z h j Features obtained after inputting graph attention for difficult features.

[0143] In an embodiment of the present application, guided by the mask attention mechanism, the difficult set can efficiently utilize the similarity matrix constructed by the mask results of the simple set to enhance the attention map, thereby improving the prediction accuracy of the overall mask of the difficult set target in instance segmentation.

[0144] In the embodiment of the present application, a garbage target identification device 200 based on X-ray is provided, comprising: an image acquisition module 210, for acquiring an original X-ray image of the garbage to be identified; a feature extraction module 220, for performing a feature extraction operation on the original X-ray image to obtain original feature information; a target identification module 230, for performing a target identification operation on the garbage to be identified in the original X-ray image according to the original feature information to obtain a target identification frame, wherein the target identification frame includes a simple identification frame and a difficult identification frame; a frame segmentation module 240, for performing a segmentation operation on the garbage to be identified in the difficult identification frame according to the segmentation network and the mask information of the simple identification frame to obtain a segmented identification frame; and a result confirmation module 250, for the simple identification frame and the segmented identification frame are the target identification results of the garbage to be identified. Compared with the prior art, the present application is based on a decomposition strategy from easy to difficult, can efficiently extract edge information in X-ray images, and effectively solve the problems of missed detection and false detection caused by overlapping occlusion between various targets in X-ray garbage images, can realize the identification of various types of garbage in the bag without opening the garbage bag, and improve the recognition accuracy of X-ray images.

[0145] Continue reading Figure 7 , showing Figure 6 The structural diagram of a specific implementation of the feature extraction module 220 is shown in FIG. 2 . For the sake of convenience, only the part related to the present application is shown.

[0146] In some optional implementations of this embodiment, the original feature information includes image feature information and thermal feature information, and the feature extraction module 220 includes: an image feature extraction submodule 221 and a thermal feature extraction submodule 222, wherein:

[0147] An image feature extraction submodule 221 is used to perform an image feature extraction operation on the original X-ray image according to a residual neural network to obtain image feature information;

[0148] The thermal feature extraction submodule 222 is used to perform a thermal feature extraction operation on the original X-ray image according to the global structure guidance model to obtain thermal feature information.

[0149] In an embodiment of the present application, in the feature extraction stage, a residual neural network is selected as the backbone network to perform image feature extraction on the X-ray garbage image to obtain multiple image feature maps of different scales; the algorithm of the disclosed embodiment uses a residual 101 neural network (ResNet101) in the implementation to extract feature information of different scales of the original X-ray image, and inputs the information of different scales into the feature pyramid network for subsequent processing steps.

[0150] In an embodiment of the present application, after acquiring image features using a residual neural network, a global structure guidance model is proposed in the disclosed embodiment for features such as penetration and occlusion in X-ray images. The model design is universal and can be easily inserted into existing top-down deep learning methods to extract global contour information.

[0151] Specifically, the global structure guides the model to first upsample the low-resolution feature map to a high-resolution feature map, and then The embodiment of the present disclosure is also inspired by the human body posture estimation method. The embodiment of the present disclosure marks the correct mask J i It is represented as a Gaussian distribution heat map centered on the contour point with variance σ. Since the global contour map contains overlapping information between different instances, the embodiment of the present disclosure proposes that the global structure guidance model is integrated into the network to extract the semantic features of the whole image to better segment the overlapping objects.

[0152] In some optional implementations of this embodiment, the target recognition module 230 includes: a feature fusion submodule and a target detection submodule, wherein:

[0153] The feature fusion submodule is used to fuse the image feature information and the thermal feature information according to the feature pyramid network to obtain feature fusion information;

[0154] The target detection submodule is used to identify the target object in the feature fusion information according to the full convolution single-stage target detection model and obtain the target recognition box.

[0155] In an embodiment of the present application, the simple recognition frame and the difficult recognition frame obtained in step S203 are input into the feature pyramid network, and the multi-layer multi-scale images are fused through the feature pyramid network, and the fused results are input into the full convolution single-stage target detection head. Full convolution single-stage target detection is a single-stage, fully convolutional target detection algorithm that predicts objects in a per-pixel manner. Based on this anchor-free frame approach, the algorithm can make more use of foreground sample training while avoiding the influence of anchor-related parameters, and can obtain a more accurate target recognition frame, which is convenient for the next step of instance segmentation image processing.

[0156] In some optional implementations of this embodiment, the target recognition module 230 further includes: a first target recognition submodule and a second target recognition submodule, wherein:

[0157] The first target recognition submodule is used to take the target recognition frame whose prediction confidence is greater than or equal to the confidence threshold as a simple recognition frame.

[0158] The second target recognition submodule is used to regard the target recognition frame whose confidence is less than the confidence threshold as a difficult recognition frame.

[0159] In the embodiment of the present application, after obtaining the target recognition frame, a set of bounding boxes B can be obtained = {B 1 , B 2 , …, B N}∈R N×5 , where N is the number of bounding boxes after non-maximum suppression, and B i =[c x , c y , w, h, s] is the target box detected from the full convolution single-stage target detection head, where (c x , c y ) represents the center coordinate of the detection box, w and h represent the width and height of the detection box respectively, and s is the prediction confidence given by the target detection head.

[0160] In the embodiment of the present application, based on the prediction confidence score of each target recognition frame, the obtained target detection frame set B is divided into N e The high confidence set B of boxes easy With N h The low confidence detection box set B of boxes hard After obtaining the simple and difficult detection box sets respectively, the disclosed embodiment plans to use the simple set to generate useful information to guide the generation of the difficult set target mask.

[0161] In some optional implementations of this embodiment, the target identification module 230 further includes: a model tuning submodule, wherein:

[0162] The model tuning submodule is used to calculate the detection loss function of the full convolution single-stage target detection model and perform tuning operations on the full convolution single-stage target detection model according to the detection loss function, where the detection loss function is expressed as:

[0163] L detect =L regression +L center +L classes

[0164] Among them, L regression represents the regression loss; L center represents the detector center point regression loss; L classes represents the detector classification loss.

[0165] In the embodiment of the present application, target detection is essentially performed based on an anchor-free method, which is more efficient, wherein full convolution single-stage target detection is selected as the target detector.

[0166] In some optional implementations of this embodiment, the X-ray based garbage target identification device 200 further includes: a network tuning submodule, wherein:

[0167] The network tuning submodule is used to calculate the segmentation loss function of the segmentation network and perform tuning operations on the segmentation network according to the segmentation loss function, where the segmentation loss function is expressed as:

[0168] L mask =λ 1 L e +λ 2 L h +λ 3 L heatmap

[0169] L e =L Occulder (M e )+L Occuldee (M e )

[0170] L h =L Occulder (M h )+L Occuldee (M h )

[0171]

[0172] Among them, L Occulder represents the mask loss function of the occluder; L Occuldee represents the occluded mask loss function; L heatmap represents the global edge loss function; J represents the edge heat map; L MSE represents the mean square error.

[0173] In the embodiment of the present application, among the variables of the above loss function, L e With L h Respectively represent the segmentation loss of the simple detection box and the difficult detection box set, L h The prediction of the global contour heat map is supervised, and L Occulder With L Occuldee Here, they represent the loss values ​​of the target detection box and mask in each region of interest. 1 ,λ 2 With λ 3 are the hyperparameters of the balanced loss function. According to the experience of the training set, the embodiments of the present disclosure adjust them to {0.5, 1.0, 0.5} respectively.

[0174] In an embodiment of the present application, the complete instance segmentation framework can be trained in an end-to-end manner by a multi-task loss function L defined as:

[0175] L=λL detect +L mask .

[0176] Generally, as the loss weight, λ=1.

[0177] To solve the above technical problems, the present application also provides a computer device. Figure 8 , Figure 8 This is a basic structural block diagram of the computer device in this embodiment.

[0178] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 300 having components 310-330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0179] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0180] The memory 310 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 can be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 can also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Of course, the memory 310 can also include both the internal storage unit of the computer device 300 and its external storage device. In this embodiment, the memory 310 is generally used to store an operating system and various application software installed on the computer device 300, such as computer-readable instructions of a garbage target identification method based on X-rays, etc. In addition, the memory 310 can also be used to temporarily store various data that have been output or are to be output.

[0181] The processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 320 is generally used to control the overall operation of the computer device 300. In this embodiment, the processor 320 is used to run the computer-readable instructions or process data stored in the memory 310, such as running the computer-readable instructions of the X-ray-based garbage target identification method.

[0182] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.

[0183] The computer device provided in the present application is based on a decomposition strategy from easy to difficult, and can efficiently extract edge information in X-ray images, and effectively solve the problems of missed detection and false detection caused by overlapping occlusions between various targets in X-ray garbage images. It can identify various types of garbage in the bag without opening the garbage bag, and improve the recognition accuracy of X-ray images.

[0184] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned X-ray-based garbage target identification method.

[0185] The computer-readable storage medium provided in the present application can efficiently extract edge information in X-ray images based on a decomposition strategy from easy to difficult, and effectively solve the problems of missed detection and false detection caused by overlapping occlusions between various targets in X-ray garbage images. It can identify various types of garbage in the garbage bag without opening the garbage bag, and improve the recognition accuracy of X-ray images.

[0186] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0187] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A garbage target identification method based on X-ray, It is characterized in that The steps include: Obtaining original X-ray images of the garbage to be identified; Performing a feature extraction operation on the original X-ray image to obtain original feature information; Performing a target recognition operation on the garbage to be recognized in the original X-ray image according to the original feature information to obtain a target recognition frame, wherein the target recognition frame includes a simple recognition frame and a difficult recognition frame; Performing a segmentation operation on the to-be-identified garbage in the difficult recognition frame according to the segmentation network and the mask information of the simple recognition frame to obtain a segmentation recognition frame; The simple recognition frame and the segmentation recognition frame are the target recognition results of the garbage to be recognized; The original feature information includes image feature information and thermal feature information. The step of performing feature extraction operation on the original X-ray image to obtain the original feature information specifically includes the following steps: Performing an image feature extraction operation on the original X-ray image according to a residual neural network to obtain the image feature information; Performing a thermal feature extraction operation on the original X-ray image according to the global structure guidance model to obtain the thermal feature information; The step of performing a target recognition operation on the garbage to be recognized in the original X-ray image according to the original feature information to obtain a target recognition frame specifically includes the following steps: Performing a fusion operation on the image feature information and the thermal feature information according to a feature pyramid network to obtain feature fusion information; Identify the target object in the feature fusion information according to the full convolution single-stage target detection model to obtain the target recognition frame; The target recognition frame also includes a prediction confidence. After the step of identifying the target object in the feature fusion information according to the full convolution single-stage target detection model and obtaining the target recognition frame, the following steps are also included: The target recognition frame whose prediction confidence is greater than or equal to the confidence threshold is used as the simple recognition frame; The target recognition frame whose confidence is less than the confidence threshold is used as the difficult recognition frame; Before the step of performing a segmentation operation on the to-be-identified garbage in the difficult recognition frame according to the segmentation network and the mask information of the simple recognition frame to obtain the segmented recognition frame, the following step is also included: The segmentation loss function of the segmentation network is calculated, and the segmentation network is tuned according to the segmentation loss function, wherein the segmentation loss function is expressed as: Among them, the Represents the mask loss function of the occluder; Represents the occluded mask loss function; represents the global edge loss function; J represents the heat map of each edge; represents the mean square error.

2. The X-ray based garbage target identification method according to claim 1, It is characterized in that Before the step of identifying the target object in the feature fusion information according to the full convolution single-stage target detection model to obtain the target recognition frame, the following steps are also included: The detection loss function of the fully convolutional single-stage target detection model is calculated, and the fully convolutional single-stage target detection model is tuned according to the detection loss function, wherein the detection loss function is expressed as: in, represents the regression loss; represents the detector center point regression loss; represents the detector classification loss.

3. A garbage target identification device based on X-ray, It is characterized in that The device is applicable to the X-ray based garbage target identification method as described in any one of claims 1 to 2, and the device comprises: An image acquisition module, used to acquire the original X-ray image of the garbage to be identified; A feature extraction module, used to perform a feature extraction operation on the original X-ray image to obtain original feature information; A target recognition module, used for performing a target recognition operation on the garbage to be recognized in the original X-ray image according to the original feature information to obtain a target recognition frame, wherein the target recognition frame includes a simple recognition frame and a difficult recognition frame; A frame segmentation module, used for performing a segmentation operation on the to-be-identified garbage of the difficult recognition frame according to the segmentation network and the mask information of the simple recognition frame to obtain a segmentation recognition frame; A result confirmation module is used for the target recognition result that the simple recognition frame and the segmented recognition frame are the garbage to be recognized.

4. The X-ray based garbage target identification device according to claim 3, It is characterized in that The original feature information includes image feature information and thermal feature information, and the feature extraction module includes: An image feature extraction submodule, used for performing an image feature extraction operation on the original X-ray image according to a residual neural network to obtain the image feature information; The thermal feature extraction submodule is used to perform a thermal feature extraction operation on the original X-ray image according to the global structure guidance model to obtain the thermal feature information.

5. A computer device, It is characterized in that The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the X-ray-based garbage target identification method as described in any one of claims 1 to 2 when executing the computer-readable instructions.

6. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the X-ray-based garbage target identification method as described in any one of claims 1 to 2 are implemented.

Citation Information

Patent Citations

  • Goods deep learning training method based on machine vision and data sampling

    CN111310519A

  • Method for automatically classifying garbage

    CN112241679A

  • Segmentation identification method and system for overlapped and shielded target object

    CN113361533A

  • Feature fusion and dense connection-based method for infrared plane object detection

    US20210174149A1