A method for inferring garbage location based on geometric appearance features

Abstract

The embodiments of the present application provide a method, apparatus, device and computer-readable storage medium for inferring the location of garbage based on geometric appearance features. The method comprises acquiring image information; extracting image features in the image information through a preset multi-layer residual convolutional neural network; completing object detection through an object detection model based on the image features; the object detection model comprises an anchor and an IOU structure; inputting the image features and the object detection results into a geometric appearance extraction model to obtain the geometric features of the object; and determining the location of the garbage based on the geometric features of the object and the results of the object detection. In this way, the location of garbage in real scenes is achieved.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of image processing, and more particularly to methods, devices, equipment, and computer-readable storage media for inferring garbage locations based on geometric appearance features. Background Art

[0002] In urban governance inspections, sanitation inspections are a heavy and important daily task, and a common type of inspection in sanitation inspections is garbage inspections. In real life, garbage often appears in piles and is in a mess.

[0003] The existing garbage classification solutions on the market are all based on the identification and classification of specific garbage categories, and are unable to detect and locate the complex and disorganized garbage in real scenarios. Summary of the invention

[0004] According to an embodiment of the present application, a solution is provided for inferring the location of garbage based on geometric appearance features.

[0005] In a first aspect of the present application, a method for inferring the location of garbage based on geometric appearance features is provided. The method comprises:

[0006] Get image information;

[0007] Extracting image features from the image information through a preset multi-layer residual convolutional neural network;

[0008] Based on the image features, object detection is performed through an object detection model; the object detection model includes an anchor and an IOU structure;

[0009] Inputting the image features and object detection results into a geometric appearance extraction model to obtain geometric features of the object;

[0010] The garbage location is determined based on the geometric features of the object and the result of the object detection.

[0011] Furthermore, the multi-layer residual convolutional neural network includes a basebone network structure.

[0012] Furthermore, the performing object detection by an object detection model based on the image features includes:

[0013] Extracting output features of the nth, n+1th and n+2th layers of the image features, fusing the output features using a pyramid structure, and obtaining a three-layer feature matrix and fused feature information outputted by each layer;

[0014] Based on the three-layer feature matrix, different width-height ratios are used to construct anchors for each layer;

[0015] The fused feature information output by each layer and the anchor of each layer are subjected to IOU operation to complete object detection.

[0016] Furthermore, performing an IOU operation on the fused feature information outputted by each layer and the anchor of each layer to complete the object detection includes:

[0017] The fused feature information output by each layer and the anchor of each layer are subjected to IOU operation, and the anchor with a preset value is selected as the object to be detected by the model to complete object detection.

[0018] Furthermore, the geometric appearance extraction model includes:

[0019]

[0020] Among them, the p j For, the color gradient information of the object surface;

[0021] Said is,the object contour point matrix;

[0022] Said is,the object contour relative matrix;

[0023] The p i is the geometric appearance feature information matrix of the object in the previous layer;

[0024] The n, δ is,the network parameter;

[0025] The W ij is the weight of the contour structure of each object point;

[0026] The q i is the geometric appearance feature information matrix of the current object.

[0027] Further, the determining of the garbage location based on the geometric features of the object and the result of the object detection includes:

[0028] The geometric features of the object and the results of object detection are mapped in high dimensions, and after semantic background removal, the dimensions are reduced to object category and position space, anchors are extracted according to preset confidence levels, objects are removed, and garbage locations are determined.

[0029] Furthermore, the multi-layer residual convolutional neural network is trained in the following manner:

[0030] Generate a training sample set, wherein the training sample includes an image and corresponding image features;

[0031] The image set in the training sample set is taken as input, and the image features corresponding to the images are taken as expected outputs. When the unification rate of the output result and the labeled result meets a preset threshold, the training of the multi-layer residual convolutional neural network is completed.

[0032] In a second aspect of the present application, a device for inferring the location of garbage based on geometric appearance features is provided. The device comprises:

[0033] An acquisition module, used for acquiring image information;

[0034] A first extraction module extracts image features from the image information through a preset multi-layer residual convolutional neural network;

[0035] A detection module, used to complete object detection based on the image features through an object detection model; the object detection model includes an anchor and an IOU structure;

[0036] A second extraction module, used for inputting the image features and object detection results into a geometric appearance extraction model to obtain geometric features of the object;

[0037] The determination module is used to determine the garbage location based on the geometric features of the object and the result of object detection.

[0038] In a third aspect of the present application, an electronic device is provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the method described above is implemented.

[0039] In a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method according to the first aspect of the present application is implemented.

[0040] The method for inferring garbage location based on geometric appearance features provided in the embodiment of the present application obtains image information; extracts image features in the image information through a preset multi-layer residual convolutional neural network; completes object detection through an object detection model based on the image features; the object detection model includes an anchor and an IOU structure; inputs the image features and the object detection results into a geometric appearance extraction model to obtain the geometric features of the object; determines the garbage location based on the geometric features of the object and the result of the object detection, thereby realizing the location of complex and disorganized garbage in real scenes.

[0041] It should be understood that the contents described in the Summary of the Invention are not intended to limit the key or important features of the embodiments of the present application, nor are they intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The above and other features, advantages and aspects of the embodiments of the present application will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0043] Figure 1 The system architecture diagram involved in the method provided in the embodiment of the present application is shown.

[0044] Figure 2 A flow chart of a method for inferring garbage location based on geometric appearance features according to an embodiment of the present application is shown;

[0045] Figure 3 A block diagram of a device for inferring garbage location based on geometric appearance features according to an embodiment of the present application is shown;

[0046] Figure 4 A schematic diagram of the structure of a terminal device or server suitable for implementing an embodiment of the present application is shown. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0048] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0049] Figure 1 An exemplary system architecture 100 is shown to which an embodiment of the method for inferring garbage location based on geometric appearance features or the apparatus for inferring garbage location based on geometric appearance features of the present application can be applied.

[0050] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0051] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as model training applications, video recognition applications, web browser applications, social platform software, etc.

[0052] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4) players, laptop computers and desktop computers, etc. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, multiple software or software modules used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0053] When the terminals 101, 102, and 103 are hardware, a video acquisition device may be installed thereon. The video acquisition device may be any device capable of acquiring video, such as a camera, a sensor, etc. A user may use the video acquisition device on the terminals 101, 102, and 103 to acquire video.

[0054] The server 105 may be a server that provides various services, such as a background server that processes data displayed on the terminal devices 101, 102, and 103. The background server may analyze the received data and feed back the processing result (such as the recognition result) to the terminal device.

[0055] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, multiple software or software modules for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0056] It should be understood that Figure 1The number of terminal devices, networks and servers in the above is only illustrative. According to the implementation requirements, there may be any number of terminal devices, networks and servers. In particular, when the target data does not need to be acquired remotely, the above system architecture may not include a network, but only include terminal devices or servers.

[0057] like Figure 2 , which is a flow chart of a method for inferring garbage location based on geometric appearance features according to an embodiment of the present application. Figure 2 As can be seen from the figure, the method for inferring garbage location based on geometric appearance features in this embodiment includes the following steps:

[0058] S210, acquiring image information.

[0059] In this embodiment, the execution body (eg Figure 1 The server shown) can obtain image information through wired or wireless connection.

[0060] Furthermore, the execution subject may obtain an electronic device (eg Figure 1 The image information may be the image information sent by the terminal device shown in the figure, or the image information may be the image information pre-stored locally.

[0061] Usually, the image information includes images of garbage dumping sites that appear in piles and in a disorderly manner.

[0062] S220, extracting image features from the image information through a preset multi-layer residual convolutional neural network.

[0063] The multi-layer residual convolutional neural network may be a basebone network structure, which may be trained in the following manner:

[0064] Generate a training sample set, wherein the training sample includes an image and corresponding image features;

[0065] The image set in the training sample set is taken as input, and the image features corresponding to the images are taken as expected outputs. When the unification rate of the output result and the labeled result meets a preset threshold, the training of the multi-layer residual convolutional neural network is completed.

[0066] In some embodiments, the structure of the multi-layer residual convolutional neural network is as follows:

[0067]

[0068] The extraction formula is: y = x + w (ax + b):

[0069] Wherein, y is the feature information of the current layer;

[0070] The x is the feature information output by the previous layer of network;

[0071] The a and b are network parameters;

[0072] The w is the weight of the current layer.

[0073] In some embodiments, image information is input into a trained multi-layer residual convolutional neural network to obtain image features.

[0074] S230: Based on the image features, complete object detection through an object detection model.

[0075] The object detection model includes anchor and IOU structure.

[0076] In some embodiments, the output features of the nth, n+1th and n+2th layers of the image features are extracted, and the output features are fused using a pyramid structure to obtain a three-layer feature matrix and fused feature information output by each layer; wherein n is a positive integer greater than 0, preferably 3, that is, the output features of the 3rd, 4th, and 5th layer network structures of the basebone network structure are extracted.

[0077] Based on the three-layer feature matrix, different width-height ratios are used to construct anchors for each layer;

[0078] The fused feature information output by each layer and the anchor of each layer are subjected to IOU operation, and the anchor with a preset value is selected as the object for model detection to complete object detection; wherein, the preset value is preferably 0.6, that is, the anchor with a confidence level of 0.6 is selected as the object for model detection.

[0079] S240, inputting the image features and object detection results into a geometric appearance extraction model to obtain geometric features of the object.

[0080] In some embodiments, the input information includes the following three parts:

[0081] The first part is the feature information after pyramid fusion of basebone layers 3, 4, and 5;

[0082] The second part is the anchor information of the object detected in each layer;

[0083] The third part is the prior information of conventional geometric rules, including color, geometric feature points and relative relationship matrix (common sense information such as appearance physical characteristics).

[0084] Furthermore, the above three types of input information are input into the following model to extract geometric features in the image:

[0085]

[0086] Among them, the p j The color gradient information of the object surface is used to describe the color change of the object;

[0087] Said The object contour point matrix is ​​formed after semantic segmentation, and the pixel positions outside the semantic segmentation are combined into a two-dimensional point matrix, which is then mapped by the network in a high-dimensional manner to form a contour point matrix.

[0088] Said The object contour relative matrix is ​​determined by the high-dimensional mapping of the peripheral contour geometric relationship. The contour geometric relationship is constructed by high-dimensional fusion according to the geometric rules of objects in real life plus the contour point matrix;

[0089] The p i The geometric appearance feature information matrix of the object in the previous layer is used to prevent the model from collapsing when deepening the network layer of the model.

[0090] The n, δ is,the network parameter, which can be preset according to the actual application,scenario;

[0091] The W ij The weight of the contour structure of each object point can be pre-set according to the actual application scenario;

[0092] The q i is the geometric appearance feature information matrix of the current object, which is used to input into the next layer of the network, namely the geometric features of the object.

[0093] S250, determining the garbage location based on the geometric features of the object and the result of object detection.

[0094] In some embodiments, the geometric features of the object and the results of object detection are mapped in high dimensions. After semantic background removal, the contour geometric appearance feature matrix is ​​mapped and then reduced in dimension to the object category and position space. Anchors are extracted according to a certain confidence level (for example, 0.6, which is pre-set according to the application scenario) and objects are removed to obtain garbage location information.

[0095] According to the embodiments of the present disclosure, the following technical effects are achieved:

[0096] Based on common sense information such as the geometric shapes and physical features of existing objects in real life, combined with the object features in image detection, the geometric feature defects of the objects are inferred, thereby forming a judgment on the incomplete areas of the image and identifying the location of garbage.

[0097] That is, the objects and background in the image information are eliminated, and an algorithm that integrates the three elements of the object's color, contour points, and contour relative matrix is ​​used to determine whether the object conforms to conventional geometric appearance features, and the garbage location information is used to filter out the complex and disorganized garbage positioning in real scenes.

[0098] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0099] The above is an introduction to the method embodiment. The following is a further explanation of the scheme described in this application through an apparatus embodiment.

[0100] Figure 3 FIG. 3 is a block diagram of an apparatus 300 for inferring garbage location based on geometric appearance features according to an embodiment of the present application. Figure 3 As shown, the device 300 includes:

[0101] An acquisition module 310 is used to acquire image information;

[0102] A first extraction module 320 extracts image features from the image information through a preset multi-layer residual convolutional neural network;

[0103] A detection module 330, configured to perform object detection based on the image features through an object detection model; the object detection model includes an anchor and an IOU structure;

[0104] A second extraction module 340 is used to input the image features and object detection results into a geometric appearance extraction model to obtain geometric features of the object;

[0105] The determination module 350 is used to determine the garbage location based on the geometric features of the object and the result of object detection.

[0106] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0107] Figure 4 A schematic diagram of the structure of a terminal device or a server suitable for implementing an embodiment of the present application is shown.

[0108] like Figure 4As shown, the terminal device or server 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 402 or the program loaded from the storage part 408 to the random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the system 400 are also stored. The CPU 401, the ROM 402, and the RAM 403 are connected to each other through a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0109] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed, so that a computer program read therefrom is installed into the storage section 408 as needed.

[0110] In particular, according to an embodiment of the present application, the above method flow steps can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 409, and / or installed from the removable medium 411. When the computer program is executed by the central processing unit (CPU) 401, the above-mentioned functions defined in the system of the present application are executed.

[0111] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0112] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of the code, and the aforementioned module, program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0113] The units or modules involved in the embodiments described in the present application may be implemented by software or hardware. The units or modules described may also be arranged in a processor. The names of these units or modules do not, in some cases, constitute limitations on the units or modules themselves.

[0114] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable storage medium stores one or more programs, and when the above programs are used by one or more processors to execute the method described in the present application.

[0115] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of application involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the aforementioned application concept. For example, the above features are replaced with (but not limited to) technical features with similar functions applied in the present application.

Claims

1. A method for inferring garbage location based on geometric appearance features, characterized in that: include: Get image information; Extracting image features from the image information through a preset multi-layer residual convolutional neural network; Based on the image features, object detection is performed through an object detection model; the object detection model includes an anchor and an IOU structure; The image features and object detection results are input into a geometric appearance extraction model to obtain geometric features of the object; wherein the input information includes: fused feature information after pyramid fusion of the nth, n+1th and n+2th layers of the image features, object anchor information detected at each layer, and conventional geometric rule prior information, wherein the conventional geometric rule prior information includes color, geometric feature points and relative relationship matrix; Determine the location of the garbage based on the geometric features of the object and the result of the object detection; The geometric appearance extraction model includes: Among them, the p j For, the color gradient information of the object surface; Said is,the object contour point matrix; Said is,the object contour relative matrix; The p i is the geometric appearance feature information matrix of the object in the previous layer; The n, δ is,the network parameter; The Wij is the weight of the contour structure of each object point; The qi is the geometric appearance feature information matrix of the current object.

2. The method according to claim 1, characterized in that The multi-layer residual convolutional neural network includes a basebone network structure.

3. The method according to claim 2, characterized in that The performing of object detection by an object detection model based on the image features includes: Extracting output features of the nth, n+1th and n+2th layers of the image features, fusing the output features using a pyramid structure, and obtaining a three-layer feature matrix and fused feature information outputted by each layer; Based on the three-layer feature matrix, different width-height ratios are used to construct anchors for each layer; The fused feature information output by each layer and the anchor of each layer are subjected to IOU operation to complete object detection.

4. The method according to claim 3, characterized in that The fusion feature information outputted by each layer and the anchor of each layer are subjected to IOU operation to complete the object detection, which includes: The fused feature information output by each layer and the anchor of each layer are subjected to IOU operation, and the anchor with a preset value is selected as the object to be detected by the model to complete object detection.

5. The method according to claim 1, characterized in that The determining of the garbage location based on the geometric features of the object and the result of the object detection includes: The geometric features of the object and the results of object detection are mapped in high dimensions, and after semantic background removal, the dimensions are reduced to object category and position space, anchors are extracted according to preset confidence levels, objects are removed, and garbage locations are determined.

6. The method according to claim 5, characterized in that The multi-layer residual convolutional neural network is trained in the following way: Generate a training sample set, wherein the training sample includes an image and corresponding image features; The image set in the training sample set is taken as input, and the image features corresponding to the images are taken as expected outputs. When the unification rate of the output result and the labeled result meets a preset threshold, the training of the multi-layer residual convolutional neural network is completed.

7. A device for inferring the location of garbage based on geometric appearance features, characterized in that: include: An acquisition module, used for acquiring image information; A first extraction module extracts image features from the image information through a preset multi-layer residual convolutional neural network; A detection module, used to complete object detection based on the image features through an object detection model; the object detection model includes an anchor and an IOU structure; The second extraction module is used to input the image features and object detection results into a geometric appearance extraction model to obtain geometric features of the object; wherein the input information includes: fusion feature information after pyramid fusion of the nth, n+1th and n+2th layers of the image features, object anchor information detected at each layer, and conventional geometric rule prior information, wherein the conventional geometric rule prior information includes color, geometric feature points and relative relationship matrix; A determination module, used for determining the garbage location based on the geometric features of the object and the result of the object detection; The geometric appearance extraction model includes: Among them, the p j For, the color gradient information of the object surface; Said is,the object contour point matrix; Said is,the object contour relative matrix; The p i is the geometric appearance feature information matrix of the object in the previous layer; The n, δ is,the network parameter; The Wij is the weight of the contour structure of each object point; The qi is the geometric appearance feature information matrix of the current object.

8. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images