Image recognition method, device, equipment, readable storage medium and program product

By acquiring the feature map and activation map of the image, performing feature fusion processing, and filtering out candidate areas of interest, solving the problem of inaccurate identification of multi-label image classification in the prior art, and improving the accuracy of image classification.

CN114581710BActive Publication Date: 2024-11-26TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202210210792.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-04
Publication Date
2024-11-26
Estimated Expiration
2042-03-04

AI Technical Summary

Technical Problem

In the prior art, the multi-label image classification method results in inaccurate ROI recognition due to problems such as background, lighting and angle, which reduces the image classification performance.

Method used

By acquiring the first feature map and the second feature map of the image to be identified, the probability value and activation map are determined based on the preset object category set, the feature fusion process is performed, and the candidate areas of interest are selected to improve the recognition accuracy.

Benefits of technology

It improves the recognition accuracy of multi-classification images, focuses on key areas in the image, enhances the semantic understanding and texture characteristics of feature maps after feature fusion, and improves the recognition performance of object categories.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114581710B_ABST
    Figure CN114581710B_ABST
Patent Text Reader

Abstract

The embodiments of the present application provide an image recognition method, apparatus, device, storage medium and program product, which are related to the field of artificial intelligence, and the application scenarios include but are not limited to image recognition. The method includes: determining a first feature map and a second feature map corresponding to the image to be recognized; based on the first feature map and each object category in the object category set, determining a first probability value that the image to be recognized belongs to each object category, and an activation map corresponding to each object category; performing feature fusion processing between the second feature map and the activation map of each object category to determine a feature map after feature fusion; and based on the activation map of each object category, determining at least one candidate region of interest of the image to be recognized; based on the feature map after feature fusion and at least one candidate region of interest, determining a second probability value that the image to be recognized belongs to each object category; based on each first probability value and each second probability value, determining the object category to which the image to be recognized belongs.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to an image recognition method, apparatus, device, readable storage medium, and program product. Background Art

[0002] In the prior art, multi-label image classification (multi-label classification) aims to identify multiple labels (classifications) of an image. It is a basic task in computer vision and multimedia, and is widely used in image retrieval, attribute recognition, automatic image annotation, etc. Traditional image recognition methods generate a large number of ROIs (Regions of Interest), which is not only inefficient for multi-label image classification, but also due to image-related background, lighting, angle and other issues, a large number of ROIs do not actually focus on the target object in the image, resulting in a large number of inaccurate ROIs, thereby reducing the final classification performance. Summary of the invention

[0003] In view of the shortcomings of existing methods, the present application proposes an image recognition method, device, equipment, computer-readable storage medium and computer program product to solve the problem of how to improve the recognition accuracy of multiple image classifications.

[0004] In a first aspect, the present application provides an image recognition method, comprising:

[0005] Obtain an image to be recognized;

[0006] Determine a first feature map and a second feature map corresponding to the image to be recognized, wherein a resolution of the first feature map is smaller than a resolution of the second feature map;

[0007] Based on the first feature map and each object category in the preset object category set, determine a first probability value that the image to be identified belongs to each object category, and an activation map corresponding to each object category;

[0008] Performing feature fusion processing between the second feature map and the activation map of each object category to determine a feature map after feature fusion; and determining at least one candidate region of interest of the image to be identified based on the activation map of each object category;

[0009] Determine a second probability value of the image to be identified belonging to each object category based on the feature map after feature fusion and at least one candidate region of interest;

[0010] Based on each first probability value and each second probability value, the object category to which the image to be identified belongs is determined.

[0011] In one embodiment, determining a first feature map and a second feature map corresponding to an image to be recognized includes:

[0012] Inputting the image to be recognized into the feature extraction model of the first neural network, extracting the second feature map from the fourth convolution block of the feature extraction model, and extracting the first feature map from the fifth convolution block of the feature extraction model;

[0013] Among them, the feature extraction model includes a first-layer convolution block, a second-layer convolution block, a third-layer convolution block, a fourth-layer convolution block and a fifth-layer convolution block, and there is a cascade relationship between the first-layer convolution block, the second-layer convolution block, the third-layer convolution block, the fourth-layer convolution block and the fifth-layer convolution block.

[0014] In one embodiment, based on the first feature map and each object category in a preset object category set, determining a first probability value that the image to be identified belongs to each object category includes:

[0015] The first feature map is input into the first fully connected layer and the maximum pooling layer of the first neural network, and classification processing is performed based on each object category in the object category set to obtain a first probability value corresponding to each object category in the image to be identified.

[0016] In one embodiment, based on the first feature map and each object category in a preset object category set, determining an activation map corresponding to each object category includes:

[0017] Performing dimensionality reduction processing on the first feature map to determine a feature map after dimensionality reduction, wherein the dimension of the feature map after dimensionality reduction is the same as the number of object categories in the object category set;

[0018] The reduced feature map is input into the batch normalization layer of the first neural network for batch normalization to determine the activation map corresponding to each object category.

[0019] In one embodiment, performing feature fusion processing between the second feature map and the activation map of each object category to determine the feature map after feature fusion includes:

[0020] Inputting the second feature map and the activation map of each object category into the feature fusion model of the first neural network, upsampling and linear interpolation processing are performed on the activation map of each object category to obtain a third feature map;

[0021] The second feature map and the third feature map are bitwise summed to obtain a feature map after feature fusion.

[0022] In one embodiment, determining at least one candidate region of interest of an image to be recognized based on the activation map of each object category includes:

[0023] Inputting the activation map of each object category into the candidate region of interest selection model of the first neural network, filtering out the background in the activation map of each object category, and obtaining the activation map of each object category after filtering out the background;

[0024] The first probability values ​​are sorted from large to small, and an activation map corresponding to at least one first probability value in the front order after background removal is subjected to edge extraction processing to obtain at least one candidate region of interest of the image to be identified.

[0025] In one embodiment, determining a second probability value of the image to be identified belonging to each object category based on the feature map after feature fusion and at least one candidate region of interest includes:

[0026] Inputting the feature fused feature map and at least one candidate region of interest into a candidate region of interest pooling layer of the first neural network, and performing cropping processing on the feature fused feature map to obtain a feature map of the candidate region of interest;

[0027] Based on the feature map of the candidate region of interest, a second probability value that the image to be identified belongs to each object category is determined.

[0028] In one embodiment, determining the object category to which the image to be identified belongs based on each first probability value and each second probability value includes:

[0029] For an object category in the object category set, if an average value between a first probability value corresponding to an object category and a second probability value corresponding to an object category is not less than a preset category threshold, it is determined that an object category exists in the image to be identified.

[0030] In one embodiment, before acquiring the image to be recognized, the method further includes:

[0031] Input the training sample into the second neural network to determine the value of the first loss function of the global branch prediction model and the activation map of each object category in the training sample; the second neural network includes a global branch prediction model, a local branch prediction model and a weak supervision model, the global branch prediction model includes a feature extraction model, a first fully connected layer and a maximum pooling layer, and the local branch prediction model includes a feature fusion model, a candidate region of interest selection model, a batch normalization layer, a candidate region of interest pooling layer and a second fully connected layer;

[0032] Based on inputting the activation map of each object category in the training sample into the weak supervision model, suppressing the noise of the activation map of each object category in the training sample, and determining the value of the second loss function of the weak supervision model;

[0033] Input the activation map of each object category in the training sample into the feature fusion model and the candidate region of interest selection model respectively, and determine the value of the third loss function of the local branch prediction model;

[0034] Based on the value of the first loss function, the value of the second loss function, and the value of the third loss function, updating the parameters of the second neural network;

[0035] If the sum of the values ​​of the first loss function, the second loss function and the third loss function is less than a preset loss threshold, the training of the second neural network is terminated; and based on the second neural network obtained through training, the first neural network is determined, and the first neural network does not include a weak supervision model.

[0036] In a second aspect, the present application provides an image recognition device, comprising:

[0037] A first processing module, used for acquiring an image to be identified;

[0038] A second processing module is used to determine a first feature map and a second feature map corresponding to the image to be recognized, wherein the resolution of the first feature map is smaller than the resolution of the second feature map;

[0039] a third processing module, configured to determine, based on the first feature map and each object category in a preset object category set, a first probability value that the image to be identified belongs to each object category, and an activation map corresponding to each object category;

[0040] A fourth processing module is used to perform feature fusion processing between the second feature map and the activation map of each object category to determine a feature map after feature fusion; and based on the activation map of each object category, determine at least one candidate region of interest of the image to be identified;

[0041] A fifth processing module, configured to determine a second probability value that the image to be identified belongs to each object category based on the feature map after feature fusion and at least one candidate region of interest;

[0042] The sixth processing module is used to determine the object category to which the image to be identified belongs based on each first probability value and each second probability value.

[0043] In a third aspect, the present application provides an electronic device, including: a processor, a memory, and a bus;

[0044] A bus, used to connect the processor and memory;

[0045] A memory, used for storing operation instructions;

[0046] The processor is used to execute the image recognition method of the first aspect of the present application by calling an operation instruction.

[0047] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program, which is used to execute the image recognition method of the first aspect of the present application.

[0048] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the steps of the image recognition method in the first aspect of the present application when the computer program is executed by a processor.

[0049] The technical solution provided in the embodiments of the present application has at least the following beneficial effects:

[0050] The feature map after feature fusion contains multiple scales, that is, it has high-level semantic understanding features and certain low-level image texture features at the same time; based on the activation map of each object category, at least one candidate region of interest of the image to be identified is determined, thereby focusing on the key areas in the image to be identified and improving the recognition accuracy of the candidate region of interest; based on the feature map after feature fusion and at least one candidate region of interest, the second probability value of the image to be identified belonging to each object category is determined, and based on each first probability value and each second probability value, the object category to which the image to be identified belongs is determined, thereby improving the recognition accuracy of multiple classifications (multiple labels) of the image to be identified. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.

[0052] Figure 1 A schematic diagram of the architecture of an image recognition system provided in an embodiment of the present application;

[0053] Figure 2 A flowchart of an image recognition method provided in an embodiment of the present application;

[0054] Figure 3 A schematic diagram of image recognition provided in an embodiment of the present application;

[0055] Figure 4 A schematic diagram of image recognition provided in an embodiment of the present application;

[0056] Figure 5 A schematic diagram of image recognition provided in an embodiment of the present application;

[0057] Figure 6 A schematic diagram of image recognition provided in an embodiment of the present application;

[0058] Figure 7 A schematic diagram of image recognition provided in an embodiment of the present application;

[0059] Figure 8A schematic diagram of image recognition provided in an embodiment of the present application;

[0060] Fig. 9 A schematic diagram of image recognition provided in an embodiment of the present application;

[0061] Fig.10 A schematic diagram of comparing candidate regions of interest provided in an embodiment of the present application;

[0062] Fig.11 A flowchart of another image recognition method provided in an embodiment of the present application;

[0063] Fig.12 A schematic diagram of the structure of an image recognition device provided in an embodiment of the present application;

[0064] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] The embodiments of the present application are described below in conjunction with the drawings in the present application. It should be understood that the implementation methods described below in conjunction with the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0066] It will be understood by those skilled in the art that, unless specifically stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application refer to that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude the implementation as other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the one element may be directly connected or coupled to the other element, or it may refer to that the one element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein indicates at least one of the items defined by the term, for example, "A and / or B" indicates that it is implemented as "A", or implemented as "B", or implemented as "A and B".

[0067] It can be understood that in the specific implementation of this application, data related to image recognition is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0068] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0069] The embodiment of the present application is an image recognition method provided by an image recognition system, and the image recognition method involves fields such as artificial intelligence and cloud technology. For example, the image recognition involved in the embodiment of the present application is a computer vision technology in the field of artificial intelligence; for another example, the artificial neural network in the embodiment of the present application is a machine learning technology in the field of artificial intelligence. The application scenarios of the image recognition method include but are not limited to image recognition and other scenarios, such as mobile phone albums, information flow news APP (Application), short video APP, image recognition scenes, etc.

[0070] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0071] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0072] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision that uses cameras and computers to replace human eyes to identify and measure targets, and further performs graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, automatic driving, smart transportation and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0073] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0074] In order to better understand and illustrate the solutions of the embodiments of the present application, some technical terms involved in the embodiments of the present application are briefly explained below.

[0075] Multi-label Classification: The main task of multi-label classification is to classify and identify images, so as to accurately classify samples into one or more labels. Compared with traditional classification problems, multi-label classification problems have the following characteristics: (1) The number of categories is uncertain. Some samples may have only one label, while some samples may have dozens of labels. (2) There may be a certain degree of dependency between labels. The category with tables also has chairs to a large extent. One label can represent one category, and multiple labels can represent multiple categories.

[0076] Region of Interest (ROI): Given an input image, find all possible locations where an object can be located; the output of this stage is a list of bounding boxes of possible locations of the object, which are presented in the form of boxes, circles, ellipses, irregular polygons, etc. In image processing, the processed image outlines the area to be processed in the form of boxes, circles, ellipses, irregular polygons, etc., which is called ROI; ROI is an image region selected from the image, and this region is the focus of image analysis.

[0077] Activation map: An activation map can be called a saliency map. The activation map is learned and used to express the activation value of each object category and the area with relatively large corresponding activation values. It can also be simply regarded as a sign of whether an object is salient. The more prominent and salient an object is, the whiter it is on the activation map.

[0078] Ground truth: The ground truth is the actual measurement value, which is relative to the predicted value and estimated value; the ground truth is 0 or 1, indicating whether the current sample contains an object (instance) with this label; for example, there are 2 cats and 1 person in a picture. If there is a data set with a total of 4 labels (categories), namely cat, dog, person and car, then the label of this picture should be 1010.

[0079] The solution provided in the embodiments of the present application involves artificial intelligence technology. The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0080] In order to better understand the solution provided by the embodiment of the present application, the solution is described below in conjunction with a specific application scenario.

[0081] In one embodiment, Figure 1 FIG. 1 shows a schematic diagram of the architecture of an image recognition system applicable to an embodiment of the present application. It can be understood that the image recognition method provided in the embodiment of the present application can be applied to, but not limited to, Figure 1 In the application scenario shown.

[0082] In this example, Figure 1As shown, the architecture of the image recognition system in this example may include but is not limited to a terminal 10 , a server 20 and a database 30 . The terminal 10, the server 20 and the database 30 can interact with each other through the network. The terminal 10 sends the image to be identified to the server 20, and the server 20 can also obtain the image to be identified from the database 30. One of the multiple servers 20 can be responsible for determining the first feature map and the second feature map corresponding to the image to be identified, and the resolution of the first feature map is smaller than the resolution of the second feature map. The server 20 determines the first probability value of the image to be identified belonging to each object category and the activation map corresponding to each object category based on the first feature map and each object category in the preset object category set. The server 20 performs feature fusion processing between the second feature map and the activation map of each object category to determine the feature map after feature fusion. The server 20 also determines at least one candidate region of interest of the image to be identified based on the activation map of each object category. The server 20 determines the second probability value of the image to be identified belonging to each object category based on the feature map after feature fusion and the at least one candidate region of interest. The server 20 determines the object category to which the image to be identified belongs based on each first probability value and each second probability value. The server 20 sends the object category to which the image to be identified belongs to the terminal 10. Another server 20 among the multiple servers 20 may be responsible for training the neural network in the image recognition method provided in the embodiment of the present application.

[0083] It is understandable that the above is only an example and this embodiment is not limited here.

[0084] The terminal may be a smart phone (such as an Android phone, an iOS phone, etc.), a mobile phone simulator, a tablet computer, a laptop computer, a digital broadcast receiver, a MID (Mobile Internet Devices), a PDA (Personal Digital Assistant), etc. The server may be an independent physical server, or a server cluster or distributed system consisting of multiple physical servers, or a cloud server or server cluster that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0085] Cloud computing is a computing model that distributes computing tasks on a resource pool composed of a large number of computers, allowing various application systems to obtain computing power, storage space and information services as needed. The network that provides resources is called a "cloud". From the user's perspective, the resources in the "cloud" are infinitely scalable and can be obtained at any time, used on demand, expanded at any time, and paid for by use.

[0086] As a provider of basic cloud computing capabilities, a cloud computing resource pool (referred to as a cloud platform, generally referred to as an IaaS (Infrastructure as a Service) platform) will be established, and various types of virtual resources will be deployed in the resource pool for external customers to choose to use. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices.

[0087] According to the logical function division, the PaaS (Platform as a Service) layer can be deployed on the IaaS (Infrastructure as a Service) layer, and the SaaS (Software as a Service) layer can be deployed on the PaaS layer. SaaS can also be deployed directly on IaaS. PaaS is a platform for software operation, such as databases, web containers, etc. SaaS is a variety of business software, such as web portals, SMS mass senders, etc. Generally speaking, SaaS and PaaS are upper layers relative to IaaS.

[0088] The so-called artificial intelligence cloud service is generally also called AIaaS (AI as a Service, Chinese for "AI as a Service"). This is the service mode of a mainstream artificial intelligence platform. Specifically, the AIaaS platform will split several common AI services and provide independent or packaged services in the cloud. This service model is similar to opening an AI theme mall: all developers can access and use one or more artificial intelligence services provided by the platform through the API interface. Some senior developers can also use the AI ​​framework and AI infrastructure provided by the platform to deploy and operate their own cloud artificial intelligence services.

[0089] The above network may include but is not limited to: wired network, wireless network, wherein the wired network includes: local area network, metropolitan area network and wide area network, and the wireless network includes: Bluetooth, Wi-Fi and other networks that realize wireless communication. The specific can also be determined based on the actual application scenario requirements and is not limited here.

[0090] See also Figure 2 , Figure 2 The flowchart of an image recognition method provided by an embodiment of the present application is shown, wherein the method can be executed by any electronic device, such as a server. As an optional implementation, the method can be executed by a server. For the convenience of description, in the description of some optional embodiments below, the server will be used as an example to illustrate the execution subject of the method. Figure 2 As shown, the image recognition method provided in the embodiment of the present application includes the following steps:

[0091] S201, obtaining an image to be recognized.

[0092] Specifically, the image to be identified can be a picture, an image in a video clip, etc. The image to be identified can be obtained from a mobile phone album, an information stream news APP, a short video APP, etc.; or a picture can be taken on site and the taken picture can be used as the image to be identified.

[0093] S202, determining a first feature map and a second feature map corresponding to the image to be recognized, wherein a resolution of the first feature map is smaller than a resolution of the second feature map.

[0094] Specifically, for example, the image to be recognized is 448*448*3, the first feature map is 2048*14*14, and the second feature map is 1024*28*28, where 2048 in 2048*14*14 represents the dimension of the first feature map, that is, 2048 channels, and 14*14 represents the size of the first feature map, that is, length*width, with a length of 14 and a width of 14.

[0095] S203: Based on the first feature map and each object category in a preset object category set, determine a first probability value that the image to be identified belongs to each object category, and an activation map corresponding to each object category.

[0096] Specifically, the preset object category set may include multiple object categories. For example, the object category set is the Pascal VOC 2007 dataset, which includes 20 object categories. Based on the first feature map and each of the 20 object categories, the first probability value of the image to be identified belonging to each object category and the activation map corresponding to each object category are determined, that is, a total of 20 first probability values ​​and 20 activation maps can be determined. The value range of the first probability value is 0~1. The object category can be a person, a cat, a dog, a table, etc.

[0097] S204, performing feature fusion processing between the second feature map and the activation map of each object category to determine a feature map after feature fusion; and determining at least one candidate region of interest of the image to be identified based on the activation map of each object category.

[0098] Specifically, each object category corresponds to an activation map. For example, the object category set includes 20 object categories. These 20 object categories correspond to 20 activation maps. Based on these 20 activation maps, multiple candidate regions of interest ROIs of the image to be identified can be determined.

[0099] S205, determining a second probability value that the image to be identified belongs to each object category based on the feature map after feature fusion and at least one candidate region of interest.

[0100] Specifically, the feature map after feature fusion contains multiple scales, that is, it has high-level semantic understanding features and certain low-level image texture features. For example, the object category set includes 20 object categories. Based on the feature map after feature fusion and multiple candidate regions of interest, the second probability value of the image to be identified belonging to each object category is determined, that is, a total of 20 second probability values ​​can be determined, and the value range of the second probability value is 0~1.

[0101] S206: Determine the object category to which the image to be identified belongs based on the first probability values ​​and the second probability values.

[0102] Specifically, for example, the object category set includes 20 object categories, each object category corresponds to a first probability value and a second probability value, that is, based on the first probability values ​​and the second probability values ​​corresponding to the 20 object categories, the object category to which the image to be identified belongs is determined, and the object category to which the image to be identified belongs can be one or more of these 20 object categories.

[0103] In an embodiment of the present application, the feature map after feature fusion contains multiple scales, that is, it has high-level semantic understanding features and also contains certain low-level image texture features; based on the activation map of each object category, at least one candidate region of interest of the image to be identified is determined, thereby focusing on the key area in the image to be identified and improving the recognition accuracy of the candidate region of interest; based on the feature map after feature fusion and at least one candidate region of interest, a second probability value that the image to be identified belongs to each object category is determined, and based on each first probability value and each second probability value, the object category to which the image to be identified belongs is determined, thereby improving the recognition accuracy of multiple classifications (multiple labels) of the image to be identified.

[0104] In one embodiment, determining a first feature map and a second feature map corresponding to an image to be recognized includes:

[0105] Inputting the image to be recognized into the feature extraction model of the first neural network, extracting the second feature map from the fourth convolution block of the feature extraction model, and extracting the first feature map from the fifth convolution block of the feature extraction model;

[0106] Among them, the feature extraction model includes a first-layer convolution block, a second-layer convolution block, a third-layer convolution block, a fourth-layer convolution block and a fifth-layer convolution block, and there is a cascade relationship between the first-layer convolution block, the second-layer convolution block, the third-layer convolution block, the fourth-layer convolution block and the fifth-layer convolution block.

[0107] Specifically, Figure 3 As shown, the first neural network includes a feature extraction model 301, a fully connected layer 302 (first fully connected layer), a maximum pooling layer 303, a 1*1 convolution layer 304, a batch normalization layer 305, a feature fusion model 306, a ROI selection model 307 (candidate region of interest selection model), a ROI pooling layer 308 (candidate region of interest pooling layer), a convolution block 309 and a fully connected layer 310 (second fully connected layer), wherein the feature fusion model 306 includes an upsampling layer and a linear interpolation layer.

[0108] The feature extraction model 301 may be a ResNet101 model, an AlexNet model, or a VGGNet model. If the feature extraction model 301 is a ResNet101 model, then Figure 3 As shown, the feature extraction model 301 includes conv1 (first layer convolution block), conv2 (second layer convolution block), conv3 (third layer convolution block), conv4 (fourth layer convolution block) and conv5 (fifth layer convolution block). For example, the image (image to be recognized) is 448*448*3, and the second feature map is extracted from conv4 of the feature extraction model 301, and the second feature map is 1024*28*28; the first feature map is extracted from conv5 of the feature extraction model 301, and the first feature map is 2048*14*14. The structure of the convolution block 309 can be the same as that of conv5. The network structure of the ResNet101 model is as follows Figure 4 As shown, a picture (image to be recognized) is used as the input of the ResNet101 model, and the high-level semantic features of the image to be recognized are extracted through the ResNet101 model, that is, the second feature map is extracted from conv4 of the ResNet101 model, and the second feature map is 1024*28*28, and the first feature map is extracted from conv5 of the ResNet101 model, and the first feature map is 2048*14*14.

[0109] In one embodiment, based on the first feature map and each object category in a preset object category set, determining a first probability value that the image to be identified belongs to each object category includes:

[0110] The first feature map is input into the first fully connected layer and the maximum pooling layer of the first neural network, and classification processing is performed based on each object category in the object category set to obtain a first probability value corresponding to each object category in the image to be identified.

[0111] Specifically, Figure 3 As shown, the first feature map is input into the fully connected layer 302 (first fully connected layer) and the maximum pooling layer 303, and classification processing is performed based on each object category in the object category set to obtain a global branch prediction value (first probability value) corresponding to each object category in the image (image to be identified); for example, the image is 448*448*3, and the first feature map is 2048*14*14.

[0112] In one embodiment, based on the first feature map and each object category in a preset object category set, determining an activation map corresponding to each object category includes:

[0113] Performing dimensionality reduction processing on the first feature map to determine a feature map after dimensionality reduction, wherein the dimension of the feature map after dimensionality reduction is the same as the number of object categories in the object category set;

[0114] The reduced feature map is input into the batch normalization layer of the first neural network for batch normalization to determine the activation map corresponding to each object category.

[0115] Specifically, Figure 3 As shown, for example, the first feature map 2048*14*14 is input to the 1*1 convolution layer 304 for dimensionality reduction processing to determine the feature map 20*14*14 after dimensionality reduction, and the feature map 20*14*14 after dimensionality reduction is input to the batch normalization layer 305 for batch normalization processing to obtain 20 1*14*14 feature maps (activation maps). These 20 1*14*14 feature maps can be represented by a 20*14*14, and each 1*14*14 feature map is an activation map corresponding to an object category.

[0116] In one embodiment, performing feature fusion processing between the second feature map and the activation map of each object category to determine the feature map after feature fusion includes:

[0117] Inputting the second feature map and the activation map of each object category into the feature fusion model of the first neural network, upsampling and linear interpolation processing are performed on the activation map of each object category to obtain a third feature map;

[0118] The second feature map and the third feature map are bitwise summed to obtain a feature map after feature fusion.

[0119] Specifically, Figure 3As shown, for example, 20 1*14*14 feature maps (activation maps) are input into the upsampling layer included in the feature fusion model 306, and 1024 14*14 feature maps are sampled from the 20 14*14 feature maps, and the 1024 14*14 feature maps can be represented by 1024*14*14; the 1024 14*14 feature maps are input into the linear interpolation layer included in the feature fusion model 306, and all the 1024 feature maps with a size of 14*14 are interpolated into 1024 feature maps with a size of 28*28, that is, 1024 28*28 feature maps are obtained, and the 1024 28*28 feature maps can be represented by 1024*28*28 (third feature map); the second feature map 1024*28*28 and the third feature map 1024*28*28 are bitwise summed to obtain the feature map after feature fusion.

[0120] In one embodiment, determining at least one candidate region of interest of an image to be recognized based on the activation map of each object category includes:

[0121] Inputting the activation map of each object category into the candidate region of interest selection model of the first neural network, filtering out the background in the activation map of each object category, and obtaining the activation map of each object category after filtering out the background;

[0122] The first probability values ​​are sorted from large to small, and an activation map corresponding to at least one first probability value in the front order after background removal is subjected to edge extraction processing to obtain at least one candidate region of interest of the image to be identified.

[0123] Specifically, Figure 3 As shown, for example, 20 1*14*14 feature maps (activation maps) are input into the ROI selection model 307 (candidate region of interest selection model), and the backgrounds in these 20 1*14*14 activation maps are filtered out to obtain 20 activation maps after the backgrounds are filtered out; the first probability values ​​are sorted from large to small, and the activation maps after the backgrounds are filtered out corresponding to the first 4 first probability values ​​are subjected to edge extraction processing to obtain 4 candidate regions of interest ROI of the image (image to be identified).

[0124] In one embodiment, Figure 5As shown, for example, 20 1*14*14 feature maps (activation maps, i.e., significant activation maps) are input into the nonlinear activation layer ReLU in the ROI selection model 307, and the background in these 20 1*14*14 activation maps is screened out by ReLU, and 20 activation maps after the background is screened out can be obtained. If the activation value of the pixel point in the activation map is a negative value, then the activation value corresponds to the background; if the activation value of the pixel point in the activation map is a non-negative value, then the activation value corresponds to an object, such as a person; therefore, the activation value less than 0 is screened out by ReLU, that is, the background of the activation map is screened out.

[0125] After the activation map passes through the nonlinear activation layer ReLU, the area with negative activation values ​​will be filtered out, and a feature map with only positive activation values ​​will be obtained. For example, N*14*14, N can be 20, that is, 20 activation maps after filtering out the background. Select the confidence probability (first probability value) with the highest order feature maps, for example, =4, that is, the activation maps after the background is removed corresponding to the 4 first probability values ​​in the first order; the edges of these 4 activation maps after the background is removed are extracted respectively, and ROI is selected for each activation map after the background is removed based on the total regional energy value, rather than selecting ROI based on the area size, to obtain 4 ROIs of the image respectively.

[0126] In one embodiment, Figure 6 As shown in , in order to select the key ROI area, you can choose to compare the ROI energy value in the area instead of the area size. The strategy of selecting the ROI area by area value may select an area with a large area but a small overall activation value; while the energy value, that is, the area integral of the activation value in the ROI area, can get a better area. Extract all contours in each feature map (activation map) and use contours with larger energy as candidate ROIs. As shown in Figure 6 As shown in the figure, there are input images, activation maps, edge extraction, suboptimal regions selected based on area, and optimal regions selected based on energy. The input image has two regions under the activation map of the "person" category, where the upper region has a higher energy value, but the lower region has a larger area. If the area priority strategy is adopted, the recognition of the key core feature of the face may be weakened. By adopting the energy priority strategy, the influence of a part of the large area of ​​low activation can be balanced, and only the most critical area can be focused.

[0127] In one embodiment, Figure 7 As shown in FIG. 3 , two additional branches are added between the batch normalization layer 305 and the ROI selection model 307 to perform a certain degree of floating up or down on the activation map (saliency activation map), thereby obtaining At the same time, for an activation map, we can not only select the ROI with the highest energy, but also the ROI with the second highest energy, the ROI with the third highest energy, etc., where the highest energy is greater than the second highest energy, and the second highest energy is greater than the third highest energy. In this way, we can get times ROI. times the number of candidate regions of interest ROI. Originally there were 4 ROIs. After expanding the range, more regions can be obtained, such as 16 ROIs, 24 ROIs, etc.

[0128] For example, the float is +0.1* , down to -0.1* , +0.1* and -0.1* are all variances. For example, =1+number of downslopes (e.g., -0.1 ); ; Figure 7 Add the second highest energy ROI in is 2, and the third highest energy ROI is added, then is 3; if -0.1 And add the second highest energy ROI, then =2, =2, that is =2*4*2=16ROI.

[0129] like Figure 8 As shown, there are input images, activation maps, edge extraction after the activation maps float up and down, and different candidate regions of interest ROIs corresponding to different edges.

[0130] In one embodiment, determining a second probability value of the image to be identified belonging to each object category based on the feature map after feature fusion and at least one candidate region of interest includes:

[0131] Inputting the feature fused feature map and at least one candidate region of interest into a candidate region of interest pooling layer of the first neural network, and performing cropping processing on the feature fused feature map to obtain a feature map of the candidate region of interest;

[0132] Based on the feature map of the candidate region of interest, a second probability value that the image to be identified belongs to each object category is determined.

[0133] Specifically, Figure 3As shown, for example, the feature map after feature fusion and the four candidate regions of interest ROIs are input to the candidate region of interest pooling layer 308, the feature map after feature fusion is cropped to obtain the feature map of the candidate region of interest, the feature map of the candidate region of interest can be expressed as 4*1024*7*7, the feature map of the candidate region of interest is input to the convolution block 309, and the feature map of the candidate region of interest is obtained. Figure 4 *2048*1*1, the feature Figure 4 *2048*1*1 is input to the fully connected layer 310 (the second fully connected layer) for classification processing to obtain a second probability value of the image (image to be identified) belonging to each object category.

[0134] In one embodiment, determining the object category to which the image to be identified belongs based on each first probability value and each second probability value includes:

[0135] For an object category in the object category set, if an average value between a first probability value corresponding to an object category and a second probability value corresponding to an object category is not less than a preset category threshold, it is determined that an object category exists in the image to be identified.

[0136] Specifically, for example, the object category set includes 20 object categories, each of which corresponds to a first probability value and a second probability value, and each object category corresponds to the average value between the first probability value and the second probability value. If the average value is not less than the preset category threshold, it can be determined that the object category exists in the image (image to be identified); if the average value is less than the preset category threshold, it can be determined that the object category does not exist in the image. Multiple object categories in the image can be determined, for example, there are 3 object categories in the image, namely, people, cats, and tables.

[0137] In one embodiment, before acquiring the image to be recognized, the method further includes:

[0138] Input the training sample into the second neural network to determine the value of the first loss function of the global branch prediction model and the activation map of each object category in the training sample; the second neural network includes a global branch prediction model, a local branch prediction model and a weak supervision model, the global branch prediction model includes a feature extraction model, a first fully connected layer and a maximum pooling layer, and the local branch prediction model includes a feature fusion model, a candidate region of interest selection model, a batch normalization layer, a candidate region of interest pooling layer and a second fully connected layer;

[0139] Based on inputting the activation map of each object category in the training sample into the weak supervision model, suppressing the noise of the activation map of each object category in the training sample, and determining the value of the second loss function of the weak supervision model;

[0140] Input the activation map of each object category in the training sample into the feature fusion model and the candidate region of interest selection model respectively, and determine the value of the third loss function of the local branch prediction model;

[0141] Based on the value of the first loss function, the value of the second loss function, and the value of the third loss function, updating the parameters of the second neural network;

[0142] If the sum of the values ​​of the first loss function, the second loss function and the third loss function is less than a preset loss threshold, the training of the second neural network is terminated; and based on the second neural network obtained through training, the first neural network is determined, and the first neural network does not include a weak supervision model.

[0143] Specifically, Figure 3 As shown, the second neural network includes a feature extraction model 301, a fully connected layer 302 (first fully connected layer), a maximum pooling layer 303, a 1*1 convolution layer 304, a batch normalization layer 305, a feature fusion model 306, a ROI selection model 307 (candidate region of interest selection model), a ROI pooling layer 308 (candidate region of interest pooling layer), a convolution block 309, a fully connected layer 310 (second fully connected layer) and a weak supervision model 311, wherein the feature fusion model 306 includes an upsampling layer and a linear interpolation layer. The weak supervision model 311 is used to improve the feature representation of the local area, suppress the noise ROI generated by the non-existent category, that is, suppress the noise of the activation map of each object category, wherein the noise ROI can refer to an inaccurate ROI, which does not frame the corresponding object but frames the background area or other targets.

[0144] like Fig. 9 As shown, the activation map of each object category in the training sample is input into the weak supervision model 311. For the activation map without object category (activation map with ground truth value of 0), the activation map is constrained by the sigmoid function in the weak supervision model 311, so that the activation value of each pixel in the activation map is as close to 0 as possible, and the activation map is constrained to a very low value. In this way, the weak supervision module 311 processes the activation map with ground truth value of 0, removes the activation value of each pixel in the activation map, and obtains the activation map corresponding to the background.

[0145] In one embodiment, binary cross entropy (BCE) is used as the first loss function , to train the global branch prediction model, the first loss function As shown in formula (1):

[0146] Formula (1)

[0147] in, Refers to the object category, is the ground truth value of the input training sample under the object category (existence or non-existence, The value is 0 or 1, binary classification), It is the predicted value output after passing through the global branch prediction model, and the predicted value ranges from 0 to 1. Through the first loss function, it can be ensured that under the continuous iteration and optimization of the global branch prediction model, the output prediction of the global branch prediction model gradually approaches the ground truth of the training sample.

[0148] Second loss function for weakly supervised model 311 As shown in formula (2):

[0149] Formula (2)

[0150] in, are the length and width of the activation map; represents the activation value of the pixel corresponding to the coordinate (h, w) under the selected object category, gt represents the ground truth value, refers to the sum of all non-existent object categories, Indicates the number of object categories; is a small value to prevent math field errors.

[0151] The global branch prediction model includes a feature extraction model 301 , a fully connected layer 302 (the first fully connected layer) and a maximum pooling layer 303 .

[0152] The third loss function for the local branch prediction model , as shown in formula (3):

[0153] Formula (3)

[0154] in, Refers to the current category, is the ground truth value of the input training sample under the object category (existence or absence, binary classification), It is the predicted value output after the local branch prediction model, and the predicted value ranges from 0 to 1. Through the third loss function, it can be ensured that under the continuous iteration and optimization of the local branch prediction model, the output prediction of the local branch prediction model gradually approaches the ground truth of the sample.

[0155] The local branch prediction model includes a 1*1 convolution layer 304, a batch normalization layer 305, a feature fusion model 306, a ROI selection model 307 (candidate region of interest selection model), a ROI pooling layer 308 (candidate region of interest pooling layer), a convolution block 309 and a fully connected layer 310 (second fully connected layer).

[0156] The application of the embodiments of the present application has at least the following beneficial effects:

[0157] The neural network structure is clear, and each model in the neural network has good generalization ability. Experiments on popular large-scale multi-label image classification data sets have proved that the image recognition method provided by the embodiment of the present application has excellent performance. The weak supervision model can effectively generate high-quality region suggestions for multi-label image classification tasks, thereby improving the efficiency of generating ROIs; the weak supervision achieved by the weak supervision model does not require heavy manpower overhead, and does not require additional candidate box annotation of the image. The ROI selection model can automatically learn the semantic boundary threshold of the ROI, and the multi-scale feature fusion also improves the quality of the generated ROI, thereby improving the accuracy of the generated ROI.

[0158] In order to better understand the method provided in the embodiment of the present application, the scheme of the embodiment of the present application is further explained below with reference to examples of specific application scenarios.

[0159] The application scenarios of the method provided in the embodiments of the present application include, but are not limited to, scenarios such as image recognition, such as mobile phone albums, information flow news APPs, short video APPs, image recognition scenarios, etc.

[0160] Specifically, mobile phone albums can classify thousands of pictures in users' mobile phones into landscapes, parties, group photos, buildings, portraits, etc., allowing users to find the pictures they want more easily. Information flow news apps, short video apps, etc. can identify corresponding multiple tags as features based on the browsed information and collected content to assist in recommending content that users like to watch. The image recognition function only requires taking a picture to identify plants, animals, commodities, etc. in the picture, and then display popular science information or recommend it to an e-commerce website for purchase.

[0161] In one embodiment, the generated candidate regions of interest are exemplified as follows: Fig.10 As shown, Fig.10 On the left side of the middle is the result of Selective Search. Fig.10 The middle right side is the result of the method provided in the embodiment of the present application; the result of the selected search includes a large number of inaccurate ROIs. In contrast, the result of the method provided in the embodiment of the present application basically includes accurate ROIs. Therefore, the method provided in the embodiment of the present application improves the recognition accuracy of multi-classification of images.

[0162] The model, model training, and inference process in the method provided in the embodiment of the present application are implemented on a server equipped with an Intel Xeon 8255C CPU and an NVIDIA Tesla V100 graphics card, using 8 V100 graphics cards for distributed parallel training and generating inference results. The coding uses Python 3.6.8, and the deep learning frameworks used are Pytorch 1.4.0 and torchvision 0.5.0, opencv-python version 4.5.1, numpy version 1.16.1, scikit-learn version 0.23.0, etc.

[0163] See also Fig.11 , Fig.11 The flowchart of an image recognition method provided by an embodiment of the present application is shown, wherein the method can be executed by any electronic device, such as a server. As an optional implementation, the method can be executed by a server. For the convenience of description, in the description of some optional embodiments below, the server will be used as an example to illustrate the execution subject of the method. Fig.11 As shown, the image recognition method provided in the embodiment of the present application includes the following steps:

[0164] S801, input the image to be recognized into the feature extraction model, extract the second feature map from the fourth convolution block of the feature extraction model, and extract the first feature map from the fifth convolution block of the feature extraction model.

[0165] S802, input the first feature map into the first fully connected layer and the maximum pooling layer, perform classification processing based on each object category in the object category set, and obtain a first probability value corresponding to each object category in the image to be identified.

[0166] S803: Input the first feature map to a 1*1 convolutional layer for dimensionality reduction to determine the feature map after dimensionality reduction, and input the feature map after dimensionality reduction to a batch normalization layer for batch normalization to determine an activation map corresponding to each object category.

[0167] S804, input the second feature map and the activation map of each object category into the feature fusion model, upsample and linearly interpolate the activation map of each object category to obtain a third feature map; and perform bitwise summation processing between the second feature map and the third feature map to obtain a feature map after feature fusion.

[0168] S805, inputting the activation map of each object category into the candidate region of interest selection model, filtering out the background in the activation map of each object category, and obtaining the activation map of each object category after filtering out the background; and performing edge extraction processing on the activation maps after filtering out the background corresponding to the multiple first probability values ​​sorted in front, to obtain multiple candidate regions of interest of the image to be identified.

[0169] S806, inputting the feature map after feature fusion and multiple candidate regions of interest into a candidate region of interest pooling layer, performing cropping processing on the feature map after feature fusion, and obtaining a feature map of the candidate region of interest.

[0170] S807, input the feature map of the candidate region of interest into the second fully connected layer for classification processing to determine a second probability value that the image to be identified belongs to each object category.

[0171] In one embodiment, for example, Figure 3 As shown, the feature map of the candidate region of interest is input into the convolution block 309 to obtain the feature Figure 4 *2048*1*1, the feature Figure 4 *2048*1*1 is input to the fully connected layer 310 (the second fully connected layer) for classification processing to obtain a second probability value of the image (image to be identified) belonging to each object category.

[0172] S808: Determine the object category to which the image to be identified belongs based on the first probability values ​​and the second probability values.

[0173] The application of the embodiments of the present application has at least the following beneficial effects:

[0174] The feature map after feature fusion contains multiple scales, that is, it has high-level semantic understanding features and certain low-level image texture features at the same time; based on the activation map of each object category, multiple candidate regions of interest of the image to be identified are determined, thereby focusing on the key areas in the image to be identified and improving the recognition accuracy of the candidate regions of interest; based on the feature map after feature fusion and the multiple candidate regions of interest, the second probability value of the image to be identified belonging to each object category is determined, and based on the first probability values ​​and the second probability values, the object category to which the image to be identified belongs is determined, thereby improving the recognition accuracy of multiple classifications of the image to be identified.

[0175] The present application also provides an image recognition device. The structural diagram of the image recognition device is as follows: Fig.12 As shown, the image recognition device 90 includes a first processing module 901 , a second processing module 902 , a third processing module 903 , a fourth processing module 904 , a fifth processing module 905 and a sixth processing module 906 .

[0176] The first processing module 901 is used to obtain an image to be identified;

[0177] A second processing module 902 is used to determine a first feature map and a second feature map corresponding to the image to be recognized, wherein the resolution of the first feature map is smaller than the resolution of the second feature map;

[0178] A third processing module 903 is used to determine a first probability value that the image to be identified belongs to each object category and an activation map corresponding to each object category based on the first feature map and each object category in the preset object category set;

[0179] The fourth processing module 904 is used to perform feature fusion processing between the second feature map and the activation map of each object category to determine a feature map after feature fusion; and determine at least one candidate region of interest of the image to be identified based on the activation map of each object category;

[0180] A fifth processing module 905, configured to determine a second probability value that the image to be identified belongs to each object category based on the feature map after feature fusion and at least one candidate region of interest;

[0181] The sixth processing module 906 is used to determine the object category to which the to-be-recognized image belongs based on each first probability value and each second probability value.

[0182] In one embodiment, the second processing module 902 is specifically configured to:

[0183] Inputting the image to be recognized into the feature extraction model of the first neural network, extracting the second feature map from the fourth convolution block of the feature extraction model, and extracting the first feature map from the fifth convolution block of the feature extraction model;

[0184] Among them, the feature extraction model includes a first-layer convolution block, a second-layer convolution block, a third-layer convolution block, a fourth-layer convolution block and a fifth-layer convolution block, and there is a cascade relationship between the first-layer convolution block, the second-layer convolution block, the third-layer convolution block, the fourth-layer convolution block and the fifth-layer convolution block.

[0185] In one embodiment, the third processing module 903 is specifically configured to:

[0186] The first feature map is input into the first fully connected layer and the maximum pooling layer of the first neural network, and classification processing is performed based on each object category in the object category set to obtain a first probability value corresponding to each object category in the image to be identified.

[0187] In one embodiment, the third processing module 903 is specifically configured to:

[0188] Performing dimensionality reduction processing on the first feature map to determine a feature map after dimensionality reduction, wherein the dimension of the feature map after dimensionality reduction is the same as the number of object categories in the object category set;

[0189] The reduced feature map is input into the batch normalization layer of the first neural network for batch normalization to determine the activation map corresponding to each object category.

[0190] In one embodiment, the fourth processing module 904 is specifically configured to:

[0191] Inputting the second feature map and the activation map of each object category into the feature fusion model of the first neural network, upsampling and linear interpolation processing are performed on the activation map of each object category to obtain a third feature map;

[0192] The second feature map and the third feature map are bitwise summed to obtain a feature map after feature fusion.

[0193] In one embodiment, the fourth processing module 904 is specifically configured to:

[0194] Inputting the activation map of each object category into the candidate region of interest selection model of the first neural network, filtering out the background in the activation map of each object category, and obtaining the activation map of each object category after filtering out the background;

[0195] The first probability values ​​are sorted from large to small, and an activation map corresponding to at least one first probability value in the front order after background removal is subjected to edge extraction processing to obtain at least one candidate region of interest of the image to be identified.

[0196] In one embodiment, the fifth processing module 905 is specifically configured to:

[0197] Inputting the feature fused feature map and at least one candidate region of interest into a candidate region of interest pooling layer of the first neural network, and performing cropping processing on the feature fused feature map to obtain a feature map of the candidate region of interest;

[0198] Based on the feature map of the candidate region of interest, a second probability value that the image to be identified belongs to each object category is determined.

[0199] In one embodiment, the sixth processing module 906 is specifically configured to:

[0200] For an object category in the object category set, if an average value between a first probability value corresponding to an object category and a second probability value corresponding to an object category is not less than a preset category threshold, it is determined that an object category exists in the image to be identified.

[0201] In one embodiment, the first processing module 901 is further configured to:

[0202] Input the training sample into the second neural network to determine the value of the first loss function of the global branch prediction model and the activation map of each object category in the training sample; the second neural network includes a global branch prediction model, a local branch prediction model and a weak supervision model, the global branch prediction model includes a feature extraction model, a first fully connected layer and a maximum pooling layer, and the local branch prediction model includes a feature fusion model, a candidate region of interest selection model, a batch normalization layer, a candidate region of interest pooling layer and a second fully connected layer;

[0203] Based on inputting the activation map of each object category in the training sample into the weak supervision model, suppressing the noise of the activation map of each object category in the training sample, and determining the value of the second loss function of the weak supervision model;

[0204] Input the activation map of each object category in the training sample into the feature fusion model and the candidate region of interest selection model respectively, and determine the value of the third loss function of the local branch prediction model;

[0205] Based on the value of the first loss function, the value of the second loss function, and the value of the third loss function, updating the parameters of the second neural network;

[0206] If the sum of the values ​​of the first loss function, the second loss function and the third loss function is less than a preset loss threshold, the training of the second neural network is terminated; and based on the second neural network obtained through training, the first neural network is determined, and the first neural network does not include a weak supervision model.

[0207] The application of the embodiments of the present application has at least the following beneficial effects:

[0208] The feature map after feature fusion contains multiple scales, that is, it has high-level semantic understanding features and certain low-level image texture features at the same time; based on the activation map of each object category, at least one candidate region of interest of the image to be identified is determined, thereby focusing on the key areas in the image to be identified and improving the recognition accuracy of the candidate region of interest; based on the feature map after feature fusion and at least one candidate region of interest, the second probability value of the image to be identified belonging to each object category is determined, and based on each first probability value and each second probability value, the object category to which the image to be identified belongs is determined, thereby improving the recognition accuracy of multiple classifications (multiple labels) of the image to be identified.

[0209] The present application also provides an electronic device. The structural diagram of the electronic device is as follows: Fig.13 As shown, Fig.13The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0210] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0211] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.13 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0212] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0213] The memory 4003 is used to store the computer program for executing the embodiment of the present application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the above method embodiment.

[0214] Among them, electronic equipment includes but is not limited to: servers, etc.

[0215] The application of the embodiments of the present application has at least the following beneficial effects:

[0216] The feature map after feature fusion contains multiple scales, that is, it has high-level semantic understanding features and certain low-level image texture features at the same time; based on the activation map of each object category, at least one candidate region of interest of the image to be identified is determined, thereby focusing on the key areas in the image to be identified and improving the recognition accuracy of the candidate region of interest; based on the feature map after feature fusion and at least one candidate region of interest, the second probability value of the image to be identified belonging to each object category is determined, and based on each first probability value and each second probability value, the object category to which the image to be identified belongs is determined, thereby improving the recognition accuracy of multiple classifications (multiple labels) of the image to be identified.

[0217] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0218] The embodiment of the present application also provides a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiment when executed by a processor.

[0219] Based on the same principle as the method provided in the embodiment of the present application, the embodiment of the present application further provides a computer program product or a computer program, which includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in any optional embodiment of the present application.

[0220] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the implementation order of these steps is not limited to the order indicated by the arrows. Unless clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages may be executed at the same time, and each sub-step or stage in these sub-steps or stages may also be executed at different times respectively. In different scenarios of execution time, the execution order of these sub-steps or stages may be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0221] The above is only an optional implementation method for some implementation scenarios of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of the present application, other similar implementation methods based on the technical ideas of the present application are also within the protection scope of the embodiments of the present application.

Claims

1. An image recognition method, characterized in that: include: Obtain an image to be recognized; Determine a first feature map and a second feature map corresponding to the image to be recognized, wherein a resolution of the first feature map is smaller than a resolution of the second feature map; Based on the first feature map and each object category in a preset object category set, determining a first probability value that the image to be identified belongs to each object category, and an activation map corresponding to each object category; Inputting the second feature map and the activation map of each object category into the feature fusion model of the first neural network, performing upsampling and linear interpolation processing on the activation map of each object category to obtain a third feature map; and performing bitwise sum processing between the second feature map and the third feature map to obtain a feature map after feature fusion; and determining at least one candidate region of interest of the image to be recognized based on the activation map of each object category; Determining a second probability value of the image to be identified belonging to each object category based on the feature map after the feature fusion and the at least one candidate region of interest; Based on each first probability value and each second probability value, the object category to which the image to be identified belongs is determined.

2. The method according to claim 1, characterized in that: The determining the first feature map and the second feature map corresponding to the image to be recognized includes: Inputting the image to be recognized into the feature extraction model of the first neural network, extracting a second feature map from the fourth convolution block of the feature extraction model, and extracting a first feature map from the fifth convolution block of the feature extraction model; Among them, the feature extraction model includes a first layer convolution block, a second layer convolution block, a third layer convolution block, the fourth layer convolution block and the fifth layer convolution block, and there is a cascade relationship between the first layer convolution block, the second layer convolution block, the third layer convolution block, the fourth layer convolution block and the fifth layer convolution block.

3. The method according to claim 1, characterized in that The determining, based on the first feature map and each object category in a preset object category set, a first probability value that the image to be identified belongs to each object category includes: The first feature map is input into the first fully connected layer and the maximum pooling layer of the first neural network, and classification processing is performed based on each object category in the object category set to obtain a first probability value corresponding to the existence of each object category in the image to be identified.

4. The method according to claim 1, characterized in that: The determining, based on the first feature map and each object category in a preset object category set, an activation map corresponding to each object category includes: Performing dimensionality reduction processing on the first feature map to determine a feature map after dimensionality reduction, wherein the dimension of the feature map after dimensionality reduction is the same as the number of object categories in the object category set; The feature map after dimensionality reduction is input into the batch normalization layer of the first neural network to perform batch normalization processing to determine the activation map corresponding to each object category.

5. The method according to claim 1, characterized in that The determining, based on the activation map of each object category, at least one candidate region of interest of the image to be recognized comprises: Inputting the activation map of each object category into the candidate region of interest selection model of the first neural network, filtering out the background in the activation map of each object category, and obtaining the activation map of each object category after filtering out the background; The first probability values ​​are sorted from large to small, and an activation map corresponding to at least one first probability value in the front order after background removal is subjected to edge extraction processing to obtain at least one candidate region of interest of the image to be identified.

6. The method according to claim 1, characterized in that The determining, based on the feature map after the feature fusion and the at least one candidate region of interest, a second probability value of the image to be identified belonging to each object category includes: Inputting the feature map after the feature fusion and the at least one candidate region of interest into the candidate region of interest pooling layer of the first neural network, and performing cropping processing on the feature map after the feature fusion to obtain a feature map of the candidate region of interest; Based on the feature map of the candidate region of interest, a second probability value that the image to be identified belongs to each object category is determined.

7. The method according to claim 1, characterized in that The determining, based on each first probability value and each second probability value, the object category to which the image to be identified belongs includes: For an object category in the object category set, if an average value between a first probability value corresponding to the object category and a second probability value corresponding to the object category is not less than a preset category threshold, it is determined that the object category exists in the image to be identified.

8. The method according to claim 1, characterized in that Before acquiring the image to be identified, the method further includes: Input the training sample into the second neural network to determine the value of the first loss function of the global branch prediction model and the activation map of each object category in the training sample; the second neural network includes a global branch prediction model, a local branch prediction model and a weak supervision model, the global branch prediction model includes a feature extraction model, a first fully connected layer and a maximum pooling layer, and the local branch prediction model includes a feature fusion model, a candidate region of interest selection model, a batch normalization layer, a candidate region of interest pooling layer and a second fully connected layer; Based on inputting the activation map of each object category in the training sample into the weak supervision model, suppressing the noise of the activation map of each object category in the training sample, and determining the value of the second loss function of the weak supervision model; Inputting the activation map of each object category in the training sample into the feature fusion model and the candidate region of interest selection model respectively, and determining the value of the third loss function of the local branch prediction model; Based on the value of the first loss function, the value of the second loss function, and the value of the third loss function, updating the parameters of the second neural network; If the sum of the values ​​of the first loss function, the second loss function and the third loss function is less than a preset loss threshold, the training of the second neural network is terminated; and based on the second neural network obtained through training, the first neural network is determined, and the first neural network does not include the weakly supervised model.

9. An image recognition device, characterized in that: include: A first processing module, used for acquiring an image to be identified; A second processing module, used to determine a first feature map and a second feature map corresponding to the image to be recognized, wherein a resolution of the first feature map is smaller than a resolution of the second feature map; a third processing module, configured to determine, based on the first feature map and each object category in a preset object category set, a first probability value that the image to be identified belongs to each object category, and an activation map corresponding to each object category; a fourth processing module, configured to input the second feature map and the activation map of each object category into a feature fusion model of the first neural network, perform upsampling and linear interpolation processing on the activation map of each object category to obtain a third feature map; and perform bitwise sum processing between the second feature map and the third feature map to obtain a feature map after feature fusion; and determining at least one candidate region of interest of the image to be recognized based on the activation map of each object category; A fifth processing module, configured to determine a second probability value that the image to be identified belongs to each object category based on the feature map after the feature fusion and the at least one candidate region of interest; The sixth processing module is used to determine the object category to which the image to be identified belongs based on the first probability values ​​and the second probability values.

10. The image recognition device according to claim 9, characterized in that: The second processing module is specifically used for: Inputting the image to be recognized into the feature extraction model of the first neural network, extracting a second feature map from the fourth convolution block of the feature extraction model, and extracting a first feature map from the fifth convolution block of the feature extraction model; Among them, the feature extraction model includes a first layer convolution block, a second layer convolution block, a third layer convolution block, the fourth layer convolution block and the fifth layer convolution block, and there is a cascade relationship between the first layer convolution block, the second layer convolution block, the third layer convolution block, the fourth layer convolution block and the fifth layer convolution block.

11. The image recognition device according to claim 9, characterized in that: The third processing module is specifically used for: The first feature map is input into the first fully connected layer and the maximum pooling layer of the first neural network, and classification processing is performed based on each object category in the object category set to obtain a first probability value corresponding to the existence of each object category in the image to be identified.

12. The image recognition device according to claim 9, characterized in that: The third processing module is specifically used for: Performing dimensionality reduction processing on the first feature map to determine a feature map after dimensionality reduction, wherein the dimension of the feature map after dimensionality reduction is the same as the number of object categories in the object category set; The feature map after dimensionality reduction is input into the batch normalization layer of the first neural network to perform batch normalization processing to determine the activation map corresponding to each object category.

13. The image recognition device according to claim 9, characterized in that: The fourth processing module is specifically used for: Inputting the activation map of each object category into the candidate region of interest selection model of the first neural network, filtering out the background in the activation map of each object category, and obtaining the activation map of each object category after filtering out the background; The first probability values ​​are sorted from large to small, and an activation map corresponding to at least one first probability value in the front order after background removal is subjected to edge extraction processing to obtain at least one candidate region of interest of the image to be identified.

14. The image recognition device according to claim 9, characterized in that: The fifth processing module is specifically used for: Inputting the feature map after the feature fusion and the at least one candidate region of interest into the candidate region of interest pooling layer of the first neural network, and performing cropping processing on the feature map after the feature fusion to obtain a feature map of the candidate region of interest; Based on the feature map of the candidate region of interest, a second probability value that the image to be identified belongs to each object category is determined.

15. The image recognition device according to claim 9, characterized in that: The sixth processing module is specifically used for: For an object category in the object category set, if an average value between a first probability value corresponding to the object category and a second probability value corresponding to the object category is not less than a preset category threshold, it is determined that the object category exists in the image to be identified.

16. The image recognition device according to claim 9, characterized in that: The first processing module is further used for: Input the training sample into the second neural network to determine the value of the first loss function of the global branch prediction model and the activation map of each object category in the training sample; the second neural network includes a global branch prediction model, a local branch prediction model and a weak supervision model, the global branch prediction model includes a feature extraction model, a first fully connected layer and a maximum pooling layer, and the local branch prediction model includes a feature fusion model, a candidate region of interest selection model, a batch normalization layer, a candidate region of interest pooling layer and a second fully connected layer; Based on inputting the activation map of each object category in the training sample into the weak supervision model, suppressing the noise of the activation map of each object category in the training sample, and determining the value of the second loss function of the weak supervision model; Inputting the activation map of each object category in the training sample into the feature fusion model and the candidate region of interest selection model respectively, and determining the value of the third loss function of the local branch prediction model; Based on the value of the first loss function, the value of the second loss function, and the value of the third loss function, updating the parameters of the second neural network; If the sum of the values ​​of the first loss function, the second loss function and the third loss function is less than a preset loss threshold, the training of the second neural network is terminated; and based on the second neural network obtained through training, the first neural network is determined, and the first neural network does not include the weakly supervised model.

17. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Citation Information

Patent Citations

  • Image scene recognition method and device

    CN105809146A

  • Fundus OCT image recognition method and device

    CN110555845A

  • Human body attribute recognition method and device, electronic equipment and storage medium

    CN110991380A

  • Image recognition method and device, electronic equipment and storage medium

    CN111291819A

  • Video classification method and device, electronic equipment and storage medium

    CN112131978A