Method for determining region of interest, method and device for identifying image content
By fusing the feature information between candidate boxes in the region proposal network, using Transformer model or graph convolution operation, the problem of not considering image feature connections in the candidate box feature processing in the prior art is solved, and the determination of the region of interest and Accuracy of image content recognition.
Patent Information
- Application Number
- CN202011582717.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-28
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2040-12-28
AI Technical Summary
The existing area proposal network does not consider the image feature connection between candidate boxes when processing the candidate box feature, resulting in insufficient accuracy in determining the region of interest and image content recognition.
The feature map of the image to be identified is obtained through the convolution neural network, the area proposal network is used to determine the candidate box, and the feature information of each candidate box is fused with the feature information of other candidate boxes, and the feature information is used to feature using the Transformer model or graph convolution operation. Weighted fusion, adjust boundaries and perform non-maximum suppression screening to determine the region of interest.
The richness and recognition accuracy of candidate box features are improved, and the accuracy of subsequent area of interest determination and image content recognition is enhanced.
Smart Images

Figure CN112597997B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a method for determining a region of interest, and a method and device for recognizing image content. Background Art
[0002] Image content recognition is a technology that uses machine learning models to identify and label preset content on an image. In some image content recognition technologies, in order to improve the speed of image content recognition, a region proposal network (RPN) is often used to first determine some regions of interest where content that needs to be recognized may exist on the image to be recognized, and then the image features of these regions of interest are input into the subsequent classification network for recognition to determine the categories of the image content in these regions of interest. The region proposal network usually first determines some candidate boxes from the feature map of the image to be recognized, and then determines the region of interest and obtains the image features in the region of interest based on the image features in the candidate boxes.
[0003] However, in the current region proposal network, when processing the features of the candidate boxes, usually only the image features of each candidate box are considered, without considering the possible connections between the image features of the candidate boxes. As a result, the image features corresponding to each candidate feature box are not rich enough, which affects the subsequent determination of the region of interest and the accuracy of image content recognition. Summary of the invention
[0004] In order to overcome the above-mentioned deficiencies in the prior art, one of the purposes of the present application is to provide a method for determining a region of interest, the method comprising:
[0005] Acquire a feature map of the image to be identified, wherein the feature map is obtained by extracting features of the image to be identified using a convolutional neural network;
[0006] Determine multiple candidate boxes from the feature map through a convolutional layer of a region proposal network, and obtain first feature information corresponding to each of the candidate boxes;
[0007] For each candidate frame, the first feature information of the candidate frame is fused with the first feature information of other candidate frames to obtain the second feature information of the candidate frame;
[0008] According to the second feature information of each candidate frame, performing boundary adjustment and binary classification on each candidate frame;
[0009] Performing non-maximum suppression screening processing on the candidate boxes determined as foreground in the binary classification;
[0010] The screened candidate frames are determined as regions of interest and second feature information corresponding to each region of interest is obtained.
[0011] In a possible implementation, the step of, for each candidate box, fusing the first feature information of the candidate box with the first feature information of other candidate boxes includes:
[0012] For each candidate frame, based on the relative position relationship between the other candidate frames and the candidate frame, the first feature information of the candidate frame is fused with the first feature information of other candidate frames.
[0013] In a possible implementation, the step of, for each candidate frame, fusing the first feature information of the candidate frame with the first feature information of other candidate frames according to the relative position relationship between the other candidate frames and the candidate frame, includes:
[0014] For each candidate box, input the position and first feature information of the candidate box and other candidate boxes into an encoder of a Transformer model;
[0015] The encoder performs weighted fusion on the first feature information of the candidate box and other candidate boxes according to the position relationship to obtain the second feature information of the candidate box.
[0016] In a possible implementation, the encoder includes a plurality of sequentially connected encoding layers, each of the encoding layers includes a multi-head self-attention layer, a first normalization layer, a first residual fusion threshold mechanism, a forward propagation network, a second normalization layer and a second residual fusion threshold mechanism; the step of performing weighted fusion of the first feature information of the candidate box and other candidate boxes according to the position relationship through the encoder includes:
[0017] For each of the coding layers, obtaining input data from the convolutional layer or the previous coding layer;
[0018] Performing weighted fusion on the input data through the multi-head self-attention layer;
[0019] Inputting the output result of the multi-head self-attention layer into the first normalization layer for normalization processing;
[0020] Inputting the input data and the output result of the first normalization layer into the first residual fusion threshold mechanism, and performing residual fusion processing on the first global feature and the first local feature through the first residual fusion threshold mechanism;
[0021] Inputting the output result of the first residual fusion threshold mechanism into the fully connected forward propagation network for processing;
[0022] Inputting the output result of the forward propagation network into the second normalization layer for normalization processing;
[0023] Inputting the output result of the first residual fusion threshold mechanism and the output result of the second normalization layer into the second residual fusion threshold mechanism, and performing residual fusion processing of global features and local features through the second residual fusion threshold mechanism;
[0024] Output the processing result of the second residual fusion threshold mechanism.
[0025] In a possible implementation, the step of, for each candidate box, fusing the first feature information of the candidate box with the first feature information of other candidate boxes includes:
[0026] For each candidate box, according to the positional relationship between other candidate boxes and the candidate box, the first feature information of other candidate boxes and the candidate box are fused through graph convolution operation to obtain the second feature information of the candidate box.
[0027] Another object of the present application is to provide an image content recognition method, the method comprising:
[0028] Obtain an image to be recognized;
[0029] Extracting features of the image to be identified by preselecting a trained convolutional neural network, and generating a feature map of the image to be identified;
[0030] Determine the region of interest and the second feature information corresponding to the region of interest by the region of interest determination method provided in the present application;
[0031] Inputting the feature map, the position of each of the regions of interest, and the second feature information corresponding to each of the regions of interest into a region of interest pooling layer for processing to obtain third feature information corresponding to each of the regions of interest;
[0032] The third feature information of each of the regions of interest is input into a classification model to obtain an image content category corresponding to each region of interest.
[0033] Another object of the present application is to provide a device for determining a region of interest, the method comprising:
[0034] A feature map acquisition module is used to acquire a feature map of the image to be identified, wherein the feature map is obtained by extracting features of the image to be identified using a convolutional neural network;
[0035] A candidate box determination module, used to determine multiple candidate boxes from the feature map through a convolution layer, and obtain first feature information corresponding to each of the candidate boxes;
[0036] A feature fusion module is used to, for each candidate frame, fuse the first feature information of the candidate frame with the first feature information of other candidate frames to obtain the second feature information of the candidate frame;
[0037] A candidate box screening module, used for performing boundary adjustment and binary classification on each candidate box according to the second feature information of each candidate box; performing non-maximum suppression screening processing on the candidate box determined as the foreground in the binary classification;
[0038] The region of interest output module is used to determine the screened candidate frames as regions of interest and obtain second feature information corresponding to each region of interest.
[0039] Another object of the present application is to provide an image content recognition device, the device comprising:
[0040] An image acquisition module to be identified, used to acquire an image to be identified;
[0041] A feature map extraction module is used to extract features of the image to be identified by using a pre-selected trained convolutional neural network to obtain a feature map of the image to be identified;
[0042] An area of interest determination module, configured to determine an area of interest and second feature information corresponding to the area of interest through the area of interest determination device provided in the present application;
[0043] A pooling processing module, used for inputting the feature map, the position of each of the regions of interest and the second feature information corresponding to each of the regions of interest into a region of interest pooling layer for processing, so as to obtain third feature information corresponding to each of the regions of interest;
[0044] The classification and recognition module is used to input the third feature information of each of the regions of interest into a classification model to obtain the image content category corresponding to each region of interest.
[0045] Another object of the present application is to provide an electronic device, including a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by the processor, the method for determining a region of interest or the method for recognizing image content provided in the present application is implemented.
[0046] Another object of the present application is to provide a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are executed by one or more processors, they implement the region of interest determination method or image content recognition method provided in the present application.
[0047] Compared with the prior art, this application has the following beneficial effects:
[0048] The method for determining a region of interest, the method for recognizing image content, and the device provided in the embodiments of the present application take into account the degree of intrinsic correlation that may exist between candidate frames when performing feature extraction, and fuse the features between candidate frames so that the features of each candidate frame incorporate the features of other candidate frames, thereby making it more accurate to subsequently use the features in the candidate frames to determine the region of interest or recognize image content. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0050] Figure 1 Schematic diagram of the proposed network architecture for the existing area;
[0051] Figure 2 A schematic diagram of the architecture of an improved region proposal network provided in an embodiment of the present application;
[0052] Figure 3 A schematic diagram of the steps of the method for determining a region of interest provided in an embodiment of the present application;
[0053] Figure 4 This is a schematic diagram of the Transformer model encoder architecture;
[0054] Figure 5 A schematic diagram of the architecture of an improved Transformer encoder provided in an embodiment of the present application;
[0055] Figure 6 A schematic diagram of a step flow for performing feature fusion provided in an embodiment of the present application;
[0056] Figure 7 A flowchart of an image content recognition method provided in an embodiment of the present application;
[0057] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;
[0058] Fig. 9 A schematic diagram of the functional modules of the device for determining a region of interest provided in an embodiment of the present application;
[0059] Fig.10A schematic diagram of the functional modules of the image content recognition device provided in an embodiment of the present application.
[0060] Icons: 100-electronic device; 120-machine-readable storage medium; 130-processor; 140-region of interest determination device; 141-feature map acquisition module; 142-candidate window determination module; 143-feature fusion module; 144-candidate window screening module; 145-region of interest output module; 150-image content recognition device; 151-to-be-recognized image acquisition module; 152-feature map extraction module; 153-region of interest determination module; 154-pooling processing module; 155-classification and recognition module. DETAILED DESCRIPTION
[0061] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.
[0062] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.
[0063] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0064] In the description of the present application, it should be noted that the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0065] In the description of this application, it should also be noted that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.
[0066] Please refer to Figure 1 , Figure 1It is a Faster-RCNN model used in some image content recognition technologies. The model usually includes a region proposal network, which is used to determine the areas where objects may exist on the image to be identified, determine candidate frames based on these areas, and perform foreground and background recognition screening, overlap screening and other actions based on the image features in the candidate frames. Recently, the screened candidate frames are used as regions of interest, and the features in the regions of interest are output to the subsequent parts of the Faster-RCNN model for further classification and recognition.
[0067] It is worth noting that in this implementation, the region proposal network mainly performs relatively independent recognition and feature extraction on a certain part of the image to be identified. That is to say, after the candidate frame is determined, the image features in each candidate frame can only represent the independent image features of a certain part of the image to be identified. This will result in that when the image features in a candidate frame are used to screen the region of interest and classify the image content, only the image features in the candidate frame will be considered.
[0068] The inventors have discovered through research that in actual image recognition, there may be some potential connections between the image features in each candidate frame. If the features in other related candidate frames are comprehensively considered when recognizing a certain candidate frame, the enhanced candidate frame can be enhanced, thereby improving the accuracy of subsequent recognition.
[0069] In view of this, the present embodiment provides a method for determining a region of interest, a method and apparatus for identifying image content, and for each candidate frame, the image features of the candidate frame are fused with the image features of other candidate frames, so that the image features of each candidate frame are integrated with the image features of other candidate frames, and the image features of each candidate frame can be used to identify or screen image content more accurately. The solution provided by the present embodiment is described in detail below.
[0070] This embodiment provides a method for determining an area of interest. Figure 2 This method can be regarded as a data processing process of an improved region proposal network provided in this embodiment. Please refer to Figure 3 , each step of the method is described in detail below.
[0071] Step S110, obtaining a feature map of the image to be identified, wherein the feature map is obtained by performing feature extraction on the image to be identified using a convolutional neural network.
[0072] In this embodiment, the feature map is obtained by extracting features from the image to be identified using a convolutional neural network. For example, in one possible implementation, when the region of interest determination method provided in this embodiment is used in a Faster-RCNN model, the convolutional neural network may be the first convolutional neural network of the Faster-RCNN model.
[0073] Step S120, determining multiple candidate boxes from the feature map through the convolutional layer of the region proposal network, and obtaining first feature information corresponding to each of the candidate boxes.
[0074] In this embodiment, the Figure 2 The convolution layer shown determines multiple candidate boxes according to the input feature map, and obtains the first feature information corresponding to each candidate box. The multiple candidate boxes and their first feature information can be obtained by the convolution layer according to the anchor mechanism.
[0075] Step S130: for each candidate box, the first feature information of the candidate box is fused with the first feature information of other candidate boxes to obtain the second feature information of the candidate box.
[0076] Please refer again Figure 2 In the improved region proposal network provided in this embodiment, a feature fusion model may be included, and the feature fusion model may be trained to fuse the features of a candidate box with the features of other candidate boxes.
[0077] For example, for each candidate box, the first feature information of the candidate box can be fused with the first feature information of other candidate boxes according to the degree of association between the other candidate boxes and the candidate box. In the second feature information of a candidate box, the feature information from other candidate boxes with a higher degree of association with it has a greater proportion, thereby having a greater impact on the subsequent recognition and screening of the candidate box.
[0078] Step S140 , performing boundary adjustment and binary classification on each candidate box according to the second feature information of each candidate box.
[0079] Please refer again Figure 2 In the improved region proposal network provided in this embodiment, the binary classification model can be pre-trained to determine whether the corresponding image content in each candidate frame belongs to the foreground image or the background image according to the second feature information of each candidate frame; the boundary adjustment model can be pre-trained to adjust the size and / or position of each candidate frame according to the second feature information of each candidate frame, so that the size and / or position of each candidate frame can more accurately cover the image content on the image to be identified.
[0080] Step S150, performing non-maximum suppression screening processing on the candidate boxes determined as foregrounds in the binary classification, determining the screened candidate boxes as regions of interest and obtaining second feature information corresponding to each region of interest.
[0081] Please refer again Figure 2 In the improved region proposal network provided in this embodiment, the non-maximum suppression model can be pre-trained to screen the overlapping candidate frames according to the classification confidence in the candidate frames determined as foreground, so as to screen out the best window covering the specific image content. The screened candidate frames can be used as regions of interest, and the positions of these regions of interest in the image to be identified together with the second feature information in these regions of interest can be output to other processing models for further image content recognition.
[0082] Based on the above design, in the method for determining the region of interest provided in the present embodiment, when extracting features in the candidate frames, the features between the candidate frames are fused, so that each candidate frame incorporates the features of other candidate frames, and associations with the features in other candidate frames are established, thereby improving the richness of information expressed by the features in the candidate frames, thereby improving the accuracy of subsequent determination of the region of interest based on the feature information in the candidate frames and classification and identification of the image content in the region of interest.
[0083] In a possible implementation, for each candidate box, first feature information of the candidate box is fused with first feature information of other candidate boxes according to the degree of association between the first feature information of the candidate box and other candidate boxes.
[0084] Through a pre-training process or pre-configuration, the feature fusion model can be trained or configured to obtain the degree of association between the first feature information of a candidate frame and other candidate frames, and the degree of association can represent the potential connection between the features behind the candidate windows mined in the pre-model training process. Then, according to the degree of association, the first feature information of other candidate frames is weightedly summed with the first feature information of the candidate frame, so as to obtain the second feature information of the candidate frame.
[0085] For example, when performing feature fusion in step S130, the encoder of the Transformer model may be used to perform feature fusion. For example, for each candidate box, the first feature information of the candidate box and other candidate boxes is input into the encoder of the Transformer model. Then, the encoder performs weighted fusion based on the correlation degree of the first feature information on the first feature information of the candidate box and other candidate boxes to obtain the second feature information of the candidate box.
[0086] The Transformer model is a combination of multiple parallel self-attention mechanisms. The self-attention mechanism mines the connections between features and converts a single feature into a weighted sum of the feature and other features based on the similarity between the features. A single self-attention mechanism is equivalent to mining the connections between features. The Transformer model is a collection of multiple parallel self-attention mechanisms, which is equivalent to mining the connections between features multiple times. The interaction between candidate box features is completed through the Transformer, and each candidate box incorporates the features of other candidate boxes, thereby realizing the feature enhancement process.
[0087] In this embodiment, the encoder part of the Transformer model can be mainly used.
[0088] Please refer to Figure 4 , Figure 4 The encoder structure of a Transformer model may include N encoding layers (for example, 6 encoding layers), each encoding layer mainly includes a multi-head self-attention mechanism, a fully connected forward propagation network, and some data processing layers that perform normalization and summation.
[0089] Among them, the basic building block of the multi-head self-attention mechanism is the attention model. The output result of the attention model, attention_output, is as follows
[0090] attention_output=Attention(Q,K,V) Formula (1)
[0091]
[0092] Among them, Q, K, and V are the first feature information of different candidate boxes, d k is the dimension for spatial mapping of Q, K, and V. The attention distribution can be obtained by calculating the correlation weight between Q and K based on the degree of similarity. After obtaining the attention distribution, the first feature information of the candidate box and the corresponding attention are weighted and summed to obtain the corresponding output.
[0093] The multi-head self-attention mechanism projects Q, K, and V through h different linear transformations, and finally concatenates the different attention results:
[0094] MultiHead(Q,K,V)=Concat(head 1 ,head 2 ,……,head h )W 0Formula (3)
[0095]
[0096] Among them, W 0 is the parameter for linear transformation, i = 1, 2, ..., h. The multi-head attention mechanism can realize the feature fusion between candidate boxes, and the fully connected forward propagation network is mainly used to provide nonlinear transformation of feature data.
[0097] Optionally, this embodiment also provides an encoder of an improved Transformer model, please refer to Figure 5 The encoder of the improved Transformer model may include a multi-head self-attention layer, a first normalization layer, a first residual fusion threshold mechanism, a forward propagation network, a second normalization layer, and a second residual fusion threshold mechanism. Please refer to Figure 6 The steps for the encoder of the improved Transformer model to perform feature fusion are as follows.
[0098] Step S310: For each of the coding layers, obtain input data from the convolutional layer or the previous coding layer.
[0099] Step S320, weighted fusion of the input data is performed through a multi-head self-attention layer.
[0100] Step S330, input the output result of the multi-head self-attention layer to the first normalization layer for normalization processing.
[0101] Step S340: input the input data and the output result of the first normalization layer into the first residual fusion threshold mechanism, and perform residual fusion processing of global features and local features through the first residual fusion threshold mechanism.
[0102] Step S350: input the output result of the first residual fusion threshold mechanism into a fully connected forward propagation network for processing.
[0103] Step S360: input the output result of the forward propagation network to the second normalization layer for normalization processing.
[0104] Step S370: input the output result of the first residual fusion threshold mechanism and the output result of the second normalization layer into the second residual fusion threshold mechanism, and perform residual fusion processing of global features and local features through the second residual fusion threshold mechanism.
[0105] Step S380: output the processing result of the second residual fusion threshold mechanism.
[0106] In other words, in this embodiment, a residual fusion threshold mechanism can be used to replace the simple summation mechanism in the traditional Transformer model, thereby realizing the fusion of global features and local features, which can further improve the feature extraction capability of the model and make subsequent classification and recognition based on the extracted features more accurate.
[0107] In another possible implementation, a graph convolution operation may also be used in step S130 to achieve feature fusion between candidate boxes. For example, for each candidate box, based on the association between other candidate boxes and the candidate box, the first feature information of the candidate box is fused with other candidate boxes through a graph convolution operation to obtain the second feature information of the candidate box.
[0108] Specifically, in the graph convolution operation, each candidate box is regarded as a node in the graph, and the edge relationship between the candidate boxes is obtained through some pre-set rules or pre-machine learning training (the edge indicates whether there is a relationship between the two candidate boxes, the edge means there is a relationship, and the absence of the edge means there is no relationship). The edge between nodes is also called the adjacency matrix between nodes, usually denoted as A. By determining the nodes and edges, the shape of the graph is obtained; after the shape of the graph is obtained, the graph is operated using graph convolution, which aggregates the information of the adjacent nodes of each node into the node, thereby completing the feature fusion process. The expression of graph convolution is as follows:
[0109] H L+1 =σ(AH L W L ) Formula (5)
[0110] Among them, A represents the pre-set adjacency matrix between nodes, H L represents the Lth layer of graph convolution, H L+1 Represents the L+1th layer of the graph convolution. Assuming there are N nodes in the graph (that is, there are N candidate boxes in the image), and the feature of each node is a D-dimensional vector, then H l and H L+1 are all N*D matrices; A is an N*N matrix. L represents the network parameters between the Lth layer and the L+1th layer. σ(·) is a nonlinear activation function, such as RELU.
[0111] The method for determining the region of interest provided in this embodiment can be used in Faster-RCNN to determine the region of interest and obtain features in the region of interest. For example, this embodiment also provides an image content recognition method, and each step of the method can be regarded as Figure 7 The data processing process of the improved Fster-RCNN model is shown in Figure 2. Figure 7 The image content recognition method provided in this embodiment may include the following steps.
[0112] Step S410, obtaining an image to be recognized.
[0113] Step S420: extracting features of the image to be identified by preselecting a trained convolutional neural network to generate a feature map of the image to be identified.
[0114] Step S430: The region of interest and second feature information corresponding to the region of interest are determined through the region of interest determination method provided by this embodiment.
[0115] In this embodiment, the improved region proposal network provided in this embodiment can be used in the Faster-RCNN model. The structure and data process of the improved region proposal network can be found in Figure 1 and Figure 2 The relevant description will not be repeated here.
[0116] Step S440: Input the feature map, the position of each of the regions of interest, and the second feature information corresponding to each of the regions of interest into a region of interest pooling layer for processing to obtain third feature information corresponding to each of the regions of interest.
[0117] Step S450: input the third feature information of each of the regions of interest into a classification model to obtain the image content category corresponding to each region of interest.
[0118] Since the region of interest determined by the region of interest determination method of this embodiment and its corresponding second feature information are integrated with the feature information of other candidate regions, the second feature information of each region of interest is richer, so that the classification result of the Faster-RCNN model can be more accurate when performing classification and recognition based on these second feature information.
[0119] Please refer to Figure 8 , Figure 8 The hardware structure diagram of an electronic device 100 provided in this embodiment. The electronic device 100 may include a processor 130 and a machine-readable storage medium 120. The processor 130 and the machine-readable storage medium 120 may communicate via a system bus. In addition, the machine-readable storage medium 120 stores machine-executable instructions. By reading and executing the machine-executable instructions corresponding to the region of interest determination logic or the image content recognition logic in the machine-readable storage medium 120, the processor 130 may execute the region of interest determination method or the image content recognition method described above.
[0120] The machine-readable storage medium 120 mentioned herein may be any electronic, magnetic, optical or other physical storage device that may contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium 120 may be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as optical disk, DVD, etc.), or similar storage media, or a combination thereof.
[0121] Please refer to Fig. 9 This embodiment further provides a region of interest determination device 140, which includes at least one functional module that can be stored in the machine-readable storage medium 120 in the form of software. From a functional perspective, the region of interest determination device 140 may include a feature map acquisition module 141, a candidate frame determination module, a feature fusion module 143, a candidate frame screening module, and a region of interest output module 145.
[0122] The feature map acquisition module 141 is used to acquire a feature map of the image to be identified, and the feature map is obtained by performing feature extraction on the image to be identified using a convolutional neural network.
[0123] In this embodiment, the feature map acquisition module 141 can be used to perform Figure 3 As shown in step S110, for a detailed description of the feature map acquisition module 141, please refer to the description of step S110.
[0124] The candidate box determination module is used to determine multiple candidate boxes from the feature map through a convolution layer to obtain first feature information corresponding to each of the candidate boxes.
[0125] In this embodiment, the candidate window determination module 142 may be used to perform Figure 3 As shown in step S120 , for a detailed description of the candidate window determination module 142 , please refer to the description of step S120 .
[0126] The feature fusion module 143 is used to fuse the first feature information of each candidate frame with the first feature information of other candidate frames to obtain the second feature information of the candidate frame.
[0127] In this embodiment, the feature fusion module 143 can be used to perform Figure 3 As shown in step S130, for a detailed description of the feature fusion module 143, please refer to the description of step S130.
[0128] The candidate box screening module is used to perform boundary adjustment and binary classification on each candidate box according to the second feature information of each candidate box.
[0129] In this embodiment, the candidate window screening module 144 can be used to perform Figure 3 As shown in step S140 , for a detailed description of the candidate window screening module 144 , please refer to the description of step S140 .
[0130] The region of interest output module 145 is used to perform non-maximum suppression screening processing on the candidate boxes determined as foregrounds in the binary classification; determine the screened candidate boxes as regions of interest and obtain second feature information corresponding to each region of interest.
[0131] In this embodiment, the region of interest output module 145 can be used to perform Figure 3 As shown in step S150, for a detailed description of the region of interest output module 145, please refer to the description of step S150.
[0132] Please refer to Fig.10 This embodiment further provides an image content recognition device 150, which includes at least one functional module that can be stored in a machine-readable storage medium 120 in the form of software. Functionally, the image content recognition device 150 may include a to-be-recognized image acquisition module 151, a feature map extraction module 152, an area of interest determination module 153, a pooling processing module 154, and a classification recognition module 155.
[0133] The to-be-recognized image acquisition module 151 is used to acquire the to-be-recognized image.
[0134] The feature map extraction module 152 is used to extract features of the image to be identified by using a pre-selected trained convolutional neural network to obtain a feature map of the image to be identified.
[0135] The region of interest determination module 153 is used to determine a region of interest and second feature information corresponding to the region of interest through the region of interest determination device 140 provided in this embodiment.
[0136] The pooling processing module 154 is used to input the feature map, the position of each of the regions of interest and the second feature information corresponding to each of the regions of interest into the region of interest pooling layer for processing to obtain the third feature information corresponding to each of the regions of interest.
[0137] The classification recognition module 155 is used to input the third feature information of each of the regions of interest into a classification model to obtain the image content category corresponding to each region of interest.
[0138] To sum up, the region of interest determination method, image content recognition method and device provided in the embodiments of the present application, when performing feature extraction, take into account the degree of intrinsic correlation that may exist between candidate frames, and fuse the features between candidate frames, so that the features of each candidate frame incorporate the features of other candidate frames, thereby making it more accurate to use the features in the candidate frames to determine the region of interest or recognize image content in the subsequent process.
[0139] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0140] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
[0141] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0142] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0143] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A method for determining a region of interest, It is characterized in that The method comprises: Acquire a feature map of the image to be identified, wherein the feature map is obtained by extracting features of the image to be identified using a convolutional neural network; Determine multiple candidate boxes from the feature map through a convolutional layer of a region proposal network, and obtain first feature information corresponding to each of the candidate boxes; For each candidate frame, the first feature information of the candidate frame is fused with the first feature information of other candidate frames to obtain the second feature information of the candidate frame; According to the second feature information of each candidate frame, performing boundary adjustment and binary classification on each candidate frame; A non-maximum suppression screening process is performed on the candidate frames determined as foregrounds in the binary classification, the screened candidate frames are determined as regions of interest, and second feature information corresponding to each region of interest is obtained.
2. The method according to claim 1, It is characterized in that The step of fusing the first feature information of each candidate frame with the first feature information of other candidate frames comprises: For each candidate frame, the first feature information of the candidate frame is feature-fused with the first feature information of other candidate frames according to the degree of association between the first feature information of the candidate frame and other candidate frames.
3. The method according to claim 2, It is characterized in that The step of, for each candidate frame, fusing the first feature information of the candidate frame with the first feature information of other candidate frames according to the degree of association between the first feature information of the candidate frame and other candidate frames, comprises: For each candidate box, input the first feature information of the candidate box and other candidate boxes into an encoder of a Transformer model; The encoder performs weighted fusion on the first feature information of the candidate box and other candidate boxes based on the degree of association of the first feature information to obtain the second feature information of the candidate box.
4. The method according to claim 3, It is characterized in that The encoder includes a plurality of sequentially connected encoding layers, each of which includes a multi-head self-attention layer, a first normalization layer, a first residual fusion threshold mechanism, a forward propagation network, a second normalization layer and a second residual fusion threshold mechanism; the step of performing weighted fusion based on the correlation degree of the first feature information on the first feature information of the candidate box and other candidate boxes by the encoder to obtain the second feature information of the candidate box includes: For each of the coding layers, obtaining input data from the convolutional layer or the previous coding layer; Performing weighted fusion on the input data through the multi-head self-attention layer; Inputting the output result of the multi-head self-attention layer into the first normalization layer for normalization processing; Inputting the input data and the output result of the first normalization layer into the first residual fusion threshold mechanism, and performing residual fusion processing of global features and local features through the first residual fusion threshold mechanism; Inputting the output result of the first residual fusion threshold mechanism into the fully connected forward propagation network for processing; Inputting the output result of the forward propagation network into the second normalization layer for normalization processing; Inputting the output result of the first residual fusion threshold mechanism and the output result of the second normalization layer into the second residual fusion threshold mechanism, and performing residual fusion processing of global features and local features through the second residual fusion threshold mechanism; Output the processing result of the second residual fusion threshold mechanism.
5. The method according to claim 1, It is characterized in that The step of fusing the first feature information of each candidate frame with the first feature information of other candidate frames to obtain the second feature information of the candidate frame comprises: For each candidate box, according to the association relationship between other candidate boxes and the candidate box, the first feature information of other candidate boxes and the candidate box are fused through graph convolution operation to obtain the second feature information of the candidate box.
6. A method for identifying image content, It is characterized in that The method comprises: Obtain an image to be recognized; Extracting features of the image to be identified by preselecting a trained convolutional neural network, and generating a feature map of the image to be identified; Determine the region of interest and the second feature information corresponding to the region of interest by the region of interest determination method according to any one of claims 1 to 5; Inputting the feature map, the position of each of the regions of interest, and the second feature information corresponding to each of the regions of interest into a region of interest pooling layer for processing to obtain third feature information corresponding to each of the regions of interest; The third feature information of each of the regions of interest is input into a classification model to obtain an image content category corresponding to each region of interest.
7. A device for determining a region of interest, It is characterized in that The method comprises: A feature map acquisition module is used to acquire a feature map of the image to be identified, wherein the feature map is obtained by extracting features of the image to be identified using a convolutional neural network; A candidate box determination module, used to determine multiple candidate boxes from the feature map through a convolution layer, and obtain first feature information corresponding to each of the candidate boxes; A feature fusion module is used to, for each candidate frame, fuse the first feature information of the candidate frame with the first feature information of other candidate frames to obtain the second feature information of the candidate frame; A candidate box screening module, used for adjusting the boundaries and performing binary classification on each candidate box according to the second feature information of each candidate box; The region of interest output module is used to perform non-maximum suppression screening processing on the candidate boxes determined as foregrounds in the binary classification; determine the screened candidate boxes as regions of interest and obtain second feature information corresponding to each region of interest.
8. An image content recognition device, It is characterized in that The device comprises: An image acquisition module to be identified, used to acquire an image to be identified; A feature map extraction module is used to extract features of the image to be identified by using a pre-selected trained convolutional neural network to obtain a feature map of the image to be identified; An area of interest determination module, configured to determine an area of interest and second feature information corresponding to the area of interest by using the area of interest determination device according to claim 7; A pooling processing module, used for inputting the feature map, the position of each of the regions of interest and the second feature information corresponding to each of the regions of interest into a region of interest pooling layer for processing, so as to obtain third feature information corresponding to each of the regions of interest; The classification and recognition module is used to input the third feature information of each of the regions of interest into a classification model to obtain the image content category corresponding to each region of interest.
9. An electronic device, It is characterized in that The method comprises a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by the processor, the method according to any one of claims 1 to 6 is implemented.
10. A machine-readable storage medium, It is characterized in that The machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by one or more processors, the method according to any one of claims 1 to 6 is implemented.
