Content recognition methods, apparatus, devices, storage media, and computer program products
By extracting and stitching together global and local feature representations in image recognition, the problem of low content recognition accuracy in existing technologies is solved, achieving higher recognition precision.
Patent Information
- Application Number
- CN202210934770.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-04
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2042-08-04
AI Technical Summary
In existing technologies, matching content recognition based solely on global features of an image with keywords results in low accuracy, and there are instances where the target image does not match the keywords.
Extract the content feature representation corresponding to the target content in the image to be identified, perform pooling and feature downsampling, and combine the key point feature representation of the target key point to obtain the local feature representation, which is used to identify the target content in the image.
By combining global and local feature representations, the accuracy of content recognition is improved, local feature information is effectively extracted, and the precision of content recognition is enhanced.
Smart Images

Figure CN115272768B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of machine learning, and in particular to a content recognition method, apparatus, device, storage medium, and computer program product. Background Technology
[0002] With the continuous development of internet technology, users browse a large amount of multimedia content every day, including pictures, videos, and articles. Therefore, by determining the category information contained in multimedia content and thus determining the corresponding attribute information, it is possible to better meet the browsing needs of users in different scenarios. For example, in the image search scenario, after a user enters search keywords, images that match the keywords are selected from the image library and displayed to the user as search results.
[0003] In related technologies, deep learning models are typically used to extract global features corresponding to images to build a content search library. In image search scenarios, when a user enters search keywords, the global features that match the keywords are determined in the content search library, and the images corresponding to the matched global features are directly displayed to the user as search results.
[0004] However, in related technologies, the target image that matches the keyword is determined solely based on the global features of the image. Although the global features corresponding to the target image have a high degree of matching with the keyword, there are cases where the target image and the keyword do not match, resulting in low accuracy of content recognition. Summary of the Invention
[0005] This application provides a content recognition method, apparatus, device, storage medium, and computer program product, which can improve the accuracy of content recognition. The technical solution is as follows:
[0006] On the one hand, a content recognition method is provided, the method comprising:
[0007] Acquire an image to be identified, the image to be identified including target recognition content, the target recognition content corresponding to target key points in the image to be identified, the target key points being key points extracted based on the pixel distribution pattern in the image to be identified;
[0008] Extract the content feature representation from the image to be identified that corresponds to the target content to be identified;
[0009] The content feature representation is pooled to obtain a global feature representation; and the content feature representation is downsampled to obtain a candidate local feature representation.
[0010] The key point feature representation extracted from the target key points and the candidate local feature representation are concatenated to obtain the local feature representation;
[0011] The target content in the image to be identified is identified based on the global feature representation and the local feature representation to obtain the content identification result, which is used to indicate the category corresponding to the target content.
[0012] On the other hand, a content recognition device is provided, the device comprising:
[0013] An acquisition module is used to acquire an image to be recognized, wherein the image to be recognized includes target recognition content, the target recognition content corresponds to target key points in the image to be recognized, and the target key points are key points extracted based on the pixel distribution pattern in the image to be recognized.
[0014] The extraction module is used to extract the content feature representation in the image to be identified that corresponds to the target content to be identified;
[0015] The processing module is used to perform pooling processing on the content feature representation to obtain a global feature representation; and to perform feature downsampling on the content feature representation to obtain a candidate local feature representation;
[0016] The splicing module is used to splice the key point feature representations extracted from the target key points and the candidate local feature representations to obtain local feature representations;
[0017] The recognition module is used to recognize the target content in the image to be recognized based on the global feature representation and the local feature representation, and to obtain the content recognition result, which is used to indicate the category corresponding to the target content.
[0018] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the content recognition method as described in any of the embodiments of this application above.
[0019] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the content recognition method as described in any of the embodiments of this application above.
[0020] On the other hand, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the content recognition methods described in the above embodiments.
[0021] The beneficial effects of the technical solutions provided in this application include at least the following:
[0022] When the image to be identified contains target content, the content feature representation corresponding to the target content is extracted. Pooling is then performed on the content feature representation to obtain a global feature representation. Feature downsampling is then applied to the content feature representation. The resulting candidate local feature representation is then concatenated with the keypoint feature representation extracted from the target keypoints corresponding to the target content to obtain a local feature representation. Based on the global and local feature representations, the target content in the image to be identified is then identified, ultimately yielding the content recognition result. In other words, by concatenating the candidate local features obtained through feature downsampling of the content feature representation with the keypoint feature representation to obtain a local feature representation, effective information about local features in the content feature representation can be effectively extracted. This, combined with the target keypoints, results in a local feature representation. Utilizing both global and local features for content recognition in the image to be identified improves the accuracy of content recognition. Attached Figure Description
[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0024] Figure 1 This is a schematic diagram illustrating the content recognition method provided in an exemplary embodiment of this application.
[0025] Figure 2 This is a schematic diagram of an implementation environment provided by an exemplary embodiment of this application;
[0026] Figure 3 This is a flowchart of a content recognition method provided in an exemplary embodiment of this application;
[0027] Figure 4 This is a flowchart of a content recognition method provided in another exemplary embodiment of this application;
[0028] Figure 5This is a flowchart of a content recognition method provided in another exemplary embodiment of this application;
[0029] Figure 6 This is a schematic diagram of the target area provided in an exemplary embodiment of this application;
[0030] Figure 7 This is a schematic diagram of the target area provided in another exemplary embodiment of this application;
[0031] Figure 8 This is a schematic diagram of a saliency detection model provided in an exemplary embodiment of this application;
[0032] Figure 9 This is a schematic diagram of a content recognition method provided in another exemplary embodiment of this application;
[0033] Figure 10 This is a structural block diagram of a content recognition device provided in an exemplary embodiment of this application;
[0034] Figure 11 This is a structural block diagram of a content recognition device provided in another exemplary embodiment of this application;
[0035] Figure 12 This is a schematic diagram of a server structure provided in an exemplary embodiment of this application. Detailed Implementation
[0036] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.
[0037] This is illustrative; please refer to it. Figure 1 This illustrates a schematic diagram of a content recognition method provided in an exemplary embodiment of this application, such as... Figure 1 As shown, an image to be identified 110 is obtained. The image to be identified 110 is a scenic spot image. Therefore, the image to be identified 110 contains target recognition content 111. The target recognition content 111 includes target key points (not shown in the figure). The target key points are key points extracted according to the pixel distribution pattern in the image to be identified 110.
[0038] Extract the content feature representation 120 corresponding to the target recognition content 111 in the image 110 to be recognized, perform pooling on the content feature representation 120 to obtain the global feature representation 130, and perform feature downsampling on the content feature representation 120 to obtain the candidate local feature representation 140.
[0039] The candidate local feature representation 140 and the key point feature representation 1121 obtained by feature extraction of the target key points are concatenated to obtain the local feature representation 150. The target recognition content 111 in the image to be recognized 110 is recognized based on the global feature representation 130 and the local feature representation 150 to obtain the content recognition result 160, which is implemented as "A landscape building".
[0040] The implementation environment involved in the embodiments of this application is described in an illustrative manner. Please refer to the following examples. Figure 2 The implementation environment involves a terminal 210 and a server 220, which are connected via a communication network 230.
[0041] As an illustration, terminal 210 sends a content recognition request to the server, wherein the content recognition request includes an image to be recognized, and the image to be recognized includes target recognition content. After receiving the content recognition request from terminal 210, server 220 performs content recognition on the image to be recognized and feeds back the recognized content recognition result to terminal 210.
[0042] In the process of content recognition of the image to be recognized, the server 220 extracts the content feature representation 221 corresponding to the target content in the image to be recognized. The content feature representation 221 is subjected to pooling and feature downsampling to obtain the global feature representation 222 and the candidate local feature representation 223, respectively. Key point detection is performed on the target content in the image to be recognized to obtain the target key points corresponding to the target content. The key point feature representation 224 corresponding to the target key points is extracted. The key point feature representation 224 and the candidate local feature representation 223 are concatenated to obtain the local feature representation 225. The content to be recognized is recognized based on the local feature representation 225 and the global feature representation 222, and the category 226 corresponding to the target content is determined as the content recognition result.
[0043] The aforementioned terminal 210 can be a mobile phone, tablet computer, desktop computer, portable laptop computer, smart TV, smart vehicle, or other terminal devices. This application embodiment does not limit this type of terminal device.
[0044] It is worth noting that the aforementioned server 220 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.
[0045] Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to achieve data computing, storage, processing, and sharing.
[0046] In some embodiments, the server 220 described above can also be implemented as a node in a blockchain system.
[0047] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the image to be identified involved in this application was obtained with full authorization.
[0048] This illustration demonstrates the content recognition method provided in this application. Please refer to [link / reference]. Figure 3 This document illustrates a flowchart of a content recognition method provided in an exemplary embodiment of this application. This method can be executed by a terminal, a server, or both. In this embodiment, the method is described as being executed by a server. Figure 3 As shown, the method includes the following steps.
[0049] Step 310: Obtain the image to be recognized, which includes the target recognition content.
[0050] Among them, the target recognition content corresponds to the target key points in the image to be recognized, and the target key points are key points extracted based on the pixel distribution pattern in the image to be recognized.
[0051] In illustrative terms, an image to be identified refers to an image containing target recognition content of an unknown category.
[0052] The target recognition content includes at least one of the following content types: people, animals, food, scenic spots, landmarks, etc.
[0053] Optionally, the image to be identified includes a single target recognition content; or, the image to be identified includes multiple target recognition contents. When the image to be identified includes multiple target recognition contents, the multiple target recognition contents may correspond to different contents or the same contents, without limitation.
[0054] Indicatively, target key points are feature points extracted based on the pixel distribution patterns of the image to be identified, used to represent pixels in the image that have symbolic target recognition content.
[0055] Optionally, the target key point type includes at least one of the key point types such as corner point, edge, or block, without limitation.
[0056] Optionally, the target recognition content may be labeled with a single target key point; or, the target recognition content may be labeled with multiple target key points, without limitation.
[0057] In some embodiments, a preset keypoint detector is used to extract keypoints from the image to be recognized, and the output result is used as the target keypoints of the target recognition content.
[0058] Step 320: Extract the content feature representation in the image to be identified that corresponds to the target content.
[0059] Indicatively, content feature representation is used to represent the feature vector corresponding to the target content in the image to be recognized.
[0060] Optionally, the content feature representation can be implemented as a set of feature vectors; or, the content feature representation can be implemented as a feature vector map (feature matrix), that is, the feature vector map contains multiple feature blocks (Patch blocks), each Patch block represents a feature vector, without limitation.
[0061] Optionally, a content recognition model is preset, and after the image to be recognized is input into the content recognition model, the content feature representation corresponding to the target content is directly output; or, a content recognition model is preset, and after the image to be recognized is input into the content recognition model, the candidate feature representation corresponding to the image to be recognized is output, and the content feature representation corresponding to the target content is selected from the candidate feature representations, without limitation.
[0062] Optionally, the content feature representation extraction method includes at least one of the following extraction methods:
[0063] 1. The Swing Transformer model (based on a moving window) is used for feature extraction. After the image to be recognized is input into the Swing Transformer model, the output is a feature map corresponding to the target content. Each patch block in the feature map represents a content feature representation.
[0064] 2. Feature extraction is performed using a deep residual network (ResNet). The image to be recognized is input into the ResNet, and the content feature representation corresponding to the target recognition content is obtained from the feature map output by each layer of the ResNet network.
[0065] 3. Feature extraction is performed using the Tokens-to-Tokens Vision Transformer (T2T-ViT model). The image to be recognized is input into the T2T-ViT model, and the output is a string sequence (Token sequence) corresponding to the target content, which serves as the content feature representation corresponding to the target content.
[0066] It is worth noting that the above-described methods for extracting content features are merely illustrative examples, and the embodiments of this application do not limit them.
[0067] Step 330: Perform pooling on the content feature representation to obtain the global feature representation; and perform feature downsampling on the content feature representation to obtain candidate local feature representations.
[0068] In illustration, pooling refers to downsampling the content feature representation, compressing the content feature representation, reducing the number of parameters while maintaining some invariance of the content feature representation (such as at least one of rotation invariance, translation invariance, or scaling invariance).
[0069] Optionally, the pooling process includes at least one of the following pooling types: max-pooling, mean-pooling, or generalized-mean pooling. This is not limited to any particular type. These three pooling processes will be described in detail in subsequent embodiments, and will not be elaborated upon here.
[0070] In illustration, feature downsampling refers to reducing the size of the image representation of content features to obtain a reduced feature vector, which is the candidate local feature representation.
[0071] In some embodiments, feature downsampling can be implemented as sparse sampling.
[0072] Optionally, pooling and feature downsampling can be performed simultaneously; or, pooling can be performed on the content feature representation first, followed by feature downsampling, without limitation.
[0073] Step 340: The key point feature representations and candidate local feature representations extracted from the target key points are concatenated to obtain the local feature representation.
[0074] In some embodiments, feature extraction is performed on the target key points to obtain the key point feature representations corresponding to the target key points.
[0075] Optionally, feature extraction is performed on all target key points to obtain key point feature representations corresponding to all target key points for subsequent feature concatenation; or, feature extraction is performed on a subset of target key points to obtain key point feature representations corresponding to a subset of target key points, without limitation.
[0076] In illustration, feature concatenation refers to concatenating candidate local features and keypoint feature representations into feature vectors, and using the concatenated feature vector as the local feature representation.
[0077] Optionally, the feature splicing method includes at least one of the following splicing methods:
[0078] 1. A single candidate local feature representation and a single keypoint feature representation are concatenated to obtain a single local feature representation. That is, the candidate local feature representation and the keypoint feature representation are concatenated one by one. The local feature representation contains multiple feature vectors obtained by concatenating the features.
[0079] 2. First, concatenate multiple candidate local feature representations one by one. Then, concatenate the concatenation result with the key point feature representation in turn to obtain the final local feature representation. That is, the local feature representation contains the single feature vector obtained by concatenation.
[0080] 3. Concatenate multiple candidate local features one by one to obtain the first concatenated feature representation, and concatenate multiple keypoint feature representations one by one to obtain the second concatenated feature representation. Then, concatenate the first and second concatenated feature representations to obtain the local feature representation. That is, first, concatenate multiple candidate local features and multiple keypoint feature representations separately, and then concatenate the feature vectors obtained from each concatenation again. The final concatenation result is used as the local feature representation.
[0081] It is worth noting that the above-described method of feature splicing is merely an illustrative example, and the embodiments of this application do not limit it.
[0082] Step 350: Based on global feature representation and local feature representation, the target content in the image to be recognized is identified to obtain the content recognition result.
[0083] The content recognition result is used to indicate the category corresponding to the target content.
[0084] For illustrative purposes, the content recognition result represents the category name corresponding to the target content, such as: the content recognition result for target content a is "garden"; or, the content recognition result represents the category type corresponding to the target content, such as: the content recognition result for target content b is "X garden", without limitation.
[0085] Optionally, the content recognition result includes a single target recognition content and its corresponding category, such as: target recognition content a, corresponding to category "Park A"; target recognition content b, corresponding to category "Park B"; or, the content recognition result contains multiple categories, each category corresponding to at least one target recognition content, such as: category A is "dolphin", category A includes target recognition content 1 and target recognition content 2 (that is, target recognition content 1 and target recognition content 2 are both "dolphin"), category B is "clownfish", category B includes target recognition content 3 (that is, target recognition content 3 is "clownfish"), and there is no limitation on this.
[0086] Optionally, the category corresponding to the target content can be implemented as a coarse-grained category. For example, if the image to be identified includes target content A (first amusement park) and target content B (second amusement park), the final content recognition result will show that the category corresponding to both target content A and target content B is "amusement park". Alternatively, the category corresponding to the target content can be implemented as a fine-grained category. For example, if both target content A and target content B belong to "museum", the final recognition result will show target content A as "a museum" and target content B as "b museum".
[0087] In summary, the content recognition method provided in this application, when the image to be recognized contains target recognition content, extracts the content feature representation corresponding to the target recognition content, performs pooling processing on the content feature representation to obtain a global feature representation, performs feature downsampling on the content feature representation, and concatenates the obtained candidate local feature representation with the key point feature representation extracted from the target key points in the target recognition content to obtain a local feature representation. Then, based on the global feature representation and the local feature representation, the target recognition content in the image to be recognized is identified, and finally, the content recognition result is obtained. That is, by concatenating the candidate local features obtained by feature downsampling the content feature representation with the key point feature representation to obtain the local feature representation, effective information about local features in the content feature representation can be effectively extracted. This, combined with the target key points, yields a local feature representation. Utilizing global and local features to perform content recognition on the image to be recognized improves the accuracy of content recognition.
[0088] In an optional embodiment, both candidate local feature representations and global feature representations can be obtained through various different pooling processes. For illustrative examples, please refer to [reference needed]. Figure 4 This illustrates a schematic diagram of a content recognition method provided in an exemplary embodiment of this application, such as... Figure 4 As shown, step 330 includes steps 331 and 332, and the method includes the following steps:
[0089] Step 331: Perform pooling on the content feature representation to obtain the global feature representation; and perform sparse sampling on the content feature representation to obtain the sparse sampling result.
[0090] Optionally, a portion of the content feature representations can be pooled to obtain a global feature representation; or, all content feature representations can be pooled to obtain a global feature representation, without any limitation.
[0091] First, we will explain in detail the average pooling, max pooling, and generalized mean pooling processes.
[0092] In some embodiments, pooling processing includes any one of average pooling, max pooling, or generalized mean pooling.
[0093] Mean-pooling refers to averaging the input feature representations to obtain the averaged feature vector as the global feature representation.
[0094] Max-pooling refers to selecting the feature vector with the largest vector value from the input content feature representation as the global feature representation.
[0095] Generalized-Mean Pooling (GeM) refers to a process where a learnable parameter p is predefined. For the input feature representation, the p-th power is first calculated, then the average of the vectors is taken, and finally the p-th root is taken. The result of the p-th root is used as the global feature representation. For illustration, GeM processing is based on Formula 1.
[0096] Formula 1:
[0097] From Formula 1, we know that X k For the k-th content feature, when p=1, Formula 1 can be implemented as an average evaluation process, that is, Formula 1 is equivalent to average pooling. When p approaches infinity, Formula 1 can be implemented as taking the maximum value, that is, Formula 1 is equivalent to max pooling.
[0098] To illustrate, the larger the p-value, the greater the focus on local features.
[0099] The following section explains the two methods for obtaining global feature representations.
[0100] The first method is to obtain global feature representations through single-type pooling.
[0101] In some embodiments, generalized mean pooling is performed on the content feature representation to obtain a global feature representation. That is, by performing generalized mean pooling on the content feature representation, the pooling result is used as the global feature representation.
[0102] In some embodiments, the content feature representation is subjected to average pooling to obtain a global feature representation.
[0103] In some embodiments, the content feature representation is max-pooled to obtain a global feature representation.
[0104] That is, by applying any one of the following pooling methods—max pooling, average pooling, and generalized mean pooling—to the content feature representation, a global feature representation is obtained. Global feature representation is used to indicate the corresponding single pooling process Pooling results .
[0105] The second method involves obtaining global feature representations through various pooling processes.
[0106] In some embodiments, the content feature representation is subjected to average pooling to obtain a first global feature representation; the content feature representation is subjected to max pooling to obtain a second global feature representation; the content feature representation is subjected to generalized mean pooling to obtain a third global feature representation; and the first global feature representation, the second global feature representation, and the third global feature representation are concatenated to obtain a global feature representation.
[0107] In this embodiment, the content feature representation is subjected to three different pooling processes to obtain a first global feature representation, a second global feature representation, and a third global feature representation. These features are then concatenated, and the concatenated result is used as the global feature representation. In other words, the global feature representation includes the concatenation result of the pooling results corresponding to the three pooling processes. .
[0108] Optionally, the first global feature representation, the second global feature representation, and the third global feature representation are concatenated in a fixed order (e.g., concatenated in the order of concatenation of the first global feature representation, the second global feature representation, and the third global feature representation); or, the first global feature representation, the second global feature representation, and the third global feature representation are concatenated in a random order, without limitation.
[0109] First, a detailed explanation of sparse sampling will be provided.
[0110] To illustrate, sparse sampling refers to representing content features through sparsification to obtain a sparse vector matrix, which serves as the sparse sampling result. Here, the content features are represented as a dense vector matrix, while the sparse sampling result is a sparse vector matrix, meaning it includes multiple zero elements.
[0111] In this embodiment, the content feature representation is implemented as a feature map (i.e., a feature matrix) with a size of k×k×1024. After sparse sampling of the content feature representation, n×1024 token vectors are obtained, and these n×1024 token vectors are used as the sparse sampling result. The number of token vectors is a pre-set fixed number; alternatively, the number of token vectors can be freely set according to actual needs, and there is no limitation on this.
[0112] In this embodiment, the pooling and feature downsampling processes for the content feature representations are performed simultaneously.
[0113] Step 332: Perform pooling processing on the sparse sampling results to obtain candidate local feature representations.
[0114] In some embodiments, pooling processing includes at least one of the following pooling methods: average pooling, max pooling, or generalized mean pooling.
[0115] This section provides an illustrative and detailed explanation of two methods for obtaining candidate local feature representations.
[0116] The first method involves performing single pooling on the sparse sampling results.
[0117] In a feasible scenario, max pooling is performed on the sparse sampling results, and the Token vector with the largest vector value in the sparse sampling results is selected as the candidate local feature representation.
[0118] In a feasible scenario, average pooling is performed on the sparse sampling results, and the average value of the sparse sampling results is calculated. The resulting average value vector is then used as a candidate local feature representation.
[0119] In a feasible scenario, generalized mean pooling is performed on the sparse sampling results. A learnable parameter p is set, and the sparse sampling results are pooled using the formula above. The pooled result is then used as a candidate local feature representation.
[0120] For the three different pooling methods mentioned above, that is, the candidate local feature representation includes the feature vector obtained by a single pooling process.
[0121] The second method involves applying various pooling techniques to the sparse sampling results.
[0122] In some embodiments, the sparse sampling results are subjected to average pooling to obtain a first local feature representation; the sparse sampling results are subjected to max pooling to obtain a second local feature representation; the sparse sampling results are subjected to generalized mean pooling to obtain a third local feature representation; and the first local feature representation, the second local feature representation, and the third local feature representation are concatenated to obtain a candidate local feature representation.
[0123] In this embodiment, the sparse sampling results are subjected to average pooling, max pooling, and generalized mean pooling to obtain a first local feature representation, a second local feature representation, and a third local feature representation, respectively. These features are then concatenated, and the concatenated result is used as a candidate local feature representation. That is, the current candidate local feature representation includes the concatenated result of feature vectors obtained from multiple different pooling processes.
[0124] Optionally, the sparse sampling results can be subjected to three different pooling processes simultaneously; or, the sparse sampling results can be pooled according to a specified processing order of the three pooling processes, without limitation. The specified processing order can be a pre-set fixed order; or, the specified processing order can be freely set according to actual needs.
[0125] Optionally, the first local feature representation, the second local feature representation, and the third local feature representation are concatenated in a fixed order (e.g., concatenated in the order of concatenation of the first local feature representation, the second local feature representation, and the third local feature representation); or, the first local feature representation, the second local feature representation, and the third local feature representation are concatenated in a random order, without limitation.
[0126] It is worth noting that the two pooling processes for content feature representation (including single pooling and concatenation of multiple pooling processes) and the two pooling processes for sparse sampling results (including single pooling and concatenation of multiple pooling processes) mentioned above are merely illustrative examples. In practice, any of the above pooling processes can be combined for content feature representation and sparse sampling results (i.e., including four pooling combination methods). This application does not limit this.
[0127] In summary, the content recognition method provided in this application, when the image to be recognized contains target recognition content, extracts the content feature representation corresponding to the target recognition content, performs pooling processing on the content feature representation to obtain a global feature representation, performs feature downsampling on the content feature representation, and concatenates the obtained candidate local feature representation with the key point feature representation extracted from the target key points in the target recognition content to obtain a local feature representation. Then, based on the global feature representation and the local feature representation, the target recognition content in the image to be recognized is identified, and finally, the content recognition result is obtained. That is, by concatenating the candidate local features obtained by feature downsampling the content feature representation with the key point feature representation to obtain the local feature representation, effective information about local features in the content feature representation can be effectively extracted. This, combined with the target key points, yields a local feature representation. Utilizing global and local features to perform content recognition on the image to be recognized improves the accuracy of content recognition.
[0128] In this embodiment, the sparse sampling method for the content feature representation can effectively obtain the local feature information contained in the content feature representation. The sparse sampling result obtained by sparse sampling is pooled to obtain candidate local feature representations, which are used to effectively extract the corresponding local features in the content feature representation, thereby improving the efficiency of feature extraction and the utilization rate of feature representation.
[0129] In this embodiment, two different pooling processing methods are provided for content feature representation: single pooling processing (any one of max pooling, average pooling, and generalized mean pooling) and feature concatenation after performing multiple different pooling processing (including performing max pooling, average pooling, and generalized mean pooling on the content feature representation separately, and concatenating the features based on the results of the three different pooling processing). This allows for the selection of the most suitable pooling processing method for content feature representation under different circumstances, improving the diversity of pooling processing options and thus improving the efficiency and accuracy of global feature representation acquisition.
[0130] In this embodiment, two different pooling processing methods are provided for sparse sampling results: single pooling processing (any one of max pooling, average pooling, and generalized mean pooling) and feature concatenation after performing multiple different pooling processing (including performing max pooling, average pooling, and generalized mean pooling on the sparse sampling results respectively, and concatenating features based on the results obtained from the three different pooling processing). This allows for the selection of the most suitable pooling processing method for sparse sampling results under different conditions, improving the diversity of pooling processing options and thus improving the efficiency and accuracy of candidate local feature representation acquisition.
[0131] In an optional embodiment, key point feature representation is obtained through a key point extraction algorithm, content feature representation is obtained through a content recognition model, and the content recognition result is determined by a content category library. For illustrative examples, please refer to [reference needed]. Figure 5 This illustrates a flowchart of a content recognition method provided by an exemplary embodiment of this application. Specifically, step 320 includes steps 321 and 322, step 340 includes steps 341 and 342, and step 350 includes steps 351, 352, 353, and 354. Figure 5 As shown, the method includes the following steps.
[0132] Step 310: Obtain the image to be recognized, which includes the target recognition content.
[0133] Among them, the target recognition content corresponds to the target key points in the image to be recognized, and the target key points are key points extracted based on the pixel distribution pattern in the image to be recognized.
[0134] Optionally, only one image to be recognized may be acquired at a time; or multiple images to be recognized may be acquired at the same time, without limitation.
[0135] For illustrative purposes, the image to be identified refers to an image containing target recognition content of an unknown category, such as: scenic spot images (images containing unknown scenic spot categories), celebrity photos (images containing photos of unknown celebrities), anime images (images containing unknown anime characters), etc., without limitation.
[0136] In some embodiments, target key points are feature points extracted by analyzing pixels in the image to be identified using a feature detector, based on the pixel distribution pattern.
[0137] Optionally, the target key points are obtained through at least one of the following extraction methods:
[0138] 1. Extract target key points corresponding to the target content in the image to be identified by scale invariant feature transformation (SIFT) detection. In the process of SIFT feature detection, the image to be identified is input into the SIFT feature detector, and the extreme points in the image to be identified are obtained by using the Gaussian Laplacian Pyramid scale space (DOG scale space) in the SIFT feature detector as target key points.
[0139] 2. Extract target key points corresponding to the target content in the image to be identified by SURF feature detection (Speeded Up Robust Features, based on an accelerated version of SIFT feature detection). In the process of SURF feature detection, the image to be identified is input into the SURF feature detector, and the SURF feature detector uses the determinant value of the Hessian matrix to detect key points in the image to be identified, thereby determining the target key points in the target content.
[0140] 3. Extract target key points corresponding to the target recognition content in the image to be recognized by ORB feature detection (Oriented Fast and Rotated BRIEF, ORB). Input the image to be recognized into the ORB feature detector to determine the target key points in the target recognition content.
[0141] It is worth noting that the above-described methods for extracting key target points are merely illustrative examples, and the embodiments of this application do not limit them.
[0142] In this embodiment, the target recognition content corresponds to multiple target key points in the image to be recognized.
[0143] Step 321: Input the image to be recognized into the content recognition model and output the candidate feature representation.
[0144] Among them, the content recognition model is used to extract deep features from the image to be recognized.
[0145] Optionally, a single image to be recognized may be input into the content recognition model at a time, and the candidate feature representation corresponding to the single image to be recognized may be output; or, multiple images to be recognized may be input into the content recognition model at the same time, and the candidate feature representations corresponding to the multiple images to be recognized may be output at the same time, without limitation.
[0146] In some embodiments, the candidate feature representation is implemented as a multi-dimensional feature vector map, wherein the feature vector map includes multiple patch blocks, each patch representing a feature vector.
[0147] Optionally, the content recognition model includes at least one of the Swin Transformer model, ResNet model, or T2T-ViT model, without limitation.
[0148] In this embodiment, the Swing Transformer model is used as the content recognition model. The Swing Transformer model will be briefly introduced below.
[0149] The Swin Transformer model introduces two concepts: hierarchical feature mapping and window attention transformation. The hierarchical feature mapping refers to the process where the feature representations in the Swin Transformer model are progressively merged after each layer's output, and feature downsampling is performed to establish a hierarchical feature map. This hierarchical feature map allows the Swin Transformer model to be well-suited for fine-grained feature prediction (e.g., semantic segmentation).
[0150] The non-convolutional feature downsampling method used in the Swin Transformer model is called Patch Merging. Here, "Patch" refers to the smallest unit in the feature vector map. For example, in a feature vector map with a feature size of 14x14, there are 14x14 = 196 patch blocks, that is, there are 196 feature blocks.
[0151] The Swin Transformer model uses a standard window-based multi-head self-attention (W-MSA) module, which computes attention only within each window. This transformation can lead to patches that don't belong to any window (i.e., isolated patches) and windows with incomplete patches. The Swin Transformer model employs a "cyclic shift" technique to move isolated patches into windows containing incomplete patches. After this shift, a window consists of non-adjacent patches from the original feature vector map. Therefore, a mask is applied during computation to restrict self-attention to adjacent patches.
[0152] In this embodiment, the image to be identified is input into the Swing Transformer model, and a k×k×1024 feature vector map is output at the end of the Swing Transformer model as a candidate feature representation.
[0153] Step 322: Determine the content feature representation corresponding to the target content from the candidate feature representations.
[0154] In some embodiments, the candidate feature representation includes all features in the image to be identified, such as the feature representation corresponding to the target content and the feature representation corresponding to the background content.
[0155] Background content refers to the content in the image to be identified that does not contain the main features corresponding to the target content. For example, if the image to be identified is a landscape of pyramids, then the background content in the image to be identified is a desert, and the target content is a pyramid.
[0156] In some embodiments, saliency detection is performed on the image to be identified to determine the target region in the image to be identified that corresponds to the target content to be identified; based on the target region, region analysis is performed on the candidate feature representation to obtain the content feature representation.
[0157] In illustrative terms, saliency detection is used to determine the target region corresponding to the target content and the background region corresponding to the background content in the image to be identified. In other words, saliency detection is used to divide the image to be identified into regions based on content features.
[0158] In some embodiments, a saliency detection model is preset. The image to be identified is input into the saliency detection model, and the output is a target saliency map corresponding to the image to be identified. The target saliency map includes the target region corresponding to the target content and the background region corresponding to the background content. The target saliency map is implemented as an image after enhancing the target region.
[0159] This is illustrative; please refer to it. Figure 6 This illustrates a schematic diagram of a target area provided in an exemplary embodiment of this application, such as... Figure 6 As shown, Figure 6 A schematic diagram 600 of target saliency maps obtained after saliency detection of three different images to be identified is shown, including a first image 610 and a first saliency map 611 corresponding to the first image 610, a second image 620 and a second saliency map 621 corresponding to the second image 620, and a third image 630 and a third saliency map 631 corresponding to the third image 630.
[0160] The first saliency map 611 includes a first target region (white region), the second saliency map 621 includes a second target region (white region), and the third saliency map 631 includes a third target region (white region). Figure 6 The target areas are marked with white areas, while the background areas are marked with black areas.
[0161] In this embodiment, Figure 6 The target saliency map shown indicates that the main features corresponding to the current target recognition content are relatively obvious. That is, the target area in the current target saliency map has good display completeness, and the edges of the areas corresponding to the white areas are relatively clear.
[0162] Furthermore, in this embodiment, there are cases where the main features corresponding to the target recognition content in the target saliency map are not obvious. This is illustrative; please refer to [the original text]. Figure 7This illustrates a schematic diagram of a target area provided in an exemplary embodiment of this application, such as... Figure 7 As shown, Figure 7 A schematic diagram 700 of target saliency maps obtained after saliency detection of two different images to be identified is shown, including the fourth image 710 and the fourth saliency map 711 corresponding to the fourth image 710, the fifth image 720 and the fifth saliency map 721 corresponding to the fifth image 720.
[0163] In the fourth saliency map 711 and the fifth saliency map 721, the white area is the target area and the black area is the background area. The current fourth saliency map 711 and the fifth saliency map 721 belong to the case where the main features corresponding to the target recognition content are not obvious, that is, the edge of the area corresponding to the white area is relatively blurry.
[0164] Optionally, the saliency detection model includes at least one of the following models: Visual Saliency Transformer (VST model) and EdgeGuidance Network for Salient Object Detection (EGNet model), without limitation.
[0165] In this embodiment, the VST model is described in detail.
[0166] This is illustrative; please refer to it. Figure 8 It illustrates a saliency detection model provided in one embodiment of this application, such as... Figure 8 As shown, the VST model is currently displayed. The model input of the VST model includes a first image 810 and a second image 820. The first image 810 is the image to be recognized (this image is an RGB image). Figure 8 (The color is not displayed in the image). The second image 820 is a grayscale image (RGB-D image) corresponding to the image to be identified. The first image block 811 corresponding to the first image 810 and the second image block 821 corresponding to the second image 820 are respectively input into the Transformer encoder space 830 (Transformer Encoder). In the Transformer encoder space 830, the first image block 811 and the second image block 821 are encoded into multi-level token vectors (such as T1, T2, T3) using the Token-to-Token (T2T) module. The multi-level token vectors are input into the converter 840 (Converter). The converter 840 is used to convert the multi-level token vectors from the encoder space 830 to the decoder space 850 (Transformer Decoder) for feature decoding, and outputs the target saliency map 8111 corresponding to the first image 810 and the target boundary map 8221 corresponding to the second image 820.
[0167] In the VST model, in addition to using the Transformer model structure, multi-level token vector fusion is utilized, and a novel token vector upsampling method is proposed under the Transformer structure to obtain high-resolution saliency detection results. A token vector-based multi-task decoder is also developed, which simultaneously performs saliency and boundary detection by introducing task-related token vectors and a Patch-Task-Attention mechanism.
[0168] Step 330: Perform pooling on the content feature representation to obtain the global feature representation; and perform feature downsampling on the content feature representation to obtain candidate local feature representations.
[0169] Optionally, the content feature representation can be processed using any one of max pooling, average pooling, or generalized mean pooling, and the pooling result can be used as the global feature representation; alternatively, the content feature representation can be processed using max pooling, average pooling, and generalized mean pooling respectively, and the results of the three pooling processes can be concatenated to obtain the global feature representation. There is no limitation on this approach. In this embodiment, the content feature representation is processed using generalized mean pooling, and the pooling result is used as the global feature representation.
[0170] To illustrate, sparse sampling is performed on the content feature representation to obtain the sparse sampling result. Pooling is then applied to the sparse sampling result to obtain candidate local feature representations.
[0171] In this embodiment, sparse sampling is performed on the k×k×1024 feature vector map to obtain n×1024 token vectors, and then average pooling is performed on these n×1024 token vectors to obtain local features.
[0172] Optionally, the sparse sampling results can be processed by any one of max pooling, average pooling, or generalized mean pooling, and the pooling result can be used as a candidate local feature representation; alternatively, the sparse sampling results can be processed by max pooling, average pooling, and generalized mean pooling respectively, and the results of the three pooling processes can be concatenated to obtain a candidate local feature representation. There is no limitation on this method. In this embodiment, the content feature representation is processed by average pooling, and the pooling result is used as a candidate local feature representation.
[0173] Step 341: Extract key point feature representations corresponding to the target key points using a key point extraction algorithm.
[0174] Optionally, after determining the target keypoints using the SIFT keypoint detector, the keypoint feature representation (SIFT feature representation) corresponding to the target keypoints is extracted; or, after determining the target keypoints using the SURF keypoint detector, the keypoint feature representation (SURF feature representation) corresponding to the target keypoints is extracted; or, after determining the target keypoints using the ORB keypoint detector, the keypoint feature representation (ORB feature representation) corresponding to the target keypoints is extracted, without limitation.
[0175] Optionally, at least one of the above-mentioned SIFT feature representation, SURF feature representation, or ORB feature representation may be selected as the key point feature representation.
[0176] Step 342: Concatenate the candidate local feature representation and the key point feature representation to obtain the local feature representation.
[0177] In a schematic manner, candidate local feature representations and key point feature representations are sequentially concatenated, and the concatenated result is used as the local feature representation.
[0178] Step 351: Obtain the content category library, which includes a set of n pre-defined categories, where n is a positive integer.
[0179] To illustrate, the content category library includes n pre-stored categories. Each category stores at least one candidate feature representation corresponding to a candidate image. That is, the candidate features correspond to the category. For example, the category "Poodle" stores multiple images containing poodles. Each poodle image is labeled with the feature representation corresponding to the poodle, which serves as the candidate feature representation.
[0180] In some embodiments, the content category library is obtained in advance.
[0181] Step 352: Match the global feature representation with each of the n categories in the content category library to obtain k candidates in the content category library that match the global feature representation, where 0 < k < n and k is an integer.
[0182] In some embodiments, the global feature representation is matched with n categories in the content category library to obtain global matching scores for each of the n categories. The global matching scores are used to indicate the probability that the target content belongs to a category. The global matching scores for each of the n categories are sorted to obtain matching degree ranking results. The top k categories in the matching degree ranking results are selected as k candidate categories that match the global feature representation.
[0183] To illustrate, based on the global feature representation, all candidate feature representations corresponding to each category are traversed in the content category library. Each candidate feature representation is matched with the global feature representation. Based on the matching situation between the candidate feature representation corresponding to the category and the current global feature representation, the global matching score corresponding to the category is determined. The higher the global matching score of the category, the higher the matching degree between the candidate feature representation and the global feature representation under that category. In other words, the higher the probability that the current target content belongs to that category.
[0184] The global matching scores are sorted from highest to lowest to obtain the matching degree ranking results. The top k categories in the matching degree ranking results are selected as the k candidate categories that match the global feature representation.
[0185] Step 353: Sort the k candidate categories based on local feature representations to obtain the category ranking results.
[0186] To illustrate, the top k candidate categories with the highest content matching scores to the target content are selected from the content category library using global feature representation. For these k candidate categories, they are then sorted again based on local feature representation to obtain the category ranking result.
[0187] The process involves matching the local feature representation with the candidate feature representations stored under each of the k candidate categories. Based on the matching results between the candidate feature representations and the local feature representations, a local matching score is determined for each of the k candidate categories. The local matching score indicates the matching result between the current local feature representation and the candidate feature representations under that category. A higher matching degree indicates a higher local matching score for that category. The categories are then sorted from highest to lowest based on their local matching scores to obtain the category ranking result.
[0188] Step 354: Based on the category sorting results, obtain the target category corresponding to the target recognition content.
[0189] As an illustration, the candidate category with the highest (or several relatively high) local matching score in the category ranking results is selected as the target category and used as the content recognition result.
[0190] In summary, the content recognition method provided in this application, when the image to be recognized contains target recognition content, extracts the content feature representation corresponding to the target recognition content, performs pooling processing on the content feature representation to obtain a global feature representation, performs feature downsampling on the content feature representation, and concatenates the obtained candidate local feature representation with the key point feature representation extracted from the target key points in the target recognition content to obtain a local feature representation. Then, based on the global feature representation and the local feature representation, the target recognition content in the image to be recognized is identified, and finally, the content recognition result is obtained. That is, by concatenating the candidate local features obtained by feature downsampling the content feature representation with the key point feature representation to obtain the local feature representation, effective information about local features in the content feature representation can be effectively extracted. This, combined with the target key points, yields a local feature representation. Utilizing global and local features to perform content recognition on the image to be recognized improves the accuracy of content recognition.
[0191] In this embodiment, by extracting candidate feature representations from the image to be recognized using a preset content recognition model, the candidate feature representations can contain both global and local feature information, thereby improving the accuracy of the feature representation output.
[0192] In this embodiment, the target region of the target content in the image to be identified is determined by saliency detection, and then the content feature representation corresponding to the target content is selected from the candidate feature representations. This can filter out background content that does not contain the main features, thereby improving the accuracy and efficiency of content recognition.
[0193] In this embodiment, k candidate categories are selected from the content category library through global feature representation, and the k candidate categories are reordered according to local feature representation. The content recognition result is determined based on the final category sorting result, which improves the accuracy of content recognition.
[0194] In one optional embodiment, the application scenarios corresponding to the content recognition method provided in this application are described. For illustrative purposes, please refer to [reference needed]. Figure 9 The illustration shows a schematic diagram of a content recognition method provided in an exemplary embodiment of this application, with the application of the content method in an image search scenario as an example for explanation.
[0195] The user inputs an image as the image to be identified. The image database is then searched to find the target image that matches the image to be identified, which is then used as the image search result.
[0196] like Figure 9As shown, an image 910 to be recognized is acquired, wherein the image 910 is an image input by the user, and the image 910 includes target recognition content 911, which includes multiple target key points ( Figure 9 (not shown in the image), the target keypoints are feature points detected by at least one of the three keypoint detectors: SIFT keypoint detector, ORB keypoint detector, or SURF keypoint detector.
[0197] The image 910 to be recognized is input into the content recognition model 920, and the output is the candidate feature representation 930. The content recognition model 920 is implemented as a Swing Transformer model, and the candidate feature representation 930 is implemented as a feature vector map with a feature size of k×k×1024 output by the last layer of the Swing Transformer model.
[0198] Saliency detection is performed on the image 910 to be recognized to obtain the target region 912 corresponding to the target content 911. Based on the target region 912, the content feature representation 931 corresponding to the target content 911 in the candidate feature representation 930 is determined. The saliency detection is implemented using the VST model.
[0199] Generalized mean pooling (940) and sparse sampling (950) are applied to the content feature representation (931) to obtain the global feature representation (941) and the sparse sampling result, respectively. Figure 9 (Not shown in the image).
[0200] The sparse sampling results are subjected to average pooling (960) to obtain candidate local feature representations. Figure 9 (Not shown in the figure), the candidate local feature representation and the key point feature representation extracted from the target key point (at least one of SIFT feature representation, SURF feature representation or ORB feature representation) are concatenated to obtain the local feature representation 951.
[0201] In addition, feature dimensionality reduction is performed on the results obtained from the above pooling process to remove redundant features with high correlation between feature representations.
[0202] Based on the global feature representation 941, a match is made in the category library 970 to obtain the top k candidate categories (TOP-K) with the highest global matching score with the global feature representation 941, which are then used as the k candidate categories 971.
[0203] Based on the local feature library 952 that stores local feature representations 951, the k candidate categories 971 are matched again to obtain the local matching degree scores corresponding to the k candidate categories 971. The k candidate categories are reordered according to the local matching degree scores, and finally the category with the highest local matching degree score is selected as the target category 980 for output. The target category 980 is implemented as "Great Wall".
[0204] Input the target category into the image library, select candidate images from the image library that correspond to the target category, and output them to the user.
[0205] Furthermore, the content recognition method provided in this application embodiment can also be applied to the following scenarios.
[0206] 1. Application in account recommendation. Taking a user's search for a video account as an example, the videos published by the current video account are marked with candidate locations. When the user enters the target location as the search content, the target video account is identified through the above content recognition method. The videos of the target video account marked with the corresponding target location are then weighted, thereby increasing the probability of that video being recommended to the user.
[0207] 2. Application in content recommendation. In the process of recommending content to a target user, if an image or video in the recommendation library is identified using the aforementioned content recognition method, that image or video content is recommended to the target user.
[0208] In summary, the content recognition method provided in this application, when the image to be recognized contains target recognition content, extracts the content feature representation corresponding to the target recognition content, performs pooling processing on the content feature representation to obtain a global feature representation, performs feature downsampling on the content feature representation, and concatenates the obtained candidate local feature representation with the key point feature representation extracted from the target key points in the target recognition content to obtain a local feature representation. Then, based on the global feature representation and the local feature representation, the target recognition content in the image to be recognized is identified, and finally, the content recognition result is obtained. That is, by concatenating the candidate local features obtained by feature downsampling the content feature representation with the key point feature representation to obtain the local feature representation, effective information about local features in the content feature representation can be effectively extracted. This, combined with the target key points, yields a local feature representation. Utilizing global and local features to perform content recognition on the image to be recognized improves the accuracy of content recognition.
[0209] The beneficial effects of the content recognition method provided in this application include:
[0210] 1) A structure was constructed that uses global feature representation for content recall and local feature representation for reordering;
[0211] 2) The candidate local feature representation and the key point feature representation are concatenated to obtain the local feature representation;
[0212] 3) Introduce saliency detection to avoid interference from background information.
[0213] Figure 10 This is a structural block diagram of a content recognition device provided in an exemplary embodiment of this application, such as... Figure 10 As shown, the device includes the following parts:
[0214] The acquisition module 1010 is used to acquire an image to be recognized, wherein the image to be recognized includes target recognition content, the target recognition content corresponds to target key points in the image to be recognized, and the target key points are key points extracted based on the pixel distribution pattern in the image to be recognized.
[0215] Extraction module 1020 is used to extract content feature representations in the image to be identified that correspond to the target recognition content;
[0216] Processing module 1030 is used to perform pooling processing on the content feature representation to obtain a global feature representation; and to perform feature downsampling on the content feature representation to obtain a candidate local feature representation;
[0217] The splicing module 1040 is used to splice the key point feature representation extracted from the target key points and the candidate local feature representation to obtain a local feature representation;
[0218] The recognition module 1050 is used to recognize the target content in the image to be recognized based on the global feature representation and the local feature representation, and to obtain a content recognition result, wherein the content recognition result is used to indicate the category corresponding to the target content.
[0219] In an optional embodiment, such as Figure 11 As shown, the processing module 1030 includes:
[0220] Sampling unit 1031 is used to perform sparse sampling on the content feature representation to obtain sparse sampling results;
[0221] The processing unit 1032 is used to perform pooling processing on the sparse sampling results to obtain the candidate local feature representation.
[0222] In an optional embodiment, the pooling process includes any one of average pooling, max pooling, or generalized mean pooling.
[0223] In an optional embodiment, the processing unit 1032 is further configured to perform average pooling on the sparse sampling results to obtain a first local feature representation; perform max pooling on the sparse sampling results to obtain a second local feature representation; perform generalized mean pooling on the sparse sampling results to obtain a third local feature representation; and perform feature concatenation on the first local feature representation, the second local feature representation, and the third local feature representation to obtain the candidate local feature representation.
[0224] In an optional embodiment, the splicing module 1040 is further configured to extract key point feature representations corresponding to the target key points using a key point extraction algorithm; and to splice the candidate local feature representations and the key point feature representations to obtain the local feature representations.
[0225] In an optional embodiment, the processing module 1030 is further configured to perform generalized mean pooling on the content feature representation to obtain the global feature representation.
[0226] In an optional embodiment, the processing module 1030 is further configured to perform average pooling on the content feature representation to obtain a first global feature representation; perform max pooling on the content feature representation to obtain a second global feature representation; perform generalized mean pooling on the content feature representation to obtain a third global feature representation; and concatenate the first global feature representation, the second global feature representation, and the third global feature representation to obtain the global feature representation.
[0227] In an optional embodiment, the extraction module 1020 is further configured to input the image to be identified into a content recognition model and output a candidate feature representation, wherein the content recognition model is used to perform deep feature extraction on the image to be identified; and to determine the content feature representation corresponding to the target recognition content from the candidate feature representation.
[0228] In an optional embodiment, the extraction module 1020 is further configured to perform saliency detection on the image to be identified, determine the target region in the image to be identified that corresponds to the target content to be identified, and perform region analysis on the candidate feature representation based on the target region to obtain the content feature representation.
[0229] In an optional embodiment, the identification module 1050 is further configured to acquire a content category library, which includes a pre-defined set of n categories, where n is a positive integer; match the global feature representation with each of the n categories in the content category library to obtain k candidate categories in the content category library that match the global feature representation, where 0 < k < n and k is an integer; sort the k candidate categories based on the local feature representation to obtain a category sorting result; and obtain the target category corresponding to the target identification content based on the category sorting result.
[0230] In an optional embodiment, the identification module 1050 is further configured to match the global feature representation with n categories in the content category library respectively to obtain global matching scores corresponding to the n categories respectively, wherein the global matching scores are used to indicate the probability that the target identified content belongs to the category; sort the global matching scores corresponding to the n categories respectively to obtain matching degree ranking results; and take the top k categories in the matching degree ranking results as k candidate categories that match the global feature representation.
[0231] In summary, the content recognition device provided in this application, when the image to be recognized contains target recognition content, extracts the content feature representation corresponding to the target recognition content, performs pooling processing on the content feature representation to obtain a global feature representation, performs feature downsampling on the content feature representation, and concatenates the obtained candidate local feature representation with the key point feature representation extracted from the target key points corresponding to the target recognition content to obtain a local feature representation. Then, based on the global feature representation and the local feature representation, the target recognition content in the image to be recognized is identified, and finally, the content recognition result is obtained. That is, by concatenating the candidate local features obtained by feature downsampling the content feature representation with the key point feature representation to obtain the local feature representation, effective information about local features in the content feature representation can be effectively extracted. This, combined with the target key points, yields a local feature representation. Utilizing global and local features to perform content recognition on the image to be recognized improves the accuracy of content recognition.
[0232] It should be noted that the content recognition device provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the content recognition device and the content recognition method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.
[0233] Figure 12A schematic diagram of the structure of a server provided in an exemplary embodiment of this application is shown. Specifically:
[0234] Server 1200 includes a Central Processing Unit (CPU) 1201, a system memory 1204 including Random Access Memory (RAM) 1202 and Read Only Memory (ROM) 1203, and a system bus 1205 connecting the system memory 1204 and the CPU 1201. Server 1200 also includes a mass storage device 1206 for storing an operating system 1213, application programs 1214, and other program modules 1215.
[0235] Mass storage device 1206 is connected to central processing unit 1201 via a mass storage controller (not shown) connected to system bus 1205. Mass storage device 1206 and its associated computer-readable media provide non-volatile storage for server 1200. That is, mass storage device 1206 may include computer-readable media (not shown) such as hard disk or compact disc read-only memory (CD-ROM) drives.
[0236] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The system memory 1204 and mass storage device 1206 described above can be collectively referred to as memory.
[0237] According to various embodiments of this application, server 1200 can also be connected to a remote computer on a network, such as the Internet. That is, server 1200 can be connected to network 1212 via network interface unit 1211 connected to system bus 1205, or network interface unit 1211 can be used to connect to other types of networks or remote computer systems (not shown).
[0238] The aforementioned memory also includes one or more programs, which are stored in the memory and configured to be executed by the CPU.
[0239] Embodiments of this application also provide a computer device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, code set, or instruction set. The at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement the content recognition method provided in the above-described method embodiments.
[0240] Embodiments of this application also provide a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the content recognition method provided in the above-described method embodiments.
[0241] Embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the content recognition methods described in the above embodiments.
[0242] Optionally, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM). The sequence numbers of the embodiments in this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.
[0243] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.
[0244] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A content recognition method, characterized in that, The method includes: A target image to be identified is obtained, which includes target identification content and background content. The target identification content corresponds to target key points in the target image. The target key points are key points extracted based on the pixel distribution pattern in the target image and are used to represent the pixels in the target image that symbolize the target identification content. The image to be identified is input into the content recognition model, and the output is a candidate feature representation. The content recognition model is used to perform deep feature extraction on the image to be identified, and the candidate feature representation is a multi-dimensional feature vector map, in which each feature block represents a feature vector. The image to be identified is subjected to saliency detection to determine the target region in the image to be identified that corresponds to the target recognition content. The saliency detection is used to determine the target region corresponding to the target recognition content and the background region corresponding to the background content in the image to be identified. The saliency detection includes: inputting the image to be identified and its grayscale image into a saliency detection model, and having the saliency detection model simultaneously perform saliency detection and edge detection to output a target saliency map corresponding to the image to be identified and a target boundary map corresponding to the grayscale image. The target saliency map includes the target region and the background region. Based on the target region, perform region analysis on the candidate feature representation, and obtain the content feature representation corresponding to the target identification content from the candidate feature representation, other than the feature representation corresponding to the background content; The content feature representation is pooled to obtain a global feature representation; and the content feature representation is downsampled to obtain a candidate local feature representation; the pooling process is a process of compressing the content feature representation while maintaining the invariance of the content feature representation, the invariance including at least one of rotation invariance, translation invariance and scaling invariance, and the feature downsampling process is a process of reducing the image size of the content feature representation; The key point feature representation extracted from the target key points and the candidate local feature representation are concatenated to obtain the local feature representation; The target content in the image to be identified is identified based on the global feature representation and the local feature representation to obtain the content identification result, which is used to indicate the category corresponding to the target content.
2. The method according to claim 1, characterized in that, The step of downsampling the content feature representation to obtain candidate local feature representations includes: The content feature representation is sparsely sampled to obtain the sparse sampling result; The sparse sampling results are pooled to obtain the candidate local feature representation.
3. The method according to claim 2, characterized in that, The pooling process includes any one of average pooling, max pooling, or generalized mean pooling.
4. The method according to claim 2, characterized in that, The pooling process performed on the sparse sampling results to obtain the candidate local feature representation includes: The sparse sampling results are subjected to average pooling to obtain the first local feature representation; The sparse sampling results are subjected to max pooling to obtain the second local feature representation; The sparse sampling results are subjected to generalized mean pooling to obtain a third local feature representation; The first local feature representation, the second local feature representation, and the third local feature representation are concatenated to obtain the candidate local feature representation.
5. The method according to any one of claims 1 to 4, characterized in that, The step of concatenating the key point feature representations extracted from the target key points and the candidate local feature representations to obtain local feature representations includes: Key point feature representations corresponding to the target key points are extracted using a key point extraction algorithm; The candidate local feature representation and the key point feature representation are concatenated to obtain the local feature representation.
6. The method according to any one of claims 1 to 4, characterized in that, The process of pooling the content feature representation to obtain a global feature representation includes: The content feature representation is subjected to generalized mean pooling to obtain the global feature representation.
7. The method according to any one of claims 1 to 4, characterized in that, The process of pooling the content feature representation to obtain a global feature representation includes: The content feature representation is subjected to average pooling to obtain the first global feature representation; The content feature representation is subjected to max pooling to obtain the second global feature representation; The content feature representation is subjected to generalized mean pooling to obtain a third global feature representation; The first global feature representation, the second global feature representation, and the third global feature representation are concatenated to obtain the global feature representation.
8. The method according to any one of claims 1 to 4, characterized in that, The step of identifying the target content in the image to be identified based on the global feature representation and the local feature representation to obtain the content identification result includes: Obtain a content category library, which includes a set of n predefined categories, where n is a positive integer; The global feature representation is matched with each of the n categories in the content category library to obtain k candidate categories in the content category library that match the global feature representation, where 0 < k < n and k is an integer; Based on the local feature representation, the k candidate categories are sorted to obtain the category sorting result; Based on the sorting results of the categories, the target category corresponding to the target recognition content is obtained.
9. The method according to claim 8, characterized in that, The step of matching the global feature representation with each of the n categories in the content category library to obtain k candidate categories in the content category library that match the global feature representation includes: The global feature representation is matched with n categories in the content category library to obtain global matching scores for each of the n categories. The global matching scores are used to indicate the probability that the target content belongs to the category. Sort the global matching scores corresponding to the n categories respectively to obtain the matching degree ranking result; The top k categories in the matching score ranking results are selected as the k candidate categories that match the global feature representation.
10. A content recognition device, characterized in that, The device includes: The acquisition module is used to acquire an image to be identified, which includes target recognition content and background content. The target recognition content corresponds to target key points in the image to be identified. The target key points are key points extracted based on the pixel distribution pattern in the image to be identified, and are used to represent the pixels in the image to be identified that symbolize the target recognition content. An extraction module is used to input the image to be recognized into a content recognition model and output a candidate feature representation. The content recognition model performs deep feature extraction on the image to be recognized, and the candidate feature representation is a multi-dimensional feature vector map, where each feature block represents a feature vector. The module performs saliency detection on the image to be recognized to determine the target region in the image corresponding to the target content. The saliency detection is used to determine the target region corresponding to the target content and the background region corresponding to the background content in the image to be recognized. The saliency detection includes: inputting the image to be recognized and its grayscale image into the saliency detection model, whereby the saliency detection model simultaneously performs saliency detection and edge detection, outputting a target saliency map corresponding to the image to be recognized and a target boundary map corresponding to the grayscale image. The target saliency map includes the target region and the background region. Based on the target region, the module performs region analysis on the candidate feature representation to obtain content feature representations corresponding to the target content, excluding the feature representations corresponding to the background content, from the candidate feature representations. The processing module is used to perform pooling processing on the content feature representation to obtain a global feature representation; and to perform feature downsampling on the content feature representation to obtain a candidate local feature representation; the pooling processing is a process of compressing the content feature representation while maintaining the invariance of the content feature representation, the invariance including at least one of rotation invariance, translation invariance and scaling invariance, and the feature downsampling is a process of reducing the image size of the content feature representation; The splicing module is used to splice the key point feature representations extracted from the target key points and the candidate local feature representations to obtain local feature representations; The recognition module is used to recognize the target content in the image to be recognized based on the global feature representation and the local feature representation, and to obtain the content recognition result, which is used to indicate the category corresponding to the target content.
11. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one program, which is loaded and executed by the processor to implement the content recognition method as described in any one of claims 1 to 9.
12. A computer-readable storage medium, characterized in that, The storage medium stores at least one program segment, which is loaded and executed by a processor to implement the content recognition method as described in any one of claims 1 to 9.
13. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the content recognition method as described in any one of claims 1 to 9.
Citation Information
Patent Citations
Facial expression recognition device and facial expression recognition method
CN105095827A
True and false picture recognition method and device and computer readable storage medium
CN108288073A
Image recognition method and related device and equipment
CN111507381A
Target detection method and device and electronic system
CN112766272A
Image retrieval method and system based on global and local feature rearrangement
CN112966137A
