An image recognition method, device, electronic equipment and storage medium

Abstract

The application relates to the computer technical field, and in particular to an image recognition method and device, electronic equipment and storage medium, to improve the efficiency and accuracy of cartoon face recognition. The method comprises: performing feature extraction on a to-be-recognized image to obtain at least two sampling features of different scales; performing multi-scale feature fusion of different scales based on the at least two sampling features to obtain at least one fused feature, and taking the at least one fused feature and a sampling feature conforming to a target scale in the at least two sampling features as to-be-weighted features; obtaining a target feature based on each to-be-weighted feature and a corresponding preset weight; and performing recognition on the to-be-recognized image based on the target feature to obtain an object contained in the to-be-recognized image. Since the application performs multi-scale feature fusion on sampling features of different scales, and obtains an object contained in a to-be-recognized image based on a target feature, the efficiency and accuracy of cartoon face recognition can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image recognition method, apparatus, electronic device, and storage medium. Background Technology

[0002] With increasing emphasis on intellectual property rights and brand protection, the demand for content review before the release of multimedia content is also rising. For example, reviewing cartoon faces in multimedia content to protect the copyright of multimedia content requires the identification of cartoon faces.

[0003] like Figure 1 The cartoon face images shown can be of different sizes for different cartoon characters. Because this type of cartoon face is usually drawn in a two-dimensional style, the proportions of the facial features are arbitrarily designed, making it impossible to recognize them by detecting key points of natural faces.

[0004] In related technologies, in order to improve the recognition accuracy of cartoon faces at different scales, the main approach is to enhance cartoon face images at different scales to obtain a large number of training samples, and then train the cartoon face recognition model based on these samples to achieve cartoon face recognition at different scales.

[0005] However, when using the above method to recognize cartoon faces, the model training process is complex, time-consuming, and has low recognition efficiency because a large number of training samples are introduced through the enhancement method.

[0006] Therefore, improving the efficiency and accuracy of cartoon face recognition has become an urgent problem to be solved. Summary of the Invention

[0007] This application provides an image recognition method, apparatus, electronic device, and storage medium to improve the efficiency and accuracy of cartoon face recognition.

[0008] This application provides an image recognition method, including:

[0009] Feature extraction is performed on the image to be recognized to obtain at least two sampled features with different scales;

[0010] Based on the at least two sampling features, multi-scale feature fusion is performed between the different scales to obtain at least one fused feature, and the at least one fused feature and the sampling features that conform to the target scale among the at least two sampling features are all used as features to be weighted.

[0011] Based on each feature to be weighted and its corresponding preset weight, the target feature is obtained;

[0012] Based on the target features, the image to be identified is identified to obtain at least one object contained in the image to be identified.

[0013] An image recognition device provided in this application includes:

[0014] The extraction unit is used to extract features from the image to be recognized, and obtain at least two sampled features with different scales;

[0015] The fusion unit is used to perform multi-scale feature fusion between the different scales based on the at least two sampled features to obtain at least one fused feature, and to take the at least one fused feature and the sampled features that conform to the target scale from the at least two sampled features as features to be weighted.

[0016] The weighting unit is used to obtain the target feature based on each feature to be weighted and its corresponding preset weight;

[0017] The recognition unit is used to recognize the image to be recognized based on the target features, and to obtain at least one object contained in the image to be recognized.

[0018] Optionally, the fused features can be obtained by performing multi-scale feature fusion based on the feature fusion network in the following manner:

[0019] The feature fusion network upsamples the feature with the lower scale among the two features to be fused by the upsampling layer to obtain the corresponding first intermediate feature. The two features to be fused are: the two sampled features of different scales, or the fused feature obtained in the previous round and the corresponding sampled feature.

[0020] Through the channel adaptation layer of the feature fusion network, the feature with the higher scale among the two features to be fused is unified with the upsampled feature in terms of channel number to obtain the corresponding second intermediate feature;

[0021] The first intermediate feature and the second intermediate feature are added and fused to obtain the corresponding fused feature.

[0022] Optionally, the weighting unit is specifically used for:

[0023] Each of the features to be weighted is input into the mapping network of the trained image recognition model for feature mapping to obtain the weighted features;

[0024] The weighted features are normalized to obtain the preset weights corresponding to each of the features to be weighted.

[0025] The target feature is obtained by performing feature weighting and fusion based on each preset weight and the corresponding feature to be weighted.

[0026] Optionally, the device further includes an adjustment unit for:

[0027] The scale of each feature to be weighted is adjusted to obtain each feature to be weighted at a uniform scale;

[0028] The weighting unit is specifically used for:

[0029] The target features are obtained by performing feature weighting and fusion based on each preset weight and the corresponding unified scale of the features to be weighted.

[0030] Optionally, the extraction unit is specifically used for, including:

[0031] The image to be identified is input into the feature extraction network of a trained image recognition model;

[0032] Based on multiple downsampling layers in the feature extraction network, the image to be identified is downsampled layer by layer to obtain the sampling features output by each downsampling layer. The sampling features output by different downsampling layers have different scales.

[0033] Optionally, the sampling features that conform to the target scale are the sampling features output by the last downsampling layer in the feature extraction network.

[0034] Optionally, the trained image recognition model can be obtained in the following way:

[0035] Obtain a training sample set, which includes multiple training samples. Each training sample contains a sample image and the ground truth annotation information corresponding to the sample image. The ground truth annotation information is used to mark the objects contained in the sample image. The objects contained in the sample images of the multiple training samples are of different scales.

[0036] Based on the training sample set, the image recognition model to be trained is iteratively trained to obtain the trained image recognition model; wherein, in one iteration, the following operations are performed:

[0037] The selected training samples are input into the image recognition model to be trained to obtain the predicted labeling information of the objects contained in the training samples;

[0038] Based on the difference between the predicted annotation information and the corresponding real annotation information, a loss function is constructed to adjust the parameters of the image recognition model to be trained.

[0039] An electronic device provided in this application includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of any of the above-described image recognition methods.

[0040] This application provides a computer-readable storage medium including a computer program. When the computer program is run on an electronic device, the computer program is used to cause the electronic device to perform the steps of any of the above-described image recognition methods.

[0041] This application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. When the processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of any of the above-described image recognition methods.

[0042] The beneficial effects of this application are as follows:

[0043] This application provides an image recognition method, apparatus, electronic device, and storage medium. By extracting features from the image to be recognized to obtain at least two sampling features with different scales, and then fusing multi-scale features based on these two sampling features, the application can enhance the feature dimensions of the image to be recognized at multiple scales. Furthermore, the at least one fused feature and the sampling features from the at least two sampling features that conform to the target scale are all used as features to be weighted. Then, based on each feature to be weighted and its corresponding preset weight, a target feature is obtained. The preset weight fully balances the scale differences between different features to be weighted, improving the accuracy of multi-scale image recognition. Finally, the image to be recognized is identified based on the target feature to obtain at least one object contained within it. Compared to related technologies that use a large number of training samples to train the recognition model, this method simplifies the recognition process, requiring only a small number of parameters for feature fusion. It improves recognition accuracy without requiring a large amount of time for model training, effectively improving recognition efficiency.

[0044] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0045] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0046] Figure 1 This is a schematic diagram of a cartoon face image at different scales in related technologies;

[0047] Figure 2 This is a schematic diagram of the structure of an image multi-scale enhancement method in related technologies;

[0048] Figure 3 This is an optional schematic diagram of an application scenario in an embodiment of this application;

[0049] Figure 4 This is a flowchart illustrating an implementation of an image recognition method according to an embodiment of this application;

[0050] Figure 5 This is a schematic diagram of the structure of a feature extraction network in an embodiment of this application;

[0051] Figure 6 This is a schematic diagram of the structure of a feature-weighted fusion network in an embodiment of this application;

[0052] Figure 7A This is a schematic diagram of a network structure based on a feature pyramid in an embodiment of this application;

[0053] Figure 7B This is a schematic diagram of another feature pyramid-based network structure in an embodiment of this application;

[0054] Figure 8 This is a schematic diagram of the training process of a cartoon face recognition model in an embodiment of this application;

[0055] Figure 9 This is a schematic diagram of the overall process of a cartoon face recognition method in an embodiment of this application;

[0056] Figure 10 This is a logical schematic diagram of an image recognition method according to an embodiment of this application;

[0057] Figure 11 This is a schematic diagram of the structure of an image recognition device according to an embodiment of this application;

[0058] Figure 12 This is a schematic diagram of the hardware structure of an electronic device using an embodiment of this application;

[0059] Figure 13 This is a schematic diagram of the hardware structure of another electronic device using an embodiment of this application. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0061] The following describes some of the concepts involved in the embodiments of this application.

[0062] Image scale: refers to the representation of an image at different scales. The description of objects in reality must be at the scale used to model nature. For example, a tree can only be described as a tree when observed at a distance of a few centimeters to a few meters, while at a distance of micrometers, it can only be described as a cell. When performing image recognition, the objects contained in the image need to be accurately identified at different scales.

[0063] Sampling features: refers to the features obtained by feature extraction from an image. Sampling features can be obtained through a downsampling network, which generally includes operations such as convolution calculation, non-linear activation function and pooling calculation. In this embodiment of the application, the size of the sampling feature obtained after one downsampling is reduced by half as an example. For example, if a 64*64 image is input into the downsampling network, the output feature after downsampling is 32*32.

[0064] Fusion feature: refers to the feature obtained after fusing two features to be fused. In this embodiment, the two features to be fused can be two sampled features, or a fused feature and a sampled feature. In this embodiment, an optional feature fusion method is to add the two features to be fused together. Before adding and fusing, the scale of the two features to be fused needs to be unified by upsampling, and the number of channels of the two features to be fused needs to be unified by channel adaptation.

[0065] Features to be weighted: refers to features that need to be fused based on preset weights. In this embodiment, features to be weighted include fused features and sampled features at the target scale. The scales of different features to be weighted are also different.

[0066] The embodiments of this application relate to artificial intelligence (AI), natural language processing (NLP), and machine learning (ML) technologies, and are designed based on computer vision technology and machine learning in artificial intelligence.

[0067] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence.

[0068] Artificial intelligence (AI) studies the design principles and implementation methods of various intelligent machines, enabling them to perceive, reason, and make decisions. AI technology mainly includes computer vision, natural language processing, machine learning / deep learning, autonomous driving, and intelligent transportation. With the research and advancement of AI technology, it is being researched and applied in multiple fields, such as smart homes, intelligent customer service, virtual assistants, smart speakers, intelligent marketing, autonomous driving, robotics, and smart healthcare. It is believed that with further technological development, AI will be applied in even more fields and play an increasingly important role.

[0069] Computer vision is the science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing, tracking, and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0070] Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence. Its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning. The image recognition model in the embodiments of this application is trained using machine learning or deep learning techniques. Based on the image recognition method in the embodiments of this application, the accuracy and efficiency of cartoon face recognition in images of different scales can be improved.

[0071] The design concept of the embodiments of this application is briefly introduced below:

[0072] With increasing emphasis on intellectual property rights and brand protection, the demand for content review before the release of multimedia content is also rising. For example, reviewing cartoon faces in multimedia content to protect copyright requires facial recognition. Figure 1 The cartoon face images shown can be of different sizes for different cartoon characters. Because this type of cartoon face is usually drawn in a two-dimensional style, the proportions of the facial features are arbitrarily designed, making it impossible to recognize them by detecting key points of natural faces.

[0073] Because cartoon face images have inconsistent input scales, but the cartoon face recognition model deployed in actual business applications is required to stably recognize data at various scales, related technologies mainly improve the accuracy of cartoon face recognition by means of the following methods:

[0074] Method 1: Collect a large amount of cartoon face data, increase the network depth and width of the cartoon face recognition model, thereby improving its recognition accuracy;

[0075] Method 2: Using an image pyramid approach, images are enhanced at different scales to obtain a large number of training samples. Recognition models at different scales are then introduced to extract features. Based on this, a cartoon face recognition model is trained to achieve cartoon face recognition at different scales.

[0076] However, based on the first method described above, cartoon face recognition requires enormous resources for data collection and annotation, and the data scale cannot reach the level of real face images. Furthermore, while increasing the size of the recognition model is a common practice to improve accuracy, in some applications with high forward inference speed requirements, an excessively large model leads to unacceptable time consumption. Based on the second method described above, for example... Figure 2 As shown, an image pyramid approach is used to perform multi-scale enhancement on the image to be recognized. On the one hand, this method requires multi-scale sampling of the input image, generating multiple images during runtime, significantly increasing storage consumption. On the other hand, training different models separately for images of different scales greatly increases the time consumption, resulting in low recognition efficiency. Furthermore, introducing models of different scales for feature extraction requires adjusting the weight ratios between the models, and determining the impact of weight hyperparameters on the corresponding training cycle is difficult and not easily achievable. Therefore, improving the efficiency and accuracy of cartoon face recognition has become an urgent problem to be solved.

[0077] In view of this, embodiments of this application provide an image recognition method, apparatus, electronic device, and storage medium. The method involves extracting features from an image to be recognized to obtain at least two sampling features with different scales; then, based on these at least two sampling features, performing multi-scale feature fusion between different scales to enhance the feature dimensions of the image to be recognized; furthermore, the obtained at least one fused feature and the sampling features from the at least two sampling features that conform to the target scale are all used as features to be weighted; then, based on each feature to be weighted and its corresponding preset weight, a target feature is obtained. The preset weights effectively balance the scale differences between different features to be weighted, improving the accuracy of multi-scale image recognition; finally, the image to be recognized is identified based on the target feature to obtain at least one object contained within it. Compared to related technologies that use a large number of training samples to train the recognition model, obtaining the object contained in the image to be recognized through the above method is simpler, requiring only a small number of parameters for feature fusion. While improving recognition accuracy, it eliminates the need for extensive model training, effectively improving recognition efficiency.

[0078] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0079] like Figure 3 The diagram shown is an application scenario illustration of an embodiment of this application. The application scenario diagram includes two terminal devices 310 and one server 320.

[0080] In this application embodiment, the terminal device 310 includes, but is not limited to, mobile phones, tablets, laptops, desktop computers, e-book readers, smart voice interaction devices, smart home appliances, and in-vehicle terminals. The terminal device may have an image recognition-related client installed, which can be software (such as a browser, short video software, etc.), or a webpage, mini-program, etc. The server 320 is the backend server corresponding to the software, webpage, mini-program, etc., or a server specifically used for image recognition; this application does not impose specific limitations. The server 320 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0081] It should be noted that the image recognition method in this embodiment can be executed by an electronic device, which can be a server 320 or a terminal device 310. That is, the method can be executed by the server 320 or the terminal device 310 alone, or by both the server 320 and the terminal device 310. For example, when executed by both the server 320 and the terminal device 310, the terminal device 310 acquires the image to be recognized and sends it to the server 320. The server 320 extracts features from the image to be recognized to obtain at least two sampling features with different scales. Based on the at least two sampling features, it performs multi-scale feature fusion between different scales to obtain at least one fused feature. The at least one fused feature and the sampling features that conform to the target scale from the at least two sampling features are all used as features to be weighted. Based on each feature to be weighted and its corresponding preset weight, a target feature is obtained. Based on the target feature, the image to be recognized is recognized to obtain at least one object contained in the image to be recognized. The server 320 sends the obtained objects contained in the image to be recognized to the terminal device 310 so that the terminal device 310 can perform content review on the objects in the image to be recognized.

[0082] In one alternative implementation, the terminal device 310 and the server 320 can communicate via a communication network.

[0083] In one alternative implementation, the communication network is a wired network or a wireless network.

[0084] It should be noted that, Figure 3 The examples shown are merely illustrative; in reality, the number of terminal devices and servers is unlimited and is not specifically limited in the embodiments of this application.

[0085] In this application embodiment, when there are multiple servers, the multiple servers can form a blockchain, and the server is a node on the blockchain; as disclosed in the image recognition method of this application embodiment, the image to be recognized can be stored on the blockchain.

[0086] Furthermore, the embodiments of this application can be applied to various scenarios, including not only image recognition scenarios, but also scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving. For example, when this application is applied to an artificial intelligence scenario, when a user publishes an image through a social networking site, the image recognition method in this application can be used to review the content of the image before publication to determine whether the objects contained in the image have copyright issues.

[0087] The image recognition method provided by the exemplary embodiments of this application will be described below with reference to the accompanying drawings and the application scenarios described above. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way.

[0088] See Figure 4 The diagram shown is a flowchart of an image recognition method provided in this application. Taking a server as the executing entity, the specific implementation process of this method includes the following steps S41-S44:

[0089] S41: The server extracts features from the image to be recognized and obtains at least two sampled features with different scales;

[0090] Feature extraction of the image to be recognized mainly involves extracting spatial features of the objects contained in the image. The obtained sampling features contain the spatial structure information of the image to be recognized. Taking the image to be recognized 1 as an example, there are two sampling features obtained by feature extraction: sampling feature 1 with a scale of 32*32 and sampling feature 2 with a scale of 64*64.

[0091] In one alternative implementation, step S41 is performed in the following manner:

[0092] The image to be recognized is input into the feature extraction network of the trained image recognition model; based on multiple downsampling layers in the feature extraction network, the image to be recognized is downsampled layer by layer to obtain the sampling features output by each downsampling layer.

[0093] Different downsampling layers output sampling features at different scales. For example, when the feature extraction network contains four downsampling layers, the downsampling of the image to be recognized layer by layer is as follows: First, downsampling layer 1 obtains sampling feature 1; then, sampling feature 1 is input into downsampling layer 2 for further downsampling to obtain sampling feature 2; sampling feature 2 is input into downsampling layer 3 for further downsampling to obtain sampling feature 3; and sampling feature 3 is input into downsampling layer 4 for further downsampling to obtain sampling feature 4. With each downsampling, the feature scale is halved. For example, if the image to be recognized is 256*256 pixels, then sampling feature 1 is 128*128 pixels. Sampling features 1-4 have different scales.

[0094] In this embodiment, the feature extraction network performs layer-by-layer downsampling on the image to be identified to obtain multiple sampled features, which can obtain richer features of the image to be identified, enhance the acquisition of global information of the image to be identified, and improve the recognition accuracy of the image to be identified.

[0095] See Figure 5The diagram shows the structure of a feature extraction network in this application embodiment. The network generally has four downsampling layers from the input image to the output feature. The output size of each convolutional layer inside each downsampling layer is the same. Each stage retains a fixed feature scale. After one downsampling, the feature scale is reduced by half and the number of channels is doubled. Each downsampling layer outputs a sampled feature.

[0096] S42: The server performs multi-scale feature fusion between different scales based on at least two sampled features to obtain at least one fused feature, and uses at least one fused feature and the sampled features that conform to the target scale from the at least two sampled features as features to be weighted.

[0097] The target scale can be a specified value. For example, if the target scale is 128*128, then the 128*128 sampled features will be used as weighted features. The target scale can also be a degree representation. For example, if sampled feature 1 is 128*128, sampled feature 2 is 64*64, sampled feature 3 is 32*32, and sampled feature 4 is 16*16, and the target scale is the minimum scale, then the sampled feature that meets the target scale is sampled feature 4.

[0098] In one alternative implementation, the sampling features that meet the target scale are the sampling features output by the last downsampling layer in the feature extraction network.

[0099] Specifically, taking the feature extraction network containing 4 downsampling layers as an example, the sampling feature at the target scale is the sampling feature 4 output by downsampling layer 4.

[0100] S43: The server obtains the target feature based on each feature to be weighted and its corresponding preset weight;

[0101] Specifically, each feature to be weighted corresponds to a preset weight. For example, the preset weight for feature 1 to be weighted is 0.5, the preset weight for feature 2 to be weighted is 0.3, and the preset weight for feature 3 to be weighted is 0.2. Based on each feature to be weighted and its corresponding preset weight, feature weighting and fusion are performed to obtain the target feature.

[0102] In an optional implementation, before step S43, the scale of each feature to be weighted is adjusted to obtain each feature to be weighted at a uniform scale; then step S43 is implemented in the following manner:

[0103] The target features are obtained by performing feature weighting and fusion based on each preset weight and the corresponding unified scale of the features to be weighted.

[0104] Specifically, since the scales of the features to be weighted are different, feature weighting and fusion need to be performed after the scale is unified. Convolutional layers can be used to adjust the scale of the features to be weighted, and after the scales of the features to be weighted are adjusted to a unified scale, feature weighting and fusion are performed based on the preset weights and the corresponding unified scale features to be weighted.

[0105] In this embodiment, by adjusting the scale of the features to be weighted, the scale of each feature to be weighted is unified, thereby performing feature weighting fusion, improving the accuracy of feature weighting fusion, and thus improving the recognition accuracy of the image to be recognized.

[0106] In one alternative implementation, step S43 is performed in the following manner:

[0107] First, each feature to be weighted is input into the mapping network of the trained image recognition model for feature mapping to obtain weighted features. Then, the weighted features are normalized to obtain the preset weights corresponding to each feature to be weighted. Finally, feature weighting and fusion are performed based on each preset weight and the corresponding feature to be weighted to obtain the target feature.

[0108] Specifically, the mapping network encodes the input features to be weighted, and then performs feature mapping to obtain weight features. The weight features can be normalized using the softmax (normalization) function, and the sum of each preset weight is 1.

[0109] In this embodiment, the mapping network automatically measures the weight of each feature to be weighted and automatically identifies features at the required scale, thereby improving the recognition accuracy of multi-scale images to be recognized.

[0110] See Figure 6 This is a schematic diagram of the structure of a feature weighted fusion network according to an embodiment of this application. It includes a fully connected mapping layer (i.e., mapping network), an activation layer, and a weighted fusion layer. The weighted fusion layer also includes a convolutional layer that performs scale unification on the features to be weighted. The activation function used in the activation layer can be softmax, various softmax with margins, or other activation functions such as ReLU. This application does not make specific limitations here.

[0111] Taking the input features as features to be weighted 1-3 as an example, the implementation process of the feature weighting fusion network includes: inputting features to be weighted 1, features to be weighted 2 and features to be weighted 3 into a fully connected mapping layer to obtain weight feature 1; inputting weight feature 1 into an activation layer to obtain a preset weight of 0.1 for features to be weighted 1, a preset weight of 0.3 for features to be weighted 2 and a preset weight of 0.6 for features to be weighted 3; inputting features to be weighted 1-3 and their corresponding preset weights into a weighting fusion layer; after the scale of features to be weighted 1-3 is unified by the convolutional layer of the weighting fusion layer, feature weighting fusion is performed on features to be weighted 1-3 and their corresponding preset weights to output target feature 1.

[0112] The input to the feature-weighted fusion network is the features to be weighted, which contain multiple scales, and the output of the fully connected mapping layer is a d2-dimensional vector w. fea (i.e., weighted features), where d2 is the number of input features. wfea obtains the weights of each input feature through a softmax activation layer, and then weights the activated features with the input features to obtain the final output features (i.e., target features). The feature weighting fusion network uses a fully connected mapping layer to automatically measure the weight of each feature to be weighted. During recognition, the network automatically identifies features at the required scale, improving the network's accuracy in recognizing multi-scale input images.

[0113] S44: The server identifies the image to be identified based on the target features, and obtains at least one object contained in the image to be identified.

[0114] In this embodiment, by extracting features from the image to be recognized, at least two sampling features with different scales are obtained. Based on these at least two sampling features, multi-scale feature fusion between different scales is performed, enabling multi-scale enhancement of the feature dimensions of the image to be recognized. Then, the at least one fused feature and the sampling features from the at least two sampling features that conform to the target scale are all used as features to be weighted. Next, based on each feature to be weighted and its corresponding preset weight, the target feature is obtained. The preset weight fully balances the scale differences between different features to be weighted, improving the accuracy of multi-scale image recognition. Finally, the image to be recognized is identified based on the target feature to obtain at least one object contained within it. Obtaining the object contained in the image to be recognized in the above manner is simpler than the method in related technologies that uses a large number of training samples to train the recognition model. It only requires adding a small number of parameters for feature fusion, improving recognition accuracy without consuming a large amount of time for model training, effectively improving recognition efficiency.

[0115] In one alternative implementation, step S42 is performed in the following manner:

[0116] If at least two sampled features are sampled features at two different scales, then the sampled features at two different scales are input into the feature fusion network in the trained image recognition model. Based on the feature fusion network, a multi-scale feature fusion is performed on the sampled features at two different scales to obtain the fused features.

[0117] If at least two sampling features are at least three different scales, then based on the feature fusion network in the trained image recognition model, at least two rounds of multi-scale feature fusion are performed on each sampling feature in ascending order of scale to obtain at least two different scales of fused features.

[0118] Specifically, if feature extraction of the image to be recognized yields two sampled features, then a multi-scale feature fusion is performed on the two sampled features to obtain a fused feature. If feature extraction of the image to be recognized yields at least three sampled features, taking four sampled features as an example, in ascending order of scale, they are: sampled feature 1, sampled feature 2, sampled feature 3, and sampled feature 4. First, a multi-scale feature fusion is performed on sampled feature 1 and sampled feature 2 to obtain fused feature 1. Then, a multi-scale feature fusion is performed on sampled feature 3 and fused feature 1 to obtain fused feature 2. Thus, two rounds of feature fusion are performed, yielding two fused features at different scales. Alternatively, a multi-scale feature fusion can be performed on sampled feature 4 and fused feature 2 to obtain fused feature 3. Thus, three rounds of feature fusion are performed, yielding three fused features at different scales.

[0119] It should be noted that when extracting features from the image to be recognized to obtain at least three sampling features, the method of performing at least two rounds of multi-scale feature fusion on each sampling feature in this application is not limited to the above embodiment. Other methods of combining sampling features and performing multi-scale feature fusion can be applied to the embodiments of this application.

[0120] In this embodiment, a feature fusion network is used to perform multi-scale feature fusion on the obtained sampling features, thereby achieving the ability to recognize images of different scales. The scale is expanded in the feature dimension to enhance the recognition accuracy of the cartoon recognition model for data of different scales.

[0121] In one optional implementation, if at least two sampling features include at least three sampling features of different scales, then the first round of multi-scale feature fusion is: based on the feature fusion network, multi-scale feature fusion is performed on the first two sampling features;

[0122] The subsequent rounds of multi-scale feature fusion are as follows: based on the feature fusion network, multi-scale feature fusion is performed on the fused features obtained in the previous round and the corresponding sampled features.

[0123] Specifically, if the number of sampling features is at least three, then in ascending order of scale, the first two sampling features are first subjected to multi-scale feature fusion to obtain fused features. In each subsequent round of multi-scale feature fusion, the objects are the fused features obtained in the previous round and the sampling features corresponding to the fused features obtained in the previous round. The scale of each fused feature and its corresponding sampling feature is a preset ratio, which can be a specific value. When the preset ratio is 0.5, the scale of each fused feature is 0.5 times the scale of the corresponding sampling feature. For example, if the scale of fused feature a is 16*16, then the scale of the corresponding sampling feature is 32*32.

[0124] In one optional implementation, multi-scale feature fusion is performed based on a feature fusion network to obtain fused features:

[0125] First, the feature with the lower scale among the two features to be fused is upsampled through the upsampling layer of the feature fusion network to obtain the corresponding first intermediate feature. The two features to be fused are either two sampled features of different scales, or the fused feature obtained in the previous round and the corresponding sampled feature. Then, the channel adaptation layer of the feature fusion network unifies the number of channels of the feature with the higher scale among the two features to be fused with the upsampled feature to obtain the corresponding second intermediate feature. Finally, the first intermediate feature and the second intermediate feature are added together and fused to obtain the corresponding fused feature.

[0126] Specifically, the first intermediate feature is the feature corresponding to the lower-scale feature among the two features to be fused. After upsampling the lower-scale feature once, the scale of the obtained first intermediate feature is increased to twice that of the feature before upsampling. The second intermediate feature is the feature corresponding to the higher-scale feature among the two features to be fused. It can be adapted by changing the number of channels based on a 1×1 convolutional layer. The number of channels of the second intermediate feature is consistent with the number of channels of the lower-scale feature among the two features to be fused.

[0127] In this embodiment, the scale of the two features to be fused is unified by an upsampling layer, and the number of channels of the two features to be fused is unified by a channel adaptation layer, which improves the feature fusion efficiency and thus improves the recognition efficiency of images to be recognized at different scales.

[0128] In one alternative implementation, the trained image recognition model is obtained by:

[0129] First, a training sample set is obtained, which includes multiple training samples. Each training sample contains a sample image and the corresponding ground truth annotation information. Then, the image recognition model to be trained is iteratively trained based on the training sample set to obtain the trained image recognition model.

[0130] In this context, the ground truth annotation information corresponding to the sample image is used to label the objects contained in the sample image. The sample images in multiple training samples contain objects at different scales, and a sample image can contain one or more objects, for example... Figure 1 The sample image shown is a single image containing multiple cartoon faces of different scales.

[0131] In one alternative implementation, the following operations are performed during a single loop iteration:

[0132] Step 1: Input the selected training samples into the image recognition model to be trained to obtain the predicted annotation information of the objects contained in the training samples;

[0133] Step 2: Based on the difference between the predicted annotation information and the corresponding real annotation information, construct a loss function and adjust the parameters of the image recognition model to be trained.

[0134] The loss function can be the cross-entropy loss function, or other loss functions such as the mean squared error loss function; this application does not make any specific restrictions here.

[0135] In this embodiment, the image recognition model to be trained is iteratively trained, and a loss function is constructed to adjust the model parameters. Based on the above method, the trained image recognition model can be obtained simply and efficiently, thereby enhancing the recognition accuracy of the image recognition model for images of different scales.

[0136] In one alternative implementation, during the iterative training process, if any of the following conditions are detected, the iterative training is stopped and the trained image recognition model is output:

[0137] Scenario 1: The value of the loss function is less than the preset loss threshold;

[0138] Scenario 2: The number of iterations reaches the preset threshold.

[0139] Specifically, when the value of the loss function is less than a preset loss threshold, or when the number of iterations reaches a preset threshold, the model training is complete, and the trained image recognition model is output. Furthermore, any condition for stopping iterative training during the iterative training process can be applied to the embodiments of this application.

[0140] See Figure 7AThis is a schematic diagram of a feature pyramid-based network structure in an embodiment of this application, including a feature extraction network and a feature fusion network. The feature extraction network has four downsampling stages from the input image to the output features. Each convolutional layer within each stage has a consistent output size, and each stage retains a fixed feature scale. Finally, a fully connected mapping unit maps the features output by downsampling layer 4 to features of a specified dimension d. A feature pyramid structure is introduced on top of the feature extraction network to enhance the feature dimensions at multiple scales. The input features of downsampling layers 2-4 are the output features of the previous downsampling layer. In this application, a multi-scale feature fusion module (i.e., a feature fusion network) is used to enhance the features at multiple scales. The multi-scale feature fusion module includes an upsampling module (i.e., an upsampling layer) and a channel adaptation module (i.e., a channel adaptation layer). The upsampling module upsamples the output feature maps of higher stages (downsampling layer 4, downsampling layer 3). In this application, interpolation is directly used to upsample the feature maps, and the upsampling module does not contain any parameters to be learned. The channel adaptation module adapts the output features of the current stage (downsampling layer 3, downsampling layer 2) by channel transformation, adapting the number of channels to be consistent with the upsampled features. The channel adaptation module mainly uses 1×1 convolutional layers for adaptation, which are the parameters to be learned. After the upsampling and channel adaptation modules, the features of the current stage are added and fused with the fused features of the previous stage to obtain fused features containing different scales.

[0141] The following is combined Figure 7A Taking an image 1 with an input scale of 128*128 as an example, the feature extraction network and feature fusion network in this embodiment are introduced. First, the input image 1 is downsampled by downsampling layer 1 to obtain sampled feature 1. Sampled feature 1 is input into downsampling layer 2 to obtain sampled feature 2. Sampled feature 2 is input into downsampling layer 3 to obtain sampled feature 3. Sampled feature 3 is input into downsampling layer 4 to obtain sampled feature 4. Then, sampled feature 4 and sampled feature 3 are input into the feature fusion network. The upsampling layer of the feature fusion network upsamples sampled feature 4 to obtain first intermediate feature 1. The channel adaptation layer adjusts the number of channels of sampled feature 3 to obtain second intermediate feature 1. The first intermediate feature 1 and the second intermediate feature 1 are added and fused to obtain fused feature 1. The fused feature 1 and sampled feature 2 are input into the feature fusion network. The fused feature 1 is upsampled to obtain first intermediate feature 2. The number of channels of sampled feature 2 is adjusted to obtain second intermediate feature 2. The first intermediate feature 2 and the second intermediate feature 2 are added and fused to obtain fused feature 2. Finally, fused feature 1, fused feature 2, and the sampled feature 4 with the smallest scale are all used as features to be weighted. After obtaining the features to be weighted, each feature can be input as follows: Figure 6 The feature-weighted fusion network shown performs feature-weighted fusion.

[0142] It should be noted that, Figure 7A Taking a feature extraction network containing four downsampling layers as an example, the feature extraction network in this application is described. In the embodiments of this application, the number of downsampling layers in the feature extraction network is not specifically limited, and the number of times multi-scale feature fusion is performed on the sampled features is not specifically limited.

[0143] See Figure 7B This is a schematic diagram of another feature pyramid-based network structure in an embodiment of this application. The feature extraction network contains 5 downsampling layers. After downsampling the input image 1 through the feature extraction network, the sampled features 3-5 output by downsampling layers 3-5 can be fused twice using multi-scale features, and the sampled feature 5 with the smallest scale can also be used as the feature to be weighted, finally obtaining the feature to be weighted 1-3; or, the sampled features 2-5 output by downsampling layers 2-5 can be fused three times using multi-scale features, and the sampled feature 5 with the smallest scale can also be used as the feature to be weighted, finally obtaining the feature to be weighted 1-4, and so on.

[0144] The image recognition method in this embodiment can be applied to cartoon face recognition. First, the cartoon recognition network module is initialized and trained. Then, a feature pyramid structure, a multi-scale feature integration module (feature fusion network), and a feature weighted fusion layer are added to the cartoon recognition network module. The cartoon recognition network module after the addition is then jointly fine-tuned and trained.

[0145] Step 1: Refer to Figure 8 This is a schematic diagram of the training process of a cartoon face recognition model in an embodiment of this application. First, the cartoon recognition network module is initialized and trained. The initialization training process includes the following steps:

[0146] S81: Prepare training data;

[0147] S82: Extract spatial features from cartoon face images;

[0148] S83: Calculate the objective function for face recognition;

[0149] S84: Determine whether the conditions for terminating model training are met. If yes, output the trained cartoon recognition network. If no, proceed to step S85.

[0150] S85: Optimize the objective function.

[0151] The above initialization training process can be implemented through the following modules:

[0152] (a) Training Data Preparation Module: This module reads the cartoon face training data during the training process, combines the read data into a batch, and sends it to the cartoon recognition network unit for processing. The cartoon image size is normalized to 256*256, which is the input size of the cartoon recognition network, and then the image pixel values ​​are normalized to [-1, 1].

[0153] (b) Cartoon Recognition Network Module: This module extracts spatial features from cartoon face images, and the output feature map retains the spatial structure information of the face image. This module typically uses a convolutional neural network structure, including operations such as convolution calculations, non-linear activation function calculations, and pooling calculations. During initial training, the cartoon recognition network module only includes the feature extraction network;

[0154] (c) Face Recognition Objective Function Calculation Module: This module takes the feature f output by the fully connected mapping unit of the cartoon recognition network module and the label information of the face image that generates the vector as input to calculate the objective function value. The objective function can be a classification function (such as softmax, various types of softmax with margin), or other types of objective functions can be used;

[0155] (d) Face Recognition Objective Function Optimization Module: This module trains and optimizes the entire network using gradient descent methods (such as stochastic gradient descent, stochastic gradient descent with a driving term, Adam, Adamard). Steps (a) to (d) are repeated during training until the training result meets the training termination condition. The termination condition is generally set to the number of iterations meeting a predetermined value, or the loss value calculated for the cartoon face recognition objective function being less than a predetermined value.

[0156] Step 2: Fine-tuning the feature pyramid-based network unit modules

[0157] In this step, the following steps are adopted: Figure 8 The training process in the middle will Figure 8 The structure of the cartoon recognition network module in this step is replaced with the image recognition model proposed in this application, including a feature pyramid-based network unit module (including a feature extraction network and a feature fusion network) and an output feature fusion module (i.e., a feature weighted fusion network). In this step, the backbone network of the feature pyramid-based network unit module still adopts the backbone network structure in step one, and its parameters are the parameters of the network units obtained from the initial training in step one. This feature pyramid network only adds a small number of network parameters to be updated. Therefore, in this step of training, fine-tuning is adopted, and the feature pyramid network structure can be optimized by a search method (NAS), while optimizing its connection position to the backbone network.

[0158] The feature pyramid-based network unit module in this embodiment can improve the recognition accuracy of cartoon face recognition systems for multi-scale input images. Furthermore, the feature pyramid structure network only adds a small number of computational parameters, thus its deployment model only increases runtime memory and time, making the cartoon recognition model adaptable to various complex application scenarios.

[0159] See Figure 9 This is a schematic diagram of the overall process of a cartoon face recognition method according to an embodiment of this application, used to improve the accuracy of cartoon face recognition based on a feature pyramid structure. The overall process is as follows: Figure 9 As shown, the method mainly consists of two stages: network module training and network module deployment. In the network module training stage, the cartoon face recognition network is first initialized and trained to obtain network units with preliminary recognition capabilities. Then, a feature pyramid structure is added to the cartoon face recognition network units and applied to the cartoon face recognition network. Finally, the network is fine-tuned using cartoon face data. In the network module deployment stage, only the designed recognition network needs to be exported and deployed. This method does not require modification to the original recognition process; it only needs to improve the training backbone model to enhance the accuracy of cartoon face recognition. This application's embodiment addresses the problem of cartoon faces being unable to be registered and aligned, yet exhibiting scale diversity, based on the feature pyramid structure. Multi-scale enhancement is applied to the features in the training network, thereby improving the recognition accuracy of the cartoon face recognition model at various scales.

[0160] See Figure 10 This is a logical schematic diagram of an image recognition method in an embodiment of this application. The input is an image to be recognized, a, and features are extracted from the image to be recognized, obtaining sampled features a, sampled features b, sampled features c and sampled features d. Then, multi-scale feature fusion is performed on the sampled features d to obtain features a, features b and features c to be weighted. Based on the preset weights of the features c to be weighted, feature weighting fusion is performed to obtain target feature 1. The target feature 1 is recognized to obtain the prediction annotation information of the image to be recognized, a.

[0161] This application employs a feature pyramid approach to address the multi-scale problem encountered by cartoon recognition models, ensuring the recognition accuracy of the cartoon recognition model at various scales. Simultaneously, to guarantee operational efficiency in practical deployments, it is necessary to control the time consumption and runtime resource consumption of the cartoon recognition model. This invention starts from the structure of the recognition model, utilizing the scale sampling information inherent in the cartoon recognition model itself, and introduces a feature pyramid structure to expand the scale along the feature dimension, thereby enhancing the recognition accuracy of the cartoon recognition model for data at different scales. This application only adds a small number of computational layers during the model's forward pass, which only increases memory consumption and inference time slightly, making it applicable to various real-world scenarios.

[0162] Based on the same inventive concept, embodiments of this application also provide an image recognition device. For example... Figure 11 As shown, this is a schematic diagram of the structure of the image recognition device 1100, which may include:

[0163] The extraction unit 1101 is used to extract features from the image to be recognized and obtain at least two sampled features with different scales.

[0164] The fusion unit 1102 is used to perform multi-scale feature fusion between different scales based on at least two sampled features to obtain at least one fused feature, and to take at least one fused feature and the sampled features that conform to the target scale from the at least two sampled features as features to be weighted.

[0165] The weighting unit 1103 is used to obtain the target feature based on each feature to be weighted and its corresponding preset weight;

[0166] The recognition unit 1104 is used to recognize the image to be recognized based on the target features, and to obtain at least one object contained in the image to be recognized.

[0167] Optionally, the fusion unit 1102 is specifically used for:

[0168] If at least two sampled features are sampled features at two different scales, then the sampled features at two different scales are input into the feature fusion network in the trained image recognition model. Based on the feature fusion network, a multi-scale feature fusion is performed on the sampled features at two different scales to obtain the fused features.

[0169] If at least two sampling features are at least three different scales, then based on the feature fusion network in the trained image recognition model, at least two rounds of multi-scale feature fusion are performed on each sampling feature in ascending order of scale to obtain at least two different scales of fused features.

[0170] Optionally, if at least two sampled features include at least three sampled features at different scales, then the first round of multi-scale feature fusion is: based on the feature fusion network, multi-scale feature fusion is performed on the first two sampled features;

[0171] The multi-scale feature fusion in subsequent rounds is as follows: based on the feature fusion network, the fused features obtained in the previous round and the corresponding sampled features are fused at multiple scales, where the scale of each fused feature and the corresponding sampled feature is a preset ratio.

[0172] Optionally, multi-scale feature fusion can be performed based on a feature fusion network to obtain fused features:

[0173] The upsampling layer of the feature fusion network upsamples the feature with the lower scale among the two features to be fused to obtain the corresponding first intermediate feature. The two features to be fused are: two sampled features of different scales, or the fused feature obtained in the previous round and the corresponding sampled feature.

[0174] By using the channel adaptation layer of the feature fusion network, the higher-scale feature in the two features to be fused is unified with the upsampled feature in terms of channel number to obtain the corresponding second intermediate feature.

[0175] The first intermediate feature and the second intermediate feature are added together and fused to obtain the corresponding fused feature.

[0176] Optionally, weighting unit 1103 is specifically used for:

[0177] Each feature to be weighted is input into the mapping network of the trained image recognition model to perform feature mapping and obtain the weighted features;

[0178] Normalize the weighted features to obtain the preset weights for each feature to be weighted.

[0179] The target feature is obtained by performing feature weighting and fusion based on each preset weight and the corresponding feature to be weighted.

[0180] Optionally, the device also includes an adjustment unit 1105 for:

[0181] The scale of each feature to be weighted is adjusted to obtain each feature to be weighted at a uniform scale;

[0182] Weighting unit 1103 is specifically used for:

[0183] The target features are obtained by performing feature weighting and fusion based on each preset weight and the corresponding unified scale of the features to be weighted.

[0184] Optionally, the extraction unit 1101 is specifically used for, including:

[0185] The image to be recognized is input into the feature extraction network of a trained image recognition model;

[0186] Based on multiple downsampling layers in the feature extraction network, the image to be recognized is downsampled layer by layer to obtain the sampling features output by each downsampling layer. The sampling features output by different downsampling layers have different scales.

[0187] Optionally, the sampling features that meet the target scale are the sampling features output by the last downsampling layer in the feature extraction network.

[0188] Optionally, the trained image recognition model can be obtained in the following ways:

[0189] Obtain a training sample set, which includes multiple training samples. Each training sample contains a sample image and the corresponding ground truth annotation information. The ground truth annotation information is used to label the objects contained in the sample image. The objects contained in the sample images of multiple training samples are at different scales.

[0190] The image recognition model to be trained is iteratively trained based on the training sample set to obtain a trained image recognition model; wherein, in one iteration, the following operations are performed:

[0191] The selected training samples are input into the image recognition model to be trained to obtain the predicted labeling information of the objects contained in the training samples.

[0192] Based on the difference between the predicted annotation information and the corresponding real annotation information, a loss function is constructed to adjust the parameters of the image recognition model to be trained.

[0193] In this embodiment, by extracting features from the image to be recognized, at least two sampling features with different scales are obtained. Based on these at least two sampling features, multi-scale feature fusion between different scales is performed, which can enhance the feature dimensions of the image to be recognized at multiple scales. Then, the at least one fused feature and the sampling features that conform to the target scale from the at least two sampling features are all used as features to be weighted. Next, based on each feature to be weighted and its corresponding preset weight, the target feature is obtained. The preset weight fully balances the scale differences between different features to be weighted, improving the accuracy of multi-scale image recognition. Finally, the image to be recognized is recognized based on the target feature to obtain at least one object contained within it. Obtaining the object contained in the image to be recognized in the above manner is simpler than the method in related technologies that uses a large number of training samples to train the recognition model. It only requires adding a small number of parameters for feature fusion, improving recognition accuracy without spending a lot of time on model training, effectively improving recognition efficiency.

[0194] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware components.

[0195] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0196] Based on the same inventive concept as the above-described method embodiments, this application also provides an electronic device. In one embodiment, the electronic device may be a server, such as... Figure 3 The server 320 is shown. In this embodiment, the structure of the electronic device can be as follows: Figure 12 As shown, it includes a memory 1201, a communication module 1203, and one or more processors 1202.

[0197] The memory 1201 is used to store computer programs executed by the processor 1202. The memory 1201 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and programs required to run instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0198] Memory 1201 may be volatile memory, such as random-access memory (RAM); memory 1201 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 1201 may be any other medium capable of carrying or storing a desired computer program having the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1201 may be a combination of the above-described memories.

[0199] The processor 1202 may include one or more central processing units (CPUs) or digital processing units, etc. The processor 1202 is used to implement the image recognition method described above when it calls the computer program stored in the memory 1201.

[0200] The communication module 1203 is used to communicate with terminal devices and other servers.

[0201] This application embodiment does not limit the specific connection medium between the memory 1201, communication module 1203, and processor 1202. This application embodiment... Figure 12 The memory 1201 and the processor 1202 are connected via a bus 1204, and the bus 1204 is in Figure 12 The diagram uses thick lines to describe the connections between other components; these are for illustrative purposes only and should not be considered limiting. The 1204 bus can be divided into address bus, data bus, control bus, etc. For ease of description, Figure 12It is described using only a thick line, but does not indicate that there is only one bus or one type of bus.

[0202] The memory 1201 stores a computer storage medium, which stores computer-executable instructions for implementing the image recognition method of this application embodiment. The processor 1202 is used to execute the above-described image recognition method, such as... Figure 4 As shown.

[0203] In another embodiment, the electronic device may also be other electronic devices, such as... Figure 3 The terminal device 310 is shown. In this embodiment, the electronic device can be structured as follows: Figure 13 As shown, it includes components such as: communication component 1310, memory 1320, display unit 1330, camera 1340, sensor 1350, audio circuit 1360, Bluetooth module 1370, processor 1380, etc.

[0204] The communication component 1310 is used to communicate with the server. In some embodiments, it may include a Circuit-Wireless Fidelity (WiFi) module, which is a short-range wireless transmission technology. Electronic devices can use the WiFi module to help users send and receive information.

[0205] The memory 1320 can be used to store software programs and data. The processor 1380 executes various functions of the terminal device 310 and performs data processing by running the software programs or data stored in the memory 1320. The memory 1320 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. The memory 1320 stores an operating system that enables the terminal device 310 to run. In this application, the memory 1320 may store the operating system and various application programs, and may also store a computer program that executes the image recognition method of the embodiments of this application.

[0206] The display unit 1330 can also be used to display information input by the user or information provided to the user, as well as various menus of the terminal device 310, forming a graphical user interface (GUI). Specifically, the display unit 1330 may include a display screen 1332 disposed on the front of the terminal device 310. The display screen 1332 may be configured as a liquid crystal display, a light-emitting diode, or the like. The display unit 1330 can be used to display the image recognition user interface, etc., as described in the embodiments of this application.

[0207] The display unit 1330 can also be used to receive input digital or character information and generate signal inputs related to user settings and function control of the terminal device 310. Specifically, the display unit 1330 may include a touch screen 1331 disposed on the front of the terminal device 310, which can collect touch operations of the user on or near it, such as clicking a button, dragging a scroll box, etc.

[0208] The touchscreen 1331 can be placed over the display screen 1332, or the touchscreen 1331 and the display screen 1332 can be integrated to realize the input and output functions of the terminal device 310. After integration, it can be referred to as a touch display screen. In this application, the display unit 1330 can display the application program and the corresponding operation steps.

[0209] Camera 1340 can be used to capture still images, which users can then post comments on via the application. There can be one or multiple cameras 1340. An object is projected onto a photosensitive element through a lens, generating an optical image. This photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to the processor 1380 for conversion into a digital image signal.

[0210] The terminal device may also include at least one sensor 1350, such as an accelerometer 1351, a proximity sensor 1352, a fingerprint sensor 1353, and a temperature sensor 1354. The terminal device may also be equipped with other sensors such as a gyroscope, barometer, hygrometer, thermometer, infrared sensor, light sensor, and motion sensor.

[0211] Audio circuitry 1360, speaker 1361, and microphone 1362 provide an audio interface between the user and terminal device 310. Audio circuitry 1360 converts received audio data into electrical signals, which are then transmitted to speaker 1361, where they are converted into sound signals for output. Terminal device 310 may also be equipped with volume buttons for adjusting the volume of the sound signal. Conversely, microphone 1362 converts collected sound signals into electrical signals, which are then received by audio circuitry 1360, converted into audio data, and output to communication component 1310 for transmission to, for example, another terminal device 310, or to memory 1320 for further processing.

[0212] The Bluetooth module 1370 is used to interact with other Bluetooth devices that also have a Bluetooth module via the Bluetooth protocol. For example, a terminal device can establish a Bluetooth connection with a wearable electronic device (such as a smartwatch) that also has a Bluetooth module through the Bluetooth module 1370, thereby exchanging data.

[0213] The processor 1380 is the control center of the terminal device, connecting various parts of the terminal through various interfaces and lines. It executes various functions and processes data by running or executing software programs stored in the memory 1320 and calling data stored in the memory 1320. In some embodiments, the processor 1380 may include one or more processing units; the processor 1380 may also integrate an application processor and a baseband processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the baseband processor mainly handles wireless communication. It is understood that the baseband processor may not be integrated into the processor 1380. In this application, the processor 1380 can run an operating system, applications, user interface display and touch response, as well as the image recognition method of the embodiments of this application. Furthermore, the processor 1380 is coupled to the display unit 1330.

[0214] In some possible implementations, various aspects of the image recognition method provided in this application can also be implemented as a program product, which includes a computer program. When the program product is run on an electronic device, the computer program causes the electronic device to perform the steps of the image recognition method according to the various exemplary embodiments of this application described above. For example, the electronic device can perform actions such as... Figure 4 The steps are shown in the figure.

[0215] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0216] The program product of the embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include a computer program, and may run on an electronic device. However, the program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with a command execution system, apparatus, or device.

[0217] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a readable computer program. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with a command execution system, apparatus, or device.

[0218] Computer programs contained on readable media may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0219] Computer programs for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The computer program can execute entirely on the user's electronic device, partially on the user's device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the user's electronic device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0220] It should be noted that, in practice, according to the embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0221] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing a computer-usable computer program.

[0222] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce a machine for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0223] These computer program commands may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the commands stored in the computer-readable storage medium produce an article of manufacture including command means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0224] These computer program commands can also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing the commands executed on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0225] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0226] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An image recognition method, characterized in that, The method includes: Feature extraction is performed on the cartoon face contained in the image to be recognized to obtain at least two sampled features with different scales; Based on the at least two sampling features, multi-scale feature fusion is performed between the different scales to obtain at least one fused feature. The at least one fused feature and the sampling features that conform to the target scale among the at least two sampling features are all used as features to be weighted. The sampling features that conform to the target scale are the sampling features output by the last downsampling layer in the feature extraction network. Each of the features to be weighted is input into the mapping network of the trained image recognition model. After encoding each of the features to be weighted through the mapping network, feature mapping is performed to obtain weight features. The weight features are then normalized through an activation layer to obtain the preset weights corresponding to each of the features to be weighted. The scale of each of the features to be weighted is adjusted through a weighted fusion layer to obtain each feature to be weighted at a uniform scale. Finally, feature weighted fusion is performed based on each preset weight and the corresponding uniform scaled feature to obtain the target feature. Based on the target features, the image to be identified is identified to obtain at least one cartoon face contained in the image to be identified; The step of fusing multi-scale features across different scales based on at least two sampled features to obtain at least one fused feature includes: If the at least two sampling features are sampling features of two different scales, then the two sampling features of different scales are input into the feature fusion network in the trained image recognition model, and a multi-scale feature fusion is performed on the two sampling features of different scales based on the feature fusion network to obtain the fused features; If the at least two sampling features are at least three sampling features of different scales, then based on the feature fusion network in the trained image recognition model, at least two rounds of multi-scale feature fusion are performed on each sampling feature in ascending order of scale to obtain at least two fused features of different scales.

2. The method as described in claim 1, characterized in that, If the at least two sampling features include at least three sampling features of different scales, then the first round of multi-scale feature fusion is: based on the feature fusion network, multi-scale feature fusion is performed on the first two sampling features; The subsequent rounds of multi-scale feature fusion are as follows: based on the feature fusion network, multi-scale feature fusion is performed on the fused features obtained in the previous round and the corresponding sampled features, wherein the scale of each fused feature and the corresponding sampled feature is a preset ratio.

3. The method as described in claim 1, characterized in that, The fused features are obtained by performing multi-scale feature fusion based on the feature fusion network in the following manner: The feature fusion network upsamples the feature with the lower scale among the two features to be fused by the upsampling layer to obtain the corresponding first intermediate feature. The two features to be fused are: the two sampled features of different scales, or the fused feature obtained in the previous round and the corresponding sampled feature. Through the channel adaptation layer of the feature fusion network, the feature with the higher scale among the two features to be fused is unified with the upsampled feature in terms of channel number to obtain the corresponding second intermediate feature; The first intermediate feature and the second intermediate feature are added and fused to obtain the corresponding fused feature.

4. The method as described in claim 1, characterized in that, The process involves extracting features from the cartoon face contained in the image to be recognized, obtaining at least two sampled features with different scales, including: The image to be identified is input into the feature extraction network of a trained image recognition model; Based on multiple downsampling layers in the feature extraction network, the cartoon face contained in the image to be identified is downsampled layer by layer to obtain the sampling features output by each downsampling layer. The sampling features output by different downsampling layers have different scales.

5. The method according to any one of claims 1 to 4, characterized in that, The trained image recognition model is obtained through the following method: Obtain a training sample set, which includes multiple training samples. Each training sample contains a sample image and the corresponding ground truth annotation information. The ground truth annotation information is used to mark the cartoon faces contained in the sample image. The cartoon faces contained in the sample images of the multiple training samples are of different scales. Based on the training sample set, the image recognition model to be trained is iteratively trained to obtain the trained image recognition model; wherein, in one iteration, the following operations are performed: The selected training samples are input into the image recognition model to be trained to obtain the predicted annotation information of the cartoon faces contained in the training samples. Based on the difference between the predicted annotation information and the corresponding real annotation information, a loss function is constructed to adjust the parameters of the image recognition model to be trained.

6. An image recognition device, characterized in that, include: The extraction unit is used to extract features from the cartoon face contained in the image to be recognized, and obtain at least two sampled features with different scales; The fusion unit is used to perform multi-scale feature fusion between the different scales based on the at least two sampling features to obtain at least one fused feature, and to take the at least one fused feature and the sampling features that conform to the target scale from the at least two sampling features as features to be weighted; wherein, the sampling features that conform to the target scale are the sampling features output by the last downsampling layer in the feature extraction network; The weighting unit is used to input each of the features to be weighted into the mapping network of the trained image recognition model, encode each of the features to be weighted through the mapping network, perform feature mapping to obtain weight features, normalize the weight features through the activation layer to obtain the preset weights corresponding to each of the features to be weighted, adjust the scale of each of the features to be weighted through the weighted fusion layer to obtain each feature to be weighted at a unified scale, and perform feature weighted fusion based on each preset weight and the corresponding unified scaled feature to obtain the target feature; The recognition unit is used to recognize the image to be recognized based on the target features, and to obtain at least one cartoon face contained in the image to be recognized; Specifically, the fusion unit is used for: If the at least two sampling features are sampling features of two different scales, then the two sampling features of different scales are input into the feature fusion network in the trained image recognition model, and a multi-scale feature fusion is performed on the two sampling features of different scales based on the feature fusion network to obtain the fused features; If the at least two sampling features are at least three sampling features of different scales, then based on the feature fusion network in the trained image recognition model, at least two rounds of multi-scale feature fusion are performed on each sampling feature in ascending order of scale to obtain at least two fused features of different scales.

7. The apparatus as claimed in claim 6, characterized in that, If the at least two sampling features include at least three sampling features of different scales, then the first round of multi-scale feature fusion is: based on the feature fusion network, multi-scale feature fusion is performed on the first two sampling features; The subsequent rounds of multi-scale feature fusion are as follows: based on the feature fusion network, multi-scale feature fusion is performed on the fused features obtained in the previous round and the corresponding sampled features, wherein the scale of each fused feature and the corresponding sampled feature is a preset ratio.

8. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of any one of the methods described in claims 1 to 5.

9. A computer-readable storage medium, characterized in that, It includes a computer program that, when run on an electronic device, causes the electronic device to perform the steps of any of the methods described in claims 1 to 5.

10. A computer program product, characterized in that, The method includes a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of any one of claims 1 to 5.

Images