Image recognition method, device and storage medium based on path fusion
Through the image recognition method based on path fusion, the batch normalized layer fusion characteristics of different noise intensities are used to solve the problem of insufficient robustness of deep learning models in actual business scenarios, and the accuracy and applicability of image recognition are improved.
Patent Information
- Application Number
- CN202110262494.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-03-10
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2041-03-10
AI Technical Summary
Deep learning models are vulnerable to attacks in real business scenarios, resulting in insufficient robustness, especially in scenarios involving consumer authentication. The prior art trades out natural precision and adversarial precision through adjustable adversarial training, but fixed tradeoff parameters It is difficult to adapt to scene changes, resulting in inconsistent model robustness and natural accuracy, affecting image recognition accuracy.
The image recognition method based on path fusion is adopted to extract image features through a shared convolutional layer, and feature fusion is performed using batch normalized layers of different noise intensities to generate fusion features to improve recognition accuracy and high applicability.
It improves the accuracy and robustness of image category recognition, has high applicability, is simple to operate, can effectively adapt in different scenarios, and improves the security of the model.
Smart Images

Figure CN115082873B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to an image recognition method, device and storage medium based on path fusion. Background Art
[0002] Deep learning has been widely used in many visual tasks, such as image classification, image detection, and face recognition. At present, when deep learning models are applied in actual business, attackers can change the model input by adding noise that is imperceptible to human vision / hearing to normal samples, thereby tampering with the model output. Especially in scenarios directly involving consumer identity authentication such as face recognition, if the robustness of the deep learning model is not high enough, it will bring great security risks.
[0003] The inventors of the present application have found in the process of research and practice that, in order to solve the risk problem of deep learning models, the prior art uses adjustable adversarial training (Once-for-all Adversarial Training, OAT) to train deep learning models. According to the risk level of different business scenarios, trade-off parameters are manually set to weigh the natural accuracy of the model (the accuracy of normal samples) and the adversarial accuracy (the accuracy of adversarial samples). For example, a larger trade-off parameter is set in a scenario with a higher risk and higher security requirements, in which case the natural accuracy of the model is lower but the adversarial accuracy is higher; a smaller trade-off parameter is set in a scenario with a lower risk and lower security requirements, in which case the natural accuracy of the model is higher but the adversarial accuracy is lower. However, in actual scenarios, business conditions may vary greatly over time, so fixed trade-off parameters are difficult to adapt to changing business scenarios. In addition, after introducing the trade-off parameter, setting a smaller trade-off parameter in a scenario with a lower risk and lower security requirements can reduce the damage to the natural accuracy, but the corresponding model robustness is not high. Therefore, there is still an inconsistency between the model robustness and the natural accuracy, which makes the model have low recognition accuracy for images and poor applicability. Summary of the invention
[0004] The embodiments of the present application provide an image recognition method, device and storage medium based on path fusion, which can improve the recognition accuracy of image categories, have simple operation and high applicability.
[0005] A first aspect of an embodiment of the present application provides an image recognition method based on path fusion, comprising:
[0006] Obtaining an image to be recognized, inputting the image to be recognized into an image recognition model, and extracting image features of the image to be recognized through a shared convolutional layer in the image recognition model;
[0007] Obtaining first normalized image features corresponding to the image features through a first batch of normalized layers in the image recognition model, and obtaining second normalized image features corresponding to the image features through a second batch of normalized layers in the image recognition model, wherein the first batch of normalized layers and the second batch of normalized layers are respectively trained by sample images with different noise intensities;
[0008] Obtaining a first weight and a second weight corresponding to the image feature through a pathway feature fusion module in the image recognition model, wherein the first weight is used to mark the weight occupied by the first normalized image feature, and the second weight is used to mark the weight occupied by the second normalized image feature, and the sum of the first weight and the second weight is 1;
[0009] Based on the first normalized image feature, the first weight, the second normalized image feature and the second weight, a fusion feature of the image to be identified is determined, and based on the fusion feature, an image category of the image to be identified is determined.
[0010] In combination with the first aspect, in a possible implementation manner, before inputting the image to be recognized into the image recognition model, the method further includes:
[0011] Acquire at least three first sample images including at least three noise intensities, input each first sample image into the image recognition model, and extract sample image features of each first sample image through a shared convolutional layer in the image recognition model;
[0012] Training at least three batch normalization layers in the image recognition model according to the sample image features of the first sample images to obtain at least three batch normalization layers after training, wherein the sample image features of the sample images of one noise intensity are used to train one batch normalization layer;
[0013] The first batch of normalized layers and the second batch of normalized layers of the image recognition model are determined from the at least three batch normalized layers, wherein the first batch of normalized layers are trained by sample images with the smallest noise intensity among the at least three sample images, and the second batch of normalized layers are trained by sample images with the largest noise intensity among the at least three sample images.
[0014] In combination with the first aspect, in a possible implementation manner, after determining the first batch normalization layers and the second batch normalization layers of the image recognition model from the at least three batch normalization layers, the method further includes:
[0015] The remaining batch normalization layers except the first batch normalization layer and the second batch normalization layer in the at least three batch normalization layers are deleted to obtain the batch normalization layers of the image recognition model.
[0016] In combination with the first aspect, in a possible implementation manner, before obtaining the first weight and the second weight corresponding to the image feature through the pathway feature fusion module in the image recognition model, the method further includes:
[0017] Obtaining a second sample image for training a pathway feature fusion module of the image recognition model, inputting the second sample image into the image recognition model, and extracting sample image features of the second sample image through a shared convolutional layer in the image recognition model;
[0018] Obtaining first normalized sample image features corresponding to the sample image features through the first batch of normalized layers in the image recognition model, and obtaining second normalized sample image features corresponding to the sample image features through the second batch of normalized layers in the image recognition model;
[0019] The pathway feature fusion module is trained based on the sample image features, the first normalized sample image features, the second normalized sample image features and the image category label of the second sample image, so that the pathway feature fusion module outputs the weight corresponding to the normalized image features output by the first batch of normalization layers for any image feature, and the weight corresponding to the normalized image features output by the second batch of normalization layers.
[0020] In combination with the first aspect, in a possible implementation, the training of the pathway feature fusion module based on the sample image feature, the first normalized sample image feature, the second normalized sample image feature, and the image category label of the second sample image includes:
[0021] Obtaining a first sample weight and a second sample weight corresponding to the sample image feature through the pathway feature fusion module in the image recognition model, wherein the first sample weight is used to mark the weight occupied by the first normalized sample image feature, and the second sample weight is used to mark the weight occupied by the second normalized sample image feature, and the sum of the first sample weight and the second sample weight is 1;
[0022] Determining a fusion feature of the second sample image based on the first normalized sample image feature, the first sample weight, the second normalized sample image feature and the second sample weight, and determining an image category of the second sample image based on the fusion feature of the second sample image;
[0023] The classification loss of the image recognition model is calculated according to the image category of the second sample image and the image category label of the second sample image, and the network parameters of the pathway fusion feature module are adjusted based on the classification loss to train the pathway feature fusion module.
[0024] In combination with the first aspect, in a possible implementation manner, determining the fusion feature of the image to be identified based on the first normalized image feature, the first weight, the second normalized image feature, and the second weight includes:
[0025] Determine a first weighted image feature according to the first normalized image feature and the first weight, and determine a second weighted image feature according to the second normalized image feature and the second weight;
[0026] Generating a fusion feature of the image to be identified according to the first weighted image feature and the second weighted image feature;
[0027] Among them, the fusion features of the above-mentioned image to be identified meet the following requirements:
[0028] X=W0BN0(x)+W1BN1(x)
[0029] Among them, X is the fusion feature of the image to be identified, W0 is the first weight, BN0(x) is the first normalized image feature, W1 is the second weight, BN1(x) is the second normalized image feature, W0BN0(x) is the first weighted image feature, and W1BN1(x) is the second weighted feature.
[0030] In a second aspect, the present application provides an image recognition device based on path fusion, the device comprising:
[0031] A first acquisition module is used to acquire an image to be recognized, input the image to be recognized into an image recognition model, and extract image features of the image to be recognized through a shared convolutional layer in the image recognition model;
[0032] A second acquisition module is used to acquire first normalized image features corresponding to the image features through a first batch of normalized layers in the image recognition model, and to acquire second normalized image features corresponding to the image features through a second batch of normalized layers in the image recognition model, wherein the first batch of normalized layers and the second batch of normalized layers are respectively trained by sample images with different noise intensities;
[0033] A third acquisition module, used for acquiring a first weight and a second weight corresponding to the image feature through a pathway feature fusion module in the image recognition model, wherein the first weight is used to mark the weight occupied by the first normalized image feature, and the second weight is used to mark the weight occupied by the second normalized image feature, and the sum of the first weight and the second weight is 1;
[0034] The first determination module is used to determine the fusion feature of the image to be identified based on the first normalized image feature, the first weight, the second normalized image feature and the second weight, and determine the image category of the image to be identified based on the fusion feature.
[0035] In conjunction with the second aspect, in a possible implementation manner, the above device further includes:
[0036] a fourth acquisition module, configured to acquire at least three first sample images including at least three noise intensities, input each first sample image into the image recognition model, and extract sample image features of each first sample image through a shared convolutional layer in the image recognition model;
[0037] A first training module, used for training at least three batch normalization layers in the image recognition model according to the sample image features of the first sample images to obtain at least three batch normalization layers after training, wherein the sample image features of the sample images of one noise intensity are used for training one batch normalization layer;
[0038] The second determination module is used to determine the first batch of normalized layers and the second batch of normalized layers of the above-mentioned image recognition model from the above-mentioned at least three batch normalized layers, wherein the above-mentioned first batch of normalized layers are obtained by training the sample images with the smallest noise intensity among the above-mentioned at least three sample images, and the above-mentioned second batch of normalized layers are obtained by training the sample images with the largest noise intensity among the above-mentioned at least three sample images.
[0039] In conjunction with the second aspect, in a possible implementation manner, the above device further includes:
[0040] A deleting module is used to delete the remaining batch normalization layers except the first batch normalization layer and the second batch normalization layer in the at least three batch normalization layers to obtain the batch normalization layers of the image recognition model.
[0041] In conjunction with the second aspect, in a possible implementation manner, the above device further includes:
[0042] a fifth acquisition module, used to acquire a second sample image for training the pathway feature fusion module of the image recognition model, input the second sample image into the image recognition model, and extract sample image features of the second sample image through a shared convolutional layer in the image recognition model;
[0043] A sixth acquisition module, used to acquire first normalized sample image features corresponding to the sample image features through the first batch of normalized layers in the image recognition model, and to acquire second normalized sample image features corresponding to the sample image features through the second batch of normalized layers in the image recognition model;
[0044] The second training module is used to train the pathway feature fusion module based on the sample image features, the first normalized sample image features, the second normalized sample image features and the image category label of the second sample image, so that the pathway feature fusion module outputs the weight corresponding to the normalized image features output by the first batch of normalization layers for any image feature, and the weight corresponding to the normalized image features output by the second batch of normalization layers.
[0045] In conjunction with the second aspect, in a possible implementation manner, the second training module further includes:
[0046] A first acquisition unit, used for acquiring a first sample weight and a second sample weight corresponding to the sample image feature through a pathway feature fusion module in the image recognition model, wherein the first sample weight is used to mark the weight occupied by the first normalized sample image feature, and the second sample weight is used to mark the weight occupied by the second normalized sample image feature, and the sum of the first sample weight and the second sample weight is 1;
[0047] a first determining unit, configured to determine a fusion feature of the second sample image based on the first normalized sample image feature, the first sample weight, the second normalized sample image feature, and the second sample weight, and determine an image category of the second sample image based on the fusion feature of the second sample image;
[0048] The first adjustment unit is used to calculate the classification loss of the image recognition model according to the image category of the second sample image and the image category label of the second sample image, and adjust the network parameters of the pathway fusion feature module based on the classification loss.
[0049] In conjunction with the second aspect, in a possible implementation manner, the first determining unit is further configured to:
[0050] Determine a first weighted image feature according to the first normalized image feature and the first weight, and determine a second weighted image feature according to the second normalized image feature and the second weight;
[0051] Generating a fusion feature of the image to be identified according to the first weighted image feature and the second weighted image feature;
[0052] Among them, the fusion features of the above-mentioned image to be identified meet the following requirements:
[0053] X=W0BN0(x)+W1BN1(x)
[0054] Among them, X is the fusion feature of the image to be identified, W0 is the first weight, BN0(x) is the first normalized image feature, W1 is the second weight, BN1(x) is the second normalized image feature, W0BN0(x) is the first weighted image feature, and W1BN1(x) is the second weighted feature.
[0055] In a third aspect, the present application provides a computer device comprising: a processor, a transceiver, a memory and a network interface; the processor is connected to the memory, the transceiver and the network interface, wherein the network interface is used to provide data communication functions, the memory is used to store program codes, and the processor and the transceiver are used to call the program codes to execute the method executed by the first aspect and any possible implementation method of the first aspect in the present application.
[0056] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program, and the computer program includes program instructions. When the program instructions are executed by a processor, they execute the method performed by the above-mentioned first aspect of the present application and any possible implementation method of the first aspect.
[0057] In the present application, by acquiring an image to be recognized, the image to be recognized is input into an image recognition model, and the image features of the image to be recognized are extracted through a shared convolutional layer in the image recognition model. Then, the first batch of normalized features corresponding to the image features are obtained through the first batch of normalized layers in the trained image recognition model, and the second batch of normalized features corresponding to the image features are obtained through the second batch of normalized layers in the trained image recognition model. And the first weight and the second weight corresponding to the image features are obtained through the path feature fusion module in the trained image recognition model, where the first weight is used to mark the weight occupied by the first normalized feature, and the second weight is used to mark the weight occupied by the second normalized feature, and the sum of the first weight and the second weight is 1. Finally, the fusion features of the image to be recognized are determined based on the first normalized image features, the first weight, the second normalized image features and the second weight, and the image category of the image to be recognized is determined based on the fusion features. Since the image features here will be normalized through the first batch of normalization layers and the second batch of normalization layers after training and obtain corresponding normalized features, using two different batch normalization layers for normalization processing for different images to be identified can improve the accuracy and robustness. In the present application, the above-mentioned first weight and the above-mentioned second weight are directly output by the trained path feature fusion module according to the image to be identified of any noise intensity, so it has higher applicability in actual business scenarios and is simpler to operate. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] In order to more clearly illustrate the technical solution in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0059] Figure 1 It is a schematic diagram of the network architecture provided by the embodiment of the present application;
[0060] Figure 2 is a schematic diagram of a scene of an image recognition method based on path fusion provided in an embodiment of the present application;
[0061] Figure 3 It is a flow chart of an image recognition method based on path fusion provided in an embodiment of the present application;
[0062] Figure 4 It is a structural schematic diagram of the image recognition model provided in the embodiment of the present application;
[0063] Figure 5is another schematic flow chart of the image recognition method based on path fusion provided in an embodiment of the present application;
[0064] Figure 6 It is a schematic diagram of a training process of an image recognition model provided in an embodiment of the present application;
[0065] Figure 7 is another schematic diagram of the training process of the image recognition model provided in an embodiment of the present application;
[0066] Figure 8 It is a structural schematic diagram of a multi-channel image recognition model provided in an embodiment of the present application;
[0067] Fig. 9 It is a structural schematic diagram of a single-channel image recognition model provided in an embodiment of the present application;
[0068] Fig.10 is a flow chart of a training method for a pathway fusion module provided in an embodiment of the present application;
[0069] Fig.11 is a structural schematic diagram of an image recognition device based on path fusion provided in an embodiment of the present application;
[0070] Fig.12 is another structural schematic diagram of an image recognition device based on path fusion provided in an embodiment of the present application;
[0071] Fig.13 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0072] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0073] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level technology and software-level technology. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0074] The image recognition method based on path fusion provided in the embodiment of the present application belongs to computer vision technology (CV) and machine learning (ML) under the field of artificial intelligence. Computer vision is a science that studies how to make machines "see". To put it more concretely, it refers to machine vision such as using cameras and computers to replace human eyes to identify, track and measure targets, and further do graphic processing to make computer processing more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, synchronous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition. Machine learning is a multi-field cross-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It is a field that studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.
[0075] Figure 1 This is a network architecture diagram provided by the embodiment of the present application. Figure 1 As shown, the network architecture diagram may include a business server 100 and a user terminal cluster, and the user terminal cluster may include user terminal 10a, user terminal 10b, ..., user terminal 10n, wherein there may be a communication connection between the user terminal clusters, for example, there is a communication connection between user terminal 10a and user terminal 10b, and there is a communication connection between user terminal 10b and user terminal 10n, and any user terminal in the user terminal cluster may have a communication connection with the business server 100, for example, there is a communication connection between user terminal 10a and business server 100, and there is a communication connection between user terminal 10b and business server 100.
[0076] Among them, the above-mentioned user terminal cluster (also including the above-mentioned user terminal 10a, user terminal 10b and user terminal 10n) can be integrated with the target application. Optionally, the target application may include an application with data information functions such as displaying text, images and videos. For example, the target application can be an image category detection application, which can be used for users to upload images or videos and detect the image category to which the image recognition object included in the image or video belongs. Or the target application can also be a face-swiping payment application, which can be used to view the images or videos collected by the user terminal camera and identify whether the image category in the camera shooting picture is a person's face; if the above-mentioned image category is confirmed to be a person's face, it is further confirmed whether the image category of the above-mentioned person's face is consistent with the target image category of the person's face corresponding to the payment account. If the image category of the person's face in the camera shooting picture is consistent with the target image category of the person's face corresponding to the payment account, the payment transaction is carried out, and if it is not consistent, the transaction is intercepted. Optionally, the target application can also be an attendance punch-in application based on face recognition, which can be used to view the images or videos collected by the user terminal camera and identify whether the image category in the camera shooting picture is a person's face. If the above image category is confirmed to be a human face, it is further confirmed whether the image category of the above human face is consistent with the target image category of the human face corresponding to the login account. If the image category of the human face in the camera shot is consistent with the target image category of the human face corresponding to the login account, attendance punching is performed. If it is not consistent, attendance punching cannot be performed. The business server 100 in the present application can collect business data such as images or videos uploaded by these applications. Optionally, the business data may include images to be identified uploaded by users. For the convenience of explanation, the image to be identified is directly used as the business data for example. The business server 100 can determine the first normalized image feature, the first weight, the second normalized image feature and the second weight of the above image to be identified from these images to be identified, and determine the fusion feature of the above image to be identified based on the first normalized image feature, the first weight, the second normalized image feature and the second weight. Subsequently, the business server 100 determines the image category of the above image to be identified based on the above fusion feature, and returns the above image category to the user terminal. Optionally, the above user terminal can be in the above Figure 1 Any one of the user terminals selected from the user terminal cluster of the corresponding embodiment, for example, the user terminal may be the user terminal 10b, and the user may view the image category of the image to be identified on the display page of the user terminal 10b.
[0077] It is understandable that the method provided in the embodiment of the present application can be executed by a computer device, and the computer device includes but is not limited to a terminal or a server. The business server 100 in the embodiment of the present application can be a computer device, and the user terminal in the user terminal cluster can also be a computer device, which is not limited here. The above-mentioned business server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The above-mentioned terminal may include: smart phones, tablet computers, laptops, desktop computers, smart TVs, smart speakers, desktop computers, smart watches, etc., which carry image recognition functions (for example, identifying the facial area of a person in an image), but are not limited thereto. Among them, the user terminal and the business server can be directly or indirectly connected by wired or wireless communication, and this application is not limited here.
[0078] See also Figure 2 , Figure 2 Schematic diagram of a scene of an image recognition method based on path fusion provided in an embodiment of the present application. Figure 2As shown, when user A uses the target application (such as a face-scanning payment application) in the user terminal, user A uploads an image 20a through the user terminal 10b, and the image 20a includes an object to be identified (such as a person), and then the user terminal 10b sends an image recognition request for the image 20a to the business server 100. Specifically, the business server 100 can detect and collect that the user has uploaded an image 20a to be identified that includes a portrait, and input the above image 20a to be identified into the image recognition model, and extract the image features of the above image to be identified based on the shared convolutional layer in the above image recognition model. Subsequently, the business server 100 obtains the normalized image features corresponding to the above image features through the first batch of normalized layers in the above image recognition model, and obtains the second normalized image features corresponding to the above image features through the second batch of normalized layers in the above image recognition model, wherein the above first batch of normalized layers and the above second batch of normalized layers are respectively trained by sample images with different noise intensities. Then, the first weight and the second weight corresponding to the above image feature are obtained through the path feature fusion module in the above image recognition model, the above first weight is used to mark the weight occupied by the above first normalized image feature, the above second weight is used to mark the weight occupied by the above second normalized image feature, and the sum of the first weight and the second weight is 1. Finally, the business server 100 can determine the fusion feature of the above image to be identified based on the above first normalized image feature, the above first weight, the above second normalized image feature and the above second weight through the above image recognition model, and determine the image category of the above image to be identified based on the above fusion feature. Optionally, the business server 100 can return the image category of the above image to be identified determined based on the image recognition model to the user terminal 10b, and the user terminal 10b can determine whether the face verification is passed based on the image category returned by the business server 100. If the verification is passed, the payment link is entered, and then user A can view the display page of successful payment on the user terminal 10b; if the verification fails, the transaction is intercepted, and user A can view the display page of failed payment on the user terminal 10b.
[0079] Among them, when the business server 100 obtains the first normalized features and the second normalized features of the above-mentioned image to be recognized through the image recognition model, it can be obtained through the first batch of normalized layers and the second batch of normalized layers of the image recognition model. When the business server 100 obtains the first weight and the second weight through the image recognition model, it can be obtained through the pathway feature fusion module. Therefore, in order to improve the prediction accuracy of the first batch of normalized layers, the second batch of normalized layers and the pathway feature fusion module in the above-mentioned image recognition model for the above-mentioned first normalized features, the second normalized features, the first weight and the second weight, the first batch of normalized layers, the second batch of normalized layers and the pathway feature fusion module in the above-mentioned image recognition model can be trained and adjusted so that the first batch of normalized layers, the second batch of normalized layers and the pathway feature fusion module after training and adjustment are optimized. For the specific process of obtaining the first batch of normalized layers and the second batch of normalized layers after training, please refer to the subsequent Figure 5 The description of steps S201-S203 in the corresponding embodiment. For the specific process of obtaining the trained pathway feature fusion module, please refer to the subsequent Figure 7 The description of steps S301-S303 in the corresponding embodiment.
[0080] Optionally, if the user terminal 10b has a trained image recognition model stored locally, the user terminal 10b can identify the image category of the above-mentioned image to be identified locally. Among them, since the training of the first batch of normalization layers, the second batch of normalization layers and the path feature fusion module in the above-mentioned image recognition model involves a large amount of offline calculations, the image recognition model of the user terminal can be sent to the user terminal 10b after the business server 100 completes the training. The specific details can be determined according to the actual application scenario and are not limited here.
[0081] For further understanding, please see Figure 3 , Figure 3 is a flow chart of an image recognition method based on path fusion provided in an embodiment of the present application. The method can be performed by a user terminal (for example, the above Figure 1 or Figure 2 The user terminal shown in the figure) can also be executed by the user terminal and the service server (as shown in the figure above) Figure 1 or Figure 2 The business server 100 in the corresponding embodiment is executed together. For ease of understanding, this embodiment takes the method executed by the above-mentioned user terminal as an example for explanation. Among them, the image recognition method based on path fusion can at least include the following steps S101-step S104:
[0082] S101, obtaining an image to be recognized, inputting the image to be recognized into an image recognition model, and extracting image features of the image to be recognized through a shared convolutional layer in the image recognition model.
[0083] For some possible implementations, see Figure 4 , Figure 4 The structural diagram of the image recognition model provided for this application is as follows: Figure 4 As shown, the role of the shared convolution layer is to extract features from the input image to be identified. Through the several shared convolution layers in the above image recognition model, the convolution features of different dimensions of the above image to be identified are obtained respectively, and the features extracted by each shared convolution layer can be abstracted and combined into higher-order features in the next layer. Based on the above convolution features, each convolution layer can output a corresponding feature map, and a point on the feature map corresponds to an area of the input image. The image features extracted by each shared convolution layer can also exist in a three-dimensional form, that is, they can be regarded as three-dimensional features composed of several two-dimensional images superimposed together. Among them, each two-dimensional image is a feature map. If it is a grayscale image, then only one feature map is superimposed on each other. If it is a color image, there are usually three different feature maps (red, green, and blue) superimposed on each other. Optionally, there will be several convolution kernels between the layers of each shared convolution layer, and the feature map generated by each shared convolution layer is the convolution result of the feature map corresponding to the previous shared convolution layer and the convolution kernel between the two layers, that is, the number of convolution kernels between the layers of each shared convolution layer will generate the same number of feature maps. As the depth of the image recognition model increases, the features extracted by each shared convolutional layer become more and more detailed, and the size of the feature map becomes smaller and smaller, so the next convolutional layer can increase the number of feature maps to more fully extract the features of the previous layer until the output can fully represent the image features of the above-mentioned image to be recognized.
[0084] S102, obtaining first normalized image features and second normalized image features corresponding to the image features through the first batch of normalized layers and the second batch of normalized layers in the image recognition model.
[0085] In some feasible implementations, the first normalized image features corresponding to the image features are obtained through the first batch of normalized layers in the image recognition model, and the second normalized image features corresponding to the image features are obtained through the second batch of normalized layers in the image recognition model. The first batch of normalized layers and the second batch of normalized layers are respectively obtained by training sample images with different noise intensities, wherein the specific process of training sample images with different noise intensities and obtaining the trained first batch of normalized layers and the second batch of normalized layers can be referred to in the subsequent Figure 5 The corresponding step S203 in the embodiment will not be described in detail here.
[0086] Specifically, the batch normalization (BN) layer is used to calculate the mean and variance of the input image features for normalization, thereby alleviating the problem of slow network learning or even difficulty in learning caused by the scattered distribution of features in deep networks to a certain extent. Therefore, a batch normalization layer is connected in series after several shared convolutional layers of the image recognition model to accelerate model training and improve model accuracy. See Figure 4 ,like Figure 4 As shown, in an optional embodiment of the present application, the image features corresponding to the image to be identified of any noise intensity are input into the first batch of normalized layers and the second batch of normalized layers after training to obtain the corresponding first normalized image features BN0(x) and second normalized image features BN1(x).
[0087] S103, obtaining a first weight and a second weight corresponding to the image feature through a pathway feature fusion module in the image recognition model.
[0088] For some possible implementations, see Figure 4 ,like Figure 4 As shown, the above-mentioned image features are input into the trained pathway feature fusion module, and the first weight W0 and the second weight W1 corresponding to the above-mentioned image features are obtained, wherein the above-mentioned first weight W0 is used to mark the weight occupied by the above-mentioned first normalized image feature, and the above-mentioned second weight W1 is used to mark the weight occupied by the above-mentioned second normalized image feature, and the sum of the above-mentioned first weight W0 and the above-mentioned second weight W1 is 1. Wherein, when training the above-mentioned pathway feature fusion module, it is necessary to fix other components in the image model to maintain the respective performances of the first batch of normalized layers and the second batch of normalized layers, and at the same time, enable the trained pathway feature fusion module to further extract features according to the acquired image features and make the generated first weight W0 for marking the first normalized feature and the second weight W1 for marking the second normalized feature more accurate. Wherein, for the specific method of training the pathway feature fusion module, please refer to the subsequent Figure 7 The steps S301-S303 in the corresponding embodiment are not described in detail here.
[0089] S104, determining fusion features of the image to be identified, and determining the image category of the image to be identified based on the fusion features.
[0090] In some feasible implementations, a first weighted image feature is determined based on the first normalized image feature and the first weight, a second weighted image feature is determined based on the second normalized image feature and the second weight, and a fusion feature of the image to be identified is generated based on the first weighted image feature and the second weighted image feature, and the image category of the image to be identified is determined based on the fusion feature. The accuracy of image category prediction is improved by performing image recognition on the image to be identified through fusion features.
[0091] For details, see Figure 4 ,like Figure 4 As shown, the fusion features of the above-mentioned image to be identified satisfy:
[0092] X=W0BN0(x)+W1BN1(x)
[0093] Among them, X is the fusion feature of the image to be identified, W0 is the first weight, BN0(x) is the first normalized image feature, W1 is the second weight, BN1(x) is the second normalized image feature, W0BN0(x) is the first weighted image feature, and W1BN1(x) is the second weighted feature.
[0094] In the present application, by acquiring an image to be recognized, the image to be recognized is input into an image recognition model, and the image features of the image to be recognized are extracted through a shared convolutional layer in the image recognition model. Then, the first batch of normalized features corresponding to the image features are obtained through the first batch of normalized layers in the trained image recognition model, and the second batch of normalized features corresponding to the image features are obtained through the second batch of normalized layers in the trained image recognition model. And the first weight and the second weight corresponding to the image features are obtained through the path feature fusion module in the trained image recognition model, where the first weight is used to mark the weight occupied by the first normalized feature, and the second weight is used to mark the weight occupied by the second normalized feature, and the sum of the first weight and the second weight is 1. Finally, the first weighted image feature is determined according to the first normalized image feature and the first weight, the second weighted image feature is determined according to the second normalized image feature and the second weight, and the fusion feature of the image to be recognized is generated according to the first weighted image feature and the second weighted image feature, and then the image category of the image to be recognized is determined based on the fusion feature. Since the image features here will be normalized through the first batch of normalization layers and the second batch of normalization layers after training and obtain corresponding normalized features, using two different batch normalization layers for normalization processing for different images to be identified can improve the accuracy and robustness of the image recognition model. In the present application, the above-mentioned first weight and the above-mentioned second weight are directly output by the trained pathway feature fusion module according to the image to be identified of any noise intensity, so it has higher applicability in actual business scenarios and is simpler to operate.
[0095] For some possible implementations, please refer to Figure 5 , Figure 5 is a flow chart of a training method for a batch normalization layer provided in an embodiment of the present application. The method may be performed by a user terminal (for example, the above Figure 1 or Figure 2 The user terminal shown in the figure) can also be executed by the user terminal and the service server (as shown in the figure above) Figure 1 or Figure 2 The business server 100 in the corresponding embodiment is jointly executed. For ease of understanding, this embodiment takes the method executed by the above-mentioned user terminal as an example for explanation. Among them, the training method of the batch normalization layer may at least include the following steps S201-S203:
[0096] S201, obtaining at least three first sample images including at least three noise intensities, and extracting sample image features of each first sample image through a shared convolutional layer in an image recognition model.
[0097] For some possible implementations, see Figure 6 , Figure 6 The following is a schematic diagram of the training process of the image recognition model provided in this application. Figure 6 As shown, the obtained at least three first sample images including at least three noise intensities include normal sample images (sample images with noise intensity equal to 0) and adversarial sample images (sample images with noise intensity greater than 0), where the adversarial sample images refer to sample images with a certain noise intensity added to the normal sample images by a certain method. The training at this time is called adversarial training, that is, the adversarial sample images are also added to the training sample set as training samples, and the normal sample images and adversarial sample images are trained at the same time. As the number of training times of the image recognition model gradually increases, the robustness of the image recognition model to the adversarial sample images can be improved, and the image recognition model can have a natural accuracy not lower than that of the ordinary training model and a higher adversarial accuracy. The natural accuracy here refers to the recognition accuracy of the image recognition model for the normal sample images, and the adversarial accuracy refers to the recognition accuracy of the image recognition model for the adversarial sample images. In an optional embodiment of the present application, the above process of generating adversarial sample images by adding a certain noise intensity to the normal sample images by a certain method can be achieved by iterative attack. The adversarial sample images obtained by iterative attack and the normal sample images are simultaneously input into the shared convolutional layer in the image recognition model and the sample image features of the above sample images are extracted.
[0098] Specifically, a gradient-based attack (Project Gradient Descent, PGD) can be used to obtain adversarial samples. That is, multiple iterative attacks are performed on the input normal sample image. At each step of the iterative attack, the normal sample image is input into the input layer and output layer of the iterative attack to determine the input value and output value of the normal sample image, and the loss value is determined according to the difference between the output value of the normal sample image and the true output label value of the normal sample image. The loss value is back-propagated to the input layer of the iterative attack, and the sample gradient of the normal sample image is calculated through the input value and loss value of the normal sample image. As shown in formula (1):
[0099]
[0100] Among them, in formula (1), X t is the input sample, y is the true label, L is the loss function, represents the gradient calculation, g t Represents the gradient calculation result.
[0101] In some feasible implementations, since the noise added to the normal sample image is usually limited to a certain range (determined according to the specific implementation scenario), the above sample gradient is projected to a specific threshold (determined according to the specific implementation scenario) as the noise intensity. Optionally, the noise intensity here can be expressed as gt / ||g t ||. As shown in formula (2):
[0102] X t+1 =∏ X+S (X t +∈(g t / ||g t ||)) (2)
[0103] Formula (2) indicates that each iteration is based on the accumulation of the noise of the previous step, but the accumulated value does not exceed ∈. Here, ∈ plays the role of constraining the noise intensity and also constrains the attack effect of the adversarial sample image. The larger ∈ is, the greater the noise intensity of the generated adversarial sample image is, the easier it is to be perceived by the human eye, and the corresponding attack effect is also stronger; the smaller ∈ is, the smaller the noise intensity of the generated adversarial sample image is, the less likely it is to be perceived by the human eye, and the corresponding attack effect is also worse. For the convenience of description, ∈ will be used to represent the noise intensity for example. Optional, see Figure 6 ,like Figure 6 As shown, noise intensity is added to a normal sample image to generate an adversarial sample image, and a first sample image including the adversarial sample image and the normal sample image is input into a shared convolutional layer in an image recognition model and sample image features of each of the sample images are extracted.
[0104] S202: Train at least three batch normalization layers in the image recognition model according to the sample image features of each first sample image to obtain at least three trained batch normalization layers.
[0105] In some feasible implementations, at least three batch normalization layers in the image recognition model are trained according to the sample image features of each first sample image obtained to obtain at least three batch normalization layers after training, wherein the sample image features of a sample image with a noise intensity are used to train one batch normalization layer. Optionally, the at least three batch normalization layers may be five batch normalization layers. See Figure 6 ,like Figure 6As shown, the above image recognition model includes five batch normalization layers connected in parallel, and all batch normalization layers use the same shared convolution layer and the same activation layer to form a multi-channel image recognition model. The shared convolution layer here is used to extract image features, and each batch normalization layer is used to normalize the extracted image features, and then the normalized image features are input into the activation layer for nonlinear transformation to make the learning ability of the image recognition model stronger. Since adversarial sample images with various noise intensities may be encountered in actual business scenarios, adversarial sample images with various noise intensities are added during training. The above adversarial sample images select different batch normalization layers for training according to different noise intensities to fully realize the disentanglement of sample image features with different noise intensities. The disentanglement here is also called decoupling, that is, transforming the entangled features of the original three-dimensional sample image features into a space that better represents the features, in which the transformations of different features can be separated from each other. For example, by convolving a face dataset with sample images of different noise intensities through a shared convolutional layer, and inputting different sample image features into the corresponding batch normalization layer according to the noise intensity, we can obtain separate feature representations of information such as whether each face in the above face dataset is smiling and the color of their hair.
[0106] Specifically, in an optional embodiment of the present application, the at least three batch normalization layers mentioned above can be five batch normalization layers, and the corresponding image recognition model is a five-channel image recognition model. Optionally, the at least three first sample images including at least three noise intensities mentioned above can be five first sample images including five noise intensities, wherein these five noise intensities can be {0, 1, 2, 4, 8}. Here, {0, 1, 2, 4, 8} is only an example, which can be determined according to the actual application scenario and is not limited here. Among them, the sample image features of a sample image with a noise intensity are used to train a batch normalization layer, for example, Figure 6 As shown, the corresponding sample image features with noise intensity of 0 can be input Figure 6 The batch normalization layer 1 in the training is trained, the corresponding sample image features with a noise intensity of 1 are input into the batch normalization layer 2 for training, the corresponding sample image features with a noise intensity of 2 are input into the batch normalization layer 3 for training, the corresponding sample image features with a noise intensity of 4 are input into the batch normalization layer 4 for training, and the corresponding sample image features with a noise intensity of 8 are input into the batch normalization layer 5 for training, and finally the trained batch normalization layer 1, batch normalization layer 2, batch normalization layer 3, batch normalization layer 4 and batch normalization layer 5 are obtained. In the present application, if the noise intensity of the adversarial sample image is known and the appropriate batch normalization layer is selected for training, the image recognition model can obtain the best accuracy.
[0107] S203, determining a first batch of normalized layers and a second batch of normalized layers of the image recognition model from at least three batch normalized layers.
[0108] For some possible implementations, see Figure 7 , Figure 7 Another training process diagram of the image recognition model provided in this application is as follows: Figure 7 As shown, the first batch normalization layers and the second batch normalization layers of the image recognition model are determined from at least three batch normalization layers, wherein the first batch normalization layers are obtained by training the sample images with the smallest noise intensity (e.g., noise intensity 0) among the three sample images, and in this case, the first batch normalization layers may be batch normalization layers 1. The second batch normalization layers are obtained by training the sample images with the largest noise intensity (e.g., noise intensity 8) among the three sample images, and in this case, the first batch normalization layers may be batch normalization layers 5. Optionally, after the first batch normalization layers and the second batch normalization layers of the image recognition model are determined from the at least three batch normalization layers, the remaining batch normalization layers except the first batch normalization layers and the second batch normalization layers among the at least three batch normalization layers may be shielded or deleted to obtain the batch normalization layers of the image recognition model. This is because in actual business scenarios, the noise intensity of the adversarial sample image may be any decimal, which may not completely correspond to the above five noise intensities {0, 1, 2, 4, 8}, and when the noise intensity of the adversarial sample image cannot be obtained, the accuracy of the above five-channel image recognition model is not high enough. Figure 6 The five-channel image recognition model shown in FIG. 1 is used, and the sample image features corresponding to the five noise intensities are used to train batch normalization layer 1, batch normalization layer 2, batch normalization layer 3, batch normalization layer 4, and batch normalization layer 5. After the training is completed, batch normalization layer 2, batch normalization layer 3, and batch normalization layer 4 are deleted, and batch normalization layer 1 is used as the first batch normalization layer in the image recognition model, and batch normalization layer 5 is used as the second batch normalization layer in the image recognition model. Figure 7 As shown in the figure, the first normalization layer and the second normalization layer after training can process the sample image features corresponding to all noise intensities between 0 and 8. The five-channel image recognition model with five normalization layers is constructed at the beginning of the training to improve the feature extraction ability of the shared convolutional layer for adversarial sample images of various noise intensities, while retaining Figure 6 The batch normalization layer 1 and batch normalization layer 5 in the image are deleted, and the batch normalization layer 2, batch normalization layer 3 and batch normalization layer 4 are deleted in order to realize the processing of sample image features corresponding to any noise intensity, thereby improving the accuracy of the image recognition model and expanding the actual application scenarios.
[0109] In the present application, at least three first sample images including at least three noise intensities are used to extract sample image features of each first sample image through a shared convolutional layer in an image recognition model, and then the sample image features of the above-mentioned first sample images are input into at least three batch normalization layers in the image recognition model for training. Finally, the batch normalization layer that processes the smallest noise intensity among the three sample images is used as the first batch normalization layer, and the batch normalization layer that processes the largest noise intensity among the three sample images is used as the second batch normalization layer, and the remaining batch normalization layers except the above-mentioned first batch normalization layer and the above-mentioned second batch normalization layer in the above-mentioned at least three batch normalization layers are deleted to obtain the batch normalization layers of the image recognition model. In the present application, when training the above-mentioned image recognition model, a multi-channel image recognition model is first constructed, such as Figure 8 As shown, Figure 8 A schematic diagram of the structure of the multi-channel image recognition model provided in this application, wherein Figure 8 At least three batch normalization layers are included in the process. The first sample image is input as Figure 8 The training is performed in the multi-channel image recognition model shown in Figure 1, which improves the shared convolution layer's ability to extract image features, while deleting the remaining batch normalization layers enables the first batch normalization layer and the second batch normalization layer to process sample image features corresponding to any noise intensity. Fig. 9 As shown, Fig. 9 The schematic diagram of the structure of the single-channel image recognition model provided in this application. By training the multi-channel image recognition model, the natural accuracy and adversarial accuracy of the first normalization layer can be better than Fig. 9 The batch normalization layer in the single-pass image recognition model shown can also make the natural accuracy and adversarial accuracy of the second batch normalization layer better than that of Fig. 9 The batch normalization layer in the single-channel image recognition model shown makes the multi-channel image recognition model better than Fig. 9 The single-channel image recognition model shown is simple to operate and highly applicable.
[0110] For some possible implementations, please refer to Fig.10 , Fig.10 is a flow chart of a training method for a pathway fusion module provided in an embodiment of the present application. The method may be performed by a user terminal (for example, the above Figure 1 or Figure 2 The user terminal shown in the figure) can also be executed by the user terminal and the service server (as shown in the figure above) Figure 1 or Figure 2 The training method of the path fusion module may include at least the following steps S301 to S305:
[0111] S301, obtaining a second sample image of a pathway feature fusion module for training an image recognition model, and extracting sample image features of the second sample image through a shared convolutional layer in the image recognition model.
[0112] In some feasible implementations, the second sample image includes an adversarial sample with any noise intensity. Specifically, the specific implementation of obtaining the sample image features of the second sample image in step S301 can be found in the above Figure 5 The description of the sample image features of the first sample image obtained in step S201 in the corresponding embodiment will not be repeated here.
[0113] S302, obtaining first normalized sample image features and second normalized sample image features corresponding to sample image features through the first batch of normalized layers and the second batch of normalized layers in the image recognition model.
[0114] In some feasible implementation modes, the normalization parameters obtained after training the first batch of normalization layers and the second batch of normalization layers can be obtained. Among them, the first batch of normalization layers normalizes the sample image features according to the corresponding normalization parameters and obtains the first normalized sample image features; the second batch of normalization layers normalizes the sample image features according to the corresponding normalization parameters and obtains the second normalized sample image features. Therefore, the first normalized sample image features corresponding to the sample image features can be obtained through the first batch of normalization layers in the image recognition model, and the second normalized sample image features corresponding to the sample image features can be obtained through the second batch of normalization layers in the image recognition model. Here, the first batch of normalization layers can be Figure 6 The batch normalization layer 1 is shown, and the second batch normalization layer can be Figure 6 The batch normalization layer 5 shown, that is, when the pathway feature fusion module is trained, the batch normalization layer in the image recognition model has been trained, and the batch normalization layer in the image recognition model at this time is the first batch normalization layer and the second batch normalization layer. At this time, the first batch normalization layer and the second batch normalization layer and other components in the image model are fixed, which can maintain the respective performance of the first batch normalization layer and the second batch normalization layer and other component modules, while improving the accuracy of the weight of the normalized feature output by the trained pathway feature fusion module, which is simple to operate and highly applicable.
[0115] S303: training a pathway feature fusion module based on the sample image feature, the first normalized sample image feature, the second normalized sample image feature, and the image category label of the second sample image.
[0116] In some feasible implementations, the first sample weight and the second sample weight corresponding to the sample image feature are obtained through the pathway feature fusion module in the image recognition model, the first sample weight is used to mark the weight occupied by the first normalized sample image feature, the second sample weight is used to mark the weight occupied by the second normalized sample image feature, and the sum of the first sample weight and the second sample weight is 1. The fusion feature of the second sample image is determined based on the first normalized sample image feature, the first sample weight, the second normalized sample image feature and the second sample weight.
[0117] The fusion feature of the second sample image satisfies:
[0118] X 样 =W 样0 BN 样0 (x)+W 样1 BN 样1 (x)
[0119] Where, X 样 is the fusion feature of the second sample image, W 样0 is the weight of the first sample, BN 样0 (x) is the first normalized sample image feature, W 样1 is the second sample weight mentioned above, BN 样1 (x) is the second normalized sample image feature mentioned above.
[0120] In some feasible implementations, the image category of the second sample image is determined based on the fusion features of the second sample image, and the classification loss of the image recognition model is calculated according to the image category of the second sample image and the image category label of the second sample image, so as to adjust the network parameters of the pathway fusion feature module based on the classification loss. Optionally, if the image category of the second sample image is determined to be for image classification, the cross entropy loss function can be used to calculate the classification loss of the image category of the second sample image and the image category label of the second sample image; if the image category of the second sample image is determined to be for face recognition, the Arcface loss function can be used to calculate the classification loss of the image category of the second sample image and the image category label of the second sample image. Finally, the network parameters of the pathway fusion feature module are adjusted based on the classification loss, so that the pathway feature fusion module outputs the weight corresponding to the normalized image feature output by the first batch of normalization layers and the weight corresponding to the normalized image feature output by the second batch of normalization layers. Among them, when training the above-mentioned pathway feature fusion module, it is necessary to fix other components in the image model to maintain the respective performance of the first batch of normalized layers and the second batch of normalized layers, and at the same time enable the trained pathway feature fusion module to further extract features according to the acquired image features and make the generated first weight for marking the first normalized feature and the second weight for marking the second normalized feature more accurate.
[0121] In the present application, the sample image features of the second sample image are extracted by the shared convolution layer in the image recognition model, the first normalized sample image features and the second normalized sample image features corresponding to the sample image features are obtained by the first batch of normalized layers and the second batch of normalized layers in the image recognition model, the first sample weight and the second sample weight corresponding to the above sample image features are obtained by the path feature fusion module in the image recognition model, the fusion features of the above second sample image are determined based on the above first normalized sample image features, the above first sample weight, the above second normalized sample image features and the above second sample weight, and the image category of the above second sample image is determined based on the fusion features of the above second sample image. The classification loss of the image recognition model is calculated according to the image category of the second sample image and the image category label of the second sample image, so as to adjust the network parameters of the above path fusion feature module based on the above classification loss, and finally the trained path feature fusion module can extract the image features of the image to be recognized and automatically adjust the first weight for marking the first normalized image features and the second weight for marking the second normalized image features. Therefore, it has higher applicability in actual business scenarios and is simpler to operate.
[0122] For further information, see Fig.11 , Fig.11: is a structural schematic diagram of an image recognition device based on pathway fusion provided by the present application. The image recognition device based on pathway fusion can be a computer program (including program code) running in a computer device. For example, the image recognition device based on pathway fusion is an application software; the device can be used to execute the corresponding steps in the method provided by the present application. Fig.11 As shown, the image recognition device based on path fusion includes: a first acquisition module 10 , a second acquisition module 20 , a third acquisition module 30 , and a first determination module 40 .
[0123] A first acquisition module 10 is used to acquire an image to be recognized, input the image to be recognized into an image recognition model, and extract image features of the image to be recognized through a shared convolutional layer in the image recognition model;
[0124] A second acquisition module 20, configured to acquire first normalized image features corresponding to the image features through a first batch of normalized layers in the image recognition model, and to acquire second normalized image features corresponding to the image features through a second batch of normalized layers in the image recognition model, wherein the first batch of normalized layers and the second batch of normalized layers are respectively trained by sample images with different noise intensities;
[0125] A third acquisition module 30, used for acquiring a first weight and a second weight corresponding to the image feature through a pathway feature fusion module in the image recognition model, wherein the first weight is used to mark the weight occupied by the first normalized image feature, and the second weight is used to mark the weight occupied by the second normalized image feature, and the sum of the first weight and the second weight is 1;
[0126] The first determination module 40 is used to determine the fusion feature of the image to be identified based on the first normalized image feature, the first weight, the second normalized image feature and the second weight, and determine the image category of the image to be identified based on the fusion feature.
[0127] In some feasible implementations, the first acquisition module 10 may acquire an image to be recognized, which may be an image uploaded by a user terminal or an image captured by a camera of the user terminal. The first acquisition module 10 may input the acquired image to be recognized into an image recognition model, and acquire image features of the image to be recognized through the image recognition model. Figure 4As shown, the shared convolutional layer in the above-mentioned image recognition model can perform feature extraction on the input image to be recognized to obtain the image features of the image to be recognized. The second acquisition module 20 can obtain the first normalized image features corresponding to the above-mentioned image features through the first batch normalization layers in the above-mentioned image recognition model, and obtain the second normalized image features corresponding to the above-mentioned image features through the second batch normalization layers in the above-mentioned image recognition model. The input image features are normalized by calculating the mean and variance through the batch normalization layer, thereby alleviating the problem of slow network learning speed or even difficulty in learning caused by the scattered feature distribution in the deep network to a certain extent. Therefore, a batch normalization layer is connected in series after several shared convolutional layers of the image recognition model to accelerate model training and improve model accuracy. As Figure 4 As shown, in an optional embodiment of the present application, the image features corresponding to the image to be identified of any noise intensity are input into the first batch of normalized layers and the second batch of normalized layers after training to obtain the corresponding first normalized image features BN0(x) and second normalized image features BN1(x).
[0128] In some feasible implementations, the third acquisition module 30 can obtain the first weight W0 and the second weight W1 corresponding to the image feature through the pathway feature fusion module in the image recognition model, wherein the pathway feature fusion module is a trained pathway feature fusion module. When training the pathway feature fusion module, it is necessary to fix other components in the image model to maintain the respective performances of the first batch of normalized layers and the second batch of normalized layers, and at the same time, the trained pathway feature fusion module can further extract features according to the acquired image features and make the generated first weight W0 for marking the first normalized feature and the second weight W1 for marking the second normalized feature more accurate. Figure 4 As shown, in an optional embodiment of the present application, the above-mentioned image features are input into the trained pathway feature fusion module, and the first weight W0 and the second weight W1 corresponding to the above-mentioned image features are obtained. The first determination module 40 can determine the fusion feature X of the image to be identified based on the above-mentioned first normalized image feature BN0(x), the above-mentioned first weight W0, the above-mentioned second normalized image feature BN1(x) and the above-mentioned second weight W1, and determine the image category of the image to be identified based on the above-mentioned fusion feature. In an optional embodiment of the present application, the above-mentioned fusion feature can be X=W0BN0(x)+W1BN1(x). Performing image recognition on the image to be identified by fusion features improves the accuracy of image category prediction.
[0129] In the present application, the image to be identified is acquired by the first acquisition module 10, the image to be identified is input into the image recognition model, and the image features of the image to be identified are extracted by the shared convolution layer in the image recognition model. Then the second acquisition module 20 acquires the first batch of normalized features corresponding to the image features through the first batch of normalized layers in the trained image recognition model, and acquires the second batch of normalized features corresponding to the image features through the second batch of normalized layers in the trained image recognition model. Then the third acquisition module 30 acquires the first weight and the second weight corresponding to the image features through the path feature fusion module in the trained image recognition model, where the first weight is used to mark the weight occupied by the first normalized feature, and the second weight is used to mark the weight occupied by the second normalized feature, and the sum of the first weight and the second weight is 1. Finally, the first determination module 40 determines the fusion features of the image to be identified based on the first normalized image features, the first weight, the second normalized image features and the second weight, and determines the image category of the image to be identified based on the fusion features. Since the image features here will be normalized through the first batch of normalization layers and the second batch of normalization layers after training and obtain corresponding normalized features, using two different batch normalization layers for normalization processing for different images to be identified can improve the accuracy and robustness of the image recognition model. In the present application, the above-mentioned first weight and the above-mentioned second weight are directly output by the trained pathway feature fusion module according to the image to be identified of any noise intensity, so it has higher applicability in actual business scenarios and is simpler to operate.
[0130] In a possible implementation, please refer to Fig.12 , the above-mentioned device also includes:
[0131] A fourth acquisition module 50, configured to acquire at least three first sample images including at least three noise intensities, input each first sample image into the image recognition model, and extract sample image features of each first sample image through a shared convolutional layer in the image recognition model;
[0132] A first training module 60 is used to train at least three batch normalization layers in the image recognition model according to the sample image features of the first sample images to obtain at least three batch normalization layers after training, wherein the sample image features of the sample images of one noise intensity are used to train one batch normalization layer;
[0133] The second determination module 70 is used to determine the first batch of normalized layers and the second batch of normalized layers of the above-mentioned image recognition model from the above-mentioned at least three batch normalized layers, wherein the above-mentioned first batch of normalized layers are obtained by training the sample images with the smallest noise intensity among the above-mentioned at least three sample images, and the above-mentioned second batch of normalized layers are obtained by training the sample images with the largest noise intensity among the above-mentioned at least three sample images.
[0134] In a possible implementation manner, the above device further includes:
[0135] The deleting module 80 is used to delete the remaining batch normalization layers except the first batch normalization layer and the second batch normalization layer in the at least three batch normalization layers to obtain the batch normalization layers of the image recognition model.
[0136] In some feasible implementations, the fourth acquisition module 50 may acquire at least three first sample images including at least three noise intensities including normal sample images (sample images with noise intensity equal to 0) and adversarial sample images (sample images with noise intensity greater than 0). The adversarial sample images here refer to sample images obtained by adding a certain noise intensity to normal sample images through a certain method. That is, the adversarial sample images are also added to the training sample set as training samples, and the normal sample images and adversarial sample images are trained at the same time. As the number of training times of the image recognition model gradually increases, the robustness of the image recognition model to the adversarial sample images can be improved, and at the same time, the image recognition model has a natural accuracy that is not lower than that of the ordinary training model and a higher adversarial accuracy. In an optional embodiment of the present application, the above-mentioned process of generating adversarial sample images by adding a certain noise intensity to the normal sample images through a certain method can be achieved through an iterative attack. The adversarial sample images obtained through the iterative attack are simultaneously input into the shared convolutional layer in the image recognition model together with the normal sample images, and the sample image features of the above-mentioned sample images are extracted. Optionally, such as Figure 6 As shown, noise intensity is added to a normal sample image to generate an adversarial sample image, and a first sample image including the adversarial sample image and the normal sample image is input into a shared convolutional layer in an image recognition model and sample image features of each of the sample images are extracted.
[0137] In some feasible implementations, the first training module 60 may train at least three batch normalization layers in the image recognition model according to the sample image features of each first sample image obtained to obtain at least three batch normalization layers after training, wherein the sample image features of a sample image with a noise intensity are used to train one batch normalization layer. Optionally, the at least three batch normalization layers may be five batch normalization layers. Figure 6As shown, the above-mentioned image recognition model includes five batch normalization layers connected in parallel, and all batch normalization layers use the same shared convolutional layer and the same activation layer, thereby forming a multi-channel image recognition model. The shared convolutional layer here is used to extract image features, and each batch normalization layer is used to normalize the extracted image features, and then the normalized image features are input into the activation layer for nonlinear transformation to make the learning ability of the image recognition model stronger. Since adversarial sample images with various noise intensities may be encountered in actual business scenarios, adversarial sample images with various noise intensities are added during training. The above-mentioned adversarial sample images select different batch normalization layers for training according to different noise intensities, so as to fully realize the disentanglement of sample image features with different noise intensities. In an optional embodiment of the present application, such as Figure 6 As shown, the corresponding sample image features with noise intensity of 0 can be input Figure 6 The batch normalization layer 1 in the training is trained, the corresponding sample image features with a noise intensity of 1 are input into the batch normalization layer 2 for training, the corresponding sample image features with a noise intensity of 2 are input into the batch normalization layer 3 for training, the corresponding sample image features with a noise intensity of 4 are input into the batch normalization layer 4 for training, and the corresponding sample image features with a noise intensity of 8 are input into the batch normalization layer 5 for training, and finally the trained batch normalization layer 1, batch normalization layer 2, batch normalization layer 3, batch normalization layer 4 and batch normalization layer 5 are obtained. In the present application, if the noise intensity of the adversarial sample image is known and the appropriate batch normalization layer is selected for training, the image recognition model can obtain the best accuracy.
[0138] In some feasible implementations, the second determination module 70 may determine the first batch normalization layer and the second batch normalization layer of the image recognition model from the at least three batch normalization layers. Figure 7 ,like Figure 7 As shown, the first batch normalization layers and the second batch normalization layers of the image recognition model are determined from at least three batch normalization layers, wherein the first batch normalization layers are obtained by training the sample images with the smallest noise intensity (e.g., noise intensity 0) among the three sample images, and in this case, the first batch normalization layers may be batch normalization layers 1. The second batch normalization layers are obtained by training the sample images with the largest noise intensity (e.g., noise intensity 8) among the three sample images, and in this case, the first batch normalization layers may be batch normalization layers 5.
[0139] In some feasible implementations, after the deletion module 80 determines the first batch normalization layers and the second batch normalization layers of the image recognition model among the at least three batch normalization layers, it may delete the remaining batch normalization layers except the first batch normalization layers and the second batch normalization layers among the at least three batch normalization layers to obtain the batch normalization layers of the image recognition model. This is because in actual business scenarios, the noise intensity of the adversarial sample image may be an arbitrary decimal, which may not fully correspond to the above five noise intensities {0, 1, 2, 4, 8}, and when the noise intensity of the adversarial sample image cannot be obtained, the accuracy of using the above five-channel image recognition model is not high enough. Therefore, in an optional embodiment of the present application, first construct the following Figure 6 The five-channel image recognition model shown in FIG. 1 is used, and the sample image features corresponding to the five noise intensities are used to train batch normalization layer 1, batch normalization layer 2, batch normalization layer 3, batch normalization layer 4, and batch normalization layer 5. After the training is completed, batch normalization layer 2, batch normalization layer 3, and batch normalization layer 4 are deleted, and batch normalization layer 1 is used as the first batch normalization layer in the image recognition model, and batch normalization layer 5 is used as the second batch normalization layer in the image recognition model. Figure 7 As shown in the figure, the first normalization layer and the second normalization layer after training can process the sample image features corresponding to all noise intensities between 0 and 8. The five-channel image recognition model with five normalization layers is constructed at the beginning of the training to improve the feature extraction ability of the shared convolutional layer for adversarial sample images of various noise intensities, while retaining Figure 6 The batch normalization layer 1 and batch normalization layer 5 in the image are deleted, and the batch normalization layer 2, batch normalization layer 3 and batch normalization layer 4 are deleted in order to realize the processing of sample image features corresponding to any noise intensity, thereby improving the accuracy of the image recognition model and expanding the actual application scenarios.
[0140] In the present application, at least three first sample images including at least three noise intensities acquired by the fourth acquisition module 50 are used to extract sample image features of each first sample image through a shared convolutional layer in the image recognition model, and then the sample image features of the above-mentioned first sample images are input into at least three batch normalization layers in the image recognition model for training through the first training module 60. Finally, the batch normalization layer that processes the smallest noise intensity among the three sample images is used as the first batch normalization layer through the second determination module 70, and the batch normalization layer that processes the largest noise intensity among the three sample images is used as the second batch normalization layer, and the remaining batch normalization layers except the above-mentioned first batch normalization layer and the above-mentioned second batch normalization layer in the at least three batch normalization layers are deleted through the deletion module 80 to obtain the batch normalization layers of the image recognition model. In the present application, when training the above-mentioned image recognition model, a multi-channel image recognition model is first constructed, such as Figure 8 As shown, Figure 8 A schematic diagram of the structure of the multi-channel image recognition model provided in this application, wherein Figure 8 At least three batch normalization layers are included in the process. The first sample image is input as Figure 8 The training is performed in the multi-channel image recognition model shown in Figure 1, which improves the shared convolution layer's ability to extract image features, while deleting the remaining batch normalization layers enables the first batch normalization layer and the second batch normalization layer to process sample image features corresponding to any noise intensity. Fig. 9 As shown, Fig. 9 The schematic diagram of the structure of the single-channel image recognition model provided in this application. By training the multi-channel image recognition model, the natural accuracy and adversarial accuracy of the first normalization layer can be better than Fig. 9 The batch normalization layer in the single-pass image recognition model shown can also make the natural accuracy and adversarial accuracy of the second batch normalization layer better than that of Fig. 9 The batch normalization layer in the single-channel image recognition model shown makes the multi-channel image recognition model better than Fig. 9 The single-channel image recognition model shown is simple to operate and highly applicable.
[0141] In a possible implementation manner, the above device further includes:
[0142] A fifth acquisition module 90, used to acquire a second sample image for training the pathway feature fusion module of the image recognition model, input the second sample image into the image recognition model, and extract sample image features of the second sample image through a shared convolutional layer in the image recognition model;
[0143] The sixth acquisition module 100 is used to acquire the first normalized sample image features corresponding to the sample image features through the first batch of normalized layers in the image recognition model, and acquire the second normalized sample image features corresponding to the sample image features through the second batch of normalized layers in the image recognition model;
[0144] The second training module 110 is used to train the pathway feature fusion module based on the sample image features, the first normalized sample image features, the second normalized sample image features and the image category label of the second sample image, so that the pathway feature fusion module outputs the weight corresponding to the normalized image features output by the first batch of normalization layers for any image feature, and the weight corresponding to the normalized image features output by the second batch of normalization layers.
[0145] In a possible implementation manner, the second training module 110 further includes:
[0146] A first acquisition unit 1101 is used to acquire a first sample weight and a second sample weight corresponding to the sample image feature through a pathway feature fusion module in the image recognition model, wherein the first sample weight is used to mark the weight occupied by the first normalized sample image feature, and the second sample weight is used to mark the weight occupied by the second normalized sample image feature, and the sum of the first sample weight and the second sample weight is 1;
[0147] A first determining unit 1102 is used to determine a fusion feature of the second sample image based on the first normalized sample image feature, the first sample weight, the second normalized sample image feature and the second sample weight, and determine an image category of the second sample image based on the fusion feature of the second sample image;
[0148] The first adjustment unit 1103 is used to calculate the classification loss of the image recognition model according to the image category of the second sample image and the image category label of the second sample image, and adjust the network parameters of the pathway fusion feature module based on the classification loss.
[0149] In a possible implementation manner, the first determining unit 1102 is further configured to:
[0150] Determine a first weighted image feature according to the first normalized image feature and the first weight, and determine a second weighted image feature according to the second normalized image feature and the second weight;
[0151] Generating a fusion feature of the image to be identified according to the first weighted image feature and the second weighted image feature;
[0152] Among them, the fusion features of the above-mentioned image to be identified meet the following requirements:
[0153] X=W0BN0(x)+W1BN1(x)
[0154] Among them, X is the fusion feature of the image to be identified, W0 is the first weight, BN0(x) is the first normalized image feature, W1 is the second weight, BN1(x) is the second normalized image feature, W0BN0(x) is the first weighted image feature, and W1BN1(x) is the second weighted feature.
[0155] In some feasible embodiments, the fifth acquisition module 90 may acquire a second sample image of a pathway feature fusion module for training an image recognition model, and extract sample image features of the second sample image through a shared convolutional layer in the image recognition model, wherein the second sample image includes adversarial samples of arbitrary noise intensity. The sixth acquisition module 100 may acquire normalization parameters obtained after training the first batch of normalization layers and the second batch of normalization layers. Among them, the first batch of normalization layers normalizes the sample image features according to the corresponding normalization parameters and obtains first normalized sample image features; the second batch of normalization layers normalizes the sample image features according to the corresponding normalization parameters and obtains second normalized sample image features. Therefore, the first normalized sample image features corresponding to the sample image features may be acquired through the first batch of normalization layers in the image recognition model, and the second normalized sample image features corresponding to the sample image features may be acquired through the second batch of normalization layers in the image recognition model. Here, the first batch of normalization layers may be Figure 6 The batch normalization layer 1 is shown, and the second batch normalization layer can be Figure 6 The batch normalization layer 5 shown, that is, when the pathway feature fusion module is trained, the batch normalization layer in the image recognition model has been trained, and the batch normalization layer in the image recognition model at this time is the first batch normalization layer and the second batch normalization layer. At this time, by fixing the first batch normalization layer and the second batch normalization layer and other components in the image model, the respective performances of the first batch normalization layer and the second batch normalization layer and other component modules can be maintained, while improving the accuracy of the weight of the normalized features output by the trained pathway feature fusion module.
[0156] In some feasible implementations, the first acquisition unit 1101 in 110 in the second training module may acquire the first sample weight and the second sample weight corresponding to the sample image feature through the path feature fusion module in the image recognition model, the first sample weight is used to mark the weight occupied by the first normalized sample image feature, the second sample weight is used to mark the weight occupied by the second normalized sample image feature, and the sum of the first sample weight and the second sample weight is 1. The first determination unit 1102 in 110 in the second training module may determine the fusion feature of the second sample image based on the first normalized sample image feature, the first sample weight, the second normalized sample image feature and the second sample weight. The fusion feature of the second sample image may be X 样 =W 样0 BN 样0 (x)+W 样1 BN 样1 (x), where X 样 is the fusion feature of the second sample image, W 样0 is the weight of the first sample, BN 样0(x) is the first normalized sample image feature, W 样1 is the second sample weight mentioned above, BN 样1 (x) is the second normalized sample image feature mentioned above.
[0157] In some feasible implementations, the first adjustment unit 1103 in 110 of the second training module can determine the image category of the second sample image based on the fusion feature of the second sample image, and calculate the classification loss of the image recognition model according to the image category of the second sample image and the image category label of the second sample image, so as to adjust the network parameters of the pathway fusion feature module based on the classification loss. Optionally, if it is determined that the image category of the second sample image is for image classification, the cross entropy loss function can be used to calculate the classification loss of the image category of the second sample image and the image category label of the second sample image; if it is determined that the image category of the second sample image is for face recognition, the Arcface loss function can be used to calculate the classification loss of the image category of the second sample image and the image category label of the second sample image. Finally, the network parameters of the pathway fusion feature module are adjusted based on the classification loss, so that the pathway feature fusion module outputs the weight corresponding to the normalized image feature output by the first batch of normalization layers of any image feature, and the weight corresponding to the normalized image feature output by the second batch of normalization layers. Among them, when training the above-mentioned pathway feature fusion module, it is necessary to fix other components in the image model to maintain the respective performance of the first batch of normalized layers and the second batch of normalized layers, and at the same time enable the trained pathway feature fusion module to further extract features according to the acquired image features and make the generated first weight for marking the first normalized feature and the second weight for marking the second normalized feature more accurate.
[0158] In the present application, the second sample image of the pathway feature fusion module for training the image recognition model is obtained through the fifth acquisition module 90, and the sample image features of the second sample image are extracted through the shared convolutional layer in the image recognition model. The sixth acquisition module 100 obtains the first normalized sample image features and the second normalized sample image features corresponding to the sample image features through the first batch of normalization layers and the second batch of normalization layers in the image recognition model. The first acquisition unit 1101 in the second training module 110 obtains the first sample weight and the second sample weight corresponding to the above-mentioned sample image features through the pathway feature fusion module in the image recognition model. The first determination unit 1102 in the second training module 110 can determine the first weighted image feature according to the above-mentioned first normalized image feature and the above-mentioned first weight, determine the second weighted image feature according to the above-mentioned second normalized image feature and the above-mentioned second weight, and generate the fusion features of the above-mentioned image to be recognized according to the above-mentioned first weighted image feature and the above-mentioned second weighted image feature, and determine the image category of the above-mentioned second sample image based on the fusion features of the above-mentioned second sample image. The first adjustment unit 1103 in the second training module 110 calculates the classification loss of the image recognition model according to the image category of the second sample image and the image category label of the second sample image, so as to adjust the network parameters of the above-mentioned pathway fusion feature module based on the above-mentioned classification loss, and finally enables the trained pathway feature fusion module to extract the image features of the image to be recognized and automatically adjust the first weight for marking the first normalized image feature and the second weight for marking the second normalized image feature. Therefore, it has higher applicability in actual business scenarios and is simpler to operate.
[0159] For further information, see Fig.13 , Fig.13 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.13 As shown, the computer device 2000 can be applied to a server, which can be the above-mentioned Figure 1 The business server 100 in the corresponding embodiment; the computer device 2000 can be applied to a terminal, which can be the above-mentioned Figure 1 The user terminal 10a, user terminal 10b, ..., user terminal 10n in the corresponding embodiment; the computer device 2000 may also be the above-mentioned Figure 3The computer device in the corresponding embodiment. The computer device 2000 may include: a processor 2001, a network interface 2004 and a memory 2005. In addition, the above-mentioned computer device 2000 also includes: a transceiver 2003, and at least one communication bus 2002. Among them, the communication bus 2002 is used to realize the connection and communication between these components. The network interface 2004 may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface). The memory 2005 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 2005 may optionally also be at least one storage device located away from the aforementioned processor 2001. Fig.13 As shown, the memory 2005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.
[0160] exist Fig.13 In the computer device 2000 shown, the network interface 2004 can provide a network communication function; the processor 2001 and the transceiver 2003 can be used to call the device control application stored in the memory 2005 to achieve:
[0161] The transceiver 2003 is used to obtain the image to be recognized.
[0162] The processor 2001 is used to input the image to be recognized into the image recognition model, and extract image features of the image to be recognized through the shared convolutional layer in the image recognition model.
[0163] The processor 2001 is further configured to obtain first normalized image features corresponding to the image features through a first batch of normalized layers in the image recognition model, and obtain second normalized image features corresponding to the image features through a second batch of normalized layers in the image recognition model, wherein the first batch of normalized layers and the second batch of normalized layers are respectively trained by sample images with different noise intensities;
[0164] The processor 2001 is used to obtain a first weight and a second weight corresponding to the image feature through a pathway feature fusion module in the image recognition model, wherein the first weight is used to mark the weight occupied by the first normalized image feature, and the second weight is used to mark the weight occupied by the second normalized image feature, and the sum of the first weight and the second weight is 1;
[0165] The processor 2001 is used to determine the fusion feature of the image to be identified based on the first normalized image feature, the first weight, the second normalized image feature and the second weight, and determine the image category of the image to be identified based on the fusion feature.
[0166] In a possible implementation manner, the processor 2001 is further configured to:
[0167] According to at least three first sample images including at least three noise intensities acquired by the transceiver 2003, each first sample image is input into the image recognition model, and sample image features of each first sample image are extracted through a shared convolutional layer in the image recognition model;
[0168] Training at least three batch normalization layers in the image recognition model according to the sample image features of the first sample images to obtain at least three batch normalization layers after training, wherein the sample image features of the sample images of one noise intensity are used to train one batch normalization layer;
[0169] The first batch of normalized layers and the second batch of normalized layers of the image recognition model are determined from the at least three batch normalized layers, wherein the first batch of normalized layers are trained by sample images with the smallest noise intensity among the at least three sample images, and the second batch of normalized layers are trained by sample images with the largest noise intensity among the at least three sample images.
[0170] In a possible implementation, after determining the first batch of normalized layers and the second batch of normalized layers of the image recognition model from the at least three batch normalized layers, the processor 2001 is further configured to:
[0171] The remaining batch normalization layers except the first batch normalization layer and the second batch normalization layer in the at least three batch normalization layers are deleted to obtain the batch normalization layers of the image recognition model.
[0172] In a possible implementation manner, the processor 2001 is further configured to:
[0173] Obtaining a second sample image for training a pathway feature fusion module of the image recognition model, inputting the second sample image into the image recognition model, and extracting sample image features of the second sample image through a shared convolutional layer in the image recognition model;
[0174] Obtaining first normalized sample image features corresponding to the sample image features through the first batch of normalized layers in the image recognition model, and obtaining second normalized sample image features corresponding to the sample image features through the second batch of normalized layers in the image recognition model;
[0175] The processor 2001 is further configured to:
[0176] The pathway feature fusion module is trained based on the sample image features, the first normalized sample image features, the second normalized sample image features and the image category label of the second sample image, so that the pathway feature fusion module outputs the weight corresponding to the normalized image features output by the first batch of normalization layers for any image feature, and the weight corresponding to the normalized image features output by the second batch of normalization layers.
[0177] In a possible implementation manner, the processor 2001 is further configured to:
[0178] Obtaining a first sample weight and a second sample weight corresponding to the sample image feature through the pathway feature fusion module in the image recognition model, wherein the first sample weight is used to mark the weight occupied by the first normalized sample image feature, and the second sample weight is used to mark the weight occupied by the second normalized sample image feature, and the sum of the first sample weight and the second sample weight is 1;
[0179] The processor 2001 is further configured to:
[0180] Determining a fusion feature of the second sample image based on the first normalized sample image feature, the first sample weight, the second normalized sample image feature and the second sample weight, and determining an image category of the second sample image based on the fusion feature of the second sample image;
[0181] The classification loss of the image recognition model is calculated according to the image category of the second sample image and the image category label of the second sample image, and the network parameters of the pathway fusion feature module are adjusted based on the classification loss to train the pathway feature fusion module.
[0182] In a possible implementation, the determining of the fusion feature of the image to be identified based on the first normalized image feature, the first weight, the second normalized image feature, and the second weight includes:
[0183] Determine a first weighted image feature according to the first normalized image feature and the first weight, and determine a second weighted image feature according to the second normalized image feature and the second weight;
[0184] Generating a fusion feature of the image to be identified according to the first weighted image feature and the second weighted image feature;
[0185] Among them, the fusion features of the above-mentioned image to be identified meet the following requirements:
[0186] X=W0BN0(x)+W1BN1(x)
[0187] Among them, X is the fusion feature of the image to be identified, W0 is the first weight, BN0(x) is the first normalized image feature, W1 is the second weight, BN1(x) is the second normalized image feature, W0BN0(x) is the first weighted image feature, and W1BN1(x) is the second weighted feature.
[0188] In the present application, by obtaining an image to be recognized, the image to be recognized is input into an image recognition model, and the image features of the image to be recognized are extracted through a shared convolutional layer in the image recognition model. Then, the first batch of normalized features corresponding to the image features are obtained through the first batch of normalized layers in the trained image recognition model, and the second batch of normalized features corresponding to the image features are obtained through the second batch of normalized layers in the trained image recognition model. The first weight and the second weight corresponding to the image features are obtained through the path feature fusion module in the trained image recognition model, where the first weight is used to mark the weight occupied by the first normalized feature, and the second weight is used to mark the weight occupied by the second normalized feature, and the sum of the first weight and the second weight is 1. Finally, the first weighted image feature is determined according to the first normalized image feature and the first weight, and the second weighted image feature is determined according to the second normalized image feature and the second weight; the fusion feature of the image to be recognized is generated according to the first weighted image feature and the second weighted image feature, and the image category of the image to be recognized is determined based on the fusion feature. Since the image features here will be normalized by the first batch of normalization layers and the second batch of normalization layers after training and obtain the corresponding normalized features, using two different batch normalization layers for normalization of different images to be recognized can improve the accuracy and robustness of the image recognition model.
[0189] Among them, the training of each batch normalization layer in the image model can be carried out by obtaining at least three first sample images including at least three noise intensities, wherein the at least three first sample images include normal sample images (sample images with noise intensity equal to 0) and adversarial sample images (sample images with noise intensity greater than 0), extracting sample image features of each first sample image through a shared convolutional layer in the image recognition model, and then inputting the sample image features of each first sample image into at least three batch normalization layers in the image recognition model for training. Finally, the batch normalization layer with the smallest noise intensity among the three sample images is used as the first batch normalization layer, and the batch normalization layer with the largest noise intensity among the three sample images is used as the second batch normalization layer, and the remaining batch normalization layers except the first batch normalization layer and the second batch normalization layer among the at least three batch normalization layers are deleted to obtain the batch normalization layer of the image recognition model. Since adversarial sample images with various noise intensities may be encountered in actual business scenarios, adversarial sample images with various noise intensities are added during training. The above adversarial sample images select different batch normalization layers for training according to the different noise intensities to fully realize the disentanglement of sample image features with different noise intensities. By training the multi-channel image recognition model, the natural accuracy and adversarial accuracy of the first batch normalization layer can be better than Fig. 9 The batch normalization layer in the single-pass image recognition model shown can also make the natural accuracy and adversarial accuracy of the second batch normalization layer better than that of Fig. 9 The batch normalization layer in the single-channel image recognition model shown makes the multi-channel image recognition model better than Fig. 9 The single-pass image recognition model shown.
[0190] Among them, for the training of the pathway feature fusion module in the image model, the sample image features of the second sample image can be extracted through the shared convolution layer in the image recognition model, the first normalized sample image features and the second normalized sample image features corresponding to the sample image features can be obtained through the first batch of normalization layers and the second batch of normalization layers in the image recognition model, the first sample weight and the second sample weight corresponding to the above sample image features can be obtained through the pathway feature fusion module in the image recognition model, the fusion features of the above second sample image can be determined based on the above first normalized sample image features, the above first sample weight, the above second normalized sample image features and the above second sample weight, and the image category of the above second sample image can be determined based on the fusion features of the above second sample image. The classification loss of the image recognition model is calculated according to the image category of the second sample image and the image category label of the second sample image, so as to adjust the network parameters of the above pathway fusion feature module based on the above classification loss, and finally the trained pathway feature fusion module can extract the image features of the image to be recognized and automatically adjust the first weight used to mark the first normalized image features and the second weight used to mark the second normalized image features. In the present application, the first weight and the second weight are directly output by the trained pathway feature fusion module according to the image to be identified of any noise intensity, so they are more applicable in actual business scenarios and are simpler to operate.
[0191] In addition, it should be pointed out here that: the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the aforementioned image recognition device based on path fusion, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the aforementioned Figure 3 and / or Figure 5 and / or Fig.10 The description of the image recognition method based on path fusion in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, the program instructions can be deployed to execute on a computing device, or on multiple computing devices located in one place.
[0192] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program. The above-mentioned program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the above-mentioned computer-readable storage medium can be a control device of a power converter provided by any of the above-mentioned embodiments or an internal storage unit of the above-mentioned device, such as a hard disk or memory of an electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the electronic device. The above-mentioned computer-readable storage medium can also include a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory, etc. Further, the computer-readable storage medium can also include both the internal storage unit of the electronic device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or is to be output.
[0193] The terms "first", "second", etc. in the claims, specification and drawings of the present invention are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. Mentioning "embodiment" in this article means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present invention. Displaying the phrase at various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments. The term "and / or" used in the specification and the appended claims of the present invention refers to any combination of one or more of the items listed in association and all possible combinations, and includes these combinations.
[0194] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0195] In the embodiments provided in the present application, the disclosed circuits and methods can also be implemented in other ways. For example, the device embodiments described above are schematic, for example, the division of circuit modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed.
[0196] Each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0197] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.
Claims
1. An image recognition method based on pathway fusion, characterized in that: The method comprises: Acquire an image to be recognized, input the image to be recognized into an image recognition model, and extract image features of the image to be recognized through a shared convolutional layer in the image recognition model; Acquire first normalized image features corresponding to the image features through a first batch of normalized layers in the image recognition model, and acquire second normalized image features corresponding to the image features through a second batch of normalized layers in the image recognition model, wherein the first batch of normalized layers and the second batch of normalized layers are respectively trained by sample images with different noise intensities; Obtaining a first weight and a second weight corresponding to the image feature through a pathway feature fusion module in the image recognition model, wherein the first weight is used to mark the weight occupied by the first normalized image feature, and the second weight is used to mark the weight occupied by the second normalized image feature, and the sum of the first weight and the second weight is 1; A fusion feature of the image to be identified is determined based on the first normalized image feature, the first weight, the second normalized image feature, and the second weight, and an image category of the image to be identified is determined based on the fusion feature.
2. The method according to claim 1, characterized in that Before inputting the image to be recognized into the image recognition model, the method further includes: Acquire at least three first sample images including at least three noise intensities, input each first sample image into the image recognition model, and extract sample image features of each first sample image through a shared convolutional layer in the image recognition model; Training at least three batch normalization layers in the image recognition model according to the sample image features of the first sample images to obtain at least three batch normalization layers after training, wherein the sample image features of the sample images of one noise intensity are used to train one batch normalization layer; A first batch of normalized layers and a second batch of normalized layers of the image recognition model are determined from the at least three batch normalized layers, wherein the first batch of normalized layers are trained by sample images with the smallest noise intensity among the at least three first sample images, and the second batch of normalized layers are trained by sample images with the largest noise intensity among the at least three first sample images.
3. The method according to claim 2, characterized in that After determining the first batch of normalized layers and the second batch of normalized layers of the image recognition model from the at least three batch normalized layers, the method further includes: Deleting the remaining batch normalization layers except the first batch normalization layer and the second batch normalization layer in the at least three batch normalization layers to obtain the batch normalization layers of the image recognition model.
4. The method according to claim 2 or 3, characterized in that: Before obtaining the first weight and the second weight corresponding to the image feature through the pathway feature fusion module in the image recognition model, the method further includes: Acquire a second sample image, input the second sample image into the image recognition model, and extract sample image features of the second sample image through a shared convolutional layer in the image recognition model; Acquire first normalized sample image features corresponding to the sample image features through a first batch of normalized layers in the image recognition model, and acquire second normalized sample image features corresponding to the sample image features through a second batch of normalized layers in the image recognition model; The pathway feature fusion module is trained based on the sample image features, the first normalized sample image features, the second normalized sample image features and the image category label of the second sample image, so that the pathway feature fusion module outputs the weight corresponding to the normalized image features output by the first batch of normalization layers for any image feature, and the weight corresponding to the normalized image features output by the second batch of normalization layers.
5. The method according to claim 4, characterized in that The training of the pathway feature fusion module based on the sample image feature, the first normalized sample image feature, the second normalized sample image feature, and the image category label of the second sample image includes: Obtaining a first sample weight and a second sample weight corresponding to the sample image feature through a pathway feature fusion module in the image recognition model, wherein the first sample weight is used to mark the weight occupied by the first normalized sample image feature, and the second sample weight is used to mark the weight occupied by the second normalized sample image feature, and the sum of the first sample weight and the second sample weight is 1; Determining a fusion feature of the second sample image based on the first normalized sample image feature, the first sample weight, the second normalized sample image feature, and the second sample weight, and determining an image category of the second sample image based on the fusion feature of the second sample image; The classification loss of the image recognition model is calculated according to the image category of the second sample image and the image category label of the second sample image, and the network parameters of the pathway fusion feature module are adjusted based on the classification loss to train the pathway feature fusion module.
6. The method according to claim 5, characterized in that The determining of the fusion feature of the image to be identified based on the first normalized image feature, the first weight, the second normalized image feature and the second weight includes: Determine a first weighted image feature according to the first normalized image feature and the first weight, and determine a second weighted image feature according to the second normalized image feature and the second weight; A fusion feature of the image to be identified is generated according to the first weighted image feature and the second weighted image feature.
7. An image recognition device based on path fusion, characterized in that: The device comprises: A first acquisition module, used for acquiring an image to be recognized, inputting the image to be recognized into an image recognition model, and extracting image features of the image to be recognized through a shared convolutional layer in the image recognition model; A second acquisition module, used to acquire first normalized image features corresponding to the image features through a first batch of normalized layers in the image recognition model, and to acquire second normalized image features corresponding to the image features through a second batch of normalized layers in the image recognition model, wherein the first batch of normalized layers and the second batch of normalized layers are respectively trained by sample images with different noise intensities; a third acquisition module, configured to acquire a first weight and a second weight corresponding to the image feature through a pathway feature fusion module in the image recognition model, wherein the first weight is used to mark the weight occupied by the first normalized image feature, and the second weight is used to mark the weight occupied by the second normalized image feature, and the sum of the first weight and the second weight is 1; The first determination module is used to determine the fusion feature of the image to be identified based on the first normalized image feature, the first weight, the second normalized image feature and the second weight, and determine the image category of the image to be identified based on the fusion feature.
8. The device according to claim 7, characterized in that The device also includes: a fourth acquisition module, configured to acquire at least three first sample images including at least three noise intensities, input each first sample image into the image recognition model, and extract sample image features of each first sample image through a shared convolutional layer in the image recognition model; A first training module, used for training at least three batch normalization layers in the image recognition model according to the sample image features of the first sample images to obtain at least three batch normalization layers after training, wherein the sample image features of the sample images of one noise intensity are used for training one batch normalization layer; The second determination module is used to determine the first batch of normalized layers and the second batch of normalized layers of the image recognition model from the at least three batch normalized layers, wherein the first batch of normalized layers are trained by sample images with the smallest noise intensity among the at least three first sample images, and the second batch of normalized layers are trained by sample images with the largest noise intensity among the at least three first sample images.
9. A computer device, characterized in that: include: Processors, transceivers, memory, and network interfaces; The processor is connected to a memory, a transceiver and a network interface, wherein the network interface is used to provide a data communication function, the memory is used to store program codes, and the transceiver and the processor are used to call the program codes to execute the method described in any one of claims 1-6.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the method according to any one of claims 1 to 6 is executed.
11. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, implements the method according to any one of claims 1 to 6.
Citation Information
Patent Citations
Faster RCNN-based papillary thyroid carcinoma ultrasonic image recognition method and system
CN107451615A
Hyperspectral imaging-based gesture recognition method and apparatus
WO2020037594A1
