Artificial intelligence ethical-oriented fraud image identification method
Through the combination of residual module, U-Net network, visual state space module and text state space module, combined with channel attention and cross attention mechanism, the problem of low efficiency and accuracy of complex fraud image recognition in the prior art is solved. Achieve efficient and stable identification effect.
Patent Information
- Application Number
- CN202411153187.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-21
AI Technical Summary
Existing image recognition technology has problems with poor recognition efficiency, accuracy and stability when processing complex fraud images, especially when high-resolution images or large-scale data processing, high computational complexity, excessive memory usage, and computing efficiency. Low.
The residual module and U-Net network are used to extract shallow visual features, combined with the visual state space module and the text state space module to extract deep visual and semantic features, and the channel attention mechanism and cross attention mechanism are used to integrate features, using U-Net and The Mamba model is trained as a recognition model.
It improves the accuracy and stability of the model in fraudulent image recognition, reduces the computational complexity and memory usage, and can flexibly adjust feature weights, extract the most recognizable features, and provide accurate and stable recognition results.
Smart Images

Figure CN118968183A_ABST
Abstract
Description
Technical Field
[0001] The disclosed embodiments relate to the field of image recognition technology, and in particular to a fraud image recognition method oriented to artificial intelligence ethics. Background Art
[0002] At present, with the rapid development of artificial intelligence technology, generative artificial intelligence is widely used in fields such as text, audio, and images, and can create highly realistic content. However, this technological advancement also brings significant ethical risks, especially in the intelligent generation of fake news and tampered false identity image information, which may be used by malicious actors to carry out various fraud activities, not only damaging the interests of individuals and organizations, but also causing certain damage to the trust mechanism and information ecosystem of society. Although existing image recognition technology has made certain progress in some aspects, it still has many limitations when processing complex fraud images. The traditional Attention mechanism has a high computational complexity when capturing the global information and local details of the image, especially when processing high-resolution images or large-scale data, which easily leads to excessive memory usage and low computational efficiency.
[0003] It can be seen that there is an urgent need for an accurate, efficient and stable fraud image recognition method oriented towards artificial intelligence ethics. Summary of the invention
[0004] In view of this, the embodiments of the present disclosure provide a fraud image recognition method oriented to artificial intelligence ethics, which at least partially solves the problems of poor recognition efficiency, accuracy and stability in the prior art.
[0005] The present disclosure provides a fraud image recognition method for artificial intelligence ethics, including:
[0006] Step 1: Obtain a fraudulent image dataset, and annotate sample fraudulent images and content description information in the fraudulent image dataset according to the annotation system;
[0007] Step 2, perform different preprocessing on sample fraudulent images and content description information;
[0008] Step 3: Use the residual module and U-Net network to extract the shallow visual features of the sample fraud image;
[0009] Step 4, based on the shallow visual features, the visual state space module is used to extract the deep visual features of the sample fraudulent image;
[0010] Step 5, based on the content description information, using the text state space module to extract the semantic features of the sample fraud image;
[0011] Step 6: Combine the channel attention mechanism and the cross attention mechanism to fuse the deep visual features and semantic features to obtain the fused features;
[0012] Step 7: Combine the fusion features and the labels corresponding to the sample fraudulent images to train a recognition model based on the U-Net and Mamba models;
[0013] Step 8: Input the target fraud image into the trained recognition model to obtain the recognition result.
[0014] According to a specific implementation of the embodiment of the present disclosure, step 1 specifically includes:
[0015] Step 1.1, generating various sample fraud images related to fraud content through a generative artificial intelligence model;
[0016] Step 1.2: According to the definition rules in the annotation system, the sample fraud image is labeled with the appropriate fraud type label, and a text description corresponding to the image content is added.
[0017] According to a specific implementation of the embodiment of the present disclosure, step 2 specifically includes:
[0018] Step 2.1, performing data augmentation operations on each sample fraud image to generate image pairs with different viewing angles and perspectives, wherein the data augmentation operations include rotation, scaling, flipping, color jittering, and cropping;
[0019] Step 2.2, the content description information is divided into sentences, and each sentence is then subjected to punctuation removal, case normalization, word segmentation, stop word removal, and part-of-speech filtering.
[0020] According to a specific implementation of the embodiment of the present disclosure, step 3 specifically includes:
[0021] Step 3.1, for the sample fraud image I f , Feature extraction is performed through multiple ResNet stacked residual modules in sequence to extract the initial feature f1 of the image;
[0022] Step 3.2, performing convolution operation and maximum pooling downsampling on the initial feature f1 through the encoder of the U-Net network, wherein the encoder includes convolution operation and maximum pooling downsampling;
[0023] In step 3.3, the initial feature f1 is upsampled and convolved through the decoder of the U-Net network, and the upsampled feature map is concatenated with the feature map of the corresponding encoding path to obtain the shallow visual feature f2.
[0024] According to a specific implementation of the embodiment of the present disclosure, step 4 specifically includes:
[0025] Step 4.1, divide the shallow visual feature f2 into multiple non-overlapping feature image window blocks, and use a learnable position encoding strategy to represent the position relationship between feature maps;
[0026] Step 4.2: f in the feature map after position encoding vpos Normalize and get the normalized image feature f vnorm ;
[0027] Step 4.3, normalize the image feature f vnorm After being input into the SSM state space module, part of the features pass through the linear layer and SiLi activation function, and the other part of the features pass through the linear layer and one-dimensional convolution layer, and then enter the SSM state space module to extract features, fuse them through the activation function, and finally output them through the linear layer;
[0028] Step 4.4, local and global information are fused through residual connection, the visual feature information is converted and combined, and then normalization and multi-layer perceptron are performed to fuse the features;
[0029] Step 4.5, after passing through multiple visual state space modules, the learned deep visual features are combined to form new deep visual features
[0030] f vision =VSSM N (f vpos ),N=1,2…,8
[0031] Among them, H, W and C represent the height, width and number of channels of the image respectively, and N represents the number of visual state space modules.
[0032] According to a specific implementation of the embodiment of the present disclosure, step 5 specifically includes:
[0033] Step 5.1: Use the BERT text encoder to extract the semantic information corresponding to the content description information and obtain the preliminary semantic features f s , and then the semantic feature f s Perform position encoding to obtain the encoded semantic feature map f pos ;
[0034] Step 5.2: Encoded semantic feature map f pos Normalize and get the normalized semantic feature f norm ;
[0035] Step 5.3: normalize the semantic features f normAfter being input into the SSM state space module, part of the features pass through the linear layer and SiLi activation function, and the other part of the features pass through the linear layer and one-dimensional convolution layer, and then enter the SSM state space module to extract features, fuse them through the activation function, and finally output them through the linear layer;
[0036] Step 5.4, local and global information are fused through residual connection to convert and combine semantic feature information, and then normalization and feedforward neural network mechanism are performed to fuse features;
[0037] Step 5.5, after passing through multiple text state space modules, the learned semantic features are combined to form a new semantic feature f text ,
[0038] f text =TSSM N (f pos ),N=1,2…,8.
[0039] According to a specific implementation of the embodiment of the present disclosure, step 6 specifically includes:
[0040] Step 6.1, obtain the visual features f vision Perform global average pooling to generate a global average pooling vector f that describes the global characteristics of each channel avg ;
[0041] Step 6.2, obtain the semantic features f text Perform global maximum pooling to generate another global maximum pooling vector f that describes the global characteristics of each channel max ;
[0042] Step 6.3, the global average pooling vector f avg and the global maximum pooling vector f max Through a shared fully connected layer and convolutional layer, a fusion feature f is obtained. catt ;
[0043] Step 6.4, for the fused feature vector f catt , a cross attention mechanism is used to calculate visual features and semantic features, where the visual feature f catt As a query, the global maximum pooling vector f max As the key and value, we can get the score f after cross attention cross , thereby performing aggregation between image global information and text token information.
[0044] According to a specific implementation of the embodiment of the present disclosure, step 7 specifically includes:
[0045] Step 7.1: Take the sample fraud image I f Input the recognition model and get the predicted fraud result label pre f , and compare it with the real text label pre t Compare and calculate the classification loss;
[0046] Step 7.2: During the training of the model, a step-size decay strategy is adopted. After every fixed number of trainings, the learning rate is reduced by a fixed ratio. The reduction ratio of the learning rate is
[0047]
[0048] Where η0 represents the initial learning rate, t represents the decay rate, and s represents the decay step size;
[0049] Step 7.3, adjust the parameters of the recognition model according to the classification loss until it meets the preset requirements.
[0050] The fraud image recognition scheme for artificial intelligence ethics in the disclosed embodiment includes: step 1, obtaining a fraud image data set, and annotating sample fraud images and content description information in the fraud image data set according to the annotation system; step 2, performing different preprocessing on the sample fraud images and content description information; step 3, extracting shallow visual features of the sample fraud images using a residual module and a U-Net network; step 4, extracting deep visual features of the sample fraud images using a visual state space module based on the shallow visual features; step 5, extracting semantic features of the sample fraud images using a text state space module based on the content description information; step 6, combining a channel attention mechanism and a cross attention mechanism to fuse deep visual features and semantic features to obtain fused features; step 7, training a recognition model based on the U-Net and Mamba models in combination with the fused features and the labels corresponding to the sample fraud images; step 8, inputting the target fraud image into the trained recognition model to obtain a recognition result.
[0051] The beneficial effects of the embodiments of the present disclosure are as follows: through the scheme of the present disclosure, the SSM state space module can not only deeply extract the visual and semantic features of the image and capture the detailed feature information of the fraudulent image, but also the computational complexity of the SSM model is linear, which effectively reduces the memory occupancy rate; through the channel attention mechanism, it is ensured that the model can flexibly adjust the feature weights in different fraudulent image scenarios and extract the most recognizable features, thereby improving the overall recognition accuracy of the model; through the cross-attention mechanism, the model can better understand and process the complex information interaction in the fraudulent image, improve the overall recognition performance, and when faced with diversified and complex fraud patterns, the model can provide accurate and stable recognition results. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0053] Figure 1 A flowchart of a fraud image recognition method for artificial intelligence ethics provided in an embodiment of the present disclosure;
[0054] Figure 2 A schematic diagram of a framework of a visual state space module and a text state space module provided in an embodiment of the present disclosure;
[0055] Figure 3 A schematic diagram of the structure of an SSM state space module provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0056] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.
[0057] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.
[0058] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.
[0059] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The illustrations only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the form, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.
[0060] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.
[0061] The disclosed embodiment provides a fraud image recognition method oriented to artificial intelligence ethics, which can be applied to the fraud image recognition process in Internet security scenarios.
[0062] See also Figure 1 , is a flow chart of a fraud image recognition method for artificial intelligence ethics provided by the embodiment of the present disclosure. Figure 1 As shown, the method mainly comprises the following steps:
[0063] Step 1: Obtain a fraudulent image dataset, and annotate sample fraudulent images and content description information in the fraudulent image dataset according to the annotation system;
[0064] Furthermore, the step 1 specifically includes:
[0065] Step 1.1, generating various sample fraud images related to fraud content through a generative artificial intelligence model;
[0066] Step 1.2: According to the definition rules in the annotation system, the sample fraud image is labeled with the appropriate fraud type label, and a text description corresponding to the image content is added.
[0067] In the specific implementation, obtain the fraud image dataset, and annotate the fraud images and content description information according to the annotation system. The specific process is as follows:
[0068] A. Generate various fraud-related image data through many generative AI models such as ChatGPT, Kimichat, Sora, Wenxinyiyan and Pangu;
[0069] B. According to the definition rules in the annotation system, the image data is labeled with appropriate fraud type labels, and appropriate text is used to describe the image content;
[0070] C. Obtain a fraud dataset containing supervised information such as image labels and image content descriptions.
[0071] At the same time, transfer learning and automatic labeling can also be used. The specific process is as follows:
[0072] 1.1 Generating samples using transfer learning
[0073] Data collection and preprocessing:
[0074] Collect a preliminary dataset containing fraudulent images. If the existing dataset is not sufficient, relevant images can be extracted from public datasets.
[0075] Preprocess the image, such as resizing and normalizing, to meet the input requirements of the transfer learning model.
[0076] Choose a pre-trained model:
[0077] Choose pre-trained models suitable for image classification and detection such as YOLO, Faster R-CNN, ResNet, etc. These models have usually been trained on large-scale datasets and can recognize and classify various image features.
[0078] Transfer Learning:
[0079] Fine-tune the pre-trained model using existing fraud image data. This step adjusts the model’s weights to make it better suited to specific fraud types.
[0080] During training, data augmentation techniques (such as rotation, cropping, and color adjustment) can be used to expand the dataset and improve the generalization ability of the model.
[0081] Generate sample:
[0082] Use the trained model to make predictions on new images to generate sample images of various fraud types. These images can be real fraud scenes or synthetic fraud scenarios.
[0083] 1.2 Automatic labeling and manual correction
[0084] Automatic annotation:
[0085] The generated fraudulent images are fed into the trained model to automatically assign preliminary fraud type labels to each image.
[0086] The model can use different classifiers or detectors to label the main fraud types in the image.
[0087] Manual correction:
[0088] The results of automatic annotation are assigned to human annotators for review and correction. The annotators will check and correct the results of automatic annotation according to the detailed annotation guidelines to ensure accuracy.
[0089] Add a detailed text description to the image, including specific information about the type of fraud and a description of the image's content.
[0090] Step 2, perform different preprocessing on sample fraudulent images and content description information;
[0091] Based on the above embodiment, step 2 specifically includes:
[0092] Step 2.1, performing data augmentation operations on each sample fraud image to generate image pairs with different viewing angles and perspectives, wherein the data augmentation operations include rotation, scaling, flipping, color jittering, and cropping;
[0093] Step 2.2, the content description information is divided into sentences, and each sentence is then subjected to punctuation removal, case normalization, word segmentation, stop word removal, and part-of-speech filtering.
[0094] In specific implementation, the process of performing different preprocessing on sample fraudulent images and content description information can be shown as follows:
[0095] A. For fraudulent images, we generate image pairs with different perspectives and changes by performing multiple transformations on the original images to improve the generalization ability of the model;
[0096]
[0097] where f i Represents the original image, Transform represents the data enhancement operation, Represents an image after data augmentation.
[0098] Rotate: Rotates the image at random angles between 0 and 360 degrees, simulating viewing angles at different angles.
[0099] Scaling: Randomly scale the image with a factor between 0.6 and 1.5 to generate images of different sizes.
[0100] Flip: Perform random flips in horizontal and vertical directions.
[0101] Cropping: Randomly crop different areas of the image to simulate partial loss of perspective.
[0102] Color dithering: Modify the image's brightness, contrast, saturation and other color parameters to simulate different lighting conditions.
[0103] B. For the content description of fraudulent images, first perform sentence segmentation, then remove punctuation, normalize the case, segment words, remove stop words, and filter parts of speech for each sentence. The Jieba word segmentation tool is used for word segmentation and part of speech filtering, and the Harbin Institute of Technology stop word list is used to remove stop words.
[0104] Step 3: Use the residual module and U-Net network to extract the shallow visual features of the sample fraud image;
[0105] Furthermore, the step 3 specifically includes:
[0106] Step 3.1, for the sample fraud image I f , Feature extraction is performed through multiple ResNet stacked residual modules in sequence to extract the initial feature f1 of the image;
[0107] Step 3.2, performing convolution operation and maximum pooling downsampling on the initial feature f1 through the encoder of the U-Net network, wherein the encoder includes convolution operation and maximum pooling downsampling;
[0108] In step 3.3, the initial feature f1 is upsampled and convolved through the decoder of the U-Net network, and the upsampled feature map is concatenated with the feature map of the corresponding encoding path to obtain the shallow visual feature f2.
[0109] In specific implementation, the process of extracting shallow visual features of fraudulent images using ResNet and U-Net networks may include:
[0110] A. For a given fraudulent image I f ∈R H×W×C , where H, W and C represent the height, width and number of channels of the image respectively, which are used for preliminary feature processing.
[0111] f0=Init(I f )
[0112] B. Feature extraction is performed through multiple stacked residual modules in sequence to extract image I f Shallow features Where C0 is the number of feature map channels. The residual module contains multiple convolutional layers, normalization layers, and skip connections. The model directly transmits information through skip connections to alleviate the gradient vanishing problem in deep networks.
[0113] f1=ResNet(f0)
[0114] C. Through a series of encoder parts consisting of convolution operations and maximum pooling downsampling, the spatial size of the input image is gradually reduced, while the depth of the image feature map is gradually increased to gradually extract deeper features.
[0115] D. The decoding part mainly consists of a series of upsampling operations and convolution operations, which are used to gradually restore the spatial dimension and resolution of the image, concatenate the upsampled feature map with the feature map of the corresponding encoding path, and allow the model to use the features extracted in the encoding stage in the decoding stage to enhance the detailed feature information of the image.
[0116] f2=UNet(f1).
[0117] Optionally, local feature descriptors can be used to extract shallow features in the image by calculating the color histogram or texture features of the local area. The specific process is:
[0118] 1. Local area division
[0119] Split the image into multiple local regions (such as fixed-size windows).
[0120] 2. Calculate the color histogram
[0121] Extract Color Channels: Extract channels in RGB or other color spaces from each region.
[0122] Construct histogram: Calculate the histogram of each color channel to show the color distribution.
[0123] Normalization: Normalize the histogram.
[0124] 3. Calculate texture features
[0125] Select a texture descriptor: Extract texture features using methods such as Gray Level Co-occurrence Matrix (GLCM), Local Binary Patterns (LBP), or Gabor filters.
[0126] Calculate features: Generate texture feature vectors of local areas and normalize them.
[0127] 4. Feature Aggregation and Representation
[0128] Generate feature vector: Aggregate the color and texture features of local areas into the overall feature representation of the image.
[0129] Feature selection: Select the most important features and perform dimensionality reduction.
[0130] These steps help extract useful shallow features from local regions of the image for image classification or recognition tasks.
[0131] Step 4, based on the shallow visual features, the visual state space module is used to extract the deep visual features of the sample fraudulent image;
[0132] Based on the above embodiment, step 4 specifically includes:
[0133] Step 4.1, divide the shallow visual feature f2 into multiple non-overlapping feature image window blocks, and use a learnable position encoding strategy to represent the position relationship between feature maps;
[0134] Step 4.2: f in the feature map after position encoding vpos Normalize and get the normalized image feature fvnorm ;
[0135] Step 4.3, normalize the image feature f vnorm After being input into the SSM state space module, part of the features pass through the linear layer and SiLi activation function, and the other part of the features pass through the linear layer and one-dimensional convolution layer, and then enter the SSM state space module to extract features, fuse them through the activation function, and finally output them through the linear layer;
[0136] Step 4.4, local and global information are fused through residual connection, the visual feature information is converted and combined, and then normalization and multi-layer perceptron are performed to fuse the features;
[0137] Step 4.5, after passing through multiple visual state space modules, the learned deep visual features are combined to form new deep visual features
[0138] f vision =VSSM N (f vpos ),N=1,2…,8
[0139] Among them, H, W and C represent the height, width and number of channels of the image respectively, and N represents the number of visual state space modules.
[0140] In the specific implementation, the specific framework of the visual state space module and the text state space module is as follows Figure 2 As shown in Figure 2, the process of extracting deep features of an image using the visual state space module is as follows:
[0141] A. The extracted image feature map f2 is divided into M non-overlapping image window blocks, and the size of each image window block is n×n.
[0142]
[0143] Where n represents the width and height of the image window block, and M represents the number of image window blocks.
[0144] B. The model uses a learnable position encoding method to better represent the positional relationship between images. During the training process, the position encoding parameters are updated according to the gradient calculated by the loss function.
[0145] C. Add f to the feature map after position encoding vpos The normalization module is used to effectively improve the model stability and training efficiency, thereby obtaining the normalized image feature f vnorm .
[0146] D. Normalize the image features f vnormInput into the SSM state space module, part of the features pass through the linear layer and SiLi activation function, and the other part of the features pass through the linear layer and one-dimensional convolution layer, and then enter the SSM state space module for feature extraction, and finally fused through the activation function, and then output through the linear layer. The structure of the state space module is as follows Figure 3 shown.
[0147] In the state-space module, the state equation can be obtained by discretizing the linear equations of the continuous-time system:
[0148] h t =Ah t-1 +Bx t
[0149] y t =Ch t-1 +Dx t
[0150] Among them, h t represents the hidden state at the tth time step, x t Represents input information, y t Represents output information, A is the state transfer matrix, which describes the dynamic relationship between states; B is the input matrix, which describes how input x affects state h; C is the output matrix, which describes how state h is transformed into output y; D represents the influence of input x on output y.
[0151] In the Mamba model, the discretization of the state equation uses the zero-order hold rule, and the converted state transfer matrices A and B can be expressed as
[0152]
[0153] Among them, Δ represents the discretized time step, exp is the exponential function, A and B are the discrete state transfer matrices, and I is the identity matrix.
[0154] Through the above state equation and the initial hidden state, the change of the hidden state in the Mamba model is iteratively calculated. The change of the hidden state can be expressed as:
[0155]
[0156] y k =Ch k +Dx k
[0157]
[0158] in is the structured convolution kernel, M represents the length of the input sequence, and * represents the convolution operation.
[0159] E. The local and global information are fused through residual connection to convert and combine the visual feature information; then normalization and multi-layer perceptron are performed to fuse the features.
[0160] F. After passing through the visual state space module of multiple images, the model combines the learned deep visual features to form new features
[0161] f vision =VSSM N (f vpos ),N=1,2…,8.
[0162] Optionally, in addition to using the visual state space module, deep visual features of the sample fraudulent image can be further extracted based on shallow visual features in other ways. For example:
[0163] 1. Convolutional Neural Network (CNN)
[0164] Feature extraction: Use pre-trained convolutional neural networks (such as VGG, ResNet, etc.) to extract deep features of the image. These networks have been trained on large-scale datasets and can automatically learn effective visual features.
[0165] Transfer learning: fine-tuning a pre-trained model to adapt it to a specific fraud image recognition task.
[0166] 2. Deep feature fusion
[0167] Fusion of shallow features and deep features: Fusion of shallow visual features (such as color histogram and texture features) with deep features (extracted by CNN) to enhance the feature representation capability of the model.
[0168] Feature fusion methods: You can combine shallow and deep features using concatenation, weighted averaging, or other fusion methods.
[0169] 3. Autoencoder
[0170] Unsupervised learning: Use autoencoders to perform unsupervised learning on images, automatically learning deep features from shallow features. The autoencoder can compress shallow features into low-dimensional deep features through the encoder part, and then reconstruct the image through the decoder.
[0171] Feature Compression: The learned encoder output is used as a deep feature representation of the image.
[0172] Step 5, based on the content description information, using the text state space module to extract the semantic features of the sample fraud image;
[0173] Furthermore, the step 5 specifically includes:
[0174] Step 5.1: Use the BERT text encoder to extract the semantic information corresponding to the content description information and obtain the preliminary semantic features f s , and then the semantic feature f s Perform position encoding to obtain the encoded semantic feature map f pos ;
[0175] Step 5.2: Encoded semantic feature map f pos Normalize and get the normalized semantic feature f norm ;
[0176] Step 5.3, normalize the semantic features f norm After being input into the SSM state space module, part of the features pass through the linear layer and SiLi activation function, and the other part of the features pass through the linear layer and one-dimensional convolution layer, and then enter the SSM state space module to extract features, fuse them through the activation function, and finally output them through the linear layer;
[0177] Step 5.4, local and global information are fused through residual connection to convert and combine semantic feature information, and then normalization and feedforward neural network mechanism are performed to fuse features;
[0178] Step 5.5, after passing through multiple text state space modules, the learned semantic features are combined to form a new semantic feature f text ,
[0179] f text =TSSM N (f pos ),N=1,2…,8.
[0180] In specific implementation, the process of extracting the semantic features of an image using the text state space module is as follows:
[0181] A. For text description information, the BERT text encoder is used to extract the semantic features of the image to obtain the preliminary semantic features f s , and then the obtained semantic feature f s Perform position encoding to obtain the encoded semantic feature map f pos .
[0182] B. Encoded semantic feature map f pos The normalization module is used to effectively improve the model stability and training efficiency, thereby obtaining the normalized semantic features f norm .
[0183] C. Normalize the semantic features f normInput into the SSM state space module, part of the features pass through the linear layer and SiLi activation function, and the other part of the features pass through the linear layer and one-dimensional convolution layer, and then enter the SSM state space module for feature extraction, and finally fused through the activation function, and then output through the linear layer. Through the above state equation and the initial hidden state, the change of the hidden state in the SSM model can be iteratively calculated. The change of the hidden state can be expressed as:
[0184]
[0185] y k =Ch k +Dx k
[0186]
[0187] in is the structured convolution kernel, M represents the length of the input sequence, and * represents the convolution operation.
[0188] D. Local and global information are fused through residual connections to convert and combine semantic feature information; then normalization and feedforward neural network mechanism are performed to fuse features.
[0189] E. After passing through multiple text state space modules, the model combines the learned semantic features to form a new semantic feature f text .
[0190] f text =TSSM N (f pos ),N=1,2…,8.
[0191] Optionally, in addition to using the text state space module, you can also use the following method:
[0192] 1. Cross-Modal Learning
[0193] Image-text alignment: By aligning images with their description text, a cross-modal model is trained to learn the association between images and text. For example, images and descriptions are mapped using an image-text embedding space so that semantically related images and texts are closer in the embedding space.
[0194] Bidirectional Network: Use a bidirectional network such as CLIP to jointly train images and text to learn shared semantic features.
[0195] 2. Text Generation Model
[0196] Description generation: Use text generation models such as GPT, BERT, etc. to generate detailed descriptions related to the image, thereby extracting rich semantic features. These descriptions can be used to further analyze the content of the image.
[0197] Description completion: If part of the description of an image is missing, a generative model can be used to complete the description to help obtain more complete semantic information.
[0198] Step 6: Combine the channel attention mechanism and the cross attention mechanism to fuse the deep visual features and semantic features to obtain the fused features;
[0199] Based on the above embodiment, step 6 specifically includes:
[0200] Step 6.1, obtain the visual features f vision Perform global average pooling to generate a global average pooling vector f that describes the global characteristics of each channel avg ;
[0201] Step 6.2, obtain the semantic features f text Perform global maximum pooling to generate another global maximum pooling vector f that describes the global characteristics of each channel max ;
[0202] Step 6.3, the global average pooling vector f avg and the global maximum pooling vector f max Through a shared fully connected layer and convolutional layer, a fusion feature f is obtained. catt ;
[0203] Step 6.4, for the fused feature vector f catt , a cross attention mechanism is used to calculate visual features and semantic features, where the visual feature f catt As a query, the global maximum pooling vector f max As the key and value, we can get the score f after cross attention cross , thereby performing aggregation between image global information and text token information.
[0204] In specific implementation, the collection channel and cross attention mechanism fuse the visual features and semantic features of the image;
[0205] A. Obtained visual features f vision Perform global average pooling to generate a vector f that describes the global characteristics of each channel avg ∈R C .
[0206]
[0207] B. Obtained semantic features f text Perform global maximum pooling to generate another vector f that describes the global characteristics of each channel max ∈R C
[0208]
[0209] C. Global average pooling vector f avg and the global maximum pooling vector f max After processing through a shared fully connected layer and convolutional layer, a fused feature representation f is obtained. catt .
[0210] f catt =σ(W2·ReLU(W1·(f avg +f max )))
[0211] D. For the fused feature vector f catt In order to further integrate the feature information between text and image, the image embedding information is transferred to the text token, and the visual features and semantic features are calculated through the cross attention mechanism, where the visual feature f catt As query, the global maximum pooling vector f max As key and value, we can get the score f after cross attention. cross , thereby performing aggregation between image global information and text token information.
[0212]
[0213] Optionally, in addition to the aggregate channel attention mechanism and the cross attention mechanism, methods can be used to effectively fuse deep visual features and semantic features, such as:
[0214] 1. Feature splicing and fusion
[0215] Feature concatenation: Visual features and semantic features are concatenated at the feature level and then further processed by fully connected layers or convolutional layers. After concatenation, the model can learn the representation of the joint features.
[0216] Weighted fusion: Apply a weighting mechanism to visual features and semantic features and calculate the weighted sum to obtain fused features. The weighting coefficients can be automatically learned through the training process.
[0217] 2. Graph Convolutional Networks (GCN)
[0218] Graph embedding: Visual features and semantic features are regarded as nodes in the graph, and a graph convolutional network is used to fuse node information. GCN is able to process graph structured data and is suitable for integrating features from different sources.
[0219] Step 7: Combine the fusion features and the labels corresponding to the sample fraudulent images to train a recognition model based on the U-Net and Mamba models;
[0220] Based on the above embodiment, step 7 specifically includes:
[0221] Step 7.1: Take the sample fraud image I f Input the recognition model and get the predicted fraud result label pre f , and compare it with the real text label pre t Compare and calculate the classification loss;
[0222] Step 7.2: During the training of the model, a step-size decay strategy is adopted. After every fixed number of trainings, the learning rate is reduced by a fixed ratio. The reduction ratio of the learning rate is
[0223]
[0224] Where η0 represents the initial learning rate, t represents the decay rate, and s represents the decay step size;
[0225] Step 7.3, adjust the parameters of the recognition model according to the classification loss until it meets the preset requirements.
[0226] In specific implementation, the fraud image I f After model recognition, the predicted fraud result label is obtained. f , and the actual text label pre t The classification loss is calculated by comparison. This loss function can effectively optimize the parameters of the model network and guide the model to optimize the recognition performance.
[0227] f rec =1 / m∑(pre f -pre t ) 2
[0228] Where m represents the number of fraudulent images.
[0229] During the training process of the model, a step size decay strategy is adopted. After a fixed number of training times, the learning rate is reduced by a fixed ratio. This method can smooth the training process and avoid premature convergence and oscillation.
[0230]
[0231] Where η0 represents the initial learning rate, t represents the decay rate, and s represents the decay step size.
[0232] Evaluate the image recognition effect of the model on the dataset to ensure that it performs well in image recognition, and dynamically adjust the model parameters and training strategies based on the evaluation results until the recognition model is trained to meet the preset requirements. For example, when the classification loss corresponding to the recognition model's recognition of sample fraudulent images is less than the preset threshold, the optimal hyperparameter value can be determined and the optimal model parameters can be saved.
[0233] Step 8: Input the target fraud image into the trained recognition model to obtain the recognition result.
[0234] In specific implementation, after obtaining a trained recognition model, the unlabeled fraudulent image acquired in real time can be input into it to obtain the recognition result corresponding to the fraudulent image, for example, the label corresponding to the fraudulent image can be output. At the same time, during the use of the trained recognition model, the precision rate, recall rate and F1 index can also be calculated in real time as the accuracy index of the model recognition, and can be updated and optimized in real time according to the accuracy index.
[0235] The fraud image recognition method for artificial intelligence ethics provided in this embodiment extracts visual feature information and semantic features of fraud images through visual and textual state space modules, effectively enhancing the richness of feature extraction in the image. At the same time, combined with channel attention and cross attention, the model can not only dynamically learn the importance of different feature channels, effectively improve the weights of important features, reduce the impact of redundant information, and conduct information interaction between visual features and semantic features, but also effectively improve the recognition ability of the model in fraud scenarios and provide more accurate and stable recognition results.
[0236] The method will be further described below in conjunction with a specific embodiment.
[0237] Step 1. Obtain a dataset of fraudulent images and annotate the fraudulent images with the type and content description according to the label system:
[0238] A. The fraud image dataset includes 12 categories of data, including forged documents, fake invoices, fake advertisements, fake medical reports, fake payment screenshots, fake QR codes, fake logistics information, fake product images, fake brand logos, fake recruitment images, fake news screenshots, and fake air tickets;
[0239] B. Appropriately label the fraudulent image data according to the label system, and add appropriate text to the content of the fraudulent image to describe the fraudulent content.
[0240] Step 2. Perform different preprocessing on the fraudulent images and content descriptions obtained in step 1:
[0241] A. For fraudulent images, multiple data augmentation techniques such as rotation, scaling, flipping, color jittering and cropping are used in the data processing stage to improve the diversity and richness of training samples and enhance the generalization ability of the model in different environments.
[0242] Suppose there are two fraudulent images d1 and d2, which are pictures of counterfeit goods and contain the logo information of counterfeit goods. The original image d1 is generated by image enhancement processing such as rotation, scaling, and flipping to generate d°1, and the original image d1 is generated by image processing combinations such as rotation, color dithering, and flipping to generate d°2. Then (d1, d°1) and (d1, d°2) are training samples generated by different data enhancement operations, thereby improving the diversity of samples.
[0243] B. For the content description information, use the Jieba word segmentation tool to segment the text description into a sequence of words. Then, filter the words in the word sequence according to the stop word list of Harbin Institute of Technology, delete and filter out the words in the stop word list, and finally compare the part of speech marked by the Jieba tool during word segmentation with the custom reserved part of speech table, and delete and filter out the words whose part of speech is not in the reserved part of speech table;
[0244] Step 3. Use ResNet and U-Net networks to extract shallow visual features of fraudulent images;
[0245] A. For a given fraudulent image I f , feature extraction is performed sequentially through multiple ResNet stacked residual modules to extract shallow features f1 of the image;
[0246] B. The encoder part consists of a series of convolution operations and maximum pooling downsampling to capture higher-level features;
[0247] C. It consists of a series of upsampling operations and convolution operations to gradually restore the spatial dimension and resolution of the image, and concatenate the upsampled feature map with the feature map of the corresponding encoding path to obtain a deeper feature f2.
[0248] Step 4. Use the visual state space module to extract deep features of the image
[0249] A. For the extracted image feature map f2, it is divided into multiple non-overlapping image window blocks, and a learnable position encoding strategy is used to better represent the positional relationship between images;
[0250] B. Input the normalization module and state space module into the position-encoded feature map to perform deep feature extraction;
[0251] C. After passing through multiple visual state space modules, the model combines the learned deep high-frequency features to form a new feature f vision ;
[0252] Step 5. Extract the semantic features of the image using the text state space module
[0253] A. Download the text encoder model and parameter configuration of the pre-trained BERT model;
[0254] B. Use the model parameter values downloaded in sub-step A to initialize the embedding layer parameters of the text encoding model, and freeze the embedding layer parameters so that they do not change during the training process. In this way, the preprocessed text is input into the model and then converted into a semantic feature vector f after passing through the embedding layer. s ;
[0255] C. Obtained semantic features f s Perform position encoding to obtain the encoded semantic feature map f pos ;
[0256] D. Encoded semantic feature map f pos The normalization module is used to obtain f norm , and then the normalized semantic feature f norm The input is sent to the state space module, where local and global information are fused through residual connections to convert and combine semantic feature information. After that, normalization and feedforward neural network mechanism are used to fuse features.
[0257] E. After passing through multiple text state space modules, the model combines the learned semantic features to form a new semantic feature f text ;
[0258] Step 6. Combine channel and cross attention mechanisms to fuse the visual and semantic features of the image
[0259] A. Obtained visual features f vision Perform global average pooling to generate a vector f that describes the global characteristics of each channel avg ;
[0260] B. Obtained semantic features f text Perform global maximum pooling to generate another vector f that describes the global characteristics of each channel max ;
[0261] C. Global average pooling vector f avg and the global maximum pooling vector f max After processing through a shared fully connected layer and convolutional layer, a fused feature representation f is obtained. catt ;
[0262] D. For the fused feature vector f catt In order to further integrate the feature information between text and image, the image embedding information is transferred to the token, and the visual features and semantic features are calculated through the cross attention mechanism to obtain the score f after cross attention. cross ;
[0263] Step 7. Combine the supervision information to train the fraud image recognition model based on the U-Net and Mamba models;
[0264] A. The fraud image recognition model uses the U-Net and Mamba models as the main architecture. After generating the visual feature vector, a Dropout (random inactivation) layer is added to reduce the overfitting problem during training, making the final training model more generalizable. Since the uneven number of samples for each label will affect the training effect, the focal loss cost function is used to assign different weights to the training samples to reduce the impact caused by the uneven number of samples for different labels;
[0265] B. In order to evaluate the recognition ability of the model, the model loss is calculated through classification loss, thereby optimizing the model parameters and structure;
[0266] C. Use the TensorBoard visualization tool to train the curve when adjusting hyperparameters, determine the best hyperparameter values, and save the best model parameters;
[0267] Step 8. Use the trained recognition model to identify fraudulent images and obtain the identified labels;
[0268] A. Input the result obtained from the preprocessing of sub-step A in step 2 into the fraud identification model for model identification;
[0269] B. Calculate the precision, recall and F1 index as the accuracy indicators for evaluating model recognition.
[0270] The present invention aims to solve the problem of fraud risk in artificial intelligence ethics and provide a fraud image recognition method for artificial intelligence ethics. The present invention uses the state space module to more deeply understand the semantic and visual features of the image, especially for complex fraud image scenes, it can identify subtle traces of fraud; at the same time, it combines the channel attention and cross attention mechanisms to efficiently perform feature fusion and information interaction, providing more accurate, stable and efficient recognition results, and providing an effective solution for fraud risk identification in artificial intelligence ethics.
[0271] The method disclosed in the present invention uses the SSM module to perform state equation modeling, which can effectively capture long-distance dependencies in images and retain the global and local features of the original image information. At the same time, compared with the traditional Attention mechanism, the calculation process of SSM is linear, which can reduce the memory usage of the model and improve the operational efficiency of the model. By combining the channel attention and cross-attention modules to fuse visual features and semantic features, the model can not only dynamically learn the importance of different feature channels, effectively improve the weight of useful features, and reduce the impact of redundant information; but also establish information interaction between text and visual features through the cross-attention mechanism, which helps the visual and semantic features to complement each other and improve the overall recognition performance of the model.
[0272] It should be understood that various parts of the present disclosure may be implemented in hardware, software, firmware, or a combination thereof.
[0273] The above is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.
Claims
1. A fraud image recognition method for artificial intelligence ethics, characterized in that: include: Step 1: Obtain a fraudulent image dataset, and annotate sample fraudulent images and content description information in the fraudulent image dataset according to the annotation system; Step 2, perform different preprocessing on sample fraudulent images and content description information; Step 3: Use the residual module and U-Net network to extract the shallow visual features of the sample fraud image; Step 4, based on the shallow visual features, the visual state space module is used to extract the deep visual features of the sample fraudulent image; Step 5, based on the content description information, using the text state space module to extract the semantic features of the sample fraud image; Step 6: Combine the channel attention mechanism and the cross attention mechanism to fuse the deep visual features and semantic features to obtain the fused features; Step 7: Combine the fusion features and the labels corresponding to the sample fraudulent images to train a recognition model based on the U-Net and Mamba models; Step 8: Input the target fraud image into the trained recognition model to obtain the recognition result.
2. The method according to claim 1, characterized in that , the step 1 specifically includes: Step 1.1, generating various sample fraud images related to fraud content through a generative artificial intelligence model; Step 1.2: According to the definition rules in the annotation system, the sample fraud image is labeled with the appropriate fraud type label, and a text description corresponding to the image content is added.
3. The method according to claim 2, characterized in that , the step 2 specifically includes: Step 2.1, performing data augmentation operations on each sample fraud image to generate image pairs with different viewing angles and perspectives, wherein the data augmentation operations include rotation, scaling, flipping, color jittering, and cropping; Step 2.2, the content description information is divided into sentences, and each sentence is then subjected to punctuation removal, case normalization, word segmentation, stop word removal, and part-of-speech filtering.
4. The method according to claim 3, characterized in that , the step 3 specifically includes: Step 3.1, for the sample fraud image I f , Feature extraction is performed through multiple ResNet stacked residual modules in sequence to extract the initial feature f1 of the image; Step 3.2, performing convolution operation and maximum pooling downsampling on the initial feature f1 through the encoder of the U-Net network, wherein the encoder includes convolution operation and maximum pooling downsampling; In step 3.3, the initial feature f1 is upsampled and convolved through the decoder of the U-Net network, and the upsampled feature map is concatenated with the feature map of the corresponding encoding path to obtain the shallow visual feature f2.
5. The method according to claim 4, characterized in that , the step 4 specifically includes: Step 4.1, divide the shallow visual feature f2 into multiple non-overlapping feature image window blocks, and use a learnable position encoding strategy to represent the position relationship between feature maps; Step 4.2: f in the feature map after position encoding vpos Normalize and get the normalized image feature f vnorm ; Step 4.3, the normalized image feature f vnorm After being input into the SSM state space module, part of the features pass through the linear layer and SiLi activation function, and the other part of the features pass through the linear layer and one-dimensional convolution layer, and then enter the SSM state space module to extract features, fuse them through the activation function, and finally output them through the linear layer; Step 4.4, local and global information are fused through residual connection, the visual feature information is converted and combined, and then normalization and multi-layer perceptron are performed to fuse the features; Step 4.5, after passing through multiple visual state space modules, the learned deep visual features are combined to form new deep visual features f vision =VSSM N (f vpos ),N=1,2…,8 Among them, H, W and C represent the height, width and number of channels of the image respectively, and N represents the number of visual state space modules.
6. The method according to claim 5, characterized in that , the step 5 specifically includes: Step 5.1: Use the BERT text encoder to extract the semantic information corresponding to the content description information and obtain the preliminary semantic features f s , and then the semantic feature f s Perform position encoding to obtain the encoded semantic feature map f pos ; Step 5.2: Encoded semantic feature map f pos Normalize and get the normalized semantic feature f norm ; Step 5.3, normalize the semantic features f norm After being input into the SSM state space module, part of the features pass through the linear layer and SiLi activation function, and the other part of the features pass through the linear layer and one-dimensional convolution layer, and then enter the SSM state space module to extract features, fuse them through the activation function, and finally output them through the linear layer; Step 5.4, local and global information are fused through residual connection to convert and combine semantic feature information, and then normalization and feedforward neural network mechanism are performed to fuse features; Step 5.5, after passing through multiple text state space modules, the learned semantic features are combined to form a new semantic feature f text , f text =TSSM N (f pos ),N=1,2…,8。 7. The method according to claim 6, characterized in that , the step 6 specifically includes: Step 6.1, obtain the visual features f vision Perform global average pooling to generate a global average pooling vector f that describes the global characteristics of each channel avg ; Step 6.2, obtain the semantic features f text Perform global maximum pooling to generate another global maximum pooling vector f that describes the global characteristics of each channel max ; Step 6.3, the global average pooling vector f avg and the global maximum pooling vector f max Through a shared fully connected layer and convolutional layer, a fusion feature f is obtained. catt ; Step 6.4, for the fused feature vector f catt , a cross attention mechanism is used to calculate visual features and semantic features, where the visual feature f catt As a query, the global maximum pooling vector f max As the key and value, we can get the score f after cross attention cross , thereby performing aggregation between image global information and text token information.
8. The method according to claim 7, characterized in that , the step 7 specifically includes: Step 7.1: Take the sample fraud image I f Input the recognition model and get the predicted fraud result label pre f , and compare it with the real text label pre t Compare and calculate the classification loss; Step 7.2: During the training of the model, a step-size decay strategy is adopted. After every fixed number of trainings, the learning rate is reduced by a fixed ratio. The reduction ratio of the learning rate is Where η0 represents the initial learning rate, t represents the decay rate, and s represents the decay step size; Step 7.3, adjust the parameters of the recognition model according to the classification loss until it meets the preset requirements.
Citation Information
Patent Citations
Deep forgery detection method based on artifact noise
CN115100128A
Brain tumor image segmentation method based on multi-scale convolution and Mama structure
CN118447244A
Brain tumor image segmentation method and device and readable storage medium
CN118505713A
Cited By
Residual aggregation enhanced neighborhood graph sampling credit card fraud detection method
CN120031570A
Table structure identification method and system based on visual state space
CN120412001A
A table structure recognition method and system based on visual state space
CN120412001B
Remote sensing image subtitle generation method and device based on channel-space attention and Mama network
CN120672902A
Method and system for identifying fraud image based on generative artificial intelligence ethics
CN122391804A
