Image detection method and device, computer equipment and storage medium

Abstract

The embodiment of the invention belongs to the field of image processing, and relates to an image detection method and device, computer equipment and a storage medium, and the method comprises the following steps: carrying out the preprocessing of image-text pair information, and obtaining standard image information and standard text information; performing feature decomposition on the standard image information to obtain a first feature image and a second feature image; inputting the first feature image and the second feature image into a preset feature extraction model for feature extraction to obtain a first image feature and a second image feature, and combining the first image feature and the second image feature into a comprehensive image feature; performing semantic feature extraction on the standard text information to obtain text semantic features; fusing the comprehensive image features and the text semantic features to obtain effective feature representation; and performing image tampering judgment on the effective feature representation according to a preset tampering detection model to obtain image tampering information and generate a tampering detection report. According to the method and the device, whether the image in the image-text pair information is tampered or not can be accurately detected.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, specifically to the field of digital medicine, and in particular to an image detection method, device, computer equipment and storage medium. Background Art

[0002] In today's digital age, image information plays a vital role in the financial and medical fields. Whether it is a credential image in a financial transaction or imaging data in a medical diagnosis, the authenticity and integrity of the image are the basis for ensuring the accuracy of decision-making. However, with the rapid development of image processing technology, image tampering has become increasingly easy and difficult to detect, which poses a severe challenge to the credibility of image information.

[0003] The traditional method of image-text information review mainly relies on manual judgment. Auditors need to refer to both the image information and the text information describing it to determine whether the image has been tampered with. However, this method has many limitations. First, due to the uneven technical understanding and experience level of auditors, it is difficult to ensure accurate identification of image tampering even with the combination of text information. Especially in the financial and medical fields, image information often contains complex expertise and details, which further increases the difficulty of manual judgment. Secondly, the manual judgment method has the problem of inefficiency. With the surge in the amount of image information, manual review is not only time-consuming and laborious, but also difficult to cope with large-scale data processing needs. This not only increases operating costs, but also limits the widespread application of image information tampering detection technology. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image detection method, apparatus, computer equipment and storage medium to solve the problem of being unable to effectively and accurately detect whether an image in image-text information has been tampered with.

[0005] In order to solve the above technical problems, the present application provides an image detection method, which adopts the following technical solution:

[0006] Acquire image-text pair information, and pre-process the image-text pair information to obtain standard image information and standard text information;

[0007] Performing feature decomposition on the standard image information to obtain a first feature image and a second feature image;

[0008] Inputting the first feature image and the second feature image into a preset feature extraction model to perform feature extraction to obtain a first image feature and a second image feature;

[0009] combining the first image feature and the second image feature into a comprehensive image feature;

[0010] Extracting semantic features from the standard text information to obtain text semantic features;

[0011] Fusing the comprehensive image features with the text semantic features to obtain effective feature representation;

[0012] The image tampering judgment is performed on the effective feature representation according to a preset tampering detection model to obtain image tampering information, and a tampering detection report is generated according to the image tampering information.

[0013] Furthermore, the image-text information includes image information and text information, and the step of preprocessing the image-text information to obtain standard image information and standard text information specifically includes:

[0014] Performing size normalization processing on the image information to obtain normalized image information;

[0015] Performing image enhancement processing on the normalized image information to obtain the standard image information;

[0016] Performing data cleaning and word segmentation processing on the text information to obtain word segmentation text information;

[0017] The word segmentation text information is represented by a word vector to obtain the standard text information.

[0018] Furthermore, the step of performing feature decomposition on the standard image information to obtain the first feature image and the second feature image specifically includes:

[0019] Inputting the standard image information into a preset convolutional neural network, performing feature map extraction on the standard image information through the convolutional neural network, and obtaining an image feature map;

[0020] The image feature image is decomposed along the channel dimension to obtain the first feature image and the second feature image.

[0021] Furthermore, the step of inputting the first feature image and the second feature image into a preset feature extraction model for feature extraction to obtain the first image feature and the second image feature specifically includes:

[0022] Obtaining a first model extraction identifier, and extracting the feature extraction model from a database according to the first model extraction identifier, wherein the feature extraction model includes a convolutional network path and a visual model path;

[0023] Performing local feature extraction on the first feature image according to the convolutional network path to obtain the first image feature;

[0024] Perform global feature extraction on the second feature image according to the visual model path to obtain the second image feature.

[0025] Furthermore, the step of extracting semantic features from the standard text information to obtain text semantic features specifically includes:

[0026] Obtaining a second model extraction identifier, and extracting a preset language processing model from a database according to the second model extraction identifier;

[0027] Performing word embedding processing on the standard text information based on the word embedding layer of the language processing model to obtain a word vector sequence;

[0028] Based on the multi-layer processing layers of the language processing model, feature extraction is performed on the word vector sequence to obtain the text semantic features.

[0029] Furthermore, the step of performing image tampering judgment on the effective feature representation according to a preset tampering detection model to obtain image tampering information, and generating a tampering detection report according to the image tampering information specifically includes:

[0030] Acquire a third model extraction identifier, and extract the tampering detection model from a database according to the third model extraction identifier;

[0031] Inputting the effective feature representation into the tampering detection model to obtain the image tampering probability;

[0032] Determining whether the image tampering probability is greater than or equal to a preset probability threshold;

[0033] If the probability of image tampering is greater than or equal to the preset probability threshold, the image in the image-text pair information is determined to be a tampered image, and the tampering type and tampering area of ​​the tampered image are identified, the tampering type, the tampering area, and the image tampering probability are integrated into the image tampering information, the corresponding tampering detection report template is obtained according to the tampering type, and the tampering detection report template is generated based on the tampering detection report template.

[0034] Furthermore, the step of obtaining a corresponding tampering detection report template according to the tampering type and generating the tampering detection report based on the tampering detection report template specifically includes:

[0035] Extracting the tampering detection report template from a template database using the tampering type as a query condition;

[0036] Matching the image tampering information with the template parameters of the tampering detection report template to obtain an information matching result;

[0037] The image tampering information is filled into the corresponding parameters of the tampering detection report template according to the information matching result to obtain the tampering detection report.

[0038] In order to solve the above technical problems, the embodiment of the present application further provides an image detection device, which adopts the following technical solution:

[0039] An information processing module, used for acquiring image-text pair information, and preprocessing the image-text pair information to obtain standard image information and standard text information;

[0040] A feature decomposition module, used for performing feature decomposition on the standard image information to obtain a first feature image and a second feature image;

[0041] A first feature extraction module, used for inputting the first feature image and the second feature image into a preset feature extraction model for feature extraction to obtain a first image feature and a second image feature;

[0042] A feature merging module, used for merging the first image feature and the second image feature into a comprehensive image feature;

[0043] A second feature extraction module is used to extract semantic features from the standard text information to obtain text semantic features;

[0044] A feature fusion module, used to fuse the comprehensive image features and the text semantic features to obtain an effective feature representation;

[0045] The result generation module is used to perform image tampering judgment on the effective feature representation according to a preset tampering detection model, obtain image tampering information, and generate a tampering detection report according to the image tampering information.

[0046] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0047] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of any one of the above image detection methods when executing the computer-readable instructions.

[0048] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0049] A computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the image detection method described in any one of the above items are implemented.

[0050] Compared with the prior art, the embodiment of the present application has the following beneficial effects: by acquiring image-text pair information, the embodiment preprocesses the image-text pair information to obtain standard image information and standard text information; performs feature decomposition on the standard image information to obtain a first feature image and a second feature image; inputs the first feature image and the second feature image into a preset feature extraction model for feature extraction to obtain a first image feature and a second image feature; merges the first image feature and the second image feature into a comprehensive image feature; performs semantic feature extraction on the standard text information to obtain text semantic features; fuses the comprehensive image features and the text semantic features to obtain an effective feature representation; performs image tampering judgment on the effective feature representation according to a preset tampering detection model to obtain image tampering information, and generates a tampering detection report according to the image tampering information. Thus, accurate detection of whether the image in the image-text pair information has been tampered with is effectively achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0053] Figure 2 A flowchart of an embodiment of an image detection method according to the present application;

[0054] Figure 3 yes Figure 2 A flowchart of a specific implementation of step S20;

[0055] Figure 4 yes Figure 2 A flowchart of a specific implementation of step S30;

[0056] Figure 5 yes Figure 2 A flowchart of a specific implementation of step S50;

[0057] Figure 6 yes Figure 2 A flowchart of a specific implementation of step S70;

[0058] Figure 7 yes Figure 6 A flowchart of a specific implementation of step S704;

[0059] Figure 8 is a schematic structural diagram of an embodiment of an image detection device according to the present application;

[0060] Fig. 9 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0062] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor does it refer to non-related or alternative embodiments that are mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0063] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0064] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0065] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0066] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0067] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0068] It should be noted that the image detection method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the image detection device is generally arranged in the server / terminal device.

[0069] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0070] Continue to refer Figure 2 , shows a flow chart of an embodiment of the image detection method according to the present application. The image detection method comprises the following steps:

[0071] Step S10, obtaining image-text pair information, and preprocessing the image-text pair information to obtain standard image information and standard text information;

[0072] In this embodiment, the image-text pair information includes image information and text information, and the image-text pair information may be medical-related image-text pair information, which includes medical image information and medical text information. Among them, the medical image information may include medical images such as X-rays, CT scans, and MRI images, and the medical text information may include diagnosis reports, treatment plans, and drug instructions. The preprocessing of the image-text pair information is to preprocess the image information and text information in the image-text pair separately. The preprocessing of the image information includes size normalization and image enhancement, and the preprocessing of the text information includes data cleaning, word segmentation, and word vector representation.

[0073] Step S20, performing feature decomposition on the standard image information to obtain a first feature image and a second feature image;

[0074] In this embodiment, the first feature image is high-frequency image data, the second feature image is low-frequency image data, and the feature decomposition includes two parts: feature map extraction and image decomposition. The feature map extraction is implemented based on a convolutional neural network, and the image decomposition is based on the channel dimension to decompose the image. Convolutional neural network (CNN) is a deep machine learning model, and its structure consists of an input layer, a convolution layer, a pooling layer, and a fully connected layer. The convolution layer is the core of CNN. It extracts local features from the input standard image information through convolution operations. The features are usually represented in the form of feature maps, which capture key information such as edges and textures in the image. Channel dimension is an important concept in image processing. For example, in the RGB color space, a color image is usually composed of three channels: red (R), green (G), and blue (B). Each channel records the distribution information of the image on the color component.

[0075] Step S30, inputting the first feature image and the second feature image into a preset feature extraction model to perform feature extraction, thereby obtaining a first image feature and a second image feature;

[0076] In this embodiment, the preset feature extraction module is a model including a convolutional network path and a visual model path, wherein the convolutional network path refers to a convolutional neural network, and the visual model path refers to a Vi T model. The convolutional neural network is used to extract local features corresponding to the high-frequency first feature image, and the Vi T model is used to extract global features corresponding to the low-frequency second feature image, thereby effectively obtaining the corresponding first image features and second image features.

[0077] Step S40, combining the first image feature and the second image feature into a comprehensive image feature;

[0078] In this embodiment, the first image feature and the second image feature can be merged by feature splicing. The first image feature and the second image feature are first matched in feature dimension, and then the first image feature and the second image feature are matched to the same dimension through linear transformation. Then, the first image feature and the second image feature after dimension matching are spliced ​​to form a longer feature vector, which is the comprehensive image feature.

[0079] Step S50, extracting semantic features from the standard text information to obtain text semantic features;

[0080] In this embodiment, the semantic feature extraction of standard text information is implemented through a preset language processing model, which uses the BERT model of Transformer. The feature extraction of the BERT model includes the following steps: converting the standard text information into a format acceptable to the BERT model, including word segmentation, adding special tags, and converting the text into the input tensor required by the model; loading the pre-trained BERT model and its related weights, which are trained on a large amount of text data and contain rich language knowledge and semantic information; inputting the pre-processed standard text information into the BERT model, the model will encode the text and generate a series of vector representations, which represent the semantic information of each word or subword in the text; extracting the required semantic features from the output of the BERT model can be achieved by selecting the vector of a specific layer or a specific position of the model output. For example, the vector corresponding to the [CLS] tag of the last layer of the BERT model can be selected as the semantic representation of the entire text.

[0081] Step S60, fusing the comprehensive image features and the text semantic features to obtain effective feature representation;

[0082] In this embodiment, the fusion of comprehensive image features and text semantic features can be achieved through feature splicing, by splicing the comprehensive image features and text semantic features into a feature vector, and then inputting the feature vector into a fully connected layer for nonlinear transformation, so as to obtain an effective feature representation.

[0083] Step S70, performing image tampering judgment on the effective feature representation according to a preset tampering detection model to obtain image tampering information, and generating a tampering detection report according to the image tampering information.

[0084] In this embodiment, the preset tampering detection model can use a fully connected neural network classifier, which includes an input layer, a hidden layer, an activation function, and an output layer, wherein the input layer is used to use the feature vector after dimensionality reduction as the input of the fully connected neural network, the hidden layer contains a certain number of neurons for extracting and converting features, the activation function is used to increase the nonlinear expression ability of the model, and the output layer contains neurons equal to the number of tampering categories, each neuron corresponds to a predicted score or probability of a tampering category, and the score of the output layer is converted into a probability distribution by using a softmax function. The fully connected neural network of the above structure is used to perform image tampering judgment on the effective feature representation to obtain image tampering information. The tampering detection report is a visual report data generated based on the image tampering information. After the tampering detection report is generated, the tampering detection report can be sent according to the preset display interface path to effectively display the tampering detection report on the display interface.

[0085] In this embodiment, the above method can be applied to a medical service system, in which image tampering detection is performed on the medical field-related image-text pair information in the medical service system, thereby effectively ensuring the authenticity and validity of the medical image. Specifically, in this embodiment, the medical service system can be one or more of a medical insurance system and a disease insurance system, and the image-text pair information is medical-related image data and text data, the image data and the text data correspond, the image-text pair information is stored in the medical insurance system and the disease insurance system and is obtained from the database of the above system, and the tampering detection report is generated by the above system through the method of this embodiment and displayed in the display interface of the above system.

[0086] This embodiment obtains image-text pair information, pre-processes the image-text pair information, obtains standard image information and standard text information; performs feature decomposition on the standard image information to obtain a first feature image and a second feature image; inputs the first feature image and the second feature image into a preset feature extraction model for feature extraction to obtain a first image feature and a second image feature; merges the first image feature and the second image feature into a comprehensive image feature; performs semantic feature extraction on the standard text information to obtain text semantic features; fuses the comprehensive image features and the text semantic features to obtain an effective feature representation; performs image tampering judgment on the effective feature representation according to a preset tampering detection model to obtain image tampering information, and generates a tampering detection report according to the image tampering information. This effectively realizes accurate detection of whether the image in the image-text pair information has been tampered with.

[0087] In some optional implementations of this embodiment, the image-text pair information includes image information and text information, and the preprocessing of the image-text pair information to obtain standard image information and standard text information includes the following steps:

[0088] Performing size normalization processing on the image information to obtain normalized image information;

[0089] In this embodiment, the image information is a data set including multiple images, and the size normalization process can adjust the image to a target size by using methods such as scaling, cropping, or padding. For example, all images can be adjusted to 224x224 pixels or other sizes suitable for model input.

[0090] Performing image enhancement processing on the normalized image information to obtain the standard image information;

[0091] In this embodiment, the image enhancement can be performed on the normalized image information by using techniques such as denoising, color correction, and contrast adjustment to obtain standard image information.

[0092] Performing data cleaning and word segmentation processing on the text information to obtain word segmentation text information;

[0093] In this embodiment, the text information is an information set including multiple text data, and data cleaning includes processing methods such as removing irrelevant characters, punctuation marks, and stop words in the text, and the text data can be cleaned by using regular expressions, string operations, and other technologies. Word segmentation is used to segment the text into separate words or phrases, which can be implemented using a rule-based word segmenter (such as ICTCLAS, Jieba word segmenter, etc.) or a machine learning-based word segmenter (such as conditional random field CRF, deep learning model, etc.).

[0094] The word segmentation text information is represented by a word vector to obtain the standard text information.

[0095] In this embodiment, each word represented by the word vector is represented as a vector of fixed length to facilitate subsequent model processing. The word segmentation text information can be processed through (Bag of Words, BOW) to obtain standard text information represented by the word vector.

[0096] In this embodiment, the image information is normalized to obtain normalized image information; the normalized image information is enhanced to obtain standard image information; the text information is cleaned and segmented to obtain segmented text information; the segmented text information is represented by word vectors to obtain standard text information. Thus, standard image information and standard text information that can be effectively processed by subsequent models are effectively obtained.

[0097] refer to Figure 3 In some optional implementations of this embodiment, step S20 includes the following steps:

[0098] Step S201, inputting the standard image information into a preset convolutional neural network, and extracting a feature map of the standard image information through the convolutional neural network to obtain an image feature map;

[0099] In this embodiment, the convolutional neural network extracts features from the input image through a series of convolutional layers, activation layers (such as ReLU), pooling layers (such as maximum pooling), etc. In the convolutional layer, the convolution kernel (or filter) slides on the input image to calculate the convolution operation, thereby extracting local features in the image. After multiple layers of convolution and pooling operations, an image feature map corresponding to the standard image information is finally obtained, which contains high-level features of the image, such as edges, textures, shapes, etc.

[0100] Step S202: Decompose the image feature image along the channel dimension to obtain the first feature image and the second feature image.

[0101] In this embodiment, the image feature map has multiple channels, which correspond to features extracted by different convolution kernels. A feature map whose channels tend to reflect high-frequency information (such as edges, details, etc.) is selected as the first feature image, and a feature map whose channels tend to reflect low-frequency information (such as smooth areas, backgrounds, etc.) is selected as the second feature image and an image decomposition operation is performed, thereby effectively obtaining a high-frequency first feature image and a low-frequency second feature image.

[0102] In this embodiment, the standard image information is input into a preset convolutional neural network, and a feature map is extracted from the standard image information through the convolutional neural network to obtain an image feature map; the image feature map is decomposed along the channel dimension to obtain a high-frequency first feature image and a low-frequency second feature image, so as to facilitate subsequent effective feature extraction processing based on the first feature image and the second feature image.

[0103] refer to Figure 4 In some optional implementations of this embodiment, step S30 includes the following steps:

[0104] Step S301, obtaining a first model extraction identifier, and extracting the feature extraction model from a database according to the first model extraction identifier, wherein the feature extraction model includes a convolutional network path and a visual model path;

[0105] In this embodiment, the feature extraction model is a model of a hybrid architecture that uses a convolutional neural network (CNN) and ViT in parallel, wherein the convolutional network path corresponds to the convolutional neural network (CNN), the visual model path corresponds to the ViT model, and the first model extraction identifier is a unique information identifier corresponding to the feature extraction model. The first model extraction identifier is used to traverse and match in the database to obtain the corresponding feature extraction model.

[0106] Step S302, extracting local features of the first feature image according to the convolutional network path to obtain the first image features;

[0107] In this embodiment, the convolution network path performs a convolution operation by applying a defined convolution kernel to the first feature image. The convolution operation is implemented by a sliding window. The convolution kernel is slid on the feature map, and the dot product between the convolution kernel and the local area of ​​the feature map is calculated at each position, thereby obtaining a new feature map. The new feature map contains the local features extracted by the convolution kernel. The result of the convolution operation is mapped to a nonlinear space through an activation function, so that more complex features can be captured. After the convolution operation and the activation function, the new feature map obtained is the first image feature.

[0108] Step S303: performing global feature extraction on the second feature image according to the visual model path to obtain the second image feature.

[0109] In this embodiment, the step of extracting global features from the second feature image through the visual model path (ViT) includes: dividing the second feature image into a series of image blocks of fixed size; flattening each image block into a one-dimensional vector, and mapping it to the embedding space of the model through a linear transformation (or called an embedding layer) to obtain an embedding representation of each block; adding a fixed position embedding vector to the embedding representation of each image block; inputting the embedding representation with position encoding into a Transformer Encoder, wherein the Transformer Encoder is composed of a plurality of stacked encoding layers, each encoding layer includes a self-attention mechanism and a feedforward neural network, and extracting global features by iteratively updating the embedding representation of each block, thereby effectively obtaining a global feature representation vector corresponding to the second feature image, and the global feature representation vector is the second image feature.

[0110] This embodiment obtains a first model extraction identifier, extracts the feature extraction model from the database according to the first model extraction identifier, wherein the feature extraction model includes a convolutional network path and a visual model path; performs local feature extraction on the first feature image according to the convolutional network path to obtain the first image feature; performs global feature extraction on the second feature image according to the visual model path to obtain the second image feature. This effectively implements local feature extraction and global feature extraction on the first feature image and the second feature image, respectively, to facilitate subsequent feature merging processing of the first image feature and the second image feature.

[0111] Continue to refer Figure 5 In some optional implementations of this embodiment, step S50 includes the following steps:

[0112] Step S501, obtaining a second model extraction identifier, and extracting a preset language processing model from a database according to the second model extraction identifier;

[0113] In this embodiment, the preset language processing model adopts the BERT model of Transformer, and the second model extraction identifier is a unique information identifier of the corresponding feature extraction model. The second model extraction identifier is used to perform traversal matching in the database to obtain the corresponding language processing model.

[0114] Step S502, performing word embedding processing on the standard text information based on the word embedding layer of the language processing model to obtain a word vector sequence;

[0115] In this embodiment, the word embedding layer of the language processing model contains three embeddings: tag embedding, position embedding, and segment embedding. Tag embedding is used to map each tag to a fixed-size vector space, position embedding is used to provide position information for the tag, because the Transformer architecture itself does not understand the order of the sequence, and segment embedding is used to distinguish different paragraphs or sentences in the input. After being processed by the word embedding layer, the tokens corresponding to each standard text information are converted into a high-dimensional vector, and this part of the high-dimensional vector constitutes the word vector sequence.

[0116] Step S503: extract features from the word vector sequence based on the multi-layer processing layers of the language processing model to obtain the text semantic features.

[0117] In this embodiment, the language processing model also includes a multi-layer Transformer Encoder, which is composed of multiple encoding layers stacked together. The multi-layer Transformer Encoder is a multi-layer processing layer. After the word vector sequence is processed by the multi-layer Transformer Encoder, the tag representation of each word vector sequence contains rich contextual information and semantic features. The tag representation of the word vector sequence is the text semantic feature.

[0118] This embodiment obtains a second model extraction identifier, extracts a preset language processing model from a database according to the second model extraction identifier; performs word embedding processing on the standard text information based on the word embedding layer of the language processing model to obtain a word vector sequence; and performs feature extraction on the word vector sequence based on the multi-layer processing layer of the language processing model, thereby effectively obtaining text semantic features that accurately represent the semantics of the standard text information, so as to facilitate subsequent feature fusion processing.

[0119] Continue to refer Figure 6 In some optional implementations of this embodiment, step S70 includes the following steps:

[0120] Step S701, obtaining a third model extraction identifier, and extracting the tampering detection model from the database according to the third model extraction identifier;

[0121] In this embodiment, the tampering detection model is a pre-trained fully connected neural network classifier, and the third model extraction identifier is a unique information identifier of the corresponding feature extraction model. The third model extraction identifier is used to perform traversal matching in the database to obtain the corresponding tampering detection model.

[0122] Step S702, inputting the effective feature representation into the tampering detection model to obtain the image tampering probability;

[0123] In this embodiment, the effective feature representation is input into a pre-trained fully connected neural network classifier to obtain the image tampering probability corresponding to the effective feature representation. The output process of the tampering detection model includes: inputting the effective feature representation into the pre-trained fully connected neural network classifier; forward propagating the effective feature representation through each layer of the classifier, each layer calculates the weighted sum of the input features and transforms it through a nonlinear activation function; in the last layer of the classifier (usually a softmax layer or a sigmoid layer), outputting a probability value, indicating the possibility that the image is tampered or not tampered with, and the probability value is the image tampering probability.

[0124] Step S703, determining whether the image tampering probability is greater than or equal to a preset probability threshold;

[0125] In this embodiment, the preset probability threshold is a preset threshold used to determine whether an image is a tampered image. The preset probability threshold is initially set to 80% and can be set and adjusted accordingly according to actual conditions.

[0126] Step S704: if the image tampering probability is greater than or equal to the preset probability threshold, the image in the image-text pair information is determined to be a tampered image, and the tampering type and tampering area of ​​the tampered image are identified, the tampering type, the tampering area, and the image tampering probability are integrated into the image tampering information, a corresponding tampering detection report template is obtained according to the tampering type, and the tampering detection report template is generated based on the tampering detection report template;

[0127] In this embodiment, the tampering type of the tampered image can be identified by a convolutional neural network, and the tampering type includes splicing, copy and paste, retouching, object removal, etc. The identification of the tampered area can be achieved by pixel-level or region-level segmentation technology, and the segmentation technology can adopt semantic segmentation, edge detection, or region-based methods. Matching query can be performed in the template database by tampering type, so as to effectively extract the corresponding tampering detection report template.

[0128] Step S705: If the image tampering probability is less than the preset probability threshold, the image in the image-text pair information is determined as a non-tampered image.

[0129] In this embodiment, a corresponding “non-tampered” mark is added to the image in the image-text pair information, so that the image in the image-text pair information is determined to be a non-tampered image.

[0130] In this embodiment, by obtaining a third model extraction identifier, the tampering detection model is extracted from the database according to the third model extraction identifier; the effective feature representation is input into the tampering detection model to obtain the image tampering probability; it is determined whether the image tampering probability is greater than or equal to a preset probability threshold; if the image tampering probability is greater than or equal to the preset probability threshold, the image in the image-text pair information is determined as a tampered image, and the tampering type and tampering area of ​​the tampered image are identified, the tampering type, the tampering area, and the image tampering probability are integrated into the image tampering information, the corresponding tampering detection report template is obtained according to the tampering type, and the tampering detection report template is generated based on the tampering detection report template; if the image tampering probability is less than the preset probability threshold, the image in the image-text pair information is determined as a non-tampered image. In this way, accurate tampering detection of the image in the image-text pair information is effectively achieved to obtain the corresponding image tampering information.

[0131] Continue to refer Figure 7 In some optional implementations of this embodiment, step S704 includes the following steps:

[0132] S7041, extracting the tampering detection report template from a template database using the tampering type as a query condition;

[0133] In this embodiment, the query condition can be a specific tampering type name (such as "splicing", "copy and paste", etc.) or an identifier related thereto. The query condition is used to perform query matching in the template database to effectively extract the corresponding tampering detection report template.

[0134] S7042, matching the image tampering information with the template parameters of the tampering detection report template to obtain an information matching result;

[0135] In this embodiment, each report template includes a series of template parameters, which are used to insert specific tampering information into the report. The template parameters can be text placeholders, image placeholders, or other forms of dynamic content. The image tampering information and the template parameters are matched to determine the parameter object that should be filled in each information item. The determined parameter object is the information matching result.

[0136] S7043: Fill the image tampering information into the corresponding parameters of the tampering detection report template according to the information matching result to obtain the tampering detection report.

[0137] In this embodiment, the image tampering information is filled into the corresponding parameters of the template one by one according to the information matching result. The filling process can be achieved by text replacement, image insertion or other types of dynamic content generation.

[0138] This embodiment extracts the tampering detection report template from the template database by taking the tampering type as a query condition; matches the image tampering information with the template parameters of the tampering detection report template to obtain an information matching result; fills the image tampering information into the corresponding parameters of the tampering detection report template according to the information matching result, thereby obtaining a tampering detection report that effectively displays the image tampering information.

[0139] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0140] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0141] Further references Figure 8 , as a response to the above Figure 1 The present application provides an embodiment of an image detection device, and the device embodiment is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0142] like Figure 8As shown, the image detection device 800 described in this embodiment includes: an information processing module 801, a feature decomposition module 802, a first feature extraction module 803, a feature merging module 804, a second feature extraction module 805, a feature fusion module 806, and a result generation module 807.

[0143] in:

[0144] The information processing module 801 is used to obtain image-text pair information, and pre-process the image-text pair information to obtain standard image information and standard text information;

[0145] A feature decomposition module 802 is used to perform feature decomposition on the standard image information to obtain a first feature image and a second feature image;

[0146] A first feature extraction module 803 is used to input the first feature image and the second feature image into a preset feature extraction model to perform feature extraction to obtain a first image feature and a second image feature;

[0147] A feature merging module 804, configured to merge the first image feature and the second image feature into a comprehensive image feature;

[0148] The second feature extraction module 805 is used to extract semantic features from the standard text information to obtain text semantic features;

[0149] A feature fusion module 806 is used to fuse the comprehensive image features and the text semantic features to obtain effective feature representation;

[0150] The result generation module 807 is used to perform image tampering judgment on the effective feature representation according to a preset tampering detection model, obtain image tampering information, and generate a tampering detection report according to the image tampering information.

[0151] By adopting the above-mentioned image detection device, this embodiment can obtain image-text pair information, pre-process the image-text pair information to obtain standard image information and standard text information; perform feature decomposition on the standard image information to obtain a first feature image and a second feature image; input the first feature image and the second feature image into a preset feature extraction model for feature extraction to obtain a first image feature and a second image feature; merge the first image feature and the second image feature into a comprehensive image feature; perform semantic feature extraction on the standard text information to obtain text semantic features; fuse the comprehensive image features and the text semantic features to obtain an effective feature representation; perform image tampering judgment on the effective feature representation according to a preset tampering detection model to obtain image tampering information, and generate a tampering detection report according to the image tampering information. This effectively realizes accurate detection of whether the image in the image-text pair information has been tampered with.

[0152] To solve the above technical problems, the present application also provides a computer device. Fig. 9 , Fig. 9 This is a basic structural block diagram of the computer device in this embodiment.

[0153] The computer device 9 includes a memory 91, a processor 92, and a network interface 93 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 9 with components 91-93, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0154] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0155] The memory 91 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 91 can be an internal storage unit of the computer device 9, such as a hard disk or memory of the computer device 9. In other embodiments, the memory 91 can also be an external storage device of the computer device 9, such as a plug-in hard disk equipped on the computer device 9, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 91 can also include both the internal storage unit of the computer device 9 and its external storage device. In this embodiment, the memory 91 is generally used to store the operating system and various application software installed on the computer device 9, such as computer-readable instructions of the image detection method, etc. In addition, the memory 91 can also be used to temporarily store various types of data that have been output or are to be output.

[0156] The processor 92 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 92 is generally used to control the overall operation of the computer device 9. In this embodiment, the processor 92 is used to run computer-readable instructions or process data stored in the memory 91, such as computer-readable instructions for running the image detection method.

[0157] The network interface 93 may include a wireless network interface or a wired network interface. The network interface 93 is generally used to establish a communication connection between the computer device 9 and other electronic devices.

[0158] By adopting the above-mentioned computer device, this embodiment can obtain image-text pair information, pre-process the image-text pair information to obtain standard image information and standard text information; perform feature decomposition on the standard image information to obtain a first feature image and a second feature image; input the first feature image and the second feature image into a preset feature extraction model for feature extraction to obtain a first image feature and a second image feature; merge the first image feature and the second image feature into a comprehensive image feature; perform semantic feature extraction on the standard text information to obtain text semantic features; fuse the comprehensive image features and the text semantic features to obtain effective feature representation; perform image tampering judgment on the effective feature representation according to a preset tampering detection model to obtain image tampering information, and generate a tampering detection report according to the image tampering information. This effectively realizes accurate detection of whether the image in the image-text pair information has been tampered with.

[0159] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the image detection method as described above.

[0160] By adopting the above-mentioned computer-readable storage medium, this embodiment can obtain image-text pair information, pre-process the image-text pair information to obtain standard image information and standard text information; perform feature decomposition on the standard image information to obtain a first feature image and a second feature image; input the first feature image and the second feature image into a preset feature extraction model for feature extraction to obtain a first image feature and a second image feature; merge the first image feature and the second image feature into a comprehensive image feature; perform semantic feature extraction on the standard text information to obtain text semantic features; fuse the comprehensive image features and the text semantic features to obtain effective feature representation; perform image tampering judgment on the effective feature representation according to a preset tampering detection model to obtain image tampering information, and generate a tampering detection report according to the image tampering information. This effectively realizes accurate detection of whether the image in the image-text pair information has been tampered with.

[0161] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0162] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

[0163] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

Claims

1. An image detection method, characterized in that: The steps include: Acquire image-text pair information, and pre-process the image-text pair information to obtain standard image information and standard text information; Performing feature decomposition on the standard image information to obtain a first feature image and a second feature image; Inputting the first feature image and the second feature image into a preset feature extraction model to perform feature extraction to obtain a first image feature and a second image feature; combining the first image feature and the second image feature into a comprehensive image feature; Extracting semantic features from the standard text information to obtain text semantic features; Fusing the comprehensive image features with the text semantic features to obtain effective feature representation; The image tampering judgment is performed on the effective feature representation according to a preset tampering detection model to obtain image tampering information, and a tampering detection report is generated according to the image tampering information.

2. The image detection method according to claim 1, characterized in that: The image-text information includes image information and text information. The step of preprocessing the image-text information to obtain standard image information and standard text information specifically includes: Performing size normalization processing on the image information to obtain normalized image information; Performing image enhancement processing on the normalized image information to obtain the standard image information; Performing data cleaning and word segmentation processing on the text information to obtain word segmentation text information; The word segmentation text information is represented by a word vector to obtain the standard text information.

3. The image detection method according to claim 1, characterized in that: The step of performing feature decomposition on the standard image information to obtain a first feature image and a second feature image specifically includes: Inputting the standard image information into a preset convolutional neural network, performing feature map extraction on the standard image information through the convolutional neural network, and obtaining an image feature map; The image feature image is decomposed along the channel dimension to obtain the first feature image and the second feature image.

4. The image detection method according to claim 1, characterized in that: The step of inputting the first feature image and the second feature image into a preset feature extraction model to extract features to obtain first image features and second image features specifically includes: Obtaining a first model extraction identifier, and extracting the feature extraction model from a database according to the first model extraction identifier, wherein the feature extraction model includes a convolutional network path and a visual model path; Performing local feature extraction on the first feature image according to the convolutional network path to obtain the first image feature; Perform global feature extraction on the second feature image according to the visual model path to obtain the second image feature.

5. The image detection method according to claim 1, characterized in that: The step of extracting semantic features from the standard text information to obtain text semantic features specifically includes: Obtaining a second model extraction identifier, and extracting a preset language processing model from a database according to the second model extraction identifier; Performing word embedding processing on the standard text information based on the word embedding layer of the language processing model to obtain a word vector sequence; Based on the multi-layer processing layers of the language processing model, feature extraction is performed on the word vector sequence to obtain the text semantic features.

6. The image detection method according to claim 1, characterized in that: The step of performing image tampering judgment on the effective feature representation according to the preset tampering detection model to obtain image tampering information, and generating a tampering detection report according to the image tampering information specifically includes: Acquire a third model extraction identifier, and extract the tampering detection model from a database according to the third model extraction identifier; Inputting the effective feature representation into the tampering detection model to obtain the image tampering probability; Determining whether the image tampering probability is greater than or equal to a preset probability threshold; If the probability of image tampering is greater than or equal to the preset probability threshold, the image in the image-text pair information is determined to be a tampered image, and the tampering type and tampering area of ​​the tampered image are identified, the tampering type, the tampering area, and the image tampering probability are integrated into the image tampering information, the corresponding tampering detection report template is obtained according to the tampering type, and the tampering detection report template is generated based on the tampering detection report template.

7. The image detection method according to claim 6, characterized in that: The step of acquiring the corresponding tampering detection report template according to the tampering type and generating the tampering detection report based on the tampering detection report template specifically includes: Extracting the tampering detection report template from a template database using the tampering type as a query condition; Matching the image tampering information with the template parameters of the tampering detection report template to obtain an information matching result; The image tampering information is filled into the corresponding parameters of the tampering detection report template according to the information matching result to obtain the tampering detection report.

8. An image detection device, characterized in that: include: An information processing module, used for acquiring image-text pair information, and preprocessing the image-text pair information to obtain standard image information and standard text information; A feature decomposition module, used for performing feature decomposition on the standard image information to obtain a first feature image and a second feature image; A first feature extraction module, used for inputting the first feature image and the second feature image into a preset feature extraction model for feature extraction to obtain a first image feature and a second image feature; A feature merging module, used for merging the first image feature and the second image feature into a comprehensive image feature; A second feature extraction module is used to extract semantic features from the standard text information to obtain text semantic features; A feature fusion module, used to fuse the comprehensive image features and the text semantic features to obtain an effective feature representation; The result generation module is used to perform image tampering judgment on the effective feature representation according to a preset tampering detection model, obtain image tampering information, and generate a tampering detection report according to the image tampering information.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the image detection method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the image detection method according to any one of claims 1 to 7 are implemented.

Images