Image tampering detection method and related apparatus

Abstract

An image tampering detection method, applicable to the technical field of artificial intelligence (AI). In the image tampering detection method, when a feature of an image in which a tampered region needs to be detected is extracted, the image feature is processed by means of a gating network, so as to obtain a confidence level of each expert network among multiple expert networks processing the current image feature, and then which expert networks among the multiple expert networks are used for targeted processing of the image feature is determined. Thus, the image processing strategy is dynamically adjusted on the basis of the tampering characteristics of images themselves, avoiding the use of a network having a fixed structure to uniformly process images of various tampering types, improving the effect of image tampering detection.

Description

A method and related apparatus for detecting image tampering

[0001] This application claims priority to Chinese Patent Application No. 202411013308.X, filed on July 25, 2024, entitled “A Method and Apparatus for Detecting Image Tampering”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI) technology, and in particular to an image tampering detection method and related apparatus. Background Technology

[0003] With the rapid development of computer technology, images, as a key medium for information transmission, play an important role in fields such as law, medicine, electronic finance, and academic research. However, due to the rapid development of image editing technology, it has become increasingly easy to tamper with images, and the tampered images are becoming more and more realistic, leading to a significant increase in the risk of image tampering.

[0004] To detect image tampering, related technologies often employ deep learning models to identify whether an image has been altered. Specifically, these technologies use a large number of tampered images as training data to train a deep learning model, enabling the model to learn how to identify tampered images.

[0005] However, in practical applications, in complex scenarios with many types of image tampering, the detection performance of current deep learning models is often poor, making it difficult to effectively identify the tampered areas in the image. Summary of the Invention

[0006] This application provides an image tampering detection method that can improve the effectiveness of image tampering detection.

[0007] Firstly, an image tampering detection method is provided, applied to detect tampered regions in an image. Specifically, this method includes: firstly, after acquiring a first image from which the tampered region to be detected is obtained, features of the first image are extracted using a feature extraction network to obtain image features. The feature extraction network can be, for example, a residual network, a convolutional neural network, or an attention network.

[0008] Then, image features are processed through a gating network to obtain the confidence score of each of the multiple expert networks. The gating network connects multiple independent expert networks, and the confidence scores of these expert networks indicate the reliability of each output obtained by the multiple expert networks in processing the image features. In other words, the gating network works in conjunction with multiple expert networks to determine which expert networks should be used to process each image feature.

[0009] Secondly, based on the confidence scores of multiple expert networks, one or more expert networks are selected to process image features, resulting in the outputs of one or more expert networks. These outputs are used to determine the tampered regions in the first image. The number of selected expert networks is determined based on the confidence scores of the multiple expert networks. In other words, the number of expert networks selected each time to process image features is not fixed, but rather needs to be determined based on the confidence scores of each expert network output by the gating network.

[0010] In this scheme, after extracting the features of the image of the tampered area to be detected, a gating network is used to process the image features to obtain the confidence level of each expert network in processing the current image features. This determines which expert networks to use to specifically process the image features. The type of expert network to process the image is dynamically selected according to the type of image tampering. By mapping the expert networks to the type of image tampering, the image processing strategy can be dynamically adjusted based on the tampering characteristics of the image itself. This avoids using a fixed network structure to uniformly process images of various tampering types, improves the effect of image tampering detection, and enhances the accuracy of identifying tampered areas in the image.

[0011] Furthermore, the number and type of expert networks used to process image features each time are determined based on the confidence level obtained by the gating network in processing image features. This ensures that the number and type of expert networks are dynamically selected based on the tampering characteristics of the image itself. As a result, in practical applications, the expert networks used to process image features can be flexibly adjusted according to dynamically changing images, thereby improving the efficiency and accuracy of image tampering detection.

[0012] In one possible implementation, selecting one or more expert networks from a plurality of expert networks to process image features can include the following two scenarios.

[0013] If any of the multiple expert networks has a confidence level greater than a preset threshold, then the expert network with a confidence level greater than the preset threshold is selected to process image features. That is, if an expert network's confidence level is greater than the preset threshold, then that expert network is selected to process image features, while expert networks with confidence levels not greater than the preset threshold are not selected.

[0014] If the confidence scores of multiple expert networks are all below a preset threshold, one or more expert networks with the highest confidence scores are selected from among the multiple expert networks to process image features.

[0015] In this scheme, the expert network to be used is determined by comparing the confidence level of the expert network with the preset threshold. This effectively selects the most suitable expert network to process image features based on the tampering characteristics of the image itself, and flexibly adjusts the number of expert networks to process image features, so that the cooperation between expert networks is better and the efficiency and accuracy of image tampering detection are improved.

[0016] In one possible implementation, a feature extraction network, a gating network, and multiple expert networks are used to construct the tampering detection model, and the parameters in the tampering detection model are updated during training based on a first loss function and a second loss function. That is, the first loss function and the second loss function jointly constrain the parameter update process of the tampering detection model, so that the parameter update objective of the tampering detection model is to minimize the values ​​of the first loss function and the second loss function as much as possible. By optimizing the parameters through the loss functions, the recognition accuracy of the tampering detection model is improved.

[0017] The first loss function is obtained based on the actual tampered regions of the training data and the predicted tampered regions output by the tamper detection model. The second loss function is obtained based on the actual tampered types of the training data and the predicted tampered types output by the classification network. The classification network is used to predict the tampered types of the training data based on the features output by the feature extraction network.

[0018] Since the input of the classification network is the output of the feature extraction network, when training the tamper detection model based on the second loss function, in order to ensure that the classification network can accurately predict the tamper type, the feature extraction network often learns how to perceive the tamper type of the image during the parameter update process. This makes the image features output by the feature extraction network easier for the gating network to identify and select an appropriate expert network based on significant tamper characteristics.

[0019] In this scheme, a classification network is introduced during the training phase of the tamper detection model to construct an additional second loss function. This implicitly constrains the tamper detection model's perception of tamper types during the training phase, making the learning process of the expert networks more professional. Furthermore, each expert network can learn to process image features under specific tamper types more specifically, thereby improving the overall accuracy of the tamper detection model.

[0020] In one possible implementation, the first image includes text content, and the tampered area to be detected in the first image is the area where the text content has been tampered with.

[0021] In one possible implementation, to obtain training data for the model, a large language model can be used to identify the semantic category corresponding to the first text in the second image. Then, based on the semantic category of the first text, the first text in the second image is modified to obtain the modified second image. The text in the modified second image corresponds to the same semantic category before and after modification, and the modified second image is used as training data for the tampering detection model. In other words, after modifying the first text in the second image, the modified text and the first text correspond to the same semantic category. The tampering detection model includes a feature extraction network, a gating network, and multiple expert networks.

[0022] In this scheme, by leveraging the text understanding capabilities of a large language model, the semantic category of the text in an image can be identified. This allows for the generation of altered text within the same semantic category when the original text in an image is modified, thereby ensuring the legitimacy of the altered image and improving the quality of the altered image used as training data.

[0023] In one possible implementation, the first text in the second image is modified based on semantic categories. Specifically, this involves generating second text using a large language model, where the second text corresponds to the same semantic category as the first text. Then, the first text is replaced with the second text to obtain the modified second image.

[0024] In this scheme, by utilizing the text generation capabilities of a large language model, it is possible to automatically generate tampered data with high semantic continuity, thereby improving the quality of the generated tampered data and providing diverse and high-quality training data for the training phase of the tampering detection model.

[0025] In one possible implementation, during the process of replacing the first text with the second text, to ensure the similarity between the second text and the first text as much as possible and to avoid the second text appearing too abruptly in the second image, the display information of the first text in the second image can be obtained first. This display information includes at least one of the following: the position, size, font format, and background color of the first text. Based on this display information, the first text can be replaced with the second text to obtain the altered second image. In the altered second image, the display information of the second text is the same as or as close as possible to the display information of the first text.

[0026] In one possible implementation, the first text in the second image is modified based on semantic categories. Specifically, this may include: determining a third text in either the second or third image based on the semantic category, where the third text corresponds to the same semantic category as the first text; and using the third image as training data for the modification detection model. Then, the first text is replaced with the third text to obtain the modified second image.

[0027] In other words, based on the semantic category corresponding to the first text, a third text with the same semantic category as the first text can be determined on the same image or another image, and then the first text can be replaced with the determined third text, thereby realizing the copy and paste of the third text.

[0028] In this scheme, by leveraging the text understanding capabilities of a large language model, similar replacement texts can be automatically found in the same image or other images for the text that needs to be tampered with, thereby automatically generating high-quality tampering data and providing diverse and high-quality training data for the training phase of the tampering detection model.

[0029] In one possible implementation, to obtain training data for the model, a large language model can be used to identify a fourth text in the second image corresponding to a preset semantic category. This preset semantic category is related to the scene to which the second image belongs. For example, if the second image is an image within a personal identity scene, the preset semantic category could be a name, age, phone number, address, or similar semantic category.

[0030] After the fourth text is identified, it can be erased from the second image to obtain a tampered second image. The tampered second image is used as training data for the tamper detection model, which includes the feature extraction network, the gating network, and the multiple expert networks.

[0031] In this solution, by using a large language model to identify text of specific semantic categories in an image and erasing the identified text, an image of the erasure type can be obtained. The erased text in the image belongs to a specific semantic category related to the scene, ensuring that meaningful text in the image can be erased and improving the quality of the tampered image.

[0032] In one possible implementation, after obtaining the outputs of one or more expert networks, the outputs are weighted and fused based on the confidence levels of the expert networks to obtain fused features. Then, a segmentation network processes the fused features to obtain the tampered region in the first image. The weighted fusion can be performed by using the confidence levels of the expert networks as weights to perform a weighted summation or a weighted average of the expert network outputs.

[0033] In one possible implementation, to facilitate the structural design of the expert network and ensure the coordination among expert networks in processing image features, multiple expert networks connected to the gating network have the same structure and different parameters.

[0034] Secondly, an image tampering detection device is provided, comprising: an extraction module for extracting features of a first image through a feature extraction network to obtain image features, wherein the first image is an image of a tampered region to be detected; a processing module for processing the image features through a gating network to obtain the confidence scores of multiple expert networks, wherein the confidence scores of the multiple expert networks are used to indicate the reliability of each output result obtained by the multiple expert networks in processing the image features; the processing module is further configured to select one or more expert networks from the multiple expert networks to process the image features based on the confidence scores of the multiple expert networks, thereby obtaining the output results of one or more expert networks, wherein the output results are used to determine the tampered region in the first image; wherein the number of the selected one or more expert networks is determined based on the confidence scores of the multiple expert networks.

[0035] In one possible implementation, the processing module is further configured to: select an expert network with a confidence level greater than the preset threshold from among the multiple expert networks to process the image features if the confidence level of any of the multiple expert networks is greater than the preset threshold; and select one or more expert networks with the highest confidence level from among the multiple expert networks to process the image features if the confidence levels of none of the multiple expert networks are greater than the preset threshold.

[0036] In one possible implementation, a feature extraction network, a gating network, and multiple expert networks are used to construct a tampering detection model. The parameters in the tampering detection model are updated during training based on a first loss function and a second loss function. The first loss function is obtained based on the actual tampered regions of the training data and the predicted tampered regions output by the tampering detection model. The second loss function is obtained based on the actual tampering types of the training data and the predicted tampering types output by the classification network. The classification network is used to predict the tampering type of the training data based on the features output by the feature extraction network.

[0037] In one possible implementation, the tampered area to be detected in the first image is the area where the text content has been tampered with.

[0038] In one possible implementation, the processing module is further configured to: identify the semantic category corresponding to the first text in the second image through a large language model; modify the first text in the second image based on the semantic category to obtain the modified second image, wherein the text in the modified second image corresponds to the same semantic category before and after modification, and the modified second image is used as training data for a tampering detection model, which includes a feature extraction network, a gating network, and multiple expert networks.

[0039] In one possible implementation, the processing module is further configured to: generate a second text based on a large language model, the second text corresponding to the same semantic category as the first text; and replace the first text with the second text to obtain the altered second image.

[0040] In one possible implementation, the processing module is further configured to: obtain display information of the first text in the second image, the display information including at least one of the position, size, font format and background color of the first text; and based on the display information, replace the first text with the second text to obtain the tampered second image.

[0041] In one possible implementation, the processing module is further configured to: determine a third text in a second or third image based on a semantic category, wherein the third text corresponds to the same semantic category as the first text, and the third image is used as training data for the tampering detection model; and replace the first text with the third text to obtain the tampered second image.

[0042] In one possible implementation, the processing module is further configured to: perform weighted fusion of the outputs of one or more expert networks based on the confidence of one or more expert networks to obtain fused features; and process the fused features through a segmentation network to obtain the tampered region in the first image.

[0043] In one possible implementation, multiple expert networks have the same structure, but with different parameters.

[0044] Thirdly, an image tampering detection device is provided, comprising: a processor and a memory; the memory is used to store computer instructions, which, when executed by the processor, cause the image tampering detection device to perform any of the methods described above.

[0045] Fourthly, a computer-readable storage medium is provided that stores instructions which, when executed on a computer, cause the computer to perform the methods of any of the above aspects.

[0046] Fifthly, a computer program product containing instructions is provided, which, when executed on a computer, enable the computer to perform the methods described above.

[0047] In a sixth aspect, a chip system is provided, the chip system including a processor and a communication interface for communicating with a module other than the chip shown, the processor for running computer programs or instructions such that an apparatus on which the chip system is mounted can perform the methods of any of the above aspects.

[0048] In a seventh aspect, a computing device is provided, which includes an image tampering detection device of the third aspect or a chip of the sixth aspect, wherein the image tampering detection device or chip system in the computing device is used to implement the operational steps of the method of any of the above aspects.

[0049] Eighthly, a computing device cluster is provided, comprising at least one computing device, wherein any one computing device is used to run a computer program or instructions, such that the computing device cluster can perform the methods of any of the above aspects. Alternatively, some or all of the computing devices are used together to run a computer program or instructions, such that the computing device cluster can perform the methods of any of the above aspects.

[0050] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0051] Figure 1 is a schematic diagram of a system architecture 100 provided in this application;

[0052] Figure 2 is a flowchart illustrating an image tampering detection method provided in this application;

[0053] Figure 3 is a schematic diagram of a process for detecting tampered areas in an image according to this application;

[0054] Figure 4 is a schematic diagram of an expert network selection based on a preset threshold provided in this application;

[0055] Figure 5 is a training diagram of a tamper detection model provided in this application;

[0056] Figure 6 is a schematic diagram of generating a tampered image according to this application;

[0057] Figure 7 is a schematic diagram of an image tampering method for generating tampering types provided in this application;

[0058] Figure 8 is a schematic diagram of an image tampering method of the same image copy and paste type provided in this application;

[0059] Figure 9 is a schematic diagram of constructing training data using a large language model according to this application;

[0060] Figure 10 is a schematic diagram of the training framework of a tamper detection model provided in this application;

[0061] Figure 11 is a schematic diagram of the structure of an image tampering detection device provided in this application;

[0062] Figure 12 is a schematic diagram of the structure of a computing device provided in this application;

[0063] Figure 13 is a schematic diagram of the structure of a computing device cluster provided in this application;

[0064] Figure 14 is a schematic diagram of another computing device cluster provided in this application;

[0065] Figure 15 is a schematic diagram of the structure of a chip provided in this application;

[0066] Figure 16 is a schematic diagram of the structure of a computer-readable storage medium provided in this application. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some, and not all, of the embodiments of this application. Those skilled in the art will understand that, with the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0068] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such descriptions can be used interchangeably where appropriate to allow embodiments to be implemented in a sequence other than that illustrated or described in this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps appearing in this application does not imply that the steps in the method flow must be performed in the chronological / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect is achieved. The division of units in this application is a logical division. In practical applications, there may be other division methods. For example, multiple units may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the shown or discussed mutual coupling, direct coupling, or communication connection may be through some interface, and the indirect coupling or communication connection between units may be electrical or other similar forms, none of which are limited in this application. Furthermore, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed among multiple circuit units. Some or all of the units can be selected to achieve the purpose of the solution in this application according to actual needs.

[0069] For ease of understanding, some technical terms involved in the embodiments of this application will be introduced below.

[0070] (1) Mixture of Experts (MoE)

[0071] A hybrid expert model is a deep learning model consisting of a gating network and multiple expert networks. The gating network determines which expert network to use to perform the task by identifying the type of the input task. The multiple expert networks are a set of networks specifically designed to handle specific tasks. Each expert network exhibits high performance in handling its corresponding type of task, and the types of tasks handled by different expert networks often differ. In other words, multiple expert networks are used to specifically address different types of tasks.

[0072] (2) Large Language Model

[0073] Large language models are deep learning models trained on massive amounts of text data that can generate natural language text or understand the meaning of language text. Large language models can handle various natural language tasks, such as text classification, question answering, and dialogue, and are an important pathway to artificial intelligence.

[0074] Specifically, large language models are a technology that has emerged in recent years. Because large language models undergo meticulous data engineering and training processes, their parameters have learned a wealth of existing natural language processing knowledge. This knowledge can now replace humans in many language-related tasks, such as having large language models write code or perform text summarization.

[0075] (3) Prompt

[0076] Prompts originated as an input format designed by researchers for downstream tasks. Their purpose is to help pre-trained models "recall" what they "learned" during pre-training, hence the name "cue word." For large language models, a prompt is the user's input, instructing the model on the task to be performed. A prompt can be a simple question, a longer text, or a set of instructions, depending on the user's specific needs. Generally, a prompt is a short text string that provides context and task-related information to help the model better understand the requirements and generate the correct output. For example, in question-answering tasks, a prompt might contain a description of the question or topic to help the large language model generate the correct answer. Furthermore, prompts are often designed by humans to help large language models better understand specific tasks or domains.

[0077] In this way, when a large language model generates content, it first processes the prompt and then outputs content based on its understanding of the prompt. The working principle of the large language model is to predict the probability of the next word appearing based on the preceding context of the user input, thereby generating the following text word by word. Therefore, differences in the user's input prompt directly affect the quality of the large language model's output. In some cases, even a difference of just a few words in the user's input prompt can result in significantly different content generated by the large language model.

[0078] (4) Transformer network

[0079] The Transformer network is a neural network model based on the self-attention mechanism, used to process sequential data. Compared to traditional recurrent neural network models, the Transformer network has better parallel performance and shorter training time, and therefore has been widely used in the field of natural language processing.

[0080] The Transformer network comprises multiple encoder and decoder layers, with each layer consisting of multiple attention mechanism modules and feedforward neural network modules. The encoder encodes the input sequence into a high-dimensional feature vector representation, while the decoder decodes this vector representation into the target sequence. The Transformer network also utilizes techniques such as residual connections and layer normalization to accelerate model convergence and improve model performance.

[0081] Specifically, the core of the Transformer network is the self-attention mechanism, which assigns a weight to each position in the input sequence and then outputs these weighted position vectors. Generally, the computation process of the self-attention mechanism includes three steps: 1. Calculate the attention weights: Calculate the attention weights between each position and other positions, i.e., the importance of each position to the other positions; 2. Calculate the weighted sum: Multiply each position vector by its attention weight and then sum them to obtain a weighted sum vector; 3. Perform a linear transformation: Apply a linear transformation to the weighted sum vector to obtain the final output vector.

[0082] (5) Multilayer Perceptron (MLP)

[0083] MLP is a deep learning model based on feedforward neural networks, consisting of multiple neuron layers, with each neuron layer fully connected to the previous one. MLPs are often used to solve various machine learning problems such as classification, regression, and clustering.

[0084] The applicant's research revealed that current deep learning models used for image tampering detection often have fixed structures, and are trained using a mixture of training data with different tampering types. However, this approach ignores the interactions between training data of different tampering types. Specifically, training a fixed-structure deep learning model with mixed training data may improve or degrade its performance, resulting in unstable detection performance. In real-world scenarios with complex tampering types, the performance degradation of these deep learning models becomes even more pronounced, making it difficult for them to effectively identify tampered regions in images in most cases.

[0085] In view of this, this application provides an image tampering detection method. After extracting the features of the image of the region to be tampered with, a gating network is used to process the image features to obtain the confidence level of each expert network in processing the current image features. This determines which expert networks to use to specifically process the image features, enabling dynamic adjustment of the image processing strategy based on the tampering characteristics of the image itself. This avoids using a fixed network structure to uniformly process images of various tampering types, improving the accuracy and efficiency of image tampering detection. Furthermore, the number and type of expert networks used to process image features each time are determined based on the confidence level obtained from the gating network's image feature processing, ensuring that the number and type of expert networks are dynamically selected based on the tampering characteristics of the image itself. This allows for flexible adjustment of the expert networks processing image features according to dynamically changing images in practical applications, improving the efficiency and accuracy of image tampering detection.

[0086] To facilitate understanding, the system architecture used in the image tampering detection method provided in this application will be introduced below.

[0087] Please refer to Figure 1, which is a schematic diagram of a system architecture 100 provided in this application. As shown in Figure 1, in this system architecture 100, the execution device 110 is used to implement the image tampering detection method provided in this application. The execution device 110 can be implemented by at least one computing instance among physical hosts (e.g., servers), mobile devices (e.g., smartphones, smart cars, or laptops), virtual machines, and containers. When the execution device 110 is implemented by a virtual machine or container, the execution device 110 actually exists in the form of a cloud computing product and can provide cloud services.

[0088] Optionally, the system architecture 100 may also include a database 120, which can be used to store images of areas to be tampered with and the tampered areas in the detected images.

[0089] Optionally, for persistent data storage, the database 120 can be implemented through a data storage system. The data storage system can be located outside the execution device 110, exchanging data with the execution device 110 via a network. For example, the data storage system could be located in the cloud, making it a cloud storage system; this application does not impose any limitations on this. Furthermore, when the execution device 110 is a physical host or a mobile device, the data storage system can also be located inside the execution device 110, such as when the data storage system exchanges data with the processor via a bus. For example, when the execution device 110 is a physical host, the data storage system may be a hard disk or a solid-state drive (SSD); when the execution device 110 is a mobile device, the data storage system may be a flash memory; this application does not impose any limitations on this.

[0090] Optionally, system architecture 100 may also include a local device 130. Users can interact with execution device 110 by operating local device 130. Local device 130 can represent any computing device, such as a client, personal computer, computer workstation, smartphone, tablet, laptop, and smart car.

[0091] Local device 130 can interact with execution device 110 through a communication network of any communication mechanism / standard. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0092] In one possible implementation, the execution device 110 is used to acquire an image of the tampered area to be detected from the user's local device 130, and to perform tamper detection on the image by executing the method provided in this application, thereby obtaining the tampered area in the image. Furthermore, after obtaining the tampered area in the image, the execution device 110 can feed back the obtained tampered area in the image to the local device 130.

[0093] In another possible implementation, the images required by the execution device 110 when executing the image tampering detection method can also be pre-stored in the database 120, and the execution device 110 obtains the images of the tampered areas to be detected by reading image data from the database 120.

[0094] Please refer to Figure 2, which is a flowchart illustrating an image tampering detection method provided in this application. As shown in Figure 2, the image tampering detection method includes the following steps 201-203.

[0095] Step 201: Extract features from the first image using a feature extraction network to obtain image features. The first image is the image of the tampered area to be detected.

[0096] In this application, the execution device first acquires a first image of the area to be tampered with. For example, when the execution device is a server, it receives an image tampering detection request sent by a user through a local device. The image tampering detection request carries the first image and is used to request tampering detection to be performed on the first image. That is, the first image is the image whose tampered area needs to be detected by the execution device. As another example, when the execution device is a local device used by a user (e.g., a mobile device such as a smartphone or laptop), the execution device receives instructions from the user to perform tampering detection on the first image in the execution device.

[0097] The execution device is equipped with a feature extraction network. After acquiring the first image, the execution device first uses the feature extraction network to extract features from the first image, thereby obtaining the image features of the first image. The image features of the first image can be represented in matrix form, for example. Furthermore, the feature extraction network can be, for example, a residual network, a convolutional neural network, or an attention network, etc. This application does not limit the specific structure of the feature extraction network, as long as it ensures that the feature extraction network can successfully extract the image features from the first image.

[0098] Optionally, the first image includes text content, and the tampered area to be detected in the first image is the area where the text content has been tampered with. For example, the first image is an image obtained by taking a picture or scanning the text content, or an image containing text content generated by a computer. For example, the first image is an invoice image, a scanned image of a contract, a scanned image of an acceptance report, or a scanned image of a medical report, etc.

[0099] In general, the first image can be an image from various scenarios where text content needs to be tampered with, such as personal identity verification scenarios, legal document review scenarios, financial record verification scenarios, academic publication inspection scenarios, and medical report verification scenarios where the authenticity of the text needs to be ensured. This application does not specifically limit the scenario to which the first image belongs.

[0100] In addition, the first image can be an image that includes text content, or it can be an image of other types that does not include text content, such as a natural image or a generated image that includes scenery. This application does not limit the specific content included in the first image.

[0101] Step 202: Process image features through a gating network to obtain the confidence scores of multiple expert networks. The confidence scores of multiple expert networks are used to indicate the credibility of each output result obtained by multiple expert networks in processing image features.

[0102] In this application, when setting the structure of the gating network, the number of expert networks connected to the gating network can be pre-defined, and the output dimension of the gating network can be determined based on the pre-defined number of expert networks, i.e., the number of confidence values ​​that the gating network needs to output. In this way, by processing image features through the gating network, it can be determined which one or more expert networks should be used to process each image feature. That is, the gating network works in conjunction with multiple expert networks to determine which expert networks should be used to process each image feature. The input to the gating network can be the image features output by the feature extraction network, and the output of the gating network is the confidence score of each expert network, indicating the reliability of the output result obtained by each expert network in processing the image features output by the feature extraction network; this is typically a probability estimate. The higher the confidence level of the expert network, the more reliable the output results obtained by the expert network in processing the current image features. In other words, the stronger the expert network's ability to process the current image features and the higher the accuracy of the output results. Conversely, the lower the confidence level of the expert network, the less reliable the output results obtained by the expert network in processing the current image features. In other words, the weaker the expert network's ability to process the current image features and the lower the accuracy of the output results.

[0103] Optionally, the gated network can be composed of trainable weight matrices. When processing image features, the gated network outputs the confidence score of each expert network by multiplying the image features by the weight matrix. Furthermore, the gated network can also integrate a softmax function to normalize the vector obtained after multiplying the image features by the weight matrix, ensuring that the confidence score output by the gated network is the normalized confidence score. Of course, the gated network can also be implemented with other structures, as long as it can output the confidence score of each expert network by processing image features; this application does not limit the specific structure of the gated network.

[0104] Optionally, to facilitate the structural design of the expert networks and ensure the coordination among expert networks in processing image features, the multiple expert networks connected to the gating network have the same structure, but different parameters. That is, the network structure and parameter count of each expert network are the same, only the parameters in the network structure are different. Of course, the structures and parameter counts of the multiple expert networks can also be different, and this application does not impose specific limitations on this.

[0105] Step 203: Based on the confidence of multiple expert networks, select one or more expert networks to process image features and obtain the output results of one or more expert networks. The output results are used to determine the tampered region in the first image. The number of selected one or more expert networks is determined based on the confidence of multiple expert networks.

[0106] After obtaining the confidence scores of each expert network output by the gating network, a subset of expert networks can be selected from among multiple expert networks based on the confidence scores of each expert network to process the image features of the first image, thereby obtaining the output results of each expert network in this subset. Furthermore, the output results of each expert network used to process image features are used to determine the tampered regions in the first image, thereby achieving tamper detection of the first image.

[0107] For example, based on the confidence scores of one or more expert networks, the outputs of these networks can be weighted and fused to obtain fused features. The weighted fusion method can be a weighted sum or a weighted average of the expert network outputs, using the expert network confidence scores as weights; this application does not limit the weighted fusion method. Then, the fused features are processed by a segmentation network to obtain the tampered region in the first image. The segmentation network can be, for example, a convolutional neural network, which obtains the tampered region of the first image based on the matrix-form fused features. Specifically, the tampered region in the first image can be represented by a mask image. The mask image is a binary image, meaning that each pixel in the mask image takes one of two fixed values. Furthermore, the mask image has the same size as the first image, and the pixel values ​​in the mask image are used to indicate whether the pixel belongs to the tampered region (e.g., a pixel value of 0 in the mask image represents that the pixel does not belong to the tampered region, and a pixel value of 1 represents that the pixel belongs to the tampered region).

[0108] It's important to note that the number of expert networks selected to process image features in each iteration is not fixed. Instead, it's determined based on the confidence score of each expert network output by the gating network. Therefore, assuming the number of expert networks is N, where N is a positive integer greater than 1 (e.g., N = 4, 5, or 6), the actual number of expert networks selected for processing image features when processing different images could be any positive integer from 1 to N, depending on the confidence score output by the gating network. Furthermore, the type of expert network selected is also not fixed and must be determined based on the confidence score output by the gating network.

[0109] In summary, after obtaining the confidence level output by the gating network, we can select an expert network with the highest confidence level to process the current image features in order to achieve better tamper detection results.

[0110] In this way, the number and type of expert networks used to process image features each time are determined based on the confidence level obtained by the gating network in processing image features. This ensures that the number and type of expert networks are dynamically selected based on the tampering characteristics of the image itself. As a result, in practical applications, the expert networks used to process image features can be flexibly adjusted according to dynamically changing images, thereby improving the efficiency and accuracy of image tampering detection.

[0111] For example, please refer to Figure 3, which is a flowchart illustrating a method for detecting tampered regions in an image according to this application. As shown in Figure 3, after the first image is input into the feature extraction network, the image features output by the feature extraction network are obtained. The gating network further processes the image features output by the feature extraction network to obtain the confidence score of each expert network among expert networks 1 to N (corresponding to the multiple expert networks mentioned above). Then, based on the confidence score of each expert network, expert network 2 and expert network N are selected to process the image features. Therefore, the gating network passes the image features output by the feature extraction network to expert network 2 and expert network N, which process the image features respectively. Finally, the output results obtained by expert network 2 and expert network N in processing the image features are fused to obtain the tampered region in the first image. The output results of expert network 2 and expert network N are both matrices, and the matrix output by expert network 2 and the matrix output by expert network N have the same dimension. In this way, after fusing the matrices output by expert network 2 and expert network N (e.g., performing matrix superposition), a matrix of the same dimension can be obtained, thus ensuring that the dimension of the fusion result of the matrix output by the expert networks is fixed (i.e., the dimension of the fusion result will not change with the number of expert networks selected).

[0112] Specifically, this application applies a hybrid expert model consisting of a gating network and multiple expert networks to image tampering detection. This allows for the adaptive selection of specific expert networks for processing images with different tampering types, avoiding the use of a fixed-structure model to uniformly process all types of tampering images, thereby improving the accuracy of image tampering detection.

[0113] Optionally, when selecting an expert network for processing image features based on the confidence level of the expert network, the expert network to be selected can be determined based on a preset threshold (i.e., a pre-set threshold).

[0114] For example, after obtaining the confidence level of each expert network in multiple expert networks, the confidence level of each expert network is compared with a preset threshold to determine the relationship between the confidence level of each expert network and the preset threshold.

[0115] If any of the multiple expert networks has a confidence level greater than a preset threshold, then the expert network with the confidence level greater than the preset threshold is selected to process the image features. That is, if an expert network's confidence level is greater than the preset threshold, then that expert network is selected to process the image features, while expert networks with confidence levels less than the preset threshold are not selected. For example, assuming the preset threshold is 0.3, and the confidence levels of the multiple expert networks are 0.4, 0.35, 0.1, and 0.15, then the expert networks with confidence levels of 0.4 and 0.35, respectively, can be selected to process the image features.

[0116] If the confidence scores of multiple expert networks are all below a preset threshold, one or more expert networks with the highest confidence scores are selected from among them to process the image features. In other words, if the confidence scores of all expert networks are below the preset threshold, one or more expert networks with the highest confidence scores (e.g., one or two expert networks with the highest confidence scores) can be selected to process the image features. For example, assuming the preset threshold is 0.3, and the confidence scores of multiple expert networks are 0.3, 0.25, 0.2, and 0.25 respectively, since the confidence score of each expert network is not greater than 0.3, the expert network with a confidence score of 0.3 (i.e., the expert network with the highest confidence score) can be selected from these four expert networks to process the image features.

[0117] The aforementioned preset threshold can be a threshold set or adjusted according to the actual scenario, such as a value of 0.3 or 0.28. This application does not limit the specific size of the preset threshold.

[0118] It should be noted that, according to the applicant's research, if the expert networks selected for different image features are fixed, it will often limit the model's adaptability to images with different characteristics. This can easily lead to the model selecting too many expert networks and wasting computing resources when dealing with relatively simple tamper detection tasks, and selecting too few expert networks when dealing with more complex tamper detection tasks, thus affecting the accuracy of image tamper detection.

[0119] Therefore, this scheme determines the expert network to be used by comparing the confidence level of the expert network with the preset threshold. This effectively selects the most suitable expert network to process image features based on the tampering characteristics of the image itself, and flexibly adjusts the number of expert networks to process image features, so that the cooperation between expert networks is better and the efficiency and accuracy of image tampering detection are improved.

[0120] For example, please refer to Figure 4, which is a schematic diagram of selecting expert networks based on a preset threshold according to this application. As shown in Figure 4, after the gating network processes the image features output by the feature extraction network, it obtains the confidence score of each expert network in expert network 1 to expert network N, i.e., confidence score 1 to confidence score N. Then, by executing a dynamic routing strategy, the confidence score of each expert network can be compared with the preset threshold to determine the relationship between the preset threshold and confidence scores 1 to N. If it is determined that only confidence score 2 and confidence score N are greater than the preset threshold, expert network 2 corresponding to confidence score 2 and expert network N corresponding to confidence score N can be selected to process the image features, thereby passing the image features to expert network 2 and expert network N.

[0121] The above describes the process of selecting a subset of expert networks to process image features based on the confidence level of each expert network among multiple expert networks, thereby achieving adaptive selection of expert networks. To ensure that the most suitable expert network can be selected based on the image tampering characteristics during practical inference applications, this application provides an optimized model training method that enables the model to effectively learn during the training phase how to flexibly select the most suitable expert network to process the image based on the tampering characteristics.

[0122] For example, the feature extraction network, gating network, and multiple expert networks described above are used to construct the tampering detection model. During the training phase of the tampering detection model, the parameters are updated based on a first loss function and a second loss function. That is, the first and second loss functions jointly constrain the parameter update process of the tampering detection model, so that the parameter update objective of the tampering detection model is to minimize the values ​​of the first and second loss functions, thereby reducing prediction error.

[0123] The first loss function is derived from the actual tampered regions in the training data and the predicted tampered regions output by the tamper detection model. In other words, the first loss function constrains the tamper detection model to learn how to identify the correct tampered regions.

[0124] The second loss function is derived from the actual tampering type of the training data and the predicted tampering type output by the classification network. The classification network is used to predict the tampering type of the training data based on the features output by the feature extraction network. The structure of the classification network is, for example, a multilayer perceptron (MLP). That is, during the training phase of the tampering detection model, an additional classification network participates in the training process. Furthermore, the input of the classification network is the output of the feature extraction network, and the output of the classification network is the predicted tampering type of the training data. Since the input of the classification network is the output of the feature extraction network, when training the tampering detection model based on the second loss function, in order to ensure that the classification network can accurately predict the tampering type, the feature extraction network often learns how to perceive the tampering type of the image during parameter updates. This allows the image features output by the feature extraction network to be easily identified by the gating network, enabling the selection of an appropriate expert network based on significant tampering characteristics.

[0125] Generally, when constructing training data, the true tampering types corresponding to the training data can be obtained, such as inpainting, generation, copy-move, or splicing tampering types. Therefore, after the classification network outputs the predicted tampering type for the training data, it can construct a second loss function based on the true and predicted tampering types. Generally, the greater the difference between the true and predicted tampering types, the larger the value of the second loss function; conversely, the smaller the difference, the smaller the value of the second loss function.

[0126] In this scheme, a classification network is introduced during the training phase of the tamper detection model to construct an additional second loss function. This implicitly constrains the tamper detection model's perception of tamper types during the training phase, making the learning process of the expert networks more professional. Furthermore, each expert network can learn to process image features under specific tamper types more specifically, thereby improving the overall accuracy of the tamper detection model.

[0127] It should be noted that the classification network described above is only used during the training phase to construct the second loss function. When the tamper detection model performs inference, the classification network is not included; that is, the classification network is not used during the inference phase.

[0128] Furthermore, the above describes the use of the first and second loss functions when training the tamper detection model. In some cases, other loss functions (such as the load balancing loss function that constrains the tamper detection model to select the expert network evenly) can also be combined with the tamper detection model during training to optimize it. This application does not limit the tamper detection model to be trained solely based on the first and second loss functions.

[0129] For example, please refer to Figure 5, which is a schematic diagram of the training of a tampering detection model provided in this application. As shown in Figure 5, the tampering detection model includes a feature extraction network, a gating network, and expert networks 1 to N. During the training phase of the tampering detection model, training data (i.e., pre-prepared images with tampered regions) is input to the feature extraction network, which extracts image features from the training data. Furthermore, the image features output by the feature extraction network are used as input to the gating network and the classification network. After processing the image features, the gating network outputs the confidence score of each expert network, thereby enabling the selection of some expert networks to process image features based on their confidence scores. The classification network processes the image features to output a predicted tampering type. Thus, based on the predicted tampered region output by the tampering detection model and the actual tampered region of the training data, a first loss function can be constructed; based on the predicted tampering type output by the classification network and the actual tampering type of the training data, a second loss function can be constructed. Finally, a fused loss function is obtained by fusing the first and second loss functions (e.g., by weighted summation of the first and second loss functions). This fused loss function is used to update the parameters of the tamper detection model. Furthermore, the fused loss function can also update the parameters of the classification network to ensure that the parameters of the classification network are updated synchronously with those of the tamper detection model, thereby improving the compatibility between the classification network and the tamper detection model.

[0130] The above describes the process of implicitly constraining the tamper detection model's perception of tamper types by incorporating a classification network during the training process, thereby improving the model's accuracy. It is understandable that the model's accuracy is not only related to the training process but also often closely related to the quality of the training data. Therefore, this application also provides a method for automatically generating high-quality training data, so as to provide a large amount of high-quality training data for the training process of the tamper detection model.

[0131] Generally, image tampering types can include erasure tampering, generation tampering, same-image copy-and-paste tampering, and cross-image copy-and-paste tampering. Erasure tampering removes the original text from the image, while generation, same-image copy-and-paste, and cross-image copy-and-paste tampering replace the original text with other text. For text replacement tampering, to ensure the quality of the tampered image, this application uses a large language model to understand the text, thereby ensuring that the tampered text has a similar meaning to the original text.

[0132] For example, during the generation of training data, the semantic category corresponding to the first text in the second image is first identified using a large language model. The second image is a pre-collected image containing text content, and it is the image before any alteration. The first text in the second image can be extracted using Optical Character Recognition (OCR); it can also be extracted using a large language model or other models. After extracting the first text from the second image, by inputting the first text into the large language model, the text understanding capabilities of the large language model can be used to determine the semantic category corresponding to the first text.

[0133] Then, based on the semantic category corresponding to the first text, the first text in the second image is modified to obtain the modified second image. The text in the modified second image corresponds to the same semantic category before and after the modification, and the modified second image is used as training data for the modification detection model. In other words, after modifying the first text in the second image, the modified text and the first text correspond to the same semantic category.

[0134] In this scheme, by leveraging the text understanding capabilities of a large language model, the semantic category of the text in an image can be identified. This allows for the generation of altered text within the same semantic category when the original text in an image is modified, thereby ensuring the legitimacy of the altered image and improving the quality of the altered image used as training data.

[0135] For example, please refer to Figure 6, which is a schematic diagram of generating a tampered image according to this application. As shown in Figure 6, the text content included in the second image is personal identification information such as name, gender, ethnicity, date, and address. After performing OCR recognition on the second image, the text recognized by the OCR can be obtained. After inputting the recognized text into a large language model, the semantic category corresponding to each text output by the large language model can be obtained. For example, if the recognized text includes "Xiaoming", "male", "Han", "2024", "7", "10", etc., the large language model can output the semantic category corresponding to each text as "name", "gender", "ethnicity", "date", "date", "date" in sequence. In this way, based on the corresponding semantic category of each recognized text, specific text in the second image can be tampered with, for example, the text "male" can be tampered with into the text "female" as shown in Figure 6.

[0136] Specifically, there are multiple ways to tamper with the first text in the second image based on the semantic category corresponding to the first text.

[0137] In one possible implementation, a second text is generated based on the semantic category of the first text using a large language model. The second text corresponds to the same semantic category as the first text. Because large language models have strong text understanding and generation capabilities, they can generate other texts (i.e., the second text) under a given semantic category.

[0138] After obtaining the second text generated by the large language model, the first text can be replaced with the second text to obtain the modified second image.

[0139] In other words, this implementation essentially replaces the first text in the second image with second text of the same semantic category, falling under the generated tampering type described above. Therefore, in this case, the label of the tampered second image can be marked as generated tampering type, thereby providing high-quality and accurately labeled training data for the tampering detection model.

[0140] For example, please refer to Figure 7, which is a schematic diagram of image tampering of a generation type provided by this application. As shown in Figure 7, for the text 1 extracted from the second image (corresponding to the first text mentioned above), text 1 is specifically "male", and the semantic category corresponding to text 1 is "gender". By inputting text 1 and its corresponding semantic category into a large language model, the large language model can be instructed to output text 2 corresponding to the same semantic category, specifically "female". In this way, by replacing text 1 with text 2 in the second image, the tampering of the second image can be achieved, and the tampering type of the tampered second image is the generation tampering type.

[0141] In this scheme, by utilizing the text generation capabilities of a large language model, it is possible to automatically generate tampered data with high semantic continuity, thereby improving the quality of the generated tampered data and providing diverse and high-quality training data for the training phase of the tampering detection model.

[0142] Optionally, during the process of replacing the first text with the second text, in order to ensure the similarity between the second text and the first text as much as possible and to avoid the second text appearing too abruptly in the second image, the display information of the first text in the second image can be obtained first. Specifically, the display information of the first text includes at least one of the following: the position, size, font format, and background color of the first text. The font format of the first text may include, for example, the font size, font color, and whether the font is bold.

[0143] Then, based on the display information of the first text, the first text is replaced with the second text to obtain the tampered second image. In the tampered second image, the display information of the second text is the same as or as close as possible to the display information of the first text. That is, the second text is set on the second image with reference to the display information of the first text, such as its position, size, font format, and background color, thereby ensuring that the display of the tampered second text is as consistent as possible with the display of the original first text.

[0144] In another possible implementation, a third text can be determined from either the second or third image based on the semantic category corresponding to the first text. The third text corresponds to the same semantic category as the first text, and the third image is used as training data for the tampering detection model. For example, the third image could be a pre-collected image containing text content, and the third image and the second image could be images collected in the same scene. Then, the first text is replaced with the third text to obtain the tampered second image.

[0145] In other words, based on the semantic category corresponding to the first text, a third text with the same semantic category as the first text can be determined on the same image or another image. Then, the first text is replaced with the determined third text, thus achieving the copy-paste of the third text. It can be understood that if the third text is determined in the second image, then the tampering type of the modified second image is same-image copy-paste; if the third text is determined in the third image, then the tampering type of the modified second image is cross-image copy-paste.

[0146] For example, please refer to Figure 8, which is a schematic diagram of image tampering of the same image copy-paste type provided by this application. As shown in Figure 8, the example shown in Figure 8 is to perform text copy-paste on the same image, thereby replacing one text on the image with another text on the same image. Specifically, for the text 3 extracted in the second image (corresponding to the first text mentioned above), text 3 is specifically "7", and the semantic category corresponding to text 3 is "date". Based on the semantic category to which text 3 belongs, other texts corresponding to the same semantic category can be found in the same image, such as text 4 (corresponding to the third text mentioned above, the third text is text with the same semantic category as the first text), text 4 is specifically "10". In this way, by replacing text 3 with text 4 in the second image, the tampering of the second image can be achieved, and the tampering type of the tampered second image belongs to the same image copy-paste type.

[0147] In this scheme, by leveraging the text understanding capabilities of a large language model, similar replacement texts can be automatically found in the same image or other images for the text that needs to be tampered with, thereby automatically generating high-quality tampering data and providing diverse and high-quality training data for the training phase of the tampering detection model.

[0148] In one possible implementation, to obtain training data for the model, a large language model can be used to identify a fourth text in the second image corresponding to a preset semantic category. This preset semantic category is related to the scene to which the second image belongs. For example, if the second image is an image within a personal identity scene, the preset semantic category could be a name, age, phone number, address, or similar semantic category.

[0149] After the fourth text is identified, it can be erased from the second image to obtain a tampered second image. The tampered second image is used as training data for the tamper detection model, which includes the feature extraction network, the gating network, and the multiple expert networks.

[0150] In this solution, by using a large language model to identify text of specific semantic categories in an image and erasing the identified text, an image of the erasure type can be obtained. The erased text in the image belongs to a specific semantic category related to the scene, ensuring that meaningful text in the image can be erased and improving the quality of the tampered image.

[0151] The above describes the implementation process of the image tampering detection method provided in this application. For ease of understanding, the specific execution process of this image tampering detection method will be detailed below with concrete examples. To implement the image tampering detection method, the following will sequentially describe the multiple stages of constructing training data, training the tampering detection model, and deploying the tampering detection model to perform image tampering detection.

[0152] Phase 1: Constructing training data.

[0153] For example, please refer to Figure 9, which is a schematic diagram of constructing training data using a large language model according to this application. As shown in Figure 9, the process of constructing training data using a large language model includes the following multiple steps.

[0154] Step 1: Use OCR to identify the pre-prepared image to be tampered with, and extract the text and text display information from the image to be tampered with.

[0155] For example, given a scanned image of an unaltered document, OCR can detect all the text in the image and its display information. This display information includes, but is not limited to, the text's position (e.g., its coordinates within the image), length, width, color, font format, and background color. Furthermore, for ease of retrieval, the text and its display information can be constructed as a dictionary data structure. The text's character content is the dictionary's field, and the display information is the corresponding content within that field. This allows for finding the text's display information within the dictionary based on the text's character content. It's important to note that when tampering with each image, a corresponding dictionary needs to be constructed for each image to facilitate subsequent searches for the text's display information.

[0156] Step 2: Determine the semantic category of the text in the image to be tampered with based on the scene to which the image belongs.

[0157] Generally, in a fixed context, the semantic categories corresponding to the text in an image to be altered are also usually fixed. Therefore, all semantic categories corresponding to the text in the image to be altered can be determined based on the context in which the image belongs. For example, in a personal identity context, the semantic categories corresponding to the text in the image to be altered include name, age, gender, address, phone number, and email address. In a financial context, the semantic categories corresponding to the text in the image to be altered include bank account number, transaction amount, and financial table. In a legal context, the semantic categories corresponding to the text in the image to be altered include contract, responsibility, rights, and obligations.

[0158] Step 3: Construct prompt words based on the text extracted from the image to be tampered with.

[0159] Generally, when using a large language model to process tasks, it is often necessary to input corresponding prompt words into the large language model to instruct it to complete a specific task (such as identifying the semantic category of text). Therefore, in this step, prompt words can be constructed by combining the text extracted from the image to be tampered with and the semantic category of the text in the current scene.

[0160] When constructing prompt words, the following rules can be followed: (1) Clear category: The prompt word should clearly indicate the type of information that needs to be identified and classified; (2) Operation instruction: It should directly indicate the operation to be performed by the large language model, such as "classify", "identify", "summarize", etc.; (3) Result formatting: Specify the format of the output result to ensure the structure and usability of the output result. Based on the above rules, three specific prompt word examples are given below.

[0161] In the context of personal identity, based on the extracted text and its corresponding semantic category, the constructed prompt can be: "Based on the text in the personal identity file, classify the extracted text into the following predefined categories: name, gender, ethnicity, date of birth, address; and provide the classification output in a structured format. Example text: Xiaoming, male, Han, 2024, 7, 10, A province, B city, C district, D street, E number."

[0162] In legal document analysis scenarios, based on the extracted text and its corresponding semantic categories, the constructed prompts can be: "Based on the text in the legal document, classify the extracted text into the following predefined categories: case number, relevant laws, key arguments, and judgment results; and provide the classification output in a structured format. Example text: XXX."

[0163] In the context of medical report processing, based on the extracted text and its corresponding semantic category, the constructed prompt can be: "Based on the medical report text, classify the extracted text into the following predefined categories: patient name, diagnosis, treatment plan, and follow-up date; and provide the classification output in a structured format. Example text: XXX".

[0164] Step 4: Based on the constructed prompt words, identify the semantic category of the text using a large language model.

[0165] After constructing the prompt words, they can be input into a large language model to obtain the semantic categories identified by the model for the extracted text. It should be noted that since some of the extracted text may not correspond to the semantic categories specified in the prompt words, the output of the large language model may only contain text with corresponding semantic categories. These texts with corresponding semantic categories can be understood as the key texts selected by the large language model. Multiple files selected by the large language model from the same image can form a set of key texts.

[0166] Step 5: Based on the text in the key text set, perform image manipulation to obtain the manipulated image as training data.

[0167] Specifically, for a set of key texts corresponding to the same image, a portion of the key texts can be randomly selected from the set, and various types of tampering can be performed on the image based on the selected key texts. The specific process of performing various types of tampering on an image will be described below.

[0168] A, Erase / Tamper Type.

[0169] For the selected key text, the corresponding display information is searched in the dictionary based on the character content of the key text to obtain the position and background color of the key text. Then, based on the position of the key text, it is erased in the image, and the erased position is filled with the background color of the key text so that the filled content is similar to the original background color, resulting in an image of the erased tamper type.

[0170] B, generates a tampering type.

[0171] Based on the selected key text, prompt words are constructed to instruct the large language model to generate other texts of the same semantic category based on the key text. For example, the prompt words could be: "Generate alternative texts of similar type and precise character length for the given key text, to be used as tampering data in document tampering detection. The alternative texts must maintain similarity to the original text in classification and style, ensuring that the replacement content is reasonable and unique." By inputting the constructed prompt words into the large language model, the alternative texts generated by the model for each key text can be obtained. Then, based on the character content of the key text, the corresponding display information is searched in the dictionary, and with reference to this information, the key text in the image is replaced with the alternative text generated by the large language model. For example, first, a new text box image is generated by combining the length, width, color, and background color of the key text, with the character content of the alternative text in the text box image; then, the new text box image is overlaid on the position of the key text in the original image, resulting in an image with the tampering type being generated.

[0172] C, same as image copy and paste type.

[0173] For a selected key text in a set of key texts, a replacement text with a similar semantic category and display information is searched within that set, where both the selected key text and the replacement text are located within the same image. Then, the text box containing the replacement text in the image is scaled so that the scaled text box is the same size as the text box containing the key text. Finally, the scaled text box is overlaid on the location of the key text, resulting in an image that has been modified using the same image copy-paste method.

[0174] For example, let text P be the key text selected in the key text set, and let text Q be the alternative text found for text P in the key text set. Text Q has similar semantic category, text length, text width, font color, and background color to text P. Thus, by scaling the text box containing text Q in the image and overlaying it onto the location of text P, the altered image can be obtained.

[0175] D, Cross-image copy and paste type.

[0176] The cross-image copy-paste type is implemented similarly to the same-image copy-paste type, except that when finding alternative text for key text, it no longer searches for alternative text in the same image, but in other images.

[0177] After obtaining the tampered image in step 5, the tampered image, the tampering location in the tampered image, and the tampering type corresponding to the tampered image can be stored to facilitate the subsequent tampering detection model training process.

[0178] Phase Two: Training the tamper detection model.

[0179] For example, please refer to Figure 10, which is a schematic diagram of the training framework of a tampering detection model provided in this application. As shown in Figure 10, the tampering detection model includes a feature extraction network, a Transformer decoder, and a segmentation network. The feature extraction network includes a backbone network for performing image feature extraction and a Transformer encoder. The Transformer decoder includes an attention mechanism module, a gating network, and multiple expert networks; essentially, the gating network and multiple expert networks replace the MLP in a conventional Transformer decoder. The backbone network can be, for example, a pre-trained network, such as a residual network, a convolutional neural network, or an attention network—a general image feature extraction network. The segmentation network can be, for example, a convolutional neural network.

[0180] In addition, during the training phase of the tamper detection model, a classification network is added to constrain the training process of the tamper detection model, which is used to predict the type of tampering of the input image based on the output of the feature extraction network.

[0181] It should be noted that in the tampering detection model shown in Figure 10, only one Transformer decoder is shown for ease of illustration. In reality, the tampering detection model may include one or more Transformer decoders connected sequentially, and the network structure of each Transformer decoder may be the same or different. For example, the multiple Transformer decoders included in the tampering detection model may all include the aforementioned attention mechanism module, gating network, and multiple expert networks, and the network structures of the multiple Transformer decoders may also be the same. Alternatively, the multiple Transformer decoders included in the tampering detection model may all include the aforementioned attention mechanism module, gating network, and multiple expert networks, but the structures of the attention mechanism module, gating network, or expert network included in different Transformer decoders may be different. For another example, any one or more Transformer decoders included in the tampering detection model may include the aforementioned attention mechanism module, gating network, and multiple expert networks, while the other Transformer decoders may have existing structures, i.e., including an attention mechanism module and an MLP.

[0182] During training, the input images used as training data are first fed into the feature extraction network, processed by the backbone network and Transformer encoder to output image features. These image features are then fed into the attention mechanism module in the Transformer decoder, which outputs weighted features. The attention mechanism module considers the influence of all other elements in the sequence when processing each element, thereby capturing richer contextual information. The role of the attention mechanism module is to improve the sensitivity of the tamper detection model to important parts of the input data, improve the quality of information representation, and thus provide more accurate data support for subsequent feature processing.

[0183] The gating network receives the features adjusted by the attention mechanism module and outputs the confidence score of each expert network. In other words, the gating network's role is to adjust the data flow based on the reliability of the data, determine the expert networks that need to perform subsequent feature processing, and optimize the model's feature processing capabilities.

[0184] Based on the confidence score of each expert network output by the gating network, the expert network used for feature processing can be determined. The selected expert network is used to process the features output by the attention mechanism module. For example, the confidence score of each expert network is compared with a given preset threshold, and the expert network with a confidence score greater than the preset threshold is selected for subsequent feature processing. If no expert network has a confidence score greater than the preset threshold, the expert network with the highest confidence score is selected for subsequent feature processing. The purpose of designing the gating network and expert networks is to enable the model to efficiently allocate computational resources according to the specific characteristics and needs of the data, improving the processing efficiency and accuracy of images under various tampering characteristics.

[0185] Secondly, when multiple expert networks have confidence scores greater than a threshold, these networks are selected. The outputs of these selected networks are then weighted and fused, with the weights corresponding to the confidence scores of each network. Finally, a segmentation network processes the fusion result to obtain the predicted tampered region of the image. The segmentation network's input is the fused output of the selected expert networks, and its output is a binary image of the same size as the input image of the tamper detection model. This binary image indicates the tampered region in the input image. Based on the segmentation network, the output of the expert networks can be converted into a binary image of the same size as the input image, thus enabling the output of the tamper detection model predicting the tampered region of the input image. This results in a predicted tampered region mask output by the tamper detection model. pred and the mask of the actual tampered area of ​​the training data gtThe first loss function L can be calculated. seg =Loss L1 (mask pred ,mask gt ).

[0186] Furthermore, after obtaining the image features output by the feature extraction network, these features are simultaneously input into the classification network to obtain the predicted tampering type output by the classification network. Based on the predicted tampering type output by the classification network and the actual tampering type of the training data, the second loss function L can be calculated. cls .

[0187] Finally, based on the tampered region prediction loss function L seg And the tamper type prediction loss function L cls Calculate the total loss function L = L seg +w1*L cls The training of the tamper detection model is completed by updating the parameters of the model and the classification network based on the total loss function. Here, w1 is a weight parameter used to balance the strength of each loss function. Phase three involves deploying the tamper detection model to perform image tamper detection.

[0188] After training the tamper detection model, it can be deployed on an execution device. The model can then be run to perform image tamper detection on various images requiring tampering detection. Specifically, after deployment, the execution device acquires the image sent by the user for tamper detection and uses it as input to the model. The device processes the input image by running the tamper detection model, outputting a binary image of the same size as the input image. This binary image indicates the tampered regions detected by the model. Therefore, based on the binary image output by the tamper detection model, the tampered regions in the input image can be determined.

[0189] The method provided in this application has been described in detail above. Next, the device provided in this application for performing the above method will be described.

[0190] Please refer to Figure 11, which is a schematic diagram of the structure of an image tampering detection device provided in this application. As shown in Figure 11, the image tampering detection device includes: an extraction module 1101, used to extract features of a first image through a feature extraction network to obtain image features, wherein the first image is an image of the tampered area to be detected; a processing module 1102, used to process the image features through a gating network to obtain the confidence scores of multiple expert networks, wherein the confidence scores of the multiple expert networks are used to indicate the credibility of each output result obtained by the multiple expert networks in processing the image features; the processing module 1102 is further used to select one or more expert networks from the multiple expert networks to process the image features based on the confidence scores of the multiple expert networks, to obtain the output results of one or more expert networks, wherein the output results are used to determine the tampered area in the first image; wherein the number of the selected one or more expert networks is determined based on the confidence scores of the multiple expert networks.

[0191] In one possible implementation, the processing module 1102 is further configured to: select an expert network with a confidence level greater than the preset threshold from among the multiple expert networks to process the image features if the confidence level of the multiple expert networks is greater than the preset threshold; and select one or more expert networks with the highest confidence level from among the multiple expert networks to process the image features if the confidence level of the multiple expert networks is not greater than the preset threshold.

[0192] In one possible implementation, a feature extraction network, a gating network, and multiple expert networks are used to construct a tampering detection model. The parameters in the tampering detection model are updated during training based on a first loss function and a second loss function. The first loss function is obtained based on the actual tampered regions of the training data and the predicted tampered regions output by the tampering detection model. The second loss function is obtained based on the actual tampering types of the training data and the predicted tampering types output by the classification network. The classification network is used to predict the tampering type of the training data based on the features output by the feature extraction network.

[0193] In one possible implementation, the tampered area to be detected in the first image is the area where the text content has been tampered with.

[0194] In one possible implementation, the processing module 1102 is further configured to: identify the semantic category corresponding to the first text in the second image through a large language model; modify the first text in the second image based on the semantic category to obtain the modified second image, wherein the text in the modified second image corresponds to the same semantic category before and after modification, and the modified second image is used as training data for a tampering detection model, wherein the tampering detection model includes a feature extraction network, a gating network and multiple expert networks.

[0195] In one possible implementation, the processing module 1102 is further configured to: generate a second text based on a large language model, the second text corresponding to the same semantic category as the first text; and replace the first text with the second text to obtain the altered second image.

[0196] In one possible implementation, the processing module 1102 is further configured to: obtain display information of the first text in the second image, the display information including at least one of the position, size, font format and background color of the first text; and replace the first text with the second text based on the display information to obtain the tampered second image.

[0197] In one possible implementation, the processing module 1102 is further configured to: determine a third text in a second or third image based on a semantic category, wherein the third text corresponds to the same semantic category as the first text, and the third image is used as training data for a tampering detection model; and replace the first text with the third text to obtain a tampered second image.

[0198] In one possible implementation, the processing module 1102 is further configured to: perform weighted fusion of the outputs of one or more expert networks based on the confidence of one or more expert networks to obtain fused features; and process the fused features through a segmentation network to obtain the tampered region in the first image.

[0199] In one possible implementation, multiple expert networks have the same structure, but with different parameters.

[0200] Both the extraction module 1101 and the processing module 1102 can be implemented in software or in hardware. For example, the implementation of the processing module 1102 will be described below. Similarly, the implementation of the extraction module 1101 can be referenced to that of the processing module 1102.

[0201] As an example of a software functional unit, processing module 1102 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, module A may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0202] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0203] As an example of a hardware functional unit, the processing module 1102 may include at least one computing device, such as a server. Alternatively, the processing module 1102 may be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system-on-chip (SoC), an offload card, an accelerator card, or any combination thereof.

[0204] The processing module 1102 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the processing module 1102 can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the processing module 1102 can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offloading cards, and accelerator cards.

[0205] Please refer to Figure 12, which is a schematic diagram of a computing device provided in this application. The computing device 1200 shown in Figure 12 can be used to execute the image tampering detection method provided in this application. As shown in Figure 12, the computing device 1200 includes: a bus 1202, a processor 1204, a memory 1206, and a communication interface 1208. The processor 1204, the memory 1206, and the communication interface 1208 communicate with each other via the bus 1202. The computing device 1200 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1200.

[0206] Bus 1202 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 12, but this does not imply that there is only one bus or one type of bus. Bus 1202 can include pathways for transmitting information between various components of computing device 1200 (e.g., memory 1206, processor 1204, communication interface 1208).

[0207] The processor 1204 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0208] The memory 1206 may include volatile memory, such as random access memory (RAM). The processor 1204 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0209] The memory 1206 stores executable program code, and the processor 1204 executes this executable program code to implement the functions of the aforementioned monitoring module and processing module, thereby realizing the model processing method described above. That is, the memory 1206 stores instructions for executing the model processing method.

[0210] The communication interface 1208 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1200 and other devices or communication networks.

[0211] It should be understood that the computing device 1200 according to this application is used to execute the image tampering detection method as shown in Figures 2 to 10, and can correspond to the subject executing the method according to this application. For the sake of brevity, it will not be described in detail here.

[0212] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0213] Please refer to Figure 13, which is a schematic diagram of a computing device cluster provided in this application. As shown in Figure 13, the computing device cluster includes at least one computing device 1200. The memory 1206 of one or more computing devices 1200 in the computing device cluster may store the same instructions for executing the image tampering detection method.

[0214] In some possible implementations, the memory 1206 of one or more computing devices 1200 in the computing device cluster may also store partial instructions for executing the image tampering detection method. In other words, a combination of one or more computing devices 1200 can jointly execute the instructions for executing the image tampering detection method.

[0215] It should be noted that the memory 1206 in different computing devices 1200 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the data processing device. That is, the instructions stored in the memory 1206 of different computing devices 1200 can implement the functions of one or more of the aforementioned monitoring and processing modules.

[0216] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 14 illustrates one possible implementation. Figure 14 is a schematic diagram of another computing device cluster structure provided in this application. As shown in Figure 14, in computing device cluster 1400, two computing devices 1200A and 1200B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 1206 in computing device 1200A stores instructions for executing the functions of the monitoring module. Simultaneously, the memory 1206 in computing device 1200B stores instructions for executing the functions of the processing module.

[0217] It should be understood that the functions of computing device 1200A shown in Figure 14 can also be performed by multiple computing devices 1200. Similarly, the functions of computing device 1200B can also be performed by multiple computing devices 1200.

[0218] It should be understood that the computing device 1200 or computing device cluster 1400 in this application can correspond to the image tampering detection device in FIG11 of this application, and can correspond to the corresponding entity executing the image tampering detection method in FIG2 of this application. Furthermore, the above-mentioned and other operations and / or functions of each module in the computing device 1200 or computing device cluster 1400 are respectively for implementing the corresponding process of the image tampering detection method in FIG2, and for the sake of brevity, will not be elaborated further here.

[0219] This application also provides a chip comprising a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in a storage unit to cause the chip within the electronic device to perform the methods described in the above embodiments. Optionally, the storage unit may be an in-chip storage unit, such as a register or cache. Alternatively, the storage unit may be an external storage unit located within a wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0220] Specifically, please refer to Figure 15, which is a schematic diagram of the structure of a chip provided in this application. The chip can be represented as a neural processing unit (NPU), a graphics processing unit (GPU), or a tensor processing unit (TPU). The following description will use an NPU 1500 as an example. The NPU 1500 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core of the NPU is the arithmetic circuit 1503, which is controlled by a controller 1504 to retrieve matrix data from memory and perform multiplication operations.

[0221] In some implementations, the arithmetic circuit 1503 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1503 is a two-dimensional pulsating array. The arithmetic circuit 1503 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1503 is a general-purpose matrix processor.

[0222] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1502 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1501 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1508.

[0223] Unified memory 1506 is used to store input and output data. Weight data is directly transferred to weight memory 1502 via Direct Memory Access Controller (DMAC) 1505. Input data is also transferred to unified memory 1506 via DMAC.

[0224] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1509.

[0225] The Bus Interface Unit (BIU) 1510 is used by the instruction fetch memory 1509 to fetch instructions from external memory, and also by the memory access controller 1505 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0226] The DMAC is mainly used to move input data from external memory DDR to unified memory 1506, or to weight data to weight memory 1502, or to input data to input memory 1501.

[0227] The vector computation unit 1507 includes multiple processing units that further process the output of the computation circuit 1503 when needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0228] In some implementations, the vector computation unit 1507 can store the processed output vector in the unified memory 1506. For example, the vector computation unit 1507 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1503, such as performing linear interpolation on the feature planes extracted by the convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1507 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as an activation input to the computation circuit 1503, for example, for use in subsequent layers of the neural network.

[0229] The instruction fetch buffer 1509 connected to the controller 1504 is used to store the instructions used by the controller 1504;

[0230] Unified memory 1506, input memory 1501, weighted memory 1502, and instruction fetch memory 1509 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0231] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0232] It should be understood that the chip in Figure 15 of this application may correspond to the image tampering detection device in Figure 11 of this application, or be deployed on the computing device 1200 or computing device cluster 1400 of this application. Furthermore, the chip in Figure 15 of this application may correspond to the corresponding entity executing the image tampering detection method in Figure 2 of this application. The above-mentioned and other operations and / or functions of each module in the chip are respectively for implementing the corresponding process of the image tampering detection method in Figure 2, which will not be elaborated here for the sake of brevity. In addition, in this application, the structure of the chip is not limited to the chip structure shown in Figure 15, and may include more or fewer hardware structures to implement the functions of the method shown in Figure 2.

[0233] Referring to Figure 16, which is a schematic diagram of the structure of a computer-readable storage medium provided in this application. This application also provides a computer-readable storage medium in which, in some embodiments, the method disclosed in Figure 2 can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art.

[0234] Figure 16 schematically illustrates a conceptual partial view of an example computer-readable storage medium arranged according to at least some of the embodiments shown herein, the example computer-readable storage medium including a computer program for executing computer processes on a computing device. In one embodiment, the computer-readable storage medium 1600 is provided using a signal bearer medium 1601. The signal bearer medium 1601 may include one or more program instructions 1602 that, when executed by one or more processors, can provide the functionality or part of the functionality described above with respect to Figure 2.

[0235] In some examples, the signal carrying medium 1601 may include a computer-readable medium 1603, such as, but not limited to, a hard disk drive, a compact disc (CD), a digital video disc (DVD), a digital magnetic tape, a memory, ROM, or RAM, etc.

[0236] In some embodiments, the signal-bearing medium 1601 may comprise a computer-recordable medium 1604, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, and so on. In some embodiments, the signal-bearing medium 1601 may comprise a communication medium 1605, such as, but not limited to, digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, and so on). Therefore, for example, the signal-bearing medium 1601 may be transmitted by a wireless communication medium 1605 (e.g., a wireless communication medium conforming to the IEEE 1202.X standard or other transmission protocols).

[0237] One or more program instructions 1602 may be, for example, computer-executable instructions or logical implementation instructions. In some examples, the computing device may be configured to provide various operations, functions, or actions in response to one or more program instructions 1602 conveyed to the computing device via a computer-readable medium 1603, a computer-recordable medium 1604, and / or a communication medium 1605.

[0238] It should be understood that the computer-readable storage medium 1600 in this application may be deployed on the image tampering detection apparatus shown in FIG11, or on the computing device 1200 or computing device cluster 1400 of this application. In this way, the image tampering detection apparatus, computing device 1200 or computing device cluster 1400 provided in this application implements the image tampering detection method shown in FIG2 by reading one or more program instructions 1602 on the computer-readable storage medium 1600.

[0239] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0240] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods of the various embodiments of this application.

[0241] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0242] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, computer instructions can be transferred from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0243] The above description is merely a specific embodiment of the present invention. Those skilled in the art can conceive of variations or substitutions based on the specific embodiments provided in this application, and all such variations or substitutions should be covered within the protection scope of this application.

Claims

1. An image tampering detection method, characterized by, The method comprises: extracting features of a first image through a feature extraction network to obtain image features, the first image being an image in which a tampered region needs to be detected; processing the image features through a gating network to obtain confidence of a plurality of expert networks, the confidence of the plurality of expert networks being used to indicate a degree of credibility of each output result obtained by the plurality of expert networks processing the image features; based on the confidence of the plurality of expert networks, selecting one or more expert networks from the plurality of expert networks to process the image features to obtain output results of the one or more expert networks, the output results being used to determine a tampered region in the first image; wherein the number of the one or more expert networks selected is determined based on the confidence of the plurality of expert networks.

2. The method of claim 1, wherein, The method further comprises: in a case where there is a confidence greater than a preset threshold in the confidence of the plurality of expert networks, selecting an expert network with a confidence greater than the preset threshold from the plurality of expert networks to process the image features; in a case where none of the confidence of the plurality of expert networks is greater than the preset threshold, selecting one or more expert networks with the highest confidence from the plurality of expert networks to process the image features.

3. The method according to claim 1 or 2, characterized in that, The feature extraction network, the gating network, and the plurality of expert networks are used to constitute a tamper detection model, and parameters in the tamper detection model are updated based on a first loss function and a second loss function during training; wherein the first loss function is obtained based on a real tampered region of training data and a predicted tampered region output by the tamper detection model, and the second loss function is obtained based on a real tampered type of the training data and a predicted tampered type output by a classification network, the classification network being used to predict the tampered type of the training data based on features output by the feature extraction network.

4. The method according to any one of claims 1 to 3, characterized in that, The tampered region to be detected in the first image is a region in which text content is tampered.

5. The method according to any one of claims 1 to 4, characterized in that, The method further comprises: identifying a semantic category corresponding to first text in a second image through a large language model; tampering with the first text in the second image based on the semantic category to obtain a tampered second image, the text in the tampered second image corresponding to the same semantic category before and after tampering, and the tampered second image being used as training data for a tamper detection model, the tamper detection model comprising the feature extraction network, the gating network, and the plurality of expert networks.

6. The method of claim 5, wherein, The tampering with the first text in the second image based on the semantic category comprises: generating second text corresponding to the same semantic category as the first text through the large language model based on the semantic category; replacing the first text with the second text to obtain the tampered second image.

7. The method of claim 6, wherein, The replacing the first text with the second text to obtain the tampered second image comprises: obtaining display information of the first text in the second image, the display information comprising at least one of a position, a size, a font format, and a background color of the first text; replacing the first text with the second text based on the display information to obtain the tampered second image.

8. The method of claim 5, wherein, The tampering of the first text in the second image based on the semantic category comprises: determining a third text in the second image or a third image based on the semantic category, the third text corresponding to the same semantic category as the first text, the third image being used as training data of the tampering detection model; replacing the first text with the third text to obtain the tampered second image.

9. The method according to any one of claims 1 to 8, characterized in that, The method further comprises: identifying a fourth text corresponding to a preset semantic category in the second image through a large language model; erasing the fourth text in the second image to obtain a tampered second image, the tampered second image being used as training data of a tampering detection model, the tampering detection model comprising the feature extraction network, the gating network, and the plurality of expert networks.

10. The method according to any one of claims 1 to 9, characterized in that, The method further comprises: weighting and fusing output results of the one or more expert networks based on confidence degrees of the one or more expert networks to obtain fused features; processing the fused features through a segmentation network to obtain a tampered region in the first image.

11. An image tampering detection apparatus characterized by comprising: comprises: a feature extraction network for extracting features of a first image to obtain image features, the first image being an image whose tampered region needs to be detected; a processing module for processing the image features through a gating network to obtain confidence degrees of a plurality of expert networks, the confidence degrees of the plurality of expert networks indicating a degree of credibility of each output result obtained by the plurality of expert networks processing the image features; The processing module is further configured to select one or more expert networks from the plurality of expert networks based on the confidence degrees of the plurality of expert networks to process the image features, and obtain output results of the one or more expert networks, the output results being used to determine a tampered region in the first image. The number of the selected one or more expert networks is determined based on the confidence degrees of the plurality of expert networks.

12. The apparatus of claim 11, wherein, The processing module is further configured to: select an expert network whose confidence degree is greater than a preset threshold from the plurality of expert networks to process the image features, if there is a confidence degree greater than the preset threshold in the confidence degrees of the plurality of expert networks; select one or more expert networks with the highest confidence degrees from the plurality of expert networks to process the image features, if none of the confidence degrees of the plurality of expert networks is greater than the preset threshold.

13. The apparatus of claim 11 or 12, wherein, The feature extraction network, the gating network, and the plurality of expert networks are used to constitute a tampering detection model, and parameters in the tampering detection model are updated based on a first loss function and a second loss function during training. The first loss function is obtained based on a real tampered region of training data and a predicted tampered region output by the tamper detection model, and the second loss function is obtained based on a real tamper type of the training data and a predicted tamper type output by a classification network, the classification network being configured to predict the tamper type of the training data based on features output by the feature extraction network.

14. The apparatus of any one of claims 11-13, wherein, The tampered region to be detected in the first image is a region in which text content is tampered.

15. The apparatus of any of claims 11-14, wherein, The processing module is further configured to: identify, by a large language model, a semantic category corresponding to the first text in the second image; tamper the first text in the second image based on the semantic category to obtain a tampered second image, the text in the tampered second image corresponding to the same semantic category before and after tampering, and the tampered second image being used as training data for a tamper detection model, the tamper detection model comprising the feature extraction network, the gating network, and the plurality of expert networks.

16. The apparatus of claim 15, wherein, The processing module is further configured to: generate, by the large language model, a second text based on the semantic category, the second text corresponding to the same semantic category as the first text; replace the first text with the second text to obtain the tampered second image.

17. The apparatus of claim 16, wherein, The processing module is further configured to: obtain display information of the first text in the second image, the display information comprising at least one of a position, a size, a font format, and a background color of the first text; replace the first text with the second text based on the display information to obtain the tampered second image.

18. The apparatus of claim 15, wherein, The processing module is further configured to: determine, based on the semantic category, a third text in the second image or a third image, the third text corresponding to the same semantic category as the first text, the third image being used as training data for the tamper detection model; replace the first text with the third text to obtain the tampered second image.

19. The apparatus of any of claims 11-18, wherein, The processing module is further configured to: identify, by a large language model, a fourth text corresponding to a preset semantic category in a second image; erase the fourth text in the second image to obtain a tampered second image, the tampered second image being used as training data for a tamper detection model, the tamper detection model comprising the feature extraction network, the gating network, and the plurality of expert networks.

20. The apparatus of any one of claims 11-18, wherein, The processing module is further configured to: weight and fuse output results of the one or more expert networks based on confidence degrees of the one or more expert networks to obtain fused features; process the fused features by a segmentation network to obtain a tampered region in the first image.

21. A computing device, comprising: A computing device comprising a memory and a processor; the memory stores code, and the processor is configured to execute the code, when the code is executed, the computing device performs the method of any one of claims 1 to 10.

22. A cluster of computing devices, characterized in that, A computing device comprising a memory and a processor; the memory stores code, and the processor is configured to execute the code, when the code is executed, the computing device performs the method of any one of claims 1 to 10. The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the operational steps of the method of any of claims 1 to 10.

23. A chip system, characterized by The chip system comprises a processor and a communication interface configured to communicate with a module outside the chip system, and the processor is configured to perform the method of any of claims 1 to 10.

24. A computer storage medium, comprising, The computer storage medium stores instructions which, when executed by a computer, cause the computer to implement the method of any of claims 1 to 10.

25. A computer program product, characterised in that, The computer program product stores instructions which, when executed by a computer, cause the computer to implement the method of any of claims 1 to 10.

Images