A multimodal large model driven academic paper image forgery detection method

Abstract

This invention relates to the fields of artificial intelligence and computer vision technology, and particularly to a multimodal, large-model-driven method, apparatus, device, and computer-readable storage medium for detecting forged academic paper images. The method includes: acquiring an academic image to be detected; analyzing the academic image using a first deep learning model to obtain visual indication information of the tampered region; performing visual semantic reasoning based on the academic image to be detected and the visual indication information of the tampered region using a second deep learning model to generate a reasonableness judgment result and semantic analysis information; and fusing the visual indication information of the tampered region and the semantic analysis information to generate a comprehensive detection report. By integrating multimodal visual detection and deep semantic understanding, the accuracy, robustness, interpretability, and interdisciplinary applicability of the detection are significantly improved.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and computer vision technology, and in particular to a method, apparatus, electronic device and computer-readable storage medium for detecting forged academic paper images driven by a multimodal large model. Background Technology

[0002] In the field of scientific research papers and academic publications, experimental images and diagrams serve as core scientific evidence, and their authenticity is the cornerstone of ensuring research integrity and the reliability of conclusions. In recent years, the rapid development of image editing and generation technologies has made it easier to forge or tamper with images, especially images generated based on complex data models. Their visual quality is highly similar to that of real images, making it difficult to effectively identify such highly concealed forgeries using traditional methods relying on human observation or simple algorithm analysis.

[0003] Current technologies primarily detect image tampering by analyzing pixel-level features, statistical patterns, or compression traces. For example, they might detect the continuity of image edges, the consistency of local illumination, or analyze abnormal patterns in the frequency domain to determine if editing is involved. However, these methods often perform poorly with images created by advanced generative models because such forged images are very similar to real images in terms of low-level features such as texture and noise distribution, making them difficult to distinguish effectively through conventional feature analysis. More importantly, existing methods typically only determine whether an image has been modified, failing to further understand the "intention behind the modification and its rationality within a scientific context." In academic practice, image optimization for legitimate purposes such as image sharpening and annotation is common and necessary, fundamentally different from malicious tampering for falsifying data or misleading conclusions. Existing detection technologies cannot combine the specific content and research logic of a paper to perform semantic analysis and judgment of the rationality of image modifications, thus easily leading to misjudgments in practical applications or failing to grasp the essence of forgery.

[0004] Therefore, there is an urgent need for a new technology that can integrate visual feature analysis and high-level semantic reasoning to solve the above problems. Summary of the Invention

[0005] This application aims to at least partially address one of the technical problems in the related art.

[0006] Therefore, the first objective of this application is to propose a multimodal large model-driven method for detecting academic paper image forgery, in order to solve the problems of existing technologies lacking robust and accurate automated detection capabilities and lacking mechanisms for deep semantic understanding and rationality judgment of image modification behavior.

[0007] The second objective of this application is to provide an apparatus.

[0008] The third objective of this application is to propose an electronic device.

[0009] The fourth objective of this application is to provide a computer-readable storage medium.

[0010] To achieve the above objectives, the first aspect of this application proposes a multimodal large model-driven method for detecting academic paper image forgery, comprising:

[0011] The academic image to be detected is acquired, and the academic image is analyzed using a first deep learning model to obtain visual indication information of the tampered area. Based on the academic image to be detected and the visual indication information of the tampered area, a second deep learning model is used to perform visual semantic reasoning to generate a reasonableness judgment result and semantic analysis information. The visual indication information of the tampered area is fused with the semantic analysis information to generate a comprehensive detection report.

[0012] Preferably, the first deep learning model is an image segmentation network based on an encoder-decoder structure, and its training process includes: Acquire sample data containing real academic images and their corresponding labeled tampered areas and tampering types to form a training sample set; A network model is constructed using an encoder-decoder structure with a high-resolution network as the backbone. The network model is trained using the training sample set. By optimizing the segmentation loss function, the model learns to identify and segment the tampered region from the dual-channel input until the model converges, thus obtaining the first deep learning model after training.

[0013] Preferably, the step of acquiring the academic image to be detected and analyzing the academic image using a first deep learning model to obtain visual indication information of the tampered region includes: Acquire the academic images to be detected; The academic image to be detected is simultaneously input into the RGB pixel domain and the frequency domain transform domain. The first deep learning model is used to extract and fuse features from the dual-channel input to generate visual indication information of the tampered area. The visual indication information of the tampered area includes a pixel-level mask map of the forged area, a confidence map, and a forgery type label.

[0014] Preferably, the second deep learning model is a multimodal large language model, and its training process includes: Based on a multimodal large language model as a pre-trained model; An image semantic understanding dataset is constructed based on original academic images, edited images, text descriptions of the edited operations, and semantic rationality labels. The low-rank adaptation method is used to freeze the preset parameters of the pre-trained model, inject a trainable low-rank matrix into the preset attention layer or feedforward network layer, train the low-rank matrix using the image semantic understanding dataset, and optimize the pre-trained model through the rationality classification loss function to obtain the trained second deep learning model.

[0015] Preferably, the step of using a second deep learning model to perform visual semantic reasoning based on the academic image to be detected and the visual indication information of the tampered area to generate a reasonableness judgment result and semantic analysis information includes: The second deep learning model is used to extract and infer visual semantic features from the original image, the fake region mask, and the locally tampered image generated by fusing the original image and the fake region mask. Visual consistency and semantic logic consistency scores are calculated, and reasonableness judgment results and semantic analysis information are generated. The reasonableness judgment results include semantic reasonableness scores, and the modification type judgment and reasonableness classification results are output.

[0016] Preferably, the semantic reasonableness score is calculated using a weighted summation method, and the calculation formula is as follows:

[0017] in, For visual consistency scoring, Score the semantic and logical consistency. These are the weighting coefficients.

[0018] Preferably, the step of fusing the visual indication information of the tampered area with the semantic analysis information to generate a comprehensive detection report includes: The forged region mask, the modification type determination, and the semantic rationality score are correlated and mapped to generate a visual image that highlights the tampered region and labels its modification category and rationality rating. Combined with the semantic analysis information, a structured integrity detection report is generated.

[0019] To achieve the above objectives, a second aspect of this application proposes a multimodal large model-driven device for detecting academic paper image forgery, comprising: The image forgery detection module acquires the academic image to be detected, analyzes the academic image using a first deep learning model, and obtains visual indication information of the tampered area. The visual semantic understanding and judgment module, based on the academic image to be detected and the visual indication information of the tampered area, uses a second deep learning model to perform visual semantic reasoning and generate a reasonableness judgment result and semantic analysis information. The result generation and output module fuses the visual indication information of the tampered area with the semantic analysis information to generate a comprehensive detection report.

[0020] To achieve the above objectives, a third aspect of this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method described in any of the preceding descriptions.

[0021] To achieve the above objectives, a fourth aspect of this application provides a computer-readable storage medium, comprising computer-executable instructions stored therein, which, when executed by a processor, are used to implement the method described in any of the above embodiments.

[0022] This application presents a multimodal, large-model-driven method for detecting academic paper image forgery. By integrating visual detection and semantic reasoning, it achieves an understanding of the intent behind academic image modification, improving the scientific rigor and accuracy of authenticity determination. By fusing pixel-domain, frequency-domain, and generation path features, it enhances the robustness and generalization ability of detecting high-fidelity AI-generated forged images. By introducing quantitative scoring and natural language interpretation, it achieves measurable, interpretable, and traceable detection results, significantly improving the practical value and credibility of the output. Furthermore, by creating a multidisciplinary semantic dataset of scientific image forgery and utilizing low-rank adaptation and other techniques for lightweight domain adaptation, the general-purpose large-model acquires a profound understanding of academic context. This solves the problems of insufficient adaptability and limited accuracy of general-purpose models on specialized images, achieving reliable detection across disciplines.

[0023] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0024] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of a first specific embodiment of a multimodal large model-driven method for detecting academic paper image forgery provided by the present invention; Figure 2 This is a structural block diagram of a multimodal large model-driven academic paper image forgery detection device provided in an embodiment of the present invention. Detailed Implementation

[0025] The core of this invention is to provide a multimodal large model-driven method, device, electronic device, and computer-readable storage medium for detecting academic paper image forgery. By integrating multimodal visual detection with deep semantic understanding, it significantly improves the accuracy, robustness, interpretability, and interdisciplinary applicability of the detection.

[0026] To enable those skilled in the art to better understand the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] Please refer to Figure 1 , Figure 1 The flowchart illustrates a first specific embodiment of a multimodal large model-driven method for detecting academic paper image forgery provided by this invention; the specific operation steps are as follows: Step S101: Obtain the academic image to be detected, analyze the academic image using the first deep learning model, and obtain visual indication information of the tampered area; Step S102: Based on the academic image to be detected and the visual indication information of the tampered area, use the second deep learning model to perform visual semantic reasoning and generate a reasonableness judgment result and semantic analysis information; Step S103: The visual indication information of the tampered area is fused with the semantic analysis information to generate a comprehensive detection report.

[0028] Based on the above embodiments, this embodiment will provide a detailed description of step S101: In one embodiment, the first deep learning model is an image segmentation network based on an encoder-decoder structure. Its training process includes: acquiring sample data containing real academic images and their corresponding labeled tampered regions and tampering types to form a training sample set; constructing a network model using an encoder-decoder structure with a high-resolution network as the backbone network; training the network model using the training sample set, and optimizing the segmentation loss function to enable the model to learn to identify and segment tampered regions from dual-channel input until the model converges, thus obtaining the first deep learning model after training.

[0029] Acquire the academic images to be detected; The academic image to be detected is simultaneously input into the RGB pixel domain and the frequency domain transform domain. The first deep learning model is used to extract and fuse features from the dual-channel input to generate visual indication information of the tampered area. The visual indication information of the tampered area includes a pixel-level mask map of the forged area, a confidence map, and a forgery type label.

[0030] Specifically, this embodiment is responsible for identifying suspicious edited or forged regions from the input academic images. To ensure the model's adaptability to various types of academic images (such as microscope images, gel maps, statistical graphs, device schematics, etc.), a segmentation and detection structure based on convolutional neural networks is adopted, combining spatial features and frequency domain information to locate forged regions.

[0031] Detection principle: During the editing or splicing process, forged images will produce anomalies in pixel distribution, edge transitions, noise textures, and compression features. Traditional visual segmentation networks can learn these latent features to predict and segment abnormal regions in the input image.

[0032] Network Structure: This method uses HR-Net (High Resolution Network) as the backbone network. The model structure includes: Encoder part: Convolutional layers are used to extract multi-scale features and capture local image artifacts and texture anomalies; Decoder part: Pixel-level positions of forged regions are reconstructed through upsampling and feature fusion; Skip connection mechanism: Fine-grained features and global context information are maintained, so that small-scale tampering can be effectively detected.

[0033] Input and Output: The model input consists of RGB (three primary color light mode) pixel domain input and DCT (discrete cosine transform) frequency domain input of the academic image. The output includes: forgery mask map M(x,y): identifies the region that may be tampered with; confidence map C(x,y): represents the model's confidence that each pixel belongs to the forgery region; forgery type label T: includes types such as splicing, cloning, local editing, and annotation overlay.

[0034] The test was conducted on an academic paper image dataset containing 7 major categories and 39 subcategories from different academic fields. The detection module was trained using publicly available and self-built academic image forgery samples. The module achieved an average IoU (Intersection over Union) of approximately 86.7% on the forgery region recognition task, and can accurately identify local forgery and minor editing behavior.

[0035] Based on the above embodiments, this embodiment will provide a detailed description of step S102: In one embodiment, the second deep learning model is a multimodal large language model, and its training process includes: Based on a multimodal large language model as a pre-trained model; An image semantic understanding dataset is constructed based on original academic images, edited images, text descriptions of the edited operations, and semantic rationality labels. The low-rank adaptation method is used to freeze the preset parameters of the pre-trained model, inject a trainable low-rank matrix into the preset attention layer or feedforward network layer, train the low-rank matrix using the image semantic understanding dataset, and optimize the pre-trained model by optimizing the joint loss function of semantic alignment and rationality classification to obtain the trained second deep learning model.

[0036] The second deep learning model is used to extract and infer visual semantic features from the original image, the fake region mask, and the locally tampered image generated by fusing the original image and the fake region mask. Visual consistency and semantic logic consistency scores are calculated, and reasonableness judgment results and semantic analysis information are generated. The reasonableness judgment results include semantic reasonableness scores, and the modification type judgment and reasonableness classification results are output.

[0037] The semantic reasonableness score is calculated using a weighted summation method, and the formula is as follows:

[0038] in, For visual consistency scoring, Score the semantic and logical consistency. These are the weighting coefficients.

[0039] Specifically, a multimodal large language model (MLLM) is used to achieve semantic-level understanding and reasonableness judgment of image modification behavior. Unlike traditional multimodal systems based on text-image alignment, this method in this embodiment only takes image information as input and does not rely on paper text, figure captions or descriptive information, but completes high semantic reasoning entirely through visual content.

[0040] Module input includes: original image The suspicious region mask M output by the detection module; the local image generated by fusing the original image and the mask. The system inputs the combined images into the visual encoder of a multimodal large model (such as LLaVA, Qwen-VL, GPT-4V, etc.) to extract high-dimensional semantic features. After lightweight fine-tuning (LoRA), the model can perform visual semantic reasoning tasks, including: determining the type of modification (brightness adjustment, annotation addition, splicing and tampering, etc.); determining the degree of impact of the modification on the overall semantics and scientific research logic; and providing a judgment on the rationality and risk level of the modification behavior.

[0041] In the feature extraction and semantic comparison process, the model first extracts the semantic embedding vectors of the original image and the modified region, and calculates the similarity between the two in the visual semantic space. If the differences are concentrated at the texture or color level, they are considered reasonable modifications; if the differences involve morphology, structure, or object replacement, the model proceeds to the next step of logical reasoning.

[0042] Semantic reasoning and logical judgment: The model automatically judges whether the modification behavior destroys the scientific semantics of the image through the contextual attention mechanism. For example, if the number of bands changes or the position is rearranged in a gel image, the model judges it as "tampering with experimental results"; if only blurring and denoising occur in the background area, it is judged as "reasonable optimization".

[0043] Reasonableness scoring mechanism: To quantify the model's judgment results, this method defines a reasonableness score (SR):

[0044] in, For visual consistency scoring, Score the semantic and logical consistency. SR is a weighting coefficient. When SR ≥ 0.7, it is considered a reasonable modification; when 0.4 ≤ SR < 0.7, it is considered a suspicious modification; when SR < 0.4, it is considered a forged modification.

[0045] Natural Language Interpretation Generation: The model automatically outputs highly readable semantic interpretation text, as shown in the following examples: "Local brightness enhancement and annotation addition were detected, which do not affect the main structure and are considered reasonable modifications." "Content replacement in the sample region was detected, which does not match the original form and is considered fraudulent modification." Based on the above embodiments, this embodiment will provide a detailed description of step S103: In one embodiment, the forged region mask, the modification type determination, and the semantic rationality score are correlated and mapped to generate a visual image that highlights the tampered region and labels its modification category and rationality rating. Combined with the semantic analysis information, a structured integrity detection report is generated.

[0046] This embodiment also briefly describes the model fine-tuning process, as follows: Data Sources: Real academic image samples from fields such as life sciences, physics, chemistry, and engineering are collected, covering types such as micrographs, experimental setup diagrams, bar charts, line graphs, and molecular structure diagrams. Forged Sample Generation: Various forged samples are constructed using manual editing (e.g., Photoshop) and mainstream image generation tools (Stable Diffusion, Midjourney, etc.), including partial splicing and cloning, partial repair and content replacement, reasonable editing (brightness enhancement, edge text legends, auxiliary boxes, auxiliary lines, etc.), and overall forgery (replacing experimental images, redrawing samples). Data Labeling: Each set of samples is labeled with "Original Image—Modified Image—Modification Type—Reasonableness Label," forming a high-quality training sample library.

[0047] A lightweight, high-efficiency fine-tuning method, LoRA (Low-Rank Adaptive), is employed, adjusting only the visual understanding and output layers to maintain the model's general visual semantic capabilities. A plausibility classification loss is used to improve the model's ability to classify fraudulent behavior. The loss function is... ,in, For the total loss, The regularization coefficient is . For model parameters, For regularization terms, The loss is for the reasonableness classification. After training, the model achieved a reasonableness classification accuracy of 93.2% on the test set, with a false positive rate controlled below 1.5%.

[0048] This embodiment provides a multimodal, large-model-driven method for detecting academic paper image forgery. By integrating visual detection and semantic reasoning, it achieves an understanding of the intent behind academic image modification, improving the scientific rigor and accuracy of authenticity determination. By fusing pixel-domain, frequency-domain, and generation path features, it enhances the robustness and generalization ability for detecting high-fidelity AI-generated forged images. By introducing quantitative scoring and natural language interpretation, it achieves measurable, interpretable, and traceable detection results, significantly improving the practical value and credibility of the output. By creating a multidisciplinary semantic dataset of scientific image forgery and utilizing low-rank adaptation and other techniques for lightweight domain adaptation, the general-purpose large-model gains a profound understanding of academic context. This solves the problems of insufficient adaptability and limited accuracy of general-purpose models on professional images, achieving reliable detection across disciplines.

[0049] Please refer to Figure 2 , Figure 2 This is a structural block diagram of a multimodal large model-driven academic paper image forgery detection device provided in an embodiment of the present invention; the specific device may include: The image forgery detection module 100 acquires the academic image to be detected, analyzes the academic image using a first deep learning model, and obtains visual indication information of the tampered area. The visual semantic understanding and judgment module 200, based on the academic image to be detected and the visual indication information of the tampered area, uses a second deep learning model to perform visual semantic reasoning and generate a reasonableness judgment result and semantic analysis information. The result generation and output module 300 fuses the visual indication information of the tampered area with the semantic analysis information to generate a comprehensive detection report.

[0050] This embodiment of the multimodal large model-driven academic paper image forgery detection device is used to implement the aforementioned multimodal large model-driven academic paper image forgery detection method. Therefore, the specific implementation of the multimodal large model-driven academic paper image forgery detection device can be found in the embodiment section of the multimodal large model-driven academic paper image forgery detection method above. For example, the image forgery detection module 100, the visual semantic understanding and judgment module 200, and the result generation and output module 300 are respectively used to implement steps S101, S102, and S103 in the aforementioned multimodal large model-driven academic paper image forgery detection method. Therefore, its specific implementation can be referred to the description of the corresponding embodiments, and will not be repeated here.

[0051] To implement the above embodiments, this application also proposes an electronic device, including: a processor and a memory communicatively connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the method provided in the foregoing embodiments.

[0052] To implement the above embodiments, this application also proposes a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods provided in the foregoing embodiments.

[0053] To implement the above embodiments, this application also proposes a computer program product, including a computer program that, when executed by a processor, implements the methods provided in the foregoing embodiments.

[0054] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0055] It should be noted that personal information collected from users should be used for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. Furthermore, such collection / sharing should only be conducted after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign an agreement / authorization that includes authorization of relevant user information before the user uses the function. In addition, any necessary steps must be taken to protect and safeguard access to such personal information data and ensure that others with access to personal information data comply with their privacy policies and procedures.

[0056] This application is intended to provide an implementation scheme for users to selectively prevent the use or access to their personal information data. Specifically, this disclosure is intended to provide hardware and / or software to prevent or block access to such personal information data. Once personal information data is no longer needed, risks can be minimized by restricting data collection and deleting data. Furthermore, where applicable, such personal information is de-identified to protect user privacy.

[0057] In the foregoing descriptions of the embodiments, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0058] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0059] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0060] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0061] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0062] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0063] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0064] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A multimodal large model-driven method for detecting forged academic paper images, characterized in that, include: The academic image to be detected is acquired, and the academic image is analyzed using a first deep learning model to obtain visual indication information of the tampered area. Based on the academic image to be detected and the visual indication information of the tampered area, a second deep learning model is used to perform visual semantic reasoning to generate a reasonableness judgment result and semantic analysis information. The visual indication information of the tampered area is fused with the semantic analysis information to generate a comprehensive detection report.

2. The method for detecting academic paper image forgery driven by a multimodal large model according to claim 1, characterized in that, The first deep learning model is an image segmentation network based on an encoder-decoder structure, and its training process includes: Acquire sample data containing real academic images and their corresponding labeled tampered areas and tampering types to form a training sample set; A network model is constructed using an encoder-decoder structure with a high-resolution network as the backbone. The network model is trained using the training sample set. By optimizing the segmentation loss function, the model learns to identify and segment the tampered region from the dual-channel input until the model converges, thus obtaining the first deep learning model after training.

3. The method for detecting academic paper image forgery driven by a multimodal large model according to claim 2, characterized in that, The process of acquiring the academic image to be detected and analyzing the academic image using a first deep learning model to obtain visual indication information of the tampered region includes: Acquire the academic images to be detected; The academic image to be detected is simultaneously input into the RGB pixel domain and the frequency domain transform domain. The first deep learning model is used to extract and fuse features from the dual-channel input to generate visual indication information of the tampered area. The visual indication information of the tampered area includes a pixel-level mask map of the forged area, a confidence map, and a forgery type label.

4. The method for detecting academic paper image forgery driven by a multimodal large model according to claim 3, characterized in that, The second deep learning model is a multimodal large language model, and its training process includes: Based on a multimodal large language model as a pre-trained model; An image semantic understanding dataset is constructed based on original academic images, edited images, text descriptions of the edited operations, and semantic rationality labels. The low-rank adaptation method is used to freeze the preset parameters of the pre-trained model, inject a trainable low-rank matrix into the preset attention layer or feedforward network layer, train the low-rank matrix using the image semantic understanding dataset, and optimize the pre-trained model through the rationality classification loss function to obtain the trained second deep learning model.

5. The method for detecting academic paper image forgery driven by a multimodal large model according to claim 4, characterized in that, The step of using a second deep learning model to perform visual semantic reasoning based on the academic image to be detected and the visual indication information of the tampered region, and generating a reasonableness judgment result and semantic analysis information, includes: The second deep learning model is used to extract and infer visual semantic features from the original image, the fake region mask, and the locally tampered image generated by fusing the original image and the fake region mask. Visual consistency and semantic logic consistency scores are calculated, and reasonableness judgment results and semantic analysis information are generated. The reasonableness judgment results include semantic reasonableness scores, and the modification type judgment and reasonableness classification results are output.

6. The method for detecting academic paper image forgery driven by a multimodal large model according to claim 5, characterized in that, The semantic reasonableness score is calculated using a weighted summation method, and the calculation formula is as follows: in, For visual consistency scoring, Score the semantic and logical consistency. These are the weighting coefficients.

7. The method for detecting academic paper image forgery driven by a multimodal large model according to claim 5, characterized in that, The step of fusing the visual indication information of the tampered area with the semantic analysis information to generate a comprehensive detection report includes: The forged region mask, the modification type determination, and the semantic rationality score are correlated and mapped to generate a visual image that highlights the tampered region and labels its modification category and rationality rating. Combined with the semantic analysis information, a structured integrity detection report is generated.

8. A multimodal large model-driven device for detecting academic paper image forgery, characterized in that, include: The image forgery detection module acquires the academic image to be detected, analyzes the academic image using a first deep learning model, and obtains visual indication information of the tampered area. The visual semantic understanding and judgment module, based on the academic image to be detected and the visual indication information of the tampered area, uses a second deep learning model to perform visual semantic reasoning and generate a reasonableness judgment result and semantic analysis information. The result generation and output module fuses the visual indication information of the tampered area with the semantic analysis information to generate a comprehensive detection report.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

Images