A text-to-image large model hallucination detection method, system and device for scene layout anomaly perception

Abstract

A scene layout anomaly perception text-to-image large model hallucination detection method, system and device, the method is: acquiring scene layout anomaly detection data and preprocessing, obtaining object instance set; Construct a scene layout anomaly perception model for text-to-image hallucination detection; Including double flow graph module, cross-modal structure alignment module and anomaly sorting module; Train the cross-modal structure alignment module and the anomaly sorting module to obtain the model weight file of the training iteration; Read the model weight file and perform hallucination detection on the text-to-image large model; The invention decouples the detection of hallucination into two dimensions of semantic context perception and geometric structure modeling of object instance through the innovative semantic-geometric double flow interaction and cross-modal alignment framework, the consistency between the two is inferred through the cross-modal interaction Transformer network, the mismatch signal of semantic and geometry is accurately captured, and the fine-grained hallucination such as unreasonable object attribute and mismatched object relationship in the generated image is accurately positioned.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of artificial intelligence and computer vision, and specifically relates to a method, system and device for detecting illusions in large-scale text-based images based on scene layout anomalies. Background Technology

[0002] In recent years, large-scale text-to-image (T2I) models, represented by Stable Diffusion 3 and FLUX.1, have made groundbreaking progress and are widely used in fields such as artistic creation, advertising design, and virtual scene construction. However, although the images generated by these models are incredibly realistic in terms of texture detail, they still frequently produce illusions that violate physical laws or logical principles when handling complex scenes. For example, the generated "dog" may have an abnormal number of limbs (extra legs), which is a layout anomaly at the object attribute level; or the model may incorrectly suspend a "cup" in mid-air, violating the laws of gravity, which is a layout anomaly at the object relationship level. Research on large-scale text-to-image model illusion detection methods for perceiving such scene layout anomalies has significant application value for improving the usability and feasibility of generated content and ensuring content security.

[0003] However, existing visual anomaly detection or hallucination assessment methods face significant challenges in handling such high-level scene layout anomalies. Unlike simple pixel noise or blurring, scene layout hallucinations often manifest as object-level logical inconsistencies. This means that while each object in an image may appear clear and correctly categorized individually, their spatial arrangement, physical form, and contextual relationships within the scene create logical conflicts. Existing technologies often lack the deep perception of the overall scene structure and the dependencies between objects, resulting in an inability to effectively distinguish between reasonable diversity and erroneous hallucinations when faced with generated images that are "textured but logically absurd," demonstrating significant limitations in adaptability.

[0004] Patent application CN202310311115 discloses a training method for an anomaly detection model. This method primarily optimizes the model by dividing image patches into positive and negative samples, calculating the contrast error between features, and combining this with a label update strategy, aiming to address the problem of excessive label noise in full-slice images. This approach performs well in handling local defects with significant texture differences in industrial quality inspection or medical images. However, this method essentially relies on the local texture statistical distribution of image patches for discrimination, focusing on the reconstruction of low-level visual features or the optimization of classification boundaries. Due to a lack of perception of the overall layout structure of the scene and the logical relationships between objects, when the Wensheng image model generates illusory images where "objects have normal appearance and texture but their spatial positions violate common sense," this method often fails to capture such high-level layout logic anomalies, leading to missed detections.

[0005] Patent application CN202510470058 discloses a zero-shot 2D anomaly detection method based on CLIP. This method leverages the generalization ability of the multimodal large model CLIP by designing learnable semantic prompts for normal and anomalous images, and extracting local image features using a diagonal attention mechanism. Anomaly detection is then determined using the cosine similarity between the features and the text. This approach effectively utilizes large-scale pre-trained semantic knowledge to achieve zero-shot detection. However, this method primarily relies on visual-semantic matching, focusing on determining the presence of image content while neglecting the rationality of the scene's spatial layout. Although it introduces local attention, the model still struggles to perceive the spatial topological relationships and physical dependencies of objects within the scene. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention aims to provide a method, system, and device for detecting illusions in large-scale generated images based on scene layout anomalies. Through an innovative semantic-geometric dual-flow interaction and cross-modal alignment framework, illusion detection is decoupled into two dimensions: semantic context awareness of object instances and geometric structure modeling. By inferring the consistency between these two dimensions through a cross-modal interactive Transformer network, the invention accurately captures semantic and geometric mismatch signals, achieving precise localization of fine-grained illusions such as unreasonable object attributes and mismatched object relationships in generated images. This effectively solves the bottlenecks of traditional methods in object-level structural understanding and logical consistency assessment. This invention fills the technological gap in detecting high-level logical and structural anomalies in generated images.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A method for detecting illusions in large-scale text-based images based on scene layout anomalies includes the following steps: Step 1: Obtain scene layout anomaly detection data and preprocess it to obtain a collection of object instances; Step 2: Based on the set of object instances obtained in Step 1, construct a scene layout anomaly perception model for the illusion detection of text-based graphs; the scene layout anomaly perception model includes a dual-flow graph module, a cross-modal structure alignment module, and an anomaly sorting module. Step 3: Train the cross-modal structure alignment module and anomaly sorting module in the scene layout anomaly perception model constructed in Step 2 to obtain the model weight file for training iterations. Step 4: Read the model weight file obtained in Step 3 and perform illusion detection on the large model of the text image.

[0008] The specific method of step 1 includes: Step 1.1: Obtain the candidate images to be detected generated by the Stable Diffusion (FLUX.1) model; Step 1.2: Read the candidate image to be detected obtained in Step 1.1 from the image processing library and normalize it; First, the color space of the candidate image to be detected is converted to RGB format. Then, the image resolution is uniformly adjusted to a uniform size and converted to floating-point Tensor format. At the same time, using parameters with a mean of [0.485, 0.456, 0.406] and a standard deviation of [0.229, 0.224, 0.225], the R, G, and B color channels of the image are standardized and normalized respectively, and the standardized image tensor is output. Step 1.3: Use the pre-trained panoramic segmentation model to perform object-level parsing on the image tensor after standardization in Step 1.2, and output the binary pixel mask and corresponding category label index of all object instances in the scene; Step 1.4: Filter the binary pixel mask output in Step 1.3, automatically remove tiny areas whose mask area is smaller than a set threshold, and combine each remaining binary pixel mask and its corresponding category label index into an object instance. All object instances constitute an object instance set.

[0009] The specific method for step 2 includes: Step 2.1: Construct a dual-flow graph module, including the construction of a geometric flow graph, the construction of a semantic flow graph, and the generation of a scene topology. This module is used to convert the set of object instances output in Step 1.4 into a dual-flow scene graph structure, which includes an initial geometric graph and an initial semantic graph. Step 2.2: Construct a cross-modal structure alignment module, including an intra-modal message passing network and a cross-modal interaction Transformer network, which is used to perform message interaction between the initial semantic graph and the initial geometry graph output in step 2.1, and to infer the layout consistency of object instances and their neighborhoods. Step 2.3: Construct an anomaly ranking module, including global feature aggregation and conditional modeling, comprehensive anomaly score calculation, and scene layout anomaly heatmap generation, which is used to quantify attribute anomalies and relationship anomalies of object instances.

[0010] The specific method of step 2.1 includes: Step 2.1.1, construct the geometric flow graph: For each object instance in the object instance set output in step 1.4, the centroid coordinates, minimum bounding rectangle coordinates, aspect ratio, and size percentage are calculated using the binary pixel mask in the object instance; at the same time, the rotation-invariant Hu moment feature is calculated and logarithmically transformed; the centroid coordinates, minimum bounding rectangle coordinates, aspect ratio, size percentage, and logarithmically transformed Hu moment feature are concatenated to obtain the initial geometric node feature vector, which is used to characterize the spatial position information and geometric attributes of the object; Step 2.1.2, construct the semantic flow graph: The standardized image tensor output from step 1.2 is input into the image encoder of the pre-trained multimodal large model CLIP to obtain the CLIP feature map. The mask average pooling strategy is adopted, and the CLIP feature map is aggregated using the binary pixel mask of each object instance in the object instance set. The visual features in the binary pixel mask region of the object instance are extracted, and the text embedding corresponding to the category label index of the object instance is combined to construct the initial semantic node feature vector. Step 2.1.3, Generate scene topology: Based on the initial geometric node feature vectors obtained in step 2.1.1, the Euclidean distance matrix between all object instances in the object instance set is calculated. The K-nearest neighbor algorithm is used to find the K nearest neighbor object instances for each object instance and establish edge connections, determining the adjacency matrix of the graph, i.e., the edge set E. Based on this, edge feature vectors are constructed: for each edge in the edge set E connecting object instance i and object instance j, the relative displacement, Euclidean distance, size ratio, and occlusion overlap rate between the initial geometric node feature vectors of object instance i and object instance j are calculated. The relative displacement, Euclidean distance, size ratio, and occlusion overlap rate are concatenated to form the initial geometric edge feature vector. Based on the initial semantic node feature vectors obtained in step 2.1.2, the appearance similarity and text embedding similarity between the initial semantic node feature vectors of object instance i and object instance j are calculated. The appearance similarity and text embedding similarity are used as the initial semantic edge feature vectors. Finally, two graph data with the same topological structure but different node attributes are output: an initial geometric graph composed of the initial geometric node feature vectors and the initial geometric edge feature vectors, and an initial semantic graph composed of the initial semantic node feature vectors and the initial semantic edge feature vectors.

[0011] The specific method of step 2.2 includes: Step 2.2.1: Construct an intramodal message passing network The intramodal message passing network consists of a projection layer and multiple stacked message passing blocks. Each message passing block sequentially includes a graph attention layer (GATv2), layer normalization, an edge update layer, and a multilayer perceptron (MLP) layer. First, the initial geometric node feature vectors and initial geometric edge feature vectors of the initial geometric graph output in step 2.1.3, along with the initial semantic node feature vectors and initial semantic edge feature vectors of the initial semantic graph, are mapped to the latent feature space through the projection layer. Then, they enter the stacked message passing blocks: in the graph attention layer (GATv2), the features of neighboring object instances are aggregated using the attention mechanism to update the features of the current node; then, they are normalized through layer normalization; finally, they enter the edge update layer. (Update) The edge features are re-encoded using the updated features of the two nodes connected by each edge to obtain the updated edge features; finally, the features are transformed through a multi-layer perceptron layer to output the updated intramodal aggregated geometric node feature vector and intramodal aggregated geometric edge feature vector, as well as the intramodal aggregated semantic node feature vector and intramodal aggregated semantic edge feature vector. Step 2.2.2: Construct a cross-modal interactive Transformer network The cross-modal interactive Transformer network consists of an embedding layer and multiple stacked cross-interaction blocks. Each cross-interaction block contains, in sequence, a self-attention layer, a layer normalization layer, a cross-attention layer, a feed-forward layer, and a multilayer perceptron (MLP) layer. First, the intra-modal aggregated geometric node feature vector and the intra-modal aggregated semantic node feature vector output from step 2.2.1 are processed through the embedding layer and then enter the cross-interaction block: in the self-attention layer... In the attention layer, long-distance dependencies within a single modality are captured. Bidirectional cross-query and edge feature bias injection are performed in the cross-attention layer: on the one hand, the geometric context is injected into the semantic representation using the intra-modal aggregated semantic node feature vector as the query vector and the intra-modal aggregated geometric node feature vector as the key and value vectors; on the other hand, the semantic context is injected into the geometric representation using the intra-modal aggregated geometric node feature vector as the query vector and the intra-modal aggregated semantic node feature vector as the key and value vectors. During this process, the intra-modal aggregated geometric edge feature vector and intra-modal aggregated semantic edge feature vector output in step 2.2.1 are used as structural biases and added to the attention score calculation in the cross-attention layer. After multiple iterations and mapping with a multilayer perceptron (MLP) layer, the final output is a depth-aligned refined semantic node feature vector and a refined geometric node feature vector. Simultaneously, the intra-modal aggregated geometric edge feature vector and the intra-modal aggregated semantic edge feature vector output in step 2.2.1 are retained as refined geometric edge feature vector and refined semantic edge feature vector.

[0012] The specific method of step 2.3 includes: Step 2.3.1, Global Feature Aggregation and Conditional Modeling First, the refined semantic node feature vectors and refined geometric node feature vectors of all object instances output in step 2.2.2 are aggregated and projected using average pooling and max pooling operations to obtain a global scene context vector. Then, a conditional probability model is constructed using a hybrid density network, which contains multiple Gaussian mixture components to predict the probability distribution of two conditions: the probability distribution of the refined semantic node feature vector given the refined geometric node feature vector and the global scene context vector; and the probability distribution of the refined geometric node feature vector given the refined semantic node feature vector and the global scene context vector. Step 2.3.2, calculate the comprehensive anomaly score. The refined semantic node feature vector and the refined geometric node feature vector output in step 2.2.2 are compared in the negative log-likelihood of the probability distribution predicted in step 2.3.1 to obtain the object-level anomaly score. Simultaneously, for each edge connecting two object instances in the graph, the probability density of the refined geometric edge feature vector between the two object instances is calculated using the refined semantic node feature vectors of these two object instances and the global scene context vector. The negative log-likelihood is then used to obtain the relation-level anomaly score. Finally, the object-level anomaly score and the relation-level anomaly score are fused using a weighted formula to obtain the comprehensive anomaly score for each object instance. Step 2.3.3: Generate a heatmap of scene layout anomalies. Based on the comprehensive anomaly score of each object instance calculated in step 2.3.2, the binary pixel mask of the object instance in the object instance set obtained in step 1.4 is used for projection; the pixel region belonging to each object instance in the object instance set is assigned the comprehensive anomaly score of the corresponding object instance to obtain the score map; the projected score map is Gaussian smoothed, and the final output is a scene layout anomaly heatmap with the same resolution as the candidate image to be detected input in step 1.1.

[0013] The specific method for step 3 includes: Step 3.1: Object-level likelihood loss and relation-level likelihood loss are used to optimize the propagation of the cross-modal structure alignment module and the anomaly sorting module in Step 2; where: Step 3.1.1: Object-level likelihood loss is used to measure whether its semantic attributes match its geometric shape. For each object instance i in the scene, predict the probability distribution of the refined semantic node feature vector given the refined geometric node feature vector and the global scene context vector, and predict the probability distribution of the refined geometric node feature vector given the refined semantic node feature vector and the global scene context vector. The object-level likelihood loss is shown in the following equation: in, and These represent the refined semantic node feature vector and the refined geometric node feature vector after cross-modal alignment, respectively. Represents the global scene context vector; and To balance the coefficients, the conditional probability density p(x|h) is modeled using a mixture density network, which assumes that the mixture density network follows a mixture distribution of K Gaussian components: in, , and These are the mixing coefficients, mean, and variance of the k-th Gaussian component, respectively, all predicted by the mixing density network based on the conditional input h. Step 3.1.2: Relational likelihood loss is used to measure the reasonableness of spatial relationships between object instances; for each pair of object instances (i, j) that are connected in the scene, the geometric relationship features of each pair of object instances (i, j) are predicted. The probability distribution under the semantic context of both parties; the relation-level likelihood loss is defined as follows: ; Step 3.1.3: The object-level likelihood loss obtained in Step 3.1.1 and the relation-level likelihood loss obtained in Step 3.1.2 are weighted and summed to obtain the total loss function, which is used for end-to-end backpropagation optimization. .

[0014] The specific method for step 4 includes: Step 4.1: Load the scene layout anomaly perception model structure, read the model weight file obtained in step 3, load the model weight file into the scene layout anomaly perception model structure, set the scene layout anomaly perception model as the inference model, and fix the parameters of the cross-modal structure alignment module and the anomaly sorting module structure. Step 4.2: The candidate images to be detected generated in Step 1.1 are processed sequentially through Steps 1.2, 1.3, and 1.4 to obtain a set of object instances. The dual-flow graph module constructs a dual-flow graph structure on the set of object instances output in Step 1, outputting an initial geometric graph and an initial semantic graph. The cross-modal structure alignment module uses this scene graph as a condition and outputs refined semantic node feature vectors and refined geometric node feature vectors after context alignment through bidirectional interactive reasoning of semantics and geometry. The anomaly ranking module calculates the object-level anomaly score of each object instance and the relation-level anomaly score between each pair of object instances based on a hybrid density network, and weighted and fused them to obtain the comprehensive anomaly score of each object instance. Finally, using the binary pixel mask of each object instance in the set of object instances output in Step 1, the corresponding comprehensive anomaly score is projected back to the pixel space and smoothed to output a scene layout anomaly heatmap.

[0015] This invention also provides a large-scale image illusion detection system for scene layout anomaly perception, comprising: The object instance collection acquisition module is used to acquire scene layout anomaly detection data and preprocess it to obtain an object instance collection. The scene layout anomaly perception model construction module is used to build a scene layout anomaly perception model based on a set of object instances for the illusion detection of text-based graphs; the scene layout anomaly perception model includes a dual-flow graph module, a cross-modal structure alignment module, and an anomaly sorting module. The scene layout anomaly perception model training module is used to train the cross-modal structure alignment module and anomaly sorting module in the scene layout anomaly perception model to obtain the model weight file for training iterations. The Wensheng Image Large Model Illusion Detection Module is used to read the model weight file and perform illusion detection on the Wensheng Image Large Model.

[0016] This invention also provides a large-scale image illusion detection device for scene layout anomaly perception, comprising: Memory: A computer program that stores the above-mentioned method for detecting illusions in large-scale text images based on scene layout anomalies, and is a computer-readable device; Processor: Used to implement the above-described method for detecting illusions in large-scale text-based images by perceiving scene layout anomalies when executing the computer program.

[0017] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the implementation of the described method for detecting illusions in large text-based images based on scene layout anomalies.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention uses the dual-stream scene graph structure of the initial geometric graph and the initial semantic graph constructed in step 2.1 to extract the feature vectors of the initial geometric nodes and the initial semantic nodes. This decouples the complex scene layout into two independent dimensions: what the object is and where the object is, and performs explicit modeling. Compared with traditional methods based on image blocks or pixel reconstruction, this object-level structured representation can ignore the interference of low-level texture noise and directly focus on the high-level scene layout logic, thus providing a solid data foundation for detecting the illusion of realistic textures but layouts that defy common sense.

[0019] 2. This invention establishes a deep semantic and geometric consistency reasoning mechanism through the cross-modal interactive Transformer network constructed in step 2.2.2 and its bidirectional cross-query and edge feature bias injection mechanism. By using each other as query vectors and key vectors, the model can determine whether the current semantic category matches its spatial form. At the same time, it explicitly introduces the intramodal aggregated edge feature vector as a structural bias to participate in attention calculation, so that the model is strictly constrained by the spatial topological relationship during reasoning. This effectively solves the bottleneck of existing technologies that are difficult to perceive the dependency relationship between objects and achieves accurate capture of hidden spatial structural anomalies.

[0020] 3. This invention achieves the inclusion of diverse normal scene layouts and the fine distinction of anomaly types through the conditional probability modeling based on hybrid density networks and the weighted fusion of object-level and relation-level anomaly scores in step 2.3. The hybrid density network, by predicting a multi-Gaussian distribution, allows the same object to have multiple reasonable spatial locations in a normal scene, reducing the false alarm rate. At the same time, it decouples the comprehensive anomaly score into object-level and relation-level anomaly scores, and uses step 2.3.3 to generate a scene layout anomaly heatmap, providing an interpretable feedback signal for the subsequent self-correction of the Wensheng graph large model.

[0021] In summary, this invention overcomes the limitations of existing technologies that focus only on low-level textures while neglecting high-level layout logic by constructing a semantic-geometric dual-stream scene graph structure and a cross-modal interactive Transformer network. Utilizing explicit decoupling of semantics and geometry and deep consistency reasoning, it accurately captures hidden spatial structural anomalies between object instances while ignoring texture noise interference. Furthermore, by combining conditional probability modeling with hybrid density networks, it not only achieves inclusiveness of normal scene diversity but also generates highly interpretable scene layout anomaly heatmaps, providing crucial technical support for the logical self-consistency evaluation and closed-loop self-correction of text-generated graph large models. Attached Figure Description

[0022] Figure 1 This is a flowchart of the implementation method of the present invention.

[0023] Figure 2 This is a diagram of the intramodal message passing network structure of the present invention.

[0024] Figure 3 This is a diagram of the cross-modal interactive network structure of the present invention.

[0025] Figure 4 This is a training flowchart of the present invention.

[0026] Figure 5 This is a flowchart of the textual image self-correction process based on hallucination detection according to the present invention.

[0027] Figure 6 This is a comparison chart of the hallucination detection effects of an embodiment of the present invention. Detailed Implementation

[0028] To make the implementation process and features of the present invention clearer and easier to understand, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] The core content of this invention includes: a dual-flow graph module, constructed through the decoupling of semantic and geometric flow graphs, respectively representing the identity information and spatial form of object instances in the scene. It extracts initial semantic node feature vectors using a pre-trained multimodal large model CLIP image encoder, and constructs initial geometric node feature vectors by combining the centroid coordinates, minimum bounding rectangle coordinates, aspect ratio, size percentage, and Hu moment features after logarithmic transformation of object instances, providing a structured scene representation for subsequent inference; and a cross-modal structure alignment module, utilizing an intramodal message passing network and a cross-modal interactive Transformer network to establish a deep interaction mechanism between the initial semantic graph and the initial geometric graph. This module employs a bidirectional cross-query strategy, that is, aggregating semantic nodes within the modality... Point feature vectors query geometric features to verify whether an object has a reasonable shape and position. At the same time, intramodal aggregated geometric node feature vectors query semantic features to verify whether the object at that position has the correct category attribute. Edge feature vectors are explicitly introduced as structural biases. This explicit alignment reasoning can accurately capture the mismatch signal between semantics and geometry, thereby discovering hidden structural illusions. The anomaly ranking module introduces a conditional probability modeling mechanism based on hybrid density networks. Considering the diversity of normal scene layouts, this module predicts the probability distribution that the refined node feature vectors should follow under the constraint of the global scene context vector. It also obtains anomaly scores by calculating the negative log-likelihood of the refined node feature vectors, realizing a refined measurement of the degree of structural illusion.

[0030] like Figure 1 As shown, a method for detecting illusions in large-scale text-based images based on scene layout anomalies includes the following steps: Step 1: Obtain scene layout anomaly detection data and preprocess it to obtain a collection of object instances; The specific method of step 1 includes: Step 1.1: Obtain the candidate images to be detected generated by the Stable Diffusion (FLUX.1) model as input data; Step 1.2: Read the candidate image to be detected obtained in Step 1.1 from the image processing library and normalize it; First, the color space of the candidate image to be detected is converted to RGB format. Then, the image resolution is uniformly adjusted to 3x640x640 and converted to floating-point Tensor format. At the same time, using parameters with mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225], the R, G, and B color channels of the image are standardized and normalized respectively, and the standardized image tensor is output. Step 1.3: Use the pre-trained panoramic segmentation model (Mask2Former is used in this example) to perform object-level parsing on the image tensor after standardization in Step 1.2, and output the binary pixel mask and corresponding category label index of all object instances in the scene. Step 1.4: Filter the binary pixel mask output in Step 1.3, automatically remove tiny areas whose mask area is smaller than a set threshold, and combine each remaining binary pixel mask and its corresponding category label index into an object instance. All object instances constitute an object instance set, which serves as the basic data for the subsequent dual flow graph module.

[0031] Step 2: Based on the set of object instances obtained in Step 1, construct a scene layout anomaly perception model for the illusion detection of text-based graphs; the scene layout anomaly perception model includes a dual-flow graph module, a cross-modal structure alignment module, and an anomaly sorting module. The specific method for step 2 includes: Step 2.1: Construct a dual-flow graph module, including the construction of a geometric flow graph, the construction of a semantic flow graph, and the generation of a scene topology. This module is used to convert the set of object instances output in Step 1.4 into a dual-flow scene graph structure, which includes an initial geometric graph and an initial semantic graph. Step 2.1.1, construct the geometric flow graph: To address the need for scene layout awareness, geometric attributes of the object instance set output in step 1.4 are specifically extracted. For each object instance in the object instance set, the centroid coordinates, minimum bounding rectangle coordinates, aspect ratio, and size percentage are calculated using the binary pixel mask in the object instance. Simultaneously, rotation-invariant Hu moment features are calculated and logarithmically transformed. The centroid coordinates, minimum bounding rectangle coordinates, aspect ratio, size percentage, and logarithmically transformed Hu moment features are concatenated to obtain an initial geometric node feature vector, which is used to characterize the spatial position information and geometric attributes of the object. Step 2.1.2, construct the semantic flow graph: The standardized image tensor output from step 1.2 is input into the image encoder of the pre-trained multimodal large model CLIP to obtain the CLIP feature map. A mask average pooling strategy is adopted, and the CLIP feature map is aggregated using the binary pixel mask of each object instance in the object instance set. Visual features within the binary pixel mask region of the object instance are extracted, and combined with the text embedding corresponding to the category label index of the object instance, an initial semantic node feature vector is constructed to represent "what the object is".

[0032] Step 2.1.3, Generate scene topology: Based on the initial geometric node feature vectors obtained in step 2.1.1, the Euclidean distance matrix between all object instances in the object instance set is calculated. The K-Nearest Neighbors (KNN, K=4 in this embodiment) algorithm is used to find the four nearest neighbor object instances for each object instance and establish edge connections, thereby determining the adjacency matrix of the graph, i.e., the edge set E. Further, edge feature vectors are constructed: for each edge in the edge set E connecting object instance i and object instance j, the relative displacement, Euclidean distance, size ratio, and occlusion overlap rate between the initial geometric node feature vectors of object instance i and object instance j are calculated. The relative displacement, Euclidean distance, size ratio, and occlusion overlap rate are concatenated to form the initial geometric edge feature vector. Based on the initial semantic node feature vector obtained in step 2.1.2, the appearance similarity and text embedding similarity between the initial semantic node feature vectors of object instance i and object instance j are calculated, and the appearance similarity and text embedding similarity are used as the initial semantic edge feature vector. Finally, two graph data with the same topology but different node attributes are output: the initial geometric graph composed of the initial geometric node feature vector and the initial geometric edge feature vector, and the initial semantic graph composed of the initial semantic node feature vector and the initial semantic edge feature vector.

[0033] Step 2.2: Construct a cross-modal structure alignment module, which includes an intra-modal message passing network and a cross-modal interaction Transformer network. It is a hybrid architecture model used for message interaction between the initial semantic graph and the initial geometry graph output in step 2.1, and to infer the layout consistency of object instances and their neighborhoods. Step 2.2.1: Construct an intramodal message passing network like Figure 2As shown, the intramodal message passing network consists of a projection layer and multiple stacked message passing blocks. Each message passing block sequentially includes a graph attention network (GATv2), layer normalization, edge update, and a multilayer perceptron (MLP) layer. First, the initial geometric node feature vectors and initial geometric edge feature vectors of the initial geometric graph output in step 2.1.3, along with the initial semantic node feature vectors and initial semantic edge feature vectors of the initial semantic graph, are mapped to the latent feature space through the projection layer. Then, they enter the stacked message passing blocks: in the graph attention network (GATv2), the features of neighboring object instances are aggregated using the attention mechanism to update the features of the current node; then, they are normalized through layer normalization; finally, they enter the edge update layer. (Update) The edge features are re-encoded using the updated features of the two nodes connected by each edge to obtain the updated edge features; finally, the features are transformed through a multi-layer perceptron layer; after the above processing, the updated intramodal aggregated geometric node feature vector and intramodal aggregated geometric edge feature vector, as well as the intramodal aggregated semantic node feature vector and intramodal aggregated semantic edge feature vector are output respectively. Step 2.2.2: Construct a cross-modal interactive Transformer network like Figure 3As shown, the cross-modal interaction Transformer network consists of an embedding layer and multiple stacked cross-interaction blocks. Each cross-interaction block sequentially includes a self-attention layer, layer normalization, cross-attention layer, feed-forward layer, and multilayer perceptron (MLP) layer. First, the intra-modal aggregated geometric node feature vectors and intra-modal aggregated semantic node feature vectors output from step 2.2.1 are processed through the embedding layer and then enter the cross-interaction block. In the self-attention layer, long-distance dependencies within a single modality are captured. In the cross-attention layer, bidirectional cross-query and edge feature bias injection are performed: on one hand, the intra-modal aggregated semantic node feature vector is used as the query vector, and the intra-modal aggregated geometric node feature vector is used as the key and value vectors, injecting geometric context into the semantic representation; on the other hand, the intra-modal aggregated geometric node feature vector is used as the query vector... Using the intramodal aggregated semantic node feature vector as the key vector and value vector, semantic context is injected into the geometric representation. In this process, the intramodal aggregated geometric edge feature vector and intramodal aggregated semantic edge feature vector output in step 2.2.1 are explicitly introduced as structural biases and added to the attention score calculation in the cross-attention layer to maintain the spatial structure constraint of the graph during cross-modal interaction. After multiple iterations and mapping with the multilayer perceptron (MLP) layer, the refined semantic node feature vector and refined geometric node feature vector are finally output after depth alignment. At the same time, the intramodal aggregated geometric edge feature vector and intramodal aggregated semantic edge feature vector output in step 2.2.1 are retained as the refined geometric edge feature vector and refined semantic edge feature vector.

[0034] Step 2.3: Construct an anomaly ranking module, including global feature aggregation and conditional modeling, comprehensive anomaly score calculation, and scene layout anomaly heatmap generation, used to quantify attribute anomalies and relationship anomalies of object instances; Step 2.3.1, Global Feature Aggregation and Conditional Modeling First, the refined semantic node feature vectors and refined geometric node feature vectors of all object instances output in step 2.2.2 are aggregated and projected using average pooling and max pooling operations to obtain a global scene context vector. Then, a conditional probability model is constructed using a hybrid density network, which contains multiple Gaussian mixture components to predict the probability distribution of two conditions: the probability distribution of the refined semantic node feature vector given the refined geometric node feature vector and the global scene context vector; and the probability distribution of the refined geometric node feature vector given the refined semantic node feature vector and the global scene context vector. Step 2.3.2, calculate the comprehensive anomaly score. The refined semantic node feature vector and the refined geometric node feature vector output in step 2.2.2 are compared in the negative log-likelihood of the probability distribution predicted in step 2.3.1 to obtain the object-level anomaly score. Simultaneously, for each edge connecting two object instances in the graph, the probability density of the refined geometric edge feature vector between the two object instances is calculated using the refined semantic node feature vectors of these two object instances and the global scene context vector. The negative log-likelihood is then used to obtain the relation-level anomaly score. Finally, the object-level anomaly score and the relation-level anomaly score are fused using a weighted formula to obtain the comprehensive anomaly score for each object instance. Step 2.3.3: Generate a heatmap of scene layout anomalies. Based on the comprehensive anomaly score of each object instance calculated in step 2.3.2, the binary pixel mask of the object instance in the object instance set obtained in step 1.4 is used for projection; the pixel region belonging to each object instance in the object instance set is assigned the comprehensive anomaly score of the corresponding object instance to obtain the score map; in order to obtain a smooth visual effect, the projected score map is Gaussian smoothed, and the final output is a scene layout anomaly heatmap with the same resolution as the candidate image to be detected input in step 1.1.

[0035] Step 3: Use the COCO 2017 dataset to train the cross-modal structure alignment module and anomaly ranking module in the scene layout anomaly perception model constructed in Step 2, obtaining the model weight file for training iterations; such as Figure 4 As shown, the training process of this invention specifically includes: first, acquiring candidate images to be detected from the normal scene training set, and obtaining the object instance set through step 1; then entering the model training stage, processing the input using the scene layout anomaly perception model to predict the probability distribution; next, calculating the Loss (i.e., the object-level likelihood loss and relation-level likelihood loss detailed in step 3.2) based on the prediction results and true features; finally, updating the parameters through backpropagation based on the calculated loss value, and iterating continuously to optimize the model weights. Specifically: Step 3.1: The scene layout anomaly perception model is implemented using the deep learning PyTorch framework. Only the cross-modal structure alignment module and the anomaly sorting module need to be trained. The AdamW optimizer is chosen to maintain the stability and regularization effect of parameter updates during training. The specific hyperparameter configuration is as follows: initial learning rate is set to 0.0001, weight decay coefficient is set to 0.0001 (1e-4), batch size is set to 32, and total training epochs are set to 30. To prevent gradient oscillations in the early stages of training, a learning rate preheating strategy is adopted, linearly increasing the learning rate in the first 5 epochs, and then gradually decreasing the learning rate using a cosine annealing strategy. The input image size is 640x640 pixels.

[0036] Step 3.2, the training objective of this invention is to maximize the log-likelihood probability of normal samples in the feature space, that is, to minimize the negative log-likelihood loss, including object-level likelihood loss and relation-level likelihood loss; The cross-modal structure alignment module and anomaly sorting module in step 2 are optimized for propagation using object-level likelihood loss and relation-level likelihood loss; where: Step 3.2.1: Object-level likelihood loss is used to measure whether its semantic attributes match its geometric shape. For each object instance i in the scene, predict the probability distribution of the refined semantic node feature vector given the refined geometric node feature vector and the global scene context vector, and predict the probability distribution of the refined geometric node feature vector given the refined semantic node feature vector and the global scene context vector. The object-level likelihood loss is shown in the following equation: in, and These represent the refined semantic node feature vector and the refined geometric node feature vector after cross-modal alignment, respectively. Represents the global scene context vector; and The balancing coefficient is 1.0. To accurately fit the complex feature distribution, the conditional probability density p(x|h) is modeled using a mixed density network, which assumes that the mixed density network follows a mixed distribution of K Gaussian components. in, , and These are the mixing coefficients, mean, and variance of the k-th Gaussian component, respectively, all predicted by the mixing density network based on the conditional input h. Step 3.2.2: Relational likelihood loss is used to measure the reasonableness of spatial relationships between object instances; for each pair of object instances (i, j) that are connected in the scene, the geometric relationship features of each pair of object instances (i, j) are predicted. The probability distribution under the semantic context of both parties; the relation-level likelihood loss is defined as follows: This loss forces the model to learn reasonable spatial interaction logic such as "people are usually above bicycles" and "cars are usually on the road".

[0037] Step 3.2.3: The object-level likelihood loss obtained in Step 3.2.1 and the relation-level likelihood loss obtained in Step 3.2.2 are weighted and summed to obtain the total loss function, which is used for end-to-end backpropagation optimization. In this embodiment, a weighting coefficient is set to balance the optimization efforts of object attributes and spatial relationships. =3.0, By minimizing this total loss, the model can gradually grasp the normal pattern of scene layout, thereby assigning higher anomaly scores to abnormal samples that deviate from this distribution during the inference phase.

[0038] Step 4: Read the model weight file obtained in Step 3 and perform illusion detection on the large model of the text image.

[0039] The specific method for step 4 includes: Step 4.1: Load the scene layout anomaly perception model structure, read the model weight file obtained in step 3, load the model weight file into the scene layout anomaly perception model structure, set the scene layout anomaly perception model as the inference model, fix the parameters of the cross-modal structure alignment module and the anomaly sorting module structure, and do not perform gradient updates.

[0040] Step 4.2: The candidate images to be detected generated in Step 1.1 are processed sequentially through Steps 1.2, 1.3, and 1.4 to obtain a set of object instances. The dual-flow graph module constructs a dual-flow graph structure on the set of object instances output in Step 1, outputting an initial geometric graph and an initial semantic graph. The cross-modal structure alignment module uses this scene graph as a condition and outputs refined semantic node feature vectors and refined geometric node feature vectors after context alignment through bidirectional interactive reasoning of semantics and geometry. The anomaly ranking module calculates the object-level anomaly score of each object instance and the relation-level anomaly score between each pair of object instances based on a hybrid density network, and weighted and fused them to obtain the comprehensive anomaly score of each object instance. Finally, using the binary pixel mask of each object instance in the set of object instances output in Step 1, the corresponding comprehensive anomaly score is projected back to the pixel space and smoothed to output a scene layout anomaly heatmap. In addition, this detection result can be used as a feedback signal. When a high-confidence anomaly is detected, the upstream textural graph model (such as Stable Diffusion) is controlled to resample and gradually generate a logically consistent final image.

[0041] This invention offers significant advantages: it proposes a large-scale image-based illusion detection method for scene layout anomalies, effectively identifying illusions with realistic textures but logical errors. This method significantly outperforms existing techniques based on pixel reconstruction or pure semantic matching in object-level illusion detection tasks. This method not only achieves more accurate judgment of object spatial relationships through an explicit cross-modal attention mechanism, supporting diverse anomaly type detection, but also outputs object-level scene layout anomaly heatmaps, clearly distinguishing whether anomalies originate from attribute errors or relationship errors, providing interpretable feedback for model optimization. Furthermore, as... Figure 5 As shown, based on the detection capabilities of this invention, a closed-loop system of generation-detection-self-correction can be constructed. In this system, the detection method of this invention acts as a discriminator. When a high-confidence hallucination is detected in the generated image, the method automatically rejects the sample and sends a feedback signal, triggering the large-scale AIGC image model to regenerate. This significantly improves the logical rationality and usability of AIGC-generated content without retraining the large-scale model.

[0042] This invention also provides a large-scale image illusion detection system for scene layout anomaly perception, comprising: The object instance collection acquisition module is used to obtain scene layout anomaly detection data and perform preprocessing in step 1 to obtain an object instance collection. The scene layout anomaly perception model construction module is used to construct a scene layout anomaly perception model based on the set of object instances obtained in step 1 in step 2, for the illusion detection of text-based graphs; the scene layout anomaly perception model includes a dual-flow graph module, a cross-modal structure alignment module, and an anomaly sorting module. The scene layout anomaly perception model training module is used to train the cross-modal structure alignment module and anomaly sorting module in the scene layout anomaly perception model constructed in step 2 in step 3, and obtain the model weight file for training iteration. The Wensheng Image Large Model Illusion Detection Module is used to read the model weight file obtained in step 3 in step 4 and perform illusion detection on the Wensheng Image Large Model.

[0043] This invention also provides a large-scale image illusion detection device for scene layout anomaly perception, comprising: Memory: A computer program that stores the above-mentioned method for detecting illusions in large-scale text images based on scene layout anomalies, and is a computer-readable device; Processor: Used to implement the above-described method for detecting illusions in large-scale text-based images by perceiving scene layout anomalies when executing the computer program.

[0044] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the implementation of the described method for detecting illusions in large text-based images based on scene layout anomalies.

[0045] Experimental Analysis 1. Experimental conditions The model training and inference of this invention and the representative methods for comparison are both implemented based on the deep learning framework PyTorch, and the experimental hardware environment is a server equipped with a single NVIDIA RTX 4090 graphics card.

[0046] 2. Experiment Content To verify the effectiveness of the proposed scene layout anomaly detection method, both the model of this invention and the representative methods used for comparison were trained on the COCO2017 dataset, and a test set containing various logical illusions was constructed. The test images were generated based on text prompts using advanced text-to-image models (including Stable Diffusion 3 and FLUX.1).

[0047] This invention selects two representative existing technologies as comparative baselines: UniAD represents a unified anomaly detection method based on feature reconstruction; PixOOD represents a method based on pixel-level out-of-distribution detection. For example... Figure 6 As shown, the ability of each method to locate anomalies in complex scenarios is evaluated through qualitative visualization comparison.

[0048] 3. Experimental Results Figure 6 This paper compares the detection performance of this invention with existing technologies on hallucinatory images generated by a large model of textual images. For the first input image, a "dog" is suspended in mid-air. Existing UniAD and PixOOD detection methods produce diffuse or fragmented noise, failing to accurately identify the abnormal object. In contrast, the scene layout anomaly heatmap generated by this invention accurately locates the key object instance of the "dog" suspended in mid-air, violating spatial logic; the heatmap edges are clear and the semantics are explicit. For the second input image, a "cow" on the left exhibits morphological distortion. Existing technologies struggle to distinguish between normal and abnormal cows, resulting in a chaotic heatmap distribution and false positives. This invention, utilizing semantic-geometric alignment reasoning, successfully identifies the abnormally shaped "cow" on the left without falsely reporting a normal object on the right. In summary, compared to existing technologies based on pixel or feature reconstruction, this invention can effectively distinguish and locate complex attribute errors and relationship confusions at the object instance level, generating a more accurate and interpretable anomaly heatmap.

Claims

1. A method for detecting hallucinations in large-scale text-based images based on scene layout anomalies, characterized in that, Includes the following steps: Step 1: Obtain scene layout anomaly detection data and preprocess it to obtain a collection of object instances; Step 2: Based on the set of object instances obtained in Step 1, construct a scene layout anomaly perception model for the illusion detection of text-based graphs; the scene layout anomaly perception model includes a dual-flow graph module, a cross-modal structure alignment module, and an anomaly sorting module. Step 3: Train the cross-modal structure alignment module and anomaly sorting module in the scene layout anomaly perception model constructed in Step 2 to obtain the model weight file for training iterations. Step 4: Read the model weight file obtained in Step 3 and perform illusion detection on the large model of the text image.

2. The method for detecting hallucinations in a large-scale text-based image based on scene layout anomalies according to claim 1, characterized in that, The specific method of step 1 includes: Step 1.1: Obtain the candidate images to be detected generated by the Stable Diffusion (FLUX.1) model; Step 1.2: Read the candidate image to be detected obtained in Step 1.1 from the image processing library and normalize it; First, the color space of the candidate image to be detected is converted to RGB format. Then, the image resolution is uniformly adjusted to a uniform size and converted to floating-point Tensor format. At the same time, using parameters with a mean of [0.485, 0.456, 0.406] and a standard deviation of [0.229, 0.224, 0.225], the R, G, and B color channels of the image are standardized and normalized respectively, and the standardized image tensor is output. Step 1.3: Use the pre-trained panoramic segmentation model to perform object-level parsing on the image tensor after standardization in Step 1.2, and output the binary pixel mask and corresponding category label index of all object instances in the scene; Step 1.4: Filter the binary pixel mask output in Step 1.3, automatically remove tiny areas whose mask area is smaller than a set threshold, and combine each remaining binary pixel mask and its corresponding category label index into an object instance. All object instances constitute an object instance set.

3. The method for detecting illusions in a large model of a text-based image based on scene layout anomalies according to claim 1, characterized in that, The specific method for step 2 includes: Step 2.1: Construct a dual-flow graph module, including the construction of a geometric flow graph, the construction of a semantic flow graph, and the generation of a scene topology. This module is used to convert the set of object instances output in Step 1.4 into a dual-flow scene graph structure, which includes an initial geometric graph and an initial semantic graph. Step 2.2: Construct a cross-modal structure alignment module, including an intra-modal message passing network and a cross-modal interaction Transformer network, which is used to perform message interaction between the initial semantic graph and the initial geometry graph output in step 2.1, and to infer the layout consistency of object instances and their neighborhoods. Step 2.3: Construct an anomaly ranking module, including global feature aggregation and conditional modeling, comprehensive anomaly score calculation, and scene layout anomaly heatmap generation, which is used to quantify attribute anomalies and relationship anomalies of object instances.

4. The method for detecting hallucinations in a large model of a text-based image based on scene layout anomalies according to claim 3, characterized in that, The specific method of step 2.1 includes: Step 2.1.1, construct the geometric flow graph: For each object instance in the object instance set output in step 1.4, the centroid coordinates, minimum bounding rectangle coordinates, aspect ratio, and size percentage are calculated using the binary pixel mask in the object instance; at the same time, the rotation-invariant Hu moment feature is calculated and logarithmically transformed; the centroid coordinates, minimum bounding rectangle coordinates, aspect ratio, size percentage, and logarithmically transformed Hu moment feature are concatenated to obtain the initial geometric node feature vector, which is used to characterize the spatial position information and geometric attributes of the object; Step 2.1.2, construct the semantic flow graph: The standardized image tensor output from step 1.2 is input into the image encoder of the pre-trained multimodal large model CLIP to obtain the CLIP feature map. The mask average pooling strategy is adopted, and the CLIP feature map is aggregated using the binary pixel mask of each object instance in the object instance set. The visual features in the binary pixel mask region of the object instance are extracted, and the text embedding corresponding to the category label index of the object instance is combined to construct the initial semantic node feature vector. Step 2.1.3, Generate scene topology: Based on the initial geometric node feature vectors obtained in step 2.1.1, the Euclidean distance matrix between all object instances in the object instance set is calculated. The K-nearest neighbor algorithm is used to find the K nearest neighbor object instances for each object instance and establish edge connections, determining the adjacency matrix of the graph, i.e., the edge set E. Based on this, edge feature vectors are constructed: for each edge in the edge set E connecting object instance i and object instance j, the relative displacement, Euclidean distance, size ratio, and occlusion overlap rate between the initial geometric node feature vectors of object instance i and object instance j are calculated. The relative displacement, Euclidean distance, size ratio, and occlusion overlap rate are concatenated to form the initial geometric edge feature vector. Based on the initial semantic node feature vectors obtained in step 2.1.2, the appearance similarity and text embedding similarity between the initial semantic node feature vectors of object instance i and object instance j are calculated. The appearance similarity and text embedding similarity are used as the initial semantic edge feature vectors. Finally, two graph data with the same topological structure but different node attributes are output: an initial geometric graph composed of the initial geometric node feature vectors and the initial geometric edge feature vectors, and an initial semantic graph composed of the initial semantic node feature vectors and the initial semantic edge feature vectors.

5. The method for detecting hallucinations in a large model of a text-based image based on scene layout anomalies according to claim 3, characterized in that, The specific method of step 2.2 includes: Step 2.2.1: Construct an intramodal message passing network The intramodal message passing network consists of a projection layer and multiple stacked message passing blocks. Each message passing block sequentially includes a graph attention network (GATv2), layer normalization, an edge update layer, and a multilayer perceptron (MLP) layer. First, the initial geometric node feature vectors and initial geometric edge feature vectors of the initial geometric graph output in step 2.1.3, along with the initial semantic node feature vectors and initial semantic edge feature vectors of the initial semantic graph, are mapped to the latent feature space through the projection layer. Then, they enter the stacked message passing blocks: in the graph attention network (GATv2), the features of neighboring object instances are aggregated using the attention mechanism to update the features of the current node; then, they are normalized through layer normalization; finally, they enter the edge update layer. (Update) The edge features are re-encoded using the updated features of the two nodes connected by each edge to obtain the updated edge features; finally, the features are transformed through a multi-layer perceptron layer to output the updated intramodal aggregated geometric node feature vector and intramodal aggregated geometric edge feature vector, as well as the intramodal aggregated semantic node feature vector and intramodal aggregated semantic edge feature vector. Step 2.2.2: Construct a cross-modal interactive Transformer network The cross-modal interactive Transformer network consists of an embedding layer and multiple stacked cross-interaction blocks. Each cross-interaction block contains, in sequence, a self-attention layer, a layer normalization layer, a cross-attention layer, a feed-forward layer, and a multilayer perceptron (MLP) layer. First, the intra-modal aggregated geometric node feature vector and the intra-modal aggregated semantic node feature vector output from step 2.2.1 are processed through the embedding layer and then enter the cross-interaction block: in the self-attention layer... In the attention layer, long-distance dependencies within a single modality are captured. Bidirectional cross-query and edge feature bias injection are performed in the cross-attention layer: on the one hand, the geometric context is injected into the semantic representation using the intra-modal aggregated semantic node feature vector as the query vector and the intra-modal aggregated geometric node feature vector as the key and value vectors; on the other hand, the semantic context is injected into the geometric representation using the intra-modal aggregated geometric node feature vector as the query vector and the intra-modal aggregated semantic node feature vector as the key and value vectors. During this process, the intra-modal aggregated geometric edge feature vector and intra-modal aggregated semantic edge feature vector output in step 2.2.1 are used as structural biases and added to the attention score calculation in the cross-attention layer. After multiple iterations and mapping with a multilayer perceptron (MLP) layer, the final output is a depth-aligned refined semantic node feature vector and a refined geometric node feature vector. Simultaneously, the intra-modal aggregated geometric edge feature vector and the intra-modal aggregated semantic edge feature vector output in step 2.2.1 are retained as refined geometric edge feature vector and refined semantic edge feature vector.

6. The method for detecting hallucinations in a large-scale text-based image based on scene layout anomalies according to claim 1, characterized in that, The specific method of step 2.3 includes: Step 2.3.1, Global Feature Aggregation and Conditional Modeling First, the refined semantic node feature vectors and refined geometric node feature vectors of all object instances output in step 2.2.2 are aggregated and projected using average pooling and max pooling operations to obtain a global scene context vector. Then, a conditional probability model is constructed using a hybrid density network, which contains multiple Gaussian mixture components to predict the probability distribution of two conditions: the probability distribution of the refined semantic node feature vector given the refined geometric node feature vector and the global scene context vector; and the probability distribution of the refined geometric node feature vector given the refined semantic node feature vector and the global scene context vector. Step 2.3.2, calculate the comprehensive anomaly score. The refined semantic node feature vector and the refined geometric node feature vector output in step 2.2.2 are compared in the negative log-likelihood of the probability distribution predicted in step 2.3.1 to obtain the object-level anomaly score. Simultaneously, for each edge connecting two object instances in the graph, the probability density of the refined geometric edge feature vector between the two object instances is calculated using the refined semantic node feature vectors of these two object instances and the global scene context vector. The negative log-likelihood is then used to obtain the relation-level anomaly score. Finally, the object-level anomaly score and the relation-level anomaly score are fused using a weighted formula to obtain the comprehensive anomaly score for each object instance. Step 2.3.3: Generate a heatmap of scene layout anomalies. Based on the comprehensive anomaly score of each object instance calculated in step 2.3.2, the binary pixel mask of the object instance in the object instance set obtained in step 1.4 is used for projection; the pixel region belonging to each object instance in the object instance set is assigned the comprehensive anomaly score of the corresponding object instance to obtain the score map; the projected score map is Gaussian smoothed, and the final output is a scene layout anomaly heatmap with the same resolution as the candidate image to be detected input in step 1.

1.

7. The method for detecting hallucinations in large-scale text-based images based on scene layout anomalies according to claim 1, characterized in that, The specific method for step 3 includes: Step 3.1: Object-level likelihood loss and relation-level likelihood loss are used to optimize the propagation of the cross-modal structure alignment module and anomaly sorting module in Step 2; where: Step 3.1.1: Object-level likelihood loss is used to measure whether its semantic attributes match its geometric shape. For each object instance i in the scene, predict the probability distribution of the refined semantic node feature vector given the refined geometric node feature vector and the global scene context vector, and predict the probability distribution of the refined geometric node feature vector given the refined semantic node feature vector and the global scene context vector. The object-level likelihood loss is shown in the following equation: in, and These represent the refined semantic node feature vector and the refined geometric node feature vector after cross-modal alignment, respectively. Represents the global scene context vector; and To balance the coefficients, the conditional probability density p(x|h) is modeled using a mixture density network, which assumes that the mixture density network follows a mixture distribution of K Gaussian components: in, , and These are the mixing coefficients, mean, and variance of the k-th Gaussian component, respectively, all predicted by the mixing density network based on the conditional input h. Step 3.1.2: Relational likelihood loss is used to measure the reasonableness of spatial relationships between object instances; for each pair of object instances (i, j) that are connected in the scene, the geometric relationship features of each pair of object instances (i, j) are predicted. The probability distribution under the semantic context of both parties; the relation-level likelihood loss is defined as follows: ； Step 3.1.3: The object-level likelihood loss obtained in Step 3.1.1 and the relation-level likelihood loss obtained in Step 3.1.2 are weighted and summed to obtain the total loss function, which is used for end-to-end backpropagation optimization. 。 8. The method for detecting hallucinations in a large model of a text-based image based on scene layout anomalies according to claim 1, characterized in that, The specific method for step 4 includes: Step 4.1: Load the scene layout anomaly perception model structure, read the model weight file obtained in step 3, load the model weight file into the scene layout anomaly perception model structure, set the scene layout anomaly perception model as the inference model, and fix the parameters of the cross-modal structure alignment module and the anomaly sorting module structure. Step 4.2: The candidate images to be detected generated in Step 1.1 are processed sequentially through Steps 1.2, 1.3, and 1.4 to obtain a set of object instances. The dual-flow graph module constructs a dual-flow graph structure on the set of object instances output in Step 1, outputting an initial geometric graph and an initial semantic graph. The cross-modal structure alignment module uses this scene graph as a condition and outputs refined semantic node feature vectors and refined geometric node feature vectors after context alignment through bidirectional interactive reasoning of semantics and geometry. The anomaly ranking module calculates the object-level anomaly score of each object instance and the relation-level anomaly score between each pair of object instances based on a hybrid density network, and weighted and fused them to obtain the comprehensive anomaly score of each object instance. Finally, using the binary pixel mask of each object instance in the set of object instances output in Step 1, the corresponding comprehensive anomaly score is projected back to the pixel space and smoothed to output a scene layout anomaly heatmap.

9. A large-scale image illusion detection system based on scene layout anomaly perception according to claim 1, characterized in that, include: The object instance collection acquisition module is used to acquire scene layout anomaly detection data and preprocess it to obtain an object instance collection. The scene layout anomaly perception model construction module is used to build a scene layout anomaly perception model based on a set of object instances for the illusion detection of text-based graphs; the scene layout anomaly perception model includes a dual-flow graph module, a cross-modal structure alignment module, and an anomaly sorting module. The scene layout anomaly perception model training module is used to train the cross-modal structure alignment module and anomaly sorting module in the scene layout anomaly perception model to obtain the model weight file for training iterations. The Wensheng Image Large Model Illusion Detection Module is used to read the model weight file and perform illusion detection on the Wensheng Image Large Model.

10. A scene layout anomaly detection device for large-scale image illusion detection, characterized in that, include: Memory: A computer program for detecting illusions in large-scale text images based on scene layout anomalies as described in any one of claims 1-8, and is a computer-readable device; Processor: configured to implement, when executing the computer program, a method for detecting illusions in a large model of a raw image based on scene layout anomalies as described in any one of claims 1-8.

Images