Patrol problem intelligent identification and data processing method and system based on multi-modal large model

Abstract

The invention discloses an intelligent inspection problem identification and data processing method and system based on a multi-modal large model, and relates to the technical field of intelligent inspection data processing, and the method comprises the steps: carrying out the preprocessing of multi-modal original data collected in the inspection process, and extracting the feature representation corresponding to each modal; inputting the feature representation into a cross-modal fusion network, and obtaining a fusion feature representation by constructing an inter-modal weight matrix; inputting the fusion feature representation into a pre-trained multi-modal large model for identification processing, adopting a multi-label classification algorithm to complete inspection problem type identification and calculate a problem risk score, and outputting a problem identification result; and constructing a correlation graph by taking a problem identification result as a node, analyzing the correlation strength between the problems by using a graph neural network, identifying a problem propagation path and an influence range, and generating a structured patrol report. According to the method, the accuracy of patrol problem identification, the scientificity of risk assessment and the structuralization and interpretability of result expression can be improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent processing technology for inspection and supervision data, and in particular to a method and system for intelligent identification and data processing of inspection and supervision issues based on a multimodal large model. Background Technology

[0002] With the continuous advancement of digital supervision and governance and the intelligent discipline inspection and supervision system, inspection and supervision work is gradually shifting from the traditional work mode that relies on human experience and post-event analysis to a technology-intensive mode characterized by data-driven, model-assisted, and intelligent analysis. Currently, inspection and supervision targets cover multi-source heterogeneous information such as business processes, financial data, meeting minutes, on-site images, video materials, and voice interviews, exhibiting obvious multimodal characteristics. Against this backdrop, how to uniformly model, collaboratively understand, and deeply analyze multimodal inspection data scattered across different carriers, time series, and semantic levels has become a key technical bottleneck restricting the efficiency of problem discovery and the accuracy of assessment. While existing technologies have made some progress in single-modal analysis, multi-source data aggregation, and preliminary correlation mining, they generally suffer from insufficient cross-modal semantic alignment capabilities, single problem identification dimensions, limited risk quantification levels, and a lack of systematic modeling of the relationships and propagation paths between problems. These limitations make it difficult to meet the integrated intelligent processing needs of "accurate problem identification—risk assessment—correlation analysis—rectification support" in complex inspection scenarios.

[0003] CN117612058A discloses a substation inspection method and system based on multimodal data fusion and augmented reality. This solution utilizes multimodal data acquisition methods such as video, image, and voice, combined with target detection, voice recognition, and equipment ledger matching, to achieve augmented reality display of equipment status queries and fault analysis results, thereby improving on-site inspection efficiency. While this technology achieves collaborative display of multimodal information and optimized human-computer interaction in engineering operation and maintenance scenarios, its multimodal fusion primarily serves equipment identification and information retrieval, lacking a unified modeling and reasoning mechanism for the deep semantic relationships of multimodal data. Furthermore, its analysis results mainly present the status of single equipment, without addressing the typological identification of inspection or patrol issues, the quantification of risk levels, or the correlation and propagation analysis between issues. This makes it difficult to apply to inspection and patrol business scenarios where problem discovery and responsibility assessment are the core objectives.

[0004] CN117688184A proposes a method and system for processing inspection clue data. This method extracts textual data from inspection reports, problem lists, rectification measures, news reports, and social media information, performs unified preprocessing and entity element extraction, and identifies correlations between different inspection cases based on association analysis algorithms, thereby improving the reliability of inspection clue processing. While this solution has some practical value in text-level entity extraction and case association analysis, its technical approach is still limited to text modal processing and does not consider the important role of unstructured data such as images, videos, and audio in identifying inspection problems. Furthermore, its association analysis is mostly based on static entity relationships, lacking the ability to quantitatively model the degree of problem risk and perform graph structure analysis of problem propagation paths and impact ranges, making it difficult to generate interpretable and traceable structured inspection analysis results.

[0005] In summary, existing technologies related to inspection and supervision generally suffer from insufficient multimodal data collaborative understanding capabilities, limited dimensions of problem identification results, and a lack of systematic support for risk assessment and problem correlation analysis. These shortcomings make it difficult to support the practical needs for panoramic, structured, and intelligent problem processing in complex inspection scenarios. To address these deficiencies, this invention proposes a method and system for intelligent identification and data processing of inspection and supervision problems based on a multimodal large-scale model. This method enables analysis of problem propagation paths and impact scope, thereby generating a structured inspection report that includes problem descriptions, evidence chains, and rectification suggestions. This effectively improves the accuracy, interpretability, and decision support capabilities of inspection and supervision problem identification. Summary of the Invention

[0006] The purpose of this section is to outline some aspects of the embodiments of the present invention and to briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section, as well as in the abstract and title of the present application, to avoid obscuring the purpose of this section, the abstract and title of the invention. Such simplifications or omissions shall not be used to limit the scope of the present invention.

[0007] In view of the shortcomings of existing inspection and supervision technologies, such as insufficient multimodal data collaborative modeling capabilities, single problem identification dimensions, lack of quantitative basis for risk assessment, and difficulty in systematically analyzing problem correlation and propagation paths, this invention is proposed.

[0008] Therefore, the problem to be solved by this invention is how to perform unified feature modeling and cross-modal fusion of heterogeneous data from multiple modalities such as images, videos, voice, and text in the context of inspection and patrol. By introducing a multimodal large model, the invention aims to achieve accurate identification and risk quantification assessment of inspection issues, and further analyze the propagation path and impact range based on the correlation between issues, thereby generating structured inspection results with interpretability and decision support capabilities.

[0009] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, embodiments of the present invention provide a method for intelligent identification and data processing of inspection and patrol issues based on a multimodal large model, comprising, The raw data of multiple modalities obtained from inspections and patrols are preprocessed to extract the feature representations corresponding to each modality. The feature representation is input into the constructed cross-modal fusion network, and the fused feature representation is obtained by calculating the inter-modal weight matrix; The fused feature representation is input into a pre-trained multimodal large model for identification processing. A multi-label classification algorithm is used to identify the patrol problem type label, calculate the problem risk score, and output the problem identification result. Construct an association graph with the problem identification results as nodes, analyze the association strength between nodes using a graph neural network model to identify the problem propagation path and scope of impact, and generate a structured inspection report.

[0010] Secondly, embodiments of the present invention provide an intelligent identification and data processing system for inspection and patrol issues based on a multimodal large model, which includes a preprocessing and feature extraction module for preprocessing the original data of multiple modalities obtained in the inspection and patrol, and extracting the feature representations corresponding to each modality; The cross-modal feature fusion module is used to input the feature representation into the constructed cross-modal fusion network and obtain the fused feature representation by calculating the inter-modal weight matrix; The multimodal problem identification and risk assessment module inputs the fused feature representation into a pre-trained multimodal large model for identification processing. It is used to identify patrol problem type labels through a multi-label classification algorithm, calculate problem risk scores, and output problem identification results. The correlation analysis and report generation module is used to construct a correlation graph with the problem identification results as nodes, analyze the correlation strength between nodes through a graph neural network model to identify the problem propagation path and scope of influence, and generate a structured inspection report.

[0011] Thirdly, embodiments of the present invention provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement any step of the above-described method for intelligent identification and data processing of inspection and patrol issues based on a multimodal large model.

[0012] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the above-described method for intelligent identification and data processing of patrol and inspection problems based on a multimodal large model.

[0013] Compared with existing technologies, the beneficial effects of this invention are as follows: By preprocessing and extracting features from multimodal raw data, standardized vector representation of heterogeneous inspection data and enhanced extraction of key discriminative information are achieved; by constructing a cross-modal fusion network and dynamically calculating the inter-modal weight matrix, complementary enhancement and conflict resolution of multi-source evidence are achieved, generating a joint representation rich in contextual semantics, enabling the system to simulate expert comprehensive judgment thinking and significantly improve the accuracy and robustness of problem identification; by constructing an association graph with problem identification results as nodes and applying a graph neural network model, a leap from discrete problem detection to systemic risk diagnosis is achieved, which can automatically reveal the problem propagation path, quantify the scope of impact, and generate a structured report integrating problem details, propagation chain, and rectification suggestions, thereby transforming inspection work from traditional manual experience-driven to a data-driven, quantifiable, and traceable intelligent decision-making model, comprehensively improving the efficiency, depth, and systematic nature of inspection work. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart of a method for intelligent identification and data processing of inspection and patrol issues based on a multimodal large model. Detailed Implementation

[0015] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0016] Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort should fall within the scope of protection of this invention.

[0017] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0018] As mentioned in the background section, existing inspection and supervision technologies generally suffer from insufficient collaborative modeling capabilities, reliance on single-modality or rule-based strategies for problem identification, lack of quantitative basis for risk assessment, and difficulty in systematically analyzing problem relationships and propagation paths when dealing with multimodal data such as images, videos, audio, and text. These issues make it difficult to meet the practical needs for accurate problem identification and comprehensive judgment in complex inspection scenarios. To address these problems, this invention provides an intelligent problem identification and data processing method for inspection and supervision based on a multimodal large-scale model.

[0019] Reference Figure 1 , Figure 1 This is a flowchart illustrating an intelligent identification and data processing method for inspection and patrol issues based on a multimodal large model, according to an embodiment of the present invention. Figure 1 As shown, a method for intelligent identification and data processing of inspection and patrol issues based on a multimodal large model includes: S1: Preprocess the raw data of multiple modalities obtained during inspections and investigations, and extract the feature representations corresponding to each modality; S2: Input the feature representation into the cross-modal fusion network, and obtain the fused feature representation by calculating the inter-modal weight matrix; S3: Input the fused feature representation into the pre-trained multimodal large model for recognition processing, identify the patrol problem type label through a multi-label classification algorithm, calculate the problem risk score, and output the problem recognition result; S4: Construct a correlation graph with problem identification results as nodes, analyze the correlation strength between nodes through a graph neural network model to identify the problem propagation path and scope of impact, and generate a structured inspection report.

[0020] In this embodiment of the application, step S1 includes: Specifically, the process involves acquiring raw data in multiple modalities generated during inspections and audits, classifying document data according to file format, including document data, image data, and audio data; and storing PDF documents, Word documents, and Excel spreadsheets into the document dataset. The image data is categorized according to the shooting scene, and the on-site environment photos, equipment close-up photos, and evidence annotation photos are stored in the image dataset respectively. The audio data was segmented according to recording duration, and the sampling rate of each segment was uniformly converted to 16kHz to generate an audio dataset. ,in This represents the i-th audio segment.

[0021] Furthermore, regarding the document dataset... The system extracts text content from various documents, converts PDF documents into editable text using OCR (Optical Character Recognition) technology, and directly reads the text content from Word documents and Excel spreadsheets to generate raw text data in a uniform format. ; For raw text data Sentence segmentation was performed, using periods, question marks, and exclamation marks as delimiters to separate the original text data. Segmented into sentence sequence ,in Represents the i-th sentence; for the sentence sequence Each sentence tᵢ in the text is segmented into Chinese words using a hybrid segmentation method that combines dictionary matching and statistical models to generate a word sequence. ,in Sentence The j-th word in the word sequence; Named entity recognition is performed, and person names are labeled using the BiLSTM-CRF model. Place Name Entities Institutional Entities and question type entity Generate entity annotation results .

[0022] It's important to note that the BiLSTM-CRF model used for named entity recognition is built upon a mature sequence labeling framework. The core of this framework is a Bidirectional Long Short-Term Memory (BiLSTM) network, which processes text sequences in parallel using two independent forward and backward LSTM layers, effectively capturing the bidirectional semantic dependencies of words within their context. The Conditional Random Field (CRF) layer acts as the top global inference module. It learns transition rules between labels (e.g., a label starting with a person's name should typically be followed by a label within that person's name, rather than a label starting with a place name), globally optimizing the local predictions of the BiLSTM output to ensure that the final generated entity label sequence is the most reasonable and coherent overall. Therefore, this model is essentially a combination of "bidirectional contextual feature extraction" and "global sequence constraint decoding." The BERT language model is built upon the encoder part of the Transformer architecture. The Transformer relies entirely on a self-attention mechanism, dynamically calculating the association strength between all elements in the input sequence, thus achieving efficient long-distance dependency modeling and deep semantic understanding. By pre-training on two core tasks—masked language modeling and next-sentence prediction—on a massive corpus, BERT enables the model to learn deep and general language representation capabilities. This results in sentence-level semantic vectors (such as 768-dimensional vectors) that are rich in lexical, syntactic, and discourse-level information.

[0023] Furthermore, the sentence sequence Input the pre-trained BERT language model to obtain each sentence. Sentence-level semantic vectors 768 represents the hidden layer dimension of the BERT model; entity annotation results Vectorize the various entities in the data, including personal name entities. Place Name Entities Institutional Entities and question type entity Mapping each onto the entity vector space to generate entity vector matrices Where n is the number of entities, For entity vector dimensions; sentence-level semantic vectors The entity vector matrix in the corresponding sentence Perform concatenation operations to generate text feature vectors that integrate sentence semantics and entity information. ; text feature vector Normalization is performed using the L2 norm normalization method to generate standardized text feature representations. Text feature representation As a feature output of the document modality.

[0024] Specifically, for image datasets The images in the dataset are standardized in size, and the resolution of all images is uniformly adjusted to 512×512 pixels. Adaptive histogram equalization is then applied to the image brightness and contrast to generate a preprocessed image dataset. ;Preprocessed image dataset Inputting into a ResNet-50 convolutional neural network extracts global feature representations of the image, specifically feature vectors extracted before the last fully connected layer of the ResNet-50. eigenvectors Includes overall semantic information of the images; preprocessed image dataset Simultaneously, a multi-scale feature extraction module is input to extract feature maps from the 2nd, 3rd, and 4th convolutional blocks of ResNet-50. , and Feature map Capture detailed texture information of the image, feature map Capture mid-level semantic information of the image, feature map Capture high-level abstract information of the image.

[0025] Furthermore, regarding the feature map Feature map and feature map Spatial pyramid pooling is performed to pool feature maps of different sizes into feature vectors of three scales: 1×1, 2×2, and 4×4, generating a multi-scale feature vector set. ; Set of multi-scale feature vectors The feature vectors at each scale are flattened and concatenated to generate a multi-scale fused vector. ,in The concatenated vector dimension; the global feature vector is calculated using the attention weight calculation module. With multi-scale fusion vector Attention weights between and Attention weight and Normalization using the softmax function satisfies ; global feature vector and multi-scale fusion vector According to attention weight and Weighted fusion is performed to generate the final visual feature vector. ; For visual feature vectors The data is aggregated to generate visual feature representations. Visual feature representation As a feature output of the image modality.

[0026] Furthermore, regarding audio datasets Each audio data segment Pre-emphasis filtering is performed using a first-order high-pass filter. The audio signal is filtered to enhance high-frequency components, generating a pre-emphasized audio signal. For pre-emphasized audio signals Perform frame segmentation, setting the frame length to 25ms and the frame shift to 10ms, to divide the continuous audio signal into overlapping short frame sequences. ,in This represents the j-th frame of audio data, where L is the total number of frames; for each frame of audio data The Hamming window function is used for windowing to reduce spectral leakage and generate windowed frame data. .

[0027] Specifically, for windowed frame data Perform a Fast Fourier Transform (FFT) to calculate the frequency domain representation and extract the Mel-frequency cepstral coefficients (MFCC) eigenvectors. MFCC eigenvectors Includes 13-dimensional Mel-Cepstral coefficients; calculates windowed frame data. First-order difference and second-order difference MFCC feature vector First-order difference and second-order difference The features are concatenated to generate a 39-dimensional enhanced MFCC feature vector. ; Enhance MFCC eigenvectors Perform time-series arrangement to generate time-series feature matrix .

[0028] Furthermore, the time series feature matrix The input is a bidirectional long short-term memory (BSSM) network. The forward LSTM layer of the BSSM network processes the temporal information from frame 1 to frame L, and the backward LSTM layer of the BSSM network processes the temporal information from frame L to frame 1, extracting the context-dependent features of the audio. The hidden state vector is extracted from the last time step of the BSSM network. Hidden state vector As a feature of speech content For pre-emphasized audio signals Short-time energy and short-time zero-crossing rate are calculated, where short-time energy reflects the amplitude variation of the audio signal, and short-time zero-crossing rate reflects the frequency characteristics of the audio signal. The statistical characteristics of short-time energy and short-time zero-crossing rate, including mean, variance, and maximum value, are concatenated to generate an environmental sound feature vector. ; to feature speech content With environmental sound feature vector The features are concatenated to generate a comprehensive audio feature vector. The comprehensive audio feature vector F_audio_i is summarized to generate an audio feature representation. Audio feature representation As a feature output of the audio modality.

[0029] It's important to note that the Bidirectional Long Short-Term Memory (LSTM) network is a specific recurrent neural network architecture. Its design aims to address the problem that traditional unidirectional networks cannot utilize "future" information. This network consists of two LSTM layers operating in opposite directions: the forward LSTM processes the sequence chronologically, encoding historical information; the backward LSTM processes it in reverse order, encoding future information. For each time point in the sequence (e.g., an audio frame), the hidden states from both directions are concatenated and fused to obtain a feature representation containing complete contextual information.

[0030] Furthermore, representing text features Visual feature representation and audio feature representation Dimension alignment is performed by mapping feature vectors of different dimensions to a unified feature space through fully connected layers. Batch normalization is then applied to the mapped modal feature representations, calculating the mean and standard deviation of each feature dimension to standardize and generate a uniformly formatted set of feature representations. Text feature representation Contains N text feature vectors, visual feature representation It contains M visual feature vectors, and the audio feature representation is... It contains P audio feature vectors.

[0031] In this embodiment of the application, step S2 includes: Specifically, the set of received feature representations A cross-modal fusion network architecture is constructed, which includes an intra-modal self-attention layer, a cross-modal interaction layer, and a fusion output layer. Hyperparameters of the cross-modal fusion network architecture are set to control the network's expressive power and generalization performance, including the number of attention heads. Hidden layer dimension and dropout rate .

[0032] Furthermore, the feature representation set The input to the intra-modal self-attention layer of the cross-modal fusion network architecture generates multimodal intra-modal augmentation features, which include text intra-modal augmentation features, visual intra-modal augmentation features, and audio intra-modal augmentation features, specifically including: Representing text features An in-modal self-attention layer is used to handle text feature representations containing N text feature vectors. The query matrix is ​​generated through a linear transformation layer. Key matrix Sum matrix Calculate the query matrix AND key matrix The dot product of the transpose is used to generate the attention score matrix; each row of the text modality attention score matrix is ​​mechanically normalized using the softmax function to generate the attention weight matrix; the attention weight matrix is ​​multiplied by the value matrix to generate the text modality enhancement features, which are then processed through residual connections and layer normalization to generate the text modality enhancement features; using the same processing flow, visual feature representations are input into the visual modality self-attention layer to generate visual modality enhancement features; audio feature representations are input into the audio modality self-attention layer to generate audio modality enhancement features.

[0033] It should be noted that the elements in the text modality attention weight matrix represent the degree of attention of the i-th text feature vector to the j-th text feature vector; the text modality augmentation features integrate the contextual dependencies between the text feature vectors in the text feature representation; the text modality augmentation features, visual modality augmentation features, and audio modality augmentation features respectively contain the contextual semantic relationships within their respective modalities.

[0034] Furthermore, based on the multimodal intramodal enhancement features, a bidirectional interaction mechanism is used to construct a cross-modal interaction module between different modalities, calculate the cross-modal attention score matrix, and generate comprehensive cross-modal features for each modality, specifically including: A text-visual cross-modal interaction module is constructed, using text-modal augmented features as query vectors and visual-modal augmented features as key vectors. A cross-modal query matrix, a cross-modal key matrix, and a cross-modal value matrix are generated through linear transformation. The dot product of the cross-modal query matrix and the transpose of the cross-modal key matrix is ​​calculated to generate a text-visual attention score matrix, where each element represents the semantic correlation between the i-th text feature vector and the j-th visual feature vector. Softmax normalization is applied to each row of the text-visual attention score matrix to generate a text-to-visual attention weight matrix. The text-to-visual attention weight matrix is ​​multiplied by the cross-modal value matrix v to generate the text-side cross-modal interaction feature, where the text-side cross-modal interaction feature F_text_cross represents the attention information of text features to visual features. A visual-text reverse cross-modal interaction module is constructed, using visual intra-modal enhancement features as query vectors and text intra-modal enhancement features as key-value vectors; a reverse cross-modal query matrix, a reverse cross-modal key matrix, and a reverse cross-modal value matrix are generated; a visual-text attention score matrix is ​​calculated, and softmax normalization is applied to generate a visual-to-text attention weight matrix; cross-modal interaction features on the visual side are calculated, where the cross-modal interaction features on the visual side represent the attention information of visual features to text features; Using the same bidirectional interaction mechanism, a text-audio cross-modal interaction module is constructed to generate text-audio interaction features on the text side and audio-text interaction features on the audio side. A visual-audio cross-modal interaction module is also constructed to generate visual-audio interaction features on the visual side and audio-visual interaction features on the audio side. For each modality, the cross-modal interaction features from other modalities are summarized. For the text modality, the text-side cross-modal interaction features are concatenated with the text-audio interaction features on the text side and dimensionality is reduced using a linear layer to generate comprehensive text cross-modal features. For the visual modality, the visual-side cross-modal interaction features are concatenated with the visual-audio interaction features on the visual side and dimensionality is reduced to generate comprehensive visual cross-modal features. For the audio modality, the audio-text interaction features on the audio side are concatenated with the audio-visual interaction features on the audio side and dimensionality is reduced to generate comprehensive audio cross-modal features.

[0035] Specifically, global average pooling is performed on the enhanced features within the multimodal domain to generate global representation vectors for each modality. These global representation vectors are then concatenated to generate a global modal representation vector. The global modal representation vector is input into a modal gating network to calculate the modal importance score vector. This vector is then normalized to generate a modal weight vector, and an intermodal weight matrix is ​​constructed. A cross-modal interaction strength index is calculated, and the intermodal weight matrix is ​​dynamically adjusted based on this index to generate the final intermodal weight matrix. The cross-modal interaction strength index is obtained by weighted summation of the maximum values ​​of the text-visual attention score matrix, the text-audio attention score matrix, and the visual-audio attention score matrix.

[0036] It should be noted that the modal gating network consists of two fully connected layers. The output dimension of the first fully connected layer is... The activation function is ReLU, the output dimension of the second fully connected layer is 3, and the activation function is sigmoid; modality importance score vector. ,in , and These represent the importance scores for the text modality, visual modality, and audio modality, respectively, with values ​​ranging from (0, 1); the inter-modal weight matrix... Intermodal weight matrix diagonal elements Set as the corresponding weight values ​​in the normalized modal weight vector, off-diagonal elements Set as , representing the interaction weight between mode i and mode j; the inter-modal weight matrix reflects the relative importance of each mode in the current inspection data and the synergistic relationship between modes.

[0037] Preferably, when the cross-modal interaction strength index is greater than a preset threshold, the weight value of the off-diagonal elements is increased, and the adjusted off-diagonal elements are calculated as follows: Where λ is the enhancement coefficient, with a value of 0.2; when the cross-modal interaction strength index is less than the preset threshold, the weight value of the off-diagonal elements is reduced, and the adjusted off-diagonal elements are calculated as follows. .

[0038] Furthermore, the intra-modal enhancement features of each modality are residually fused with the corresponding comprehensive cross-modal features, compressed into a fixed-length vector representation by global average pooling, and then weighted according to the inter-modal weight matrix to generate a fused feature vector. The fused feature vector is then input into a feedforward neural network for nonlinear transformation to generate a fused feature representation.

[0039] It should be noted that residual fusion preserves intramodal information while introducing cross-modal interaction information. The fused feature vector is input into a feedforward neural network (FFN) for nonlinear transformation. The FFN contains two fully connected layers. The first fully connected layer has an output dimension of 4d_h and uses GELU as the activation function. The second fully connected layer has an output dimension of... The final fused feature representation is generated using a feedforward neural network (FFN). fusion feature representation It integrates semantic information from text, visual, and audio modalities, as well as the interaction relationships between these modalities.

[0040] Furthermore, the fused feature representation is normalized, a confidence score is calculated, and the normalized fused feature representation, inter-modal weight matrix, and confidence score are encapsulated and output. The calculation formula for the normalization is as follows: ,in Representation of fusion features L2 norm; confidence score It is obtained by calculating the weighted average of the cosine similarity between the fused feature representation and the original features of each modality.

[0041] In this embodiment of the application, step S3 includes: Specifically, the fused feature representation is input into the pre-trained multimodal large model, and the output is a deep semantic encoded feature. The multimodal large model is built based on the Transformer architecture. The multimodal large model includes multiple encoder layers, each of which includes a multi-head self-attention sub-layer and a feedforward neural network sub-layer. The encoder layer has 12 layers, the hidden layer dimension of the multimodal large model is 768, and the number of attention heads is 12. Here, the number of attention heads is independent of the 8 heads in the S2 cross-modal fusion network, belonging to different network modules. Specifically, it includes: The normalized fused feature representation is then dimension-mapped through a linear projection layer to generate a feature representation that matches the input dimension of the multimodal large model. The weight matrix of the linear projection layer Initialize using the least squares method; Constructing position encoding vectors and represent the features With position encoding vector Element-wise summation is performed to generate input features with positional information. The positional encoding vector P_enc is generated using a sine-cosine positional encoding method, and the calculation formula is as follows: and ,in For location index, For dimension indexing; Construct modality type embedding vectors, which are generated based on the weight distribution in the inter-modality weight matrix l; add the modality type embedding vectors to the input features with positional information to generate the final model input features; In each encoder layer, the input features are transformed into query vector, key vector and value vector through linear transformation. They are then segmented according to the number of attention heads. Attention scores are calculated for each attention head and a normalization function is applied to generate attention weights. These weights are then multiplied by the value vector and concatenated to obtain the multi-head attention output. In an optional embodiment, in each encoder layer, the model input features are input into the first encoder layer of the multimodal large model. In the multi-head self-attention sublayer of the first encoder layer, the model input features are processed through three linear transformation layers to generate query vectors. Key vector Sum value vector Query vector Key vector Sum value vector All dimensions are ; query vector The query vectors are divided according to the number of attention heads h=12, generating 12 sub-query vectors. The dimension of each subquery vector is ; for key vectors Sum value vector Perform the same segmentation operation to generate 12 sub-key vectors. and 12 sub-value vectors For the j-th attention head, calculate the attention score matrix. , Key vector The transpose of the matrix is ​​used to apply the softmax function to the attention score matrix to generate the attention weight matrix. Calculate attention output .

[0042] Multi-head attention output Residual connections and layer normalization are performed, and then the input is fed into the feedforward neural network for nonlinear transformation. After residual connections and layer normalization, the output is sent to the next encoder layer. Specifically, this includes: Output of 12 attention points The multi-head attention output is generated by concatenating the data; the multi-head attention output is then linearly transformed through the output projection layer to generate the attention layer output; the attention layer output and the model input features are then residually connected and layer normalization is applied; the normalized features are then input into the feedforward neural network sublayer; the feedforward output is generated through the feedforward neural network sublayer, and the feedforward output and the normalized features are then residually connected and layer normalization is applied to generate the final output of the first layer encoder.

[0043] It should be noted that the feedforward neural network sublayer contains two fully connected layers. The first fully connected layer expands the dimension from 768 to 3072 with the activation function GELU. The second fully connected layer compresses the dimension from 3072 back to 768.

[0044] Using the same processing flow, the final output of the first encoder layer is sequentially input into the second to twelfth encoder layers for layer-by-layer encoding. Each encoder layer contains a cascaded processing of multi-head self-attention sublayers, residual connections, layer normalization, and feedforward neural network sublayers. In the l-th encoder layer (l=2, 3, ..., 12), the output of the previous layer is used as input. After multi-head self-attention calculation, residual connections, layer normalization, feedforward neural network transformation, further residual connections, and layer normalization, the output of the l-th encoder layer is generated. After layer-by-layer processing of the 12 encoder layers, deep semantic coding features containing high-level abstract semantic information are generated.

[0045] Furthermore, based on the constructed inspection problem type labeling system, deep semantic encoding features are identified to generate a set of problem type labels. The inspection problem type labeling system includes coarse-grained category labels and fine-grained problem labels, specifically including: A labeling system for inspection issues was constructed, employing a two-tiered hierarchical structure. The first tier consists of coarse-grained category labels, including five categories: disciplinary violations, financial violations, conduct issues, dereliction of duty, and other issues. The second tier comprises fine-grained issue labels. The disciplinary violations category includes eight sub-labels such as violations of political discipline, organizational discipline, and integrity discipline; the financial violations category includes six sub-labels such as misuse of funds, chaotic accounting management, and the establishment of slush funds; the conduct issues category includes five sub-labels such as formalism, bureaucracy, and hedonism; the dereliction of duty category includes four sub-labels such as inaction, slow action, and arbitrary action; and the other issues category includes three sub-labels. The total number of fine-grained issue labels is 26. It should be noted that the coarse-grained classification head contains one fully connected layer with an output dimension of 5, corresponding to 5 coarse-grained categories; each element in the coarse-grained category probability vector takes a value in the range (0, 1), representing the probability of belonging to the corresponding coarse-grained category, and the coarse-grained classification threshold is set to 0.3; the fine-grained classification head contains two fully connected layers. The first fully connected layer has an output dimension of 256 and uses ReLU as the activation function, while the output dimension of the second fully connected layer is equal to the number of fine-grained labels under the coarse-grained category. Deep semantic encoding features are input into a coarse-grained classification head. A sigmoid activation function is applied to the coarse-grained category score vector to generate a coarse-grained category probability vector. For categories whose probability values ​​in the coarse-grained category probability vector are greater than the coarse-grained classification threshold, they are marked as active categories, generating an active category set. For each active category in the active category set, a corresponding fine-grained classification head is constructed. The deep semantic encoding features are weighted with the probability values ​​of the corresponding categories in the coarse-grained category probability vector to generate category-aware features. The category-aware features are then input into the corresponding fine-grained classification head to generate a fine-grained label score vector under the coarse-grained category. The sigmoid activation function is applied to the fine-grained label score vector to generate a fine-grained label probability vector. A fine-grained classification threshold is set, and labels with probability values ​​greater than the fine-grained classification threshold are marked as predicted question type labels. The predicted question type labels under all activated categories are summarized to generate the final question type label set.

[0046] Furthermore, based on the set of problem type labels, a multi-dimensional risk quantification assessment mechanism is used to calculate the problem risk score, generating a standardized problem risk score, specifically including: Construct an issue severity assessment module that, for each issue type label in the issue type label set, retrieves the corresponding basic severity score from a predefined issue severity knowledge base. Basic severity score The severity score is determined based on a comprehensive assessment of factors such as the degree of legal and regulatory violation, the scope of social impact, and the difficulty of rectification for each problem type; the average severity score corresponding to the problem type label set is calculated; and an evidence sufficiency assessment module is constructed based on the confidence score. The evidence sufficiency score is calculated using the inter-modal weight matrix; the entropy value of the inter-modal weight matrix is ​​calculated, which reflects the uniformity of the distribution of modal weights. The lower the entropy value, the more dominant a certain mode is, and the higher the entropy value, the more balanced the multimodal information is. Set entropy threshold The entropy value is 0.8. Greater than the entropy threshold If the evidence is sufficient, then it is considered that the multimodal evidence is sufficient, and the sufficiency score of the evidence is calculated. When the entropy value Less than or equal to the entropy threshold If the condition is met, it is determined to be a single modality dominance, and the sufficiency of evidence score is calculated. Sufficiency of evidence score The value range is [0, 1.2], and the higher the value, the stronger the evidence for problem identification; a historical correlation evaluation module is constructed to query the historical inspection database and retrieve historical problem records with the same or similar tags as the currently identified problem type tag set; the number of historical repetitions is calculated. Historical repetition count Count the number of times the same or similar question types have occurred in the past 12 months; based on historical repetition counts. Calculate historical correlation score The calculation formula is: Historical correlation score The value range is [0, 1]. When the number of historical repetitions reaches 10 or more, the historical correlation score is... Take the maximum value of 1.0; Construct an issue urgency assessment module to calculate an urgency score based on predefined urgency levels of tags in the issue type tag set. The urgency of issues is divided into four levels, with major and urgent issues corresponding to different urgency scores. =1.0, the urgency score corresponding to a relatively urgent problem. =0.7, the urgency score for a typical urgent problem. =0.4, the urgency score for non-urgent issues =0.1; For cases where the problem type label set contains multiple labels, the maximum value of the urgency scores of all labels is taken as the overall urgency score; The problem risk score is calculated by weighted summation of the average severity score, evidence sufficiency score, historical relevance score, and urgency score; The problem risk score is normalized and mapped to the interval [0, 100] to generate a standardized problem risk score.

[0047] Specifically, based on standardized problem risk scores, the labels in the problem type label set are re-filtered, retaining labels with posterior probabilities greater than the calibrated classification threshold, to generate a calibrated problem type label set, which includes: Construct a confidence calibration module to perform Bayesian calibration on the predicted probability of each label in the question type label set; for the label Its original predicted probability is Calculate the prior accuracy of the label from the historical validation dataset. Among them, prior accuracy This represents the proportion of labels l_i predicted in historical data that are actually correctly verified; based on the sufficiency of evidence score. Calculate the likelihood function The likelihood function is represented by the label. The probability of observing the sufficiency of current evidence under the condition that it is true is calculated using the following formula: ,in For tags Average sufficiency of evidence in historical data, For tags The standard deviation of the sufficiency of evidence; the calibrated posterior probability is calculated using Bayes' theorem. The calculation formula is: posterior probability It integrates model prediction probabilities, historical prior knowledge, and the quality of current evidence; Set the calibrated classification threshold The labels in the problem type label set are re-filtered, retaining the posterior probability. Greater than the calibrated classification threshold The system generates a calibrated set of question type labels. A cross-modal consistency verification module is constructed. For each label in the calibrated question type label set, the original features of the text modality, visual modality, and audio modality are extracted, and their independent prediction results on the label classifier are extracted. The variance of the independent prediction probabilities of the three modalities is calculated; the variance reflects the degree of consistency between the predictions of different modalities for the label. A consistency threshold is set. When the variance is less than the consistency threshold When the cross-modal prediction consistency of the label is high, a high-confidence label is added to the label; when the variance is greater than or equal to the consistency threshold... If the discrepancy exists between modal predictions, a manual review mark is added to the label. The overall confidence score of the recognition results is calculated, taking into account the confidence score, the average posterior probability of the calibrated labels, and cross-modal consistency. The recognition results are then graded according to the overall confidence score: a score greater than 0.8 is marked as a high-quality recognition result and can be directly output; a score between 0.5 and 0.8 is marked as a medium-quality recognition result and manual review is recommended; and a score less than 0.5 is marked as a low-quality recognition result, requiring data re-collection or manual review. Specifically, based on the calibrated set of question type labels, a data structure for question identification results is constructed to generate structured question identification results, including: A data structure for the problem identification results is constructed, which includes a basic problem information field, a risk assessment field, and a quality control field. The basic problem information field records the calibrated set of problem type labels, the posterior probability of each label, and the problem description text. The risk assessment field records the standardized problem risk score, average severity score, evidence sufficiency score, historical correlation score, and urgency score. The quality control field records the overall confidence level, the cross-modal prediction variance of each label, the list of labels requiring manual review, and the quality level of the identification results. It should be noted that the problem description text is generated by extracting the sentence with the highest semantic relevance to the predicted label from the original text data; the risk assessment field provides multi-dimensional risk quantification indicators; A unique identifier is generated for each identified problem, where the unique identifier ID_problem consists of a timestamp, the inspection object code, and the problem sequence number, in the format "YYYYMMDD-ORG_CODE-SEQ". An inter-modal weight matrix is ​​associated to record the contribution of each modality to the current problem identification, providing modality source information for subsequent evidence chain construction. The attention weight distribution of deep semantic coding features is extracted to identify the data segment that contributes most to problem identification in the original dataset, generating a key evidence index list. The unique identifier, problem type label set, standardized problem risk score, comprehensive confidence score, and key evidence index list are encapsulated into a problem identification result object. The problem identification result object is serialized into JSON format to generate structured problem identification result data. The structured problem identification result data is used as the output of step S3 and passed to the subsequent association graph construction module for problem propagation path analysis and inspection report generation.

[0048] In this embodiment of the application, step S4 includes: Specifically, the process involves receiving the problem identification results, using the unique identifier in the problem identification results as the node identifier, creating problem nodes, and generating a node set. This includes: using deep semantic coding features as the node feature vector of the problem node; encoding the calibrated problem type label set into a multi-hot vector form to generate problem type attributes; normalizing the standardized problem risk score to generate risk score attributes; and using the comprehensive confidence score as the confidence score attribute. The process also involves extracting relevant personnel entities, relevant department entities, and relevant matter entities from the original data corresponding to the key evidence index list. For each extracted entity, a corresponding entity node is created. The node feature vector of the personnel entity node is generated by querying the personnel information database to obtain the job level, tenure time, and historical records. The node feature vector of the department entity node is generated by the department code, department level, and functional category code. The node feature vector of the matter entity node is generated by the matter type, occurrence time, and number of associated documents. Finally, all created problem nodes, personnel entity nodes, department entity nodes, and matter entity nodes are summarized to generate a node set.

[0049] Furthermore, for any two problem nodes in the node set, the association strength between the nodes is calculated, specifically including: for any two problem nodes, semantic similarity, type overlap, and spatiotemporal correlation are calculated. Semantic similarity is obtained by calculating the cosine similarity between the node feature vectors of the two problem nodes; type overlap is obtained by calculating the similarity coefficient of the problem type attributes of the two problem nodes; and spatiotemporal correlation is obtained by extracting the timestamp and location information from the original data corresponding to the key evidence index list and calculating the time interval and spatial distance. The association strength between the problems is calculated by combining the semantic similarity, type overlap, and spatiotemporal correlation using a weighted summation method. An association strength threshold is set. When the association strength is greater than the association strength threshold, an undirected association edge is created between the two problem nodes, and the edge weight of the undirected association edge is set to the association strength. Directed association edges are constructed between problem nodes and entity nodes, where entity nodes include personnel entity nodes, department entity nodes, and event entity nodes. The edge types of the directed association edges include personnel involved, responsible departments, and triggering events.

[0050] It should be noted that the association strength threshold is determined by analyzing verified problem association networks on historical data in a way that maximizes the F1-score.

[0051] Furthermore, directed edges are constructed between problem nodes and entity nodes. For problem nodes and personnel entity nodes, a directed edge is created when the personnel is explicitly mentioned in the evidence corresponding to the key evidence index list. The direction of the directed edge is from the problem node to the personnel entity node, and the edge type is marked as the personnel type. The edge weight of the directed edge is calculated based on the frequency of mention of the personnel in the evidence and the importance weight of the mention context. For problem nodes and department entity nodes, a directed edge is created when the department appears as the responsible party in the evidence corresponding to the key evidence index list. The direction of the directed edge is from the problem node to the department entity node, and the edge type is marked as the responsible department type. The edge weight level of the directed edge is set according to the strength of the department's responsibility description in the evidence. For problem nodes and event entity nodes, a directed edge is created when the event appears as the triggering factor for the problem in the evidence corresponding to the key evidence index list. The direction of the directed edge is from the event entity node to the problem node, and the edge type is marked as the triggering event type. The directed and undirected edges are summarized to generate an edge set, and the node set and edge set are combined to construct a relationship graph.

[0052] Specifically, a graph neural network model is constructed, and the node feature vector of each node in the association graph is initialized to the initial hidden state of the corresponding node in the graph neural network model.

[0053] Furthermore, the graph neural network model includes an input embedding layer, multiple graph convolutional layers, a graph attention layer, a graph pooling layer, and an output prediction layer. These layers are cascaded to form an end-to-end network architecture, specifically including: The initial hidden state is subjected to neighborhood aggregation. Based on the edge connections and edge weights in the edge set, the hidden states of neighboring nodes are summed according to their weights and then fused with the hidden state of the current node. After iterative aggregation through multiple graph convolutional layers, node aggregation features containing multi-hop neighborhood information are generated, specifically including: The input embedding layer receives the node feature vectors of each node in the association graph, performs dimension alignment processing on the node feature vectors of different types of nodes, and maps the node feature vectors of problem nodes, personnel entity nodes, department entity nodes and event entity nodes to a unified embedding space through a linear transformation layer to generate the initial embedding vector of each node; at the same time, the edge type of each edge in the edge set is one-hot encoded, and the edge type embedding layer generates the edge type embedding vector, and the edge type embedding vector is concatenated with the corresponding edge weight to generate the edge feature vector; The multi-layer graph convolutional layer consists of a first graph convolutional layer, a second graph convolutional layer, and a third graph convolutional layer cascaded sequentially. Each graph convolutional layer includes a neighborhood aggregation sub-layer, a feature transformation sub-layer, and a nonlinear activation sub-layer. In the neighborhood aggregation sub-layer, the set of neighboring nodes of the current node is obtained based on the edge connections in the edge set. The hidden states of each neighboring node in the neighborhood node set are weighted and summed according to the edge weights in the corresponding edge feature vectors to generate a neighborhood aggregation vector. In the feature transformation sub-layer, the hidden state of the current node is concatenated with the neighborhood aggregation vector and then input into a fully connected layer for linear transformation to generate the transformed node representation. In the nonlinear activation sub-layer, an activation function is applied to the transformed node representation and added to the hidden state of the current node through a residual connection to generate the output hidden state of this graph convolutional layer. Through the layer-by-layer processing of the first graph convolutional layer, the second graph convolutional layer, and the third graph convolutional layer, a node aggregation feature containing three-hop neighborhood information is generated. The node aggregation features are input into the graph attention layer. The node aggregation features of adjacent nodes are concatenated and then input into a single-layer feedforward neural network to calculate the attention coefficients between nodes, generating attention-enhanced node features, specifically including: The graph attention layer comprises a multi-head attention computation sublayer and an attention fusion sublayer. In the multi-head attention computation sublayer, node aggregation features are processed through multiple linear transformation layers to generate multiple sets of query vectors, key vectors, and value vectors. For each attention head, the query vector of the current node is concatenated with the key vectors of its neighboring nodes and then input into a single-layer feedforward neural network. After activation function processing, the original attention score under that attention head is generated. A normalization function is applied to the original attention score to generate normalized attention coefficients. The value vectors of the neighboring nodes are weighted and summed based on the normalized attention coefficients to generate the output vector of that attention head. In the attention fusion sublayer, the output vectors of each attention head are concatenated and then dimensionality-reduced through a linear transformation layer. After being added to the node aggregation features through residual connections and then processed by layer normalization, attention-enhanced node features are generated. The graph pooling layer consists of a hierarchical pooling sublayer and a global readout sublayer. In the hierarchical pooling sublayer, the importance score of each node is calculated based on the attention-enhanced node features. The nodes are sorted according to their importance scores, and the nodes with the highest importance scores are retained. The feature information of the removed nodes is aggregated and retained through their neighboring nodes, generating a pooled sparse graph structure and corresponding pooled node features. In the global readout sublayer, global average pooling and global max pooling operations are performed on the pooled node features. The results of global average pooling and global max pooling are concatenated to generate a graph-level representation vector. The output prediction layer includes a path prediction sublayer and an influence prediction sublayer. In the path prediction sublayer, the attention-enhanced node features of any two nodes in the association graph are concatenated and then input into a binary classification network to predict whether there is a propagation path relationship between the two nodes and generate the probability of path existence. In the influence prediction sublayer, the attention-enhanced node features of each problem node are input into a regression network to predict the comprehensive influence score of the problem node. It should be noted that the graph neural network model uses a multi-task joint loss function for parameter optimization during the training phase. The multi-task joint loss function includes a weighted sum of three loss terms: node classification loss, edge prediction loss, and influence regression loss. Dropout regularization is used between the layers of the graph neural network model to prevent overfitting. Dropout regularization randomly discards some node feature dimensions during the training phase.

[0054] Furthermore, based on the attention-enhanced node features and the direction of the directed associated edges in the edge set, the upstream and downstream nodes of each problem node are traced along the directed associated edges. The directed path sequence from the matter entity node through the problem node to the personnel entity node and department entity node is extracted. The edge weights of all directed associated edges on each directed path sequence are accumulated to calculate the path propagation strength. The directed path sequences with path propagation strength greater than the propagation strength threshold are selected to generate a problem propagation path set. Cluster analysis is performed on the paths in the problem propagation path set to identify path clusters with the same starting point or ending point. All nodes involved in the same path cluster are marked as the same propagation chain.

[0055] It should be noted that the propagation intensity threshold is based on rigorous business logic and data support, and usually comes from statistical analysis of historical inspection cases. By examining the weight distribution of edges in the verified problem chain, an empirical dividing point that can distinguish the core propagation path from general associations is determined.

[0056] Furthermore, taking each problem node in the association graph as the center, a breadth-first traversal is performed according to the edge connections in the edge set. The number of personnel entity nodes, department entity nodes, and event entity nodes reachable within a preset hop count range are counted, and the comprehensive influence score of this problem node is calculated, specifically including: The graph pooling operation is used to calculate the scope of the problem's impact. The breadth score of the problem node's impact is calculated based on the number of reachable entity nodes of each type and the edge weights of the corresponding directed edges. The comprehensive influence score of the problem node is calculated by multiplying the risk scoring attribute and the breadth score of the impact. All problem nodes in the node set are sorted in descending order according to their comprehensive influence scores to generate a problem influence ranking list. Based on the distribution of the comprehensive influence scores of the problem nodes in the problem influence ranking list, the problem is divided into three levels of impact: core impact problem, related impact problem, and peripheral impact problem.

[0057] Specifically, based on the set of problem propagation paths and comprehensive influence scores, a structured inspection report data structure is constructed, integrating problem details, propagation paths, and rectification suggestions to generate a structured inspection report, which includes: Based on the problem propagation path set, problem influence ranking list, and influence level, a structured inspection report data structure is constructed. This structure includes a report summary module, a problem details module, a propagation analysis module, and a rectification suggestion module. The report summary module records the total number of problems identified during this inspection, the distribution of problems at each influence level, the overall risk level, and key areas of concern. The problem details module, following the order of the problem influence ranking list, records for each problem node its calibrated problem type label set, standardized problem risk score, comprehensive confidence level, and information on involved personnel and departments. The propagation analysis module records the starting point, ending point, intermediate nodes, and propagation intensity of each propagation chain in the problem propagation path set, and marks the key nodes in the propagation chain. In the rectification suggestion module, categorized rectification suggestions are generated based on the characteristics of the propagation chain in the problem propagation path set. For multiple problems with common upstream entity nodes, source governance rectification suggestions are generated; for multiple problems with common downstream department entity nodes, responsibility implementation rectification suggestions are generated; and for problem nodes with core impact levels, targeted rectification suggestions are generated by matching their calibrated problem type tag set with a preset rectification measure knowledge base. The report summary module, problem details module, propagation analysis module, and rectification suggestion module are integrated to generate a structured inspection report. The structured inspection report is serialized into a standard document format and associated with the visualization data of the association graph, outputting complete inspection analysis results.

[0058] In summary, this invention achieves standardized vector representation of heterogeneous inspection data and enhanced extraction of key discriminative information by preprocessing and extracting features from multimodal raw data. By constructing a cross-modal fusion network and dynamically calculating the inter-modal weight matrix, it realizes complementary enhancement and conflict resolution of multi-source evidence, generating joint representations rich in contextual semantics. This enables the system to simulate expert comprehensive judgment thinking, significantly improving the accuracy and robustness of problem identification. By constructing an association graph with problem identification results as nodes and applying a graph neural network model, it achieves a leap from discrete problem detection to systemic risk diagnosis. It can automatically reveal the problem propagation path, quantify the scope of impact, and generate a structured report integrating problem details, propagation chains, and rectification suggestions. This transforms inspection work from traditional manual experience-driven to a data-driven, quantifiable, and traceable intelligent decision-making model, comprehensively improving the efficiency, depth, and systematic nature of inspection work.

[0059] Based on the teachings of the above embodiments, other aspects of the present invention also propose an intelligent identification and data processing system for inspection and patrol issues based on a multimodal large model, including: The preprocessing and feature extraction module is used to preprocess the raw data of various modalities obtained in the inspection and patrol, and extract the feature representations corresponding to each modality; The cross-modal feature fusion module is used to input the feature representation into the constructed cross-modal fusion network and obtain the fused feature representation by calculating the inter-modal weight matrix; The multimodal problem identification and risk assessment module takes the fused feature representation input to the pre-trained multimodal large model for identification processing. It is used to identify the patrol problem type label through a multi-label classification algorithm, calculate the problem risk score, and output the problem identification result. The correlation analysis and report generation module is used to construct a correlation graph with problem identification results as nodes. It analyzes the correlation strength between nodes through a graph neural network model to identify the problem propagation path and scope of impact, and generates a structured inspection report.

[0060] This embodiment also provides a computer device applicable to the intelligent identification and data processing method for inspection and patrol issues based on a multimodal large model, including a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the intelligent identification and data processing method for inspection and patrol issues based on a multimodal large model as proposed in the above embodiment.

[0061] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0062] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, it implements the intelligent identification and data processing method for inspection and patrol issues based on a multimodal large model as proposed in the above embodiment.

[0063] The storage medium proposed in this embodiment and the data storage method proposed in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0064] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for intelligent identification and data processing of inspection and patrol issues based on a multimodal large model, characterized in that: include, The raw data of multiple modalities obtained from inspections and patrols are preprocessed to extract the feature representations corresponding to each modality. The feature representation is input into the constructed cross-modal fusion network, and the fused feature representation is obtained by calculating the inter-modal weight matrix; The fused feature representation is input into a pre-trained multimodal large model for identification processing. A multi-label classification algorithm is used to identify the patrol problem type label, calculate the problem risk score, and output the problem identification result. Construct an association graph with the problem identification results as nodes, analyze the association strength between nodes using a graph neural network model to identify the problem propagation path and scope of impact, and generate a structured inspection report.

2. The intelligent identification and data processing method for inspection and patrol issues based on a multimodal large model as described in claim 1, characterized in that: The method for generating the structured inspection report is as follows: A graph neural network model is constructed by initializing the node feature vector of each node in the association graph to the initial hidden state of the corresponding node in the graph neural network model. The graph neural network model includes an input embedding layer, multiple graph convolutional layers, graph attention layers, graph pooling layers, and an output prediction layer. The layers are connected in a cascaded manner to form an end-to-end network architecture. The initial hidden state is subjected to a neighborhood aggregation operation. Based on the edge connection relationship and edge weight in the edge set, the hidden states of adjacent nodes are summed according to their weights and then fused with the hidden state of the current node. After iterative aggregation through multiple graph convolutional layers, a node aggregation feature containing multi-hop neighborhood information is generated. The node aggregation features are input into the graph attention layer. The node aggregation features of adjacent nodes are concatenated and then input into a single-layer feedforward neural network to calculate the attention coefficients between nodes, thereby generating attention-enhanced node features. Based on the attention-enhancing node features and the direction of the directed associated edges in the edge set, the upstream and downstream nodes of each problem node are traced along the directed associated edges. The directed path sequence from the matter entity node through the problem node to the personnel entity node and department entity node is extracted. The edge weights of all directed associated edges on each directed path sequence are accumulated to calculate the path propagation strength. The directed path sequences with the path propagation strength greater than the propagation strength threshold are selected to generate a problem propagation path set. Taking each problem node in the association graph as the center, a breadth-first traversal is performed according to the edge connection relationship in the edge set. The number of personnel entity nodes, department entity nodes, and event entity nodes that can be reached within a preset number of hops are counted, and the comprehensive influence score of the problem node is calculated. Based on the set of problem propagation paths and the comprehensive influence score, a structured inspection report data structure is constructed, which integrates problem details, propagation paths and rectification suggestions to generate a structured inspection report.

3. The intelligent identification and data processing method for inspection and patrol issues based on a multimodal large model as described in claim 2, characterized in that: The method for constructing the association map is as follows: Receive the problem identification results, use the unique identifier in the problem identification results as the node identifier, create a problem node, and generate a node set; For any two problem nodes in the node set, calculate the association strength between the nodes; Set a threshold for association strength. When the association strength is greater than the threshold, create an undirected association edge between the two problem nodes and set the edge weight of the undirected association edge to the association strength. Construct directed association edges between problem nodes and entity nodes, wherein the entity nodes include personnel entity nodes, department entity nodes, and event entity nodes; the edge types of the directed association edges include personnel involved, responsible departments, and triggering events; The directed and undirected edges are aggregated to generate an edge set, and the node set and edge set are combined to construct an association graph.

4. The intelligent identification and data processing method for inspection and patrol issues based on a multimodal large model as described in claim 3, characterized in that: The method for outputting the problem identification results is as follows: The fused feature representation is input into a pre-trained multimodal large model, and the output is a deep semantic encoded feature. Based on the constructed inspection problem type label system, the deep semantic coding features are identified to generate a set of problem type labels, wherein the inspection problem type label system includes coarse-grained category labels and fine-grained problem labels; Based on the set of problem type labels, a multi-dimensional risk quantification assessment mechanism is used to calculate the problem risk score and generate a standardized problem risk score. Based on the standardized problem risk score, the labels in the problem type label set are re-filtered, and the labels with a posterior probability greater than the calibrated classification threshold are retained to generate a calibrated problem type label set. Based on the calibrated set of question type labels, a question identification result data structure is constructed to generate structured question identification results.

5. The intelligent identification and data processing method for inspection and patrol issues based on a multimodal large model as described in claim 4, characterized in that: The multimodal large model is built based on the Transformer architecture; the multimodal large model includes multiple encoder layers, each encoder layer including a multi-head self-attention sub-layer and a feedforward neural network sub-layer.

6. The intelligent identification and data processing method for inspection and patrol issues based on a multimodal large model as described in claim 4, characterized in that: The method for obtaining the fusion feature representation is as follows: Received feature representation set Construct a cross-modal fusion network architecture; The feature representation set Input the intramodal self-attention layer of the cross-modal fusion network architecture to generate multimodal intra-enhanced features; Based on the aforementioned multimodal intramodal enhancement features, a bidirectional interaction mechanism is used to construct a cross-modal interaction module between each modality, calculate the cross-modal attention score matrix, and generate comprehensive cross-modal features for each modality. Global average pooling is performed on the multimodal intramodal enhancement features to generate global representation vectors for each modality, and the global representation vectors of each modality are concatenated to generate a global modality representation vector. The global modal representation vector is input into the modal gating network to calculate the modal importance score vector, and modal weight vectors are generated by normalization to construct the inter-modal weight matrix; The intra-modal enhancement features of each modality are residually fused with the corresponding comprehensive cross-modal features, compressed into a fixed-length vector representation by global average pooling, and then weighted according to the inter-modal weight matrix to generate a fused feature vector. The fused feature vector is then input into a feedforward neural network for nonlinear transformation to generate a fused feature representation.

7. The intelligent identification and data processing method for inspection and patrol issues based on a multimodal large model as described in claim 6, characterized in that: The cross-modal fusion network architecture includes an intra-modal self-attention layer, a cross-modal interaction layer, and a fusion output layer; the feature representation set This includes text feature representation, visual feature representation, and audio feature representation; The multimodal intra-modal enhancement features include text intra-modal enhancement features, visual intra-modal enhancement features, and audio intra-modal enhancement features.

8. A system for intelligent identification and data processing of inspection and patrol issues based on a multimodal large model, comprising the method for intelligent identification and data processing of inspection and patrol issues based on a multimodal large model as described in any one of claims 1 to 7, characterized in that: include, The preprocessing and feature extraction module is used to preprocess the raw data of various modalities obtained in the inspection and patrol, and extract the feature representations corresponding to each modality; The cross-modal feature fusion module is used to input the feature representation into the constructed cross-modal fusion network and obtain the fused feature representation by calculating the inter-modal weight matrix; The multimodal problem identification and risk assessment module inputs the fused feature representation into a pre-trained multimodal large model for identification processing. It is used to identify patrol problem type labels through a multi-label classification algorithm, calculate problem risk scores, and output problem identification results. The correlation analysis and report generation module is used to construct a correlation graph with the problem identification results as nodes, analyze the correlation strength between nodes through a graph neural network model to identify the problem propagation path and scope of influence, and generate a structured inspection report.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the intelligent identification and data processing method for patrol and inspection problems based on a multimodal large model as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the intelligent identification and data processing method for patrol and inspection problems based on a multimodal large model as described in any one of claims 1 to 7.

Images