Multi-modal text intelligent analysis system

Abstract

The invention relates to the technical field of multi-modal text analysis, in particular to a multi-modal text intelligent analysis system. According to the system, text features and image features of a multi-modal text are extracted, matrix coordinate transformation is carried out on the text features and the image features, dimension reduction processing is carried out to obtain decomposition vectors, and the spatial distance D of the decomposition vectors is calculated; judging whether the text features and the image features meet multi-modal embedded semantic representation extraction requirements or not based on the spatial distance D, and obtaining visual embedded semantic representation and text embedded semantic representation; performing clustering processing on the visual embedded semantic representation and the text embedded semantic representation to obtain a plurality of semantic representation pairs, performing alignment fusion processing on the semantic representation pairs to obtain fused semantic representation, and learning the fused semantic representation through a graph convolutional neural network to obtain a learning result, the ability to understand and analyze complex documents or multimedia contents can be improved by fusing text information and image information.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal text parsing, and specifically relates to a multimodal text intelligent parsing system. Background Art

[0002] With the rapid development of information technology, in addition to traditional pure text information, multimodal information containing text and images has gradually become the mainstream. Such multimodal text information widely exists in multiple fields such as social media, news reports, product manuals, academic papers, advertising materials, etc. However, most of the existing information processing technologies are designed for single modalities (such as pure text or pure images), and it is difficult to effectively process and understand this complex content that combines text and image information.

[0003] Currently, although some studies have attempted to fuse text and image information, most methods have problems such as simple fusion strategies, low processing efficiency, and poor adaptability to complex scenarios. For example, simple splicing or concatenation methods cannot effectively mine the deep associations between text and images; while deep learning-based fusion methods, although improving the processing effect to a certain extent, often face challenges such as high model complexity and large training data requirements.

[0004] Therefore, there is an urgent need for a multimodal text intelligent parsing system to solve the above problems. Summary of the Invention

[0005] The purpose of the present invention is to provide a multimodal text intelligent parsing system: aiming to improve the understanding and analysis capabilities of complex documents or multimedia content by fusing text information and image information, and to improve the processing efficiency and adaptability to complex scenarios.

[0006] The purpose of the present invention can be achieved by the following technical solutions: A multimodal text intelligent parsing system, the system includes: A matrix coordinate transformation module, which is used to extract the text features and image features of multimodal text, perform matrix coordinate transformation on the text features and image features to obtain a transformation matrix P, perform PCA dimensionality reduction processing on the transformation matrix P to obtain two decomposition vectors, calculate the spatial distance D between the two decomposition vectors, and determine whether the text features and image features meet the requirements for extracting multimodal embedded semantic representations based on the spatial distance D; An embedded semantic representation extraction module, which is used to extract multimodal embedded semantic representations from the text features and image features that meet the requirements for extracting multimodal embedded semantic representations respectively, to obtain a visual embedded semantic representation and a text embedded semantic representation; The fusion processing module is used to perform clustering processing on the visual embedded semantic representation and the text embedded semantic representation to obtain a number of semantic representation pairs, and perform alignment and fusion processing on the semantic representation pairs to obtain a fused semantic representation, and learn the fused semantic representation through a graph convolutional neural network to obtain a learning result.

[0007] Further, performing matrix coordinate transformation on the text feature and the image feature to obtain the transformation matrix P specifically includes the following process: Calculate the text feature and the image feature respectively for their vector averages and , where n is the number of vectors: ; ; where the value range of i is ; Concatenate the calculated text feature and image feature to form a vector set X: ; Among them, is the concatenation operation, is the transpose operation symbol; Calculate the covariance matrix A of X: ; Use the eigenvalue decomposition method to solve the eigenvalues and eigenvectors of the covariance matrix A, and obtain the matrices Q and B. Among them, the matrix Q is a matrix composed of the eigenvectors of the matrix A, and B is a diagonal array, and the elements on the diagonal are the eigenvalues; Take the first p columns of Q as the transformation matrix P.

[0008] Further, perform PCA dimensionality reduction processing on the transformation matrix P to obtain two decomposition vectors, which specifically includes the following process: (1) Calculate the covariance matrix S of the transformation matrix P; (2) Calculate the eigenvectors , , …, ; (3) Project the data into the space of the eigenvectors, and use the formula to calculate the vectors with the same dimension as the text feature, the image feature, and the transformation matrix P: ; Among them, is the value of the corresponding dimension in the original sample; (4) Calculate the two decomposition vectors and respectively: , .

[0009] Furthermore, calculate the spatial distance D between the two decomposed vectors, and determine whether the text features and image features meet the requirements for extracting multimodal embedded semantic representations based on the spatial distance D. The specific process includes the following: Calculate the spatial distance D between the two decomposed vectors: ; where, represents the i-th element of the decomposed vector , represents the i-th element of the decomposed vector; Load the spatial distance threshold. The spatial distance threshold is stored in the system and its value is set by the system. Determine whether the spatial distance D is greater than the spatial distance threshold. If so, it is determined that the text features and image features do not meet the requirements for extracting multimodal embedded semantic representations. If not, it is determined that the text features and image features meet the requirements for extracting multimodal embedded semantic representations.

[0010] Furthermore, perform multimodal embedded semantic representation extraction on the text features and image features that meet the requirements for extracting multimodal embedded semantic representations to obtain visual embedded semantic representations and text embedded semantic representations. The specific process includes the following: ; ; where, is the visual embedded semantic representation, is the text embedded semantic representation, , are the region feature information and region bounding box coordinate information extracted from the image features respectively, is the concatenation operation, is the layer normalization operator, is the L2 norm normalization, , are the linear transformation matrices, , are the learnable bias terms, is the non-linear activation function ReLU.

[0011] Furthermore, perform clustering processing on the visual embedded semantic representations and text embedded semantic representations to obtain several pairs of semantic representations. The specific process includes the following: Regard the visual embedded semantic representations and text embedded semantic representations as a data set; Set the value of K according to the size and shape of the data set; Randomly select a data point in the data set as the initial centroid; When the number of initial centroids is less than K, let be the data set, and be the initial centroids; Based on the objective function calculate the distance D(x) between each data point in the data set and the existing initial centroids; Take the data point corresponding to the maximum value in D(x) as the next initial centroid, and obtain E initial centroids in sequence; Cluster the data in the data set according to the E initial centroids to obtain E categories and the visual embedded semantic representation and text embedded semantics corresponding to each category; ; If in the neighborhood of at least one object, then mark this object as the paired object of this semantic representation, and mark this object and the paired object as a semantic representation pair; Among them, represents the neighborhood radius for defining density.

[0012] Furthermore, perform alignment and fusion processing on the semantic representation pairs to obtain the fused semantic representation, which specifically includes the following process: Delete the edges of the semantic representation pairs according to the degree of importance. Among them, represent the semantic representation pair as , where is denoted as the node set, and is denoted as the edge set, and the number of edges is ; Calculate the node centrality: The probability set of the sampled modified subset in the semantic representation pair representation is defined by the following formula: ; Among them, is the edge set in the fused semantic representation, is the probability of deleting the edge , reflecting the degree of importance of the edge ; Calculate the edge centrality: ; Among them, is the centrality of the edge , used to represent the influence of the importance between two connected nodes, , represent the node centrality metric; Calculate the fusion probability of each edge according to the centrality of each edge: Set To mitigate the impact of dense nodes, after normalization, the values are converted into probabilities through the following conversion formula: ; where is a hyperparameter, the purpose of which is to control the overall probability of deleting edges; is the maximum value of is the average value of , which is used to limit the maximum value of the probability; Each edge is incorporated into the node in the order of the fusion probability to obtain the fused semantic representation.

[0013] Furthermore, the fused semantic representation is learned through a graph convolutional neural network, and the learning results specifically include the following process: According to the task requirements and data characteristics, the fused semantic representation is constructed into a graph structure; The representation of the nodes is initialized using pre-trained word embeddings, sentence embeddings, etc., and this representation serves as the input to the graph convolutional neural network; The graph structure is iteratively learned through a multi-layer graph convolutional neural network; In each layer, the graph convolutional neural network aggregates the information of the nodes and their neighbors, updates the representation of the nodes, captures the complex relationships and semantic information in the graph structure, and obtains the learning results.

[0014] Compared with the existing solutions, the beneficial effects achieved by the present invention are as follows: The present invention extracts the text features and image features of multi-modal text, performs matrix coordinate transformation on the text features and image features, and performs dimensionality reduction processing to obtain the decomposed vectors. The spatial distance D of the decomposed vectors is calculated, and based on the spatial distance D, it is determined whether the text features and image features meet the requirements for extracting multi-modal embedded semantic representations, and the visual embedded semantic representation and text embedded semantic representation are obtained; the visual embedded semantic representation and text embedded semantic representation are clustered to obtain several semantic representation pairs, and the semantic representation pairs are subjected to alignment and fusion processing to obtain the fused semantic representation, and the fused semantic representation is learned through a graph convolutional neural network to obtain the learning results, which can improve the understanding and analysis ability of complex documents or multimedia content by fusing text information and image information.

[0015] Furthermore, the intelligent parsing system based on multi-modal text proposed by the present invention can simultaneously process the information of two modalities, text and image. Through an advanced fusion algorithm, the semantic information in the text and the visual information in the image are organically combined, so as to achieve a comprehensive parsing of multi-modal text content. This fusion method breaks through the limitations of traditional single-modal processing, making the parsing results more complete and accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

[0017] Figure 1 is a system block diagram of a multi-modal text intelligent parsing system according to an embodiment of the present invention; Figure 2 is a flowchart of the operation of the first multi-modal text intelligent parsing system according to an embodiment of the present invention; Figure 3 is a flowchart of the operation of the second multi-modal text intelligent parsing system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0018] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0019] In addition, the described features, structures, or characteristics can be combined in any suitable manner in one or more example embodiments. In the following description, many specific details are provided to give a thorough understanding of the example embodiments of the present disclosure. However, those skilled in the art will realize that the technical solutions of the present disclosure can be practiced without one or more of the specific details, or other methods, components, steps, etc. can be adopted. In other cases, well-known structures, methods, implementations, or operations are not shown or described in detail to avoid obscuring the various aspects of the present disclosure.

[0020] This embodiment provides a multi-modal text intelligent parsing system, Figure 1 is a system block diagram of a multi-modal text intelligent parsing system according to an embodiment of the present invention, as Figure 1 shown, the system includes: A matrix coordinate transformation module, configured to extract text features and image features of multi-modal text, perform matrix coordinate transformation on the text features and image features to obtain a transformation matrix P, perform PCA dimensionality reduction processing on the transformation matrix P to obtain two decomposition vectors, calculate the spatial distance D between the two decomposition vectors, and determine whether the text features and image features meet the requirements for extracting multi-modal embedded semantic representations based on the spatial distance D; An embedded semantic representation extraction module is used to perform multi-modal embedded semantic representation extraction on text features and image features that meet the requirements of multi-modal embedded semantic representation extraction respectively, so as to obtain a visual embedded semantic representation and a text embedded semantic representation; A fusion processing module is used to perform clustering processing on the visual embedded semantic representation and the text embedded semantic representation to obtain a number of semantic representation pairs, and perform alignment and fusion processing on the semantic representation pairs to obtain a fused semantic representation, and perform learning on the fused semantic representation through a graph convolutional neural network to obtain a learning result.

[0021] In summary, the present invention extracts text features and image features of multi-modal text, performs matrix coordinate transformation on the text features and image features, and performs dimensionality reduction processing to obtain decomposed vectors, calculates the spatial distance D of the decomposed vectors, and judges whether the text features and image features meet the requirements of multi-modal embedded semantic representation extraction based on the spatial distance D to obtain a visual embedded semantic representation and a text embedded semantic representation; performs clustering processing on the visual embedded semantic representation and the text embedded semantic representation to obtain a number of semantic representation pairs, and performs alignment and fusion processing on the semantic representation pairs to obtain a fused semantic representation, and performs learning on the fused semantic representation through a graph convolutional neural network to obtain a learning result, which can improve the understanding and analysis ability of complex documents or multimedia content by fusing text information and image information.

[0022] In some embodiments, performing matrix coordinate transformation on text features and image features to obtain a transformation matrix P specifically includes the following process: Calculate the text features and the image features respectively for the vector averages and , where n is the number of vectors: ; ; where the value of i is ; Concatenate the calculated text features and image features to form a vector set X: ; where is the concatenation operation, is the transpose operation symbol; Calculate the covariance matrix A of X: ; Use the eigenvalue decomposition method to solve the eigenvalues and eigenvectors of the covariance matrix A to obtain matrices Q and B. Among them, matrix Q is a matrix composed of the eigenvectors of matrix A, and B is a diagonal array, and the elements on the diagonal are eigenvalues; Take the first p columns of Q as the transformation matrix P.

[0023] In some embodiments, the process of obtaining two decomposition vectors by performing PCA dimensionality reduction on the transformation matrix P specifically includes the following steps: (1) Calculate the covariance matrix S of the transformation matrix P; (2) Calculate the eigenvectors of the covariance matrix S , ,…, ; (3) Project the data into the space of the eigenvectors, and use the formula to calculate the vector with the same dimension as the text feature, image feature, and transformation matrix P : ; where, The value is the corresponding dimension value in the original sample; (4) Calculate two decomposition vectors and : , .

[0024] In some embodiments, calculating the spatial distance D between the two decomposition vectors and determining whether the text feature and the image feature meet the requirements for multi-modal embedded semantic representation extraction based on the spatial distance D specifically includes the following process: Calculate the spatial distance D between the two decomposition vectors: ; where, represents the i-th element of the decomposition vector , represents the i-th element of the decomposition vector; Load the spatial distance threshold. The spatial distance threshold is stored in the system and its value is set by the system. Determine whether the spatial distance D is greater than the spatial distance threshold. If so, it is determined that the text feature and the image feature do not meet the requirements for multi-modal embedded semantic representation extraction. If not, it is determined that the text feature and the image feature meet the requirements for multi-modal embedded semantic representation extraction.

[0025] In some embodiments, performing multi-modal embedded semantic representation extraction on the text feature and the image feature that meet the requirements for multi-modal embedded semantic representation extraction respectively to obtain a visual embedded semantic representation and a text embedded semantic representation specifically includes the following process: ; ; where, is the visual embedded semantic representation, is the text embedded semantic representation, , They are respectively the region feature information and the region bounding box coordinate information extracted from the image features, is the concatenation operation, is the layer normalization operator, is the L2 norm normalization, 、 is the linear transformation matrix, 、 is the learnable bias term, is the non-linear activation function ReLU.

[0026] In some embodiments, Figure 2 is the flowchart of the operation of the first multi-modal text intelligent parsing system according to the embodiments of the present invention. As Figure 2 shown, the clustering process of the visual embedded semantic representation and the text embedded semantic representation to obtain a number of semantic representation pairs specifically includes the following processes: Step S201: Regarding the visual embedded semantic representation and the text embedded semantic representation as a data set, set the value of K according to the size and shape of the data set; Step S202: Randomly select a data point in the data set as the initial centroid; Specifically, let be the data set, be the initial centroid; Step S203: When the number of initial centroids is less than K, calculate the distance D(x) of each data point in the data set from the existing initial centroids based on the objective function; Specifically, the objective function is as follows: ; Step S204: Take the data point corresponding to the maximum value in D(x) as the next initial centroid, and sequentially obtain E initial centroids; Step S205: Cluster the data in the data set according to the E initial centroids to obtain E categories and the corresponding visual embedded semantics and text embedded semantics for each category, and pair them pairwise.

[0027] The specific process of pairwise pairing includes: randomly selecting the data density of the semantic representation of a category ; If in the domain has at least 1 object, then mark this object as the paired object of this semantic representation, and mark this object and the paired object as a semantic representation pair; Among them, [[ID=No.59]]represents the neighborhood radius defining the density.

[0028] In some embodiments, aligning and fusing the semantic representation pairs to obtain the fused semantic representation specifically includes the following process: Delete the edges of the semantic representation pairs according to the degree of importance, where the semantic representation pairs are represented as , where is denoted as the node set, and is denoted as the edge set, and the number of edges is ; Calculate the node centrality: Sample the modified subset in the semantic representation pair, and the probability set definition formula is as follows: ; where is the edge set in the fused semantic representation, is the probability of deleting the edge , reflecting the degree of importance of the edge ; Calculate the edge centrality: ; where is the centrality of the edge , used to represent the influence of the importance between two connected nodes, , represent the node centrality metric; Calculate the fusion probability of each edge according to the centrality of each edge : Set to reduce the influence of dense nodes. After normalization, convert the value to a probability through the conversion formula, and the conversion formula is as follows: ; where is a hyperparameter, the purpose of which is to control the overall probability of deleting edges; is 's maximum value, is 's average value, , used to limit the maximum value of the probability; Integrate each edge into the node in turn according to the size order of the fusion probability to obtain the fused semantic representation.

[0029] In some embodiments, Figure 3 is the workflow diagram of the second multi-modal text intelligent parsing system of the embodiments of the present invention. As shown in Figure 3 , learning the fused semantic representation through a graph convolutional neural network to obtain the learning result specifically includes the following process: Step S301: Construct the fused semantic representation into a graph structure according to the task requirements and data characteristics; Step S302: Initialize the representation of nodes using pre-trained word embeddings, sentence embeddings, etc., and this representation serves as the input to the graph convolutional neural network; Step S303: Iteratively learn the graph structure through a multi-layer graph convolutional neural network; Step S304: In each layer, the graph convolutional neural network aggregates the information of nodes and their neighbors, updates the representation of nodes, captures the complex relationships and semantic information in the graph structure, and obtains the learning result.

[0030] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more collections of available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0031] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0032] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be repeated here.

[0033] In several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only for some logical function divisions. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of the devices or units can be in electrical, mechanical, or other forms.

[0034] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0035] As mentioned above, the above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed in the present application can easily think of changes or substitutions, which should all be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. Multimodal text intelligent parsing system, characterized in that the system Including: A matrix coordinate transformation module, which is used to extract the text features and image features of multimodal text, perform matrix coordinate transformation on the text features and image features to obtain a transformation matrix P, perform PCA dimensionality reduction processing on the transformation matrix P to obtain two decomposition vectors, calculate the spatial distance D between the two decomposition vectors, and judge whether the text features and image features meet the requirements for extracting multimodal embedded semantic representations based on the spatial distance D; An embedded semantic representation extraction module, which is used to respectively extract multimodal embedded semantic representations from the text features and image features that meet the requirements for extracting multimodal embedded semantic representations to obtain visual embedded semantic representations and text embedded semantic representations; A fusion processing module, which is used to perform clustering processing on the visual embedded semantic representations and text embedded semantic representations to obtain several semantic representation pairs, perform alignment and fusion processing on the semantic representation pairs to obtain a fused semantic representation, and perform learning on the fused semantic representation through a graph convolutional neural network to obtain a learning result.

2. The multimodal text intelligent parsing system according to claim 1, wherein Performing matrix coordinate transformation on the text features and image features to obtain the transformation matrix P specifically includes the following process: Calculate the text features separately and the image features of the vector mean value and , where n is the number of vectors: ; ; where the value of i is ; Concatenate the calculated text features and image features to form a vector set X: ； Among them, is the concatenation operation, is the transpose operation symbol; Calculate the covariance matrix A of X: ； Use the eigenvalue decomposition method to solve the eigenvalues and eigenvectors of the covariance matrix A to obtain matrices Q and B, where matrix Q is a matrix composed of the eigenvectors of matrix A, and B is a diagonal array, and the elements on the diagonal are eigenvalues; Take the first p columns of Q as the transformation matrix P.

3. The multimodal text intelligent parsing system according to claim 2, characterized in that, Performing PCA dimensionality reduction processing on the transformation matrix P to obtain two decomposition vectors specifically includes the following process: (1) Calculate the covariance matrix S of the transformation matrix P; (2) Calculate the eigenvectors of the covariance matrix S , ,…, ; (3) Project the data into the space of feature vectors, and use the formula to calculate vectors with the same dimension for text features, image features, and transformation matrix P. : ； Among them, The value is the corresponding dimensional value in the original sample; (4) Calculate to obtain two decomposed vectors respectively and : ， 。 4. The multimodal text intelligent parsing system according to claim 3, wherein Calculating the spatial distance D between the two decomposition vectors and judging whether the text features and image features meet the requirements for extracting multimodal embedded semantic representations based on the spatial distance D specifically includes the following process: Calculate the spatial distance D between the two decomposition vectors: ； Among them, represents the i-th element of the decomposition vector , and represents the i-th element of the decomposition vector; Load the spatial distance threshold, which is stored in the system and its value is set by the system. Judge whether the spatial distance D is greater than the spatial distance threshold. If so, it is determined that the text features and image features do not meet the requirements for extracting multimodal embedded semantic representations. If not, it is determined that the text features and image features meet the requirements for extracting multimodal embedded semantic representations.

5. The multimodal text intelligent parsing system according to claim 1, wherein Respectively extracting multimodal embedded semantic representations from the text features and image features that meet the requirements for extracting multimodal embedded semantic representations to obtain visual embedded semantic representations and text embedded semantic representations specifically includes the following process: ； ； Among them, is a visual embedded semantic representation, is a text embedded semantic representation, , are respectively the region feature information and region bounding box coordinate information extracted from the image features, is a concatenation operation, is a layer normalization operator, is L2-norm normalization, , are linear transformation matrices, , are learnable bias terms, is the non-linear activation function ReLU.

6. The multimodal text intelligent parsing system according to claim 1, wherein Performing clustering processing on the visual embedded semantic representations and text embedded semantic representations to obtain several semantic representation pairs specifically includes the following process: Regard the visual embedded semantic representations and text embedded semantic representations as a data set; Set the value of K according to the size and shape of the data set; Randomly select a data point in the data set as the initial centroid; When the number of initial centroids is less than K, let be the data set, be the initial centroids; Based on the objective function Calculate the distance D(x) between each data point in the dataset and the existing initial centroids; Take the data point corresponding to the maximum value in D(x) as the next initial centroid, and sequentially obtain E initial centroids; Perform clustering processing on the data in the data set according to the E initial centroids to obtain E categories and the corresponding visual embedded semantic representations and text semantics for each category; The data density of the semantic representation randomly selected from one category ; If there is at least one object in the field, then mark this object as the paired object of this semantic representation, and mark this object and the paired object as a semantic representation pair; Among them, represents the neighborhood radius for defining the density.

7. The multimodal text intelligent parsing system according to claim 1, wherein Align and fuse the semantic representation pairs to obtain the fused semantic representation, which specifically includes the following processes: Edges of the semantic representation pair are deleted according to the degree of importance, where the semantic representation pair is expressed as , where is denoted as the node set, and is denoted as the edge set, and the number of edges is ; Calculating node centrality: Sampling a modified subset in the semantic representation pair The probability set definition formula is as follows: ； Among them, is the edge set in the fused semantic representation, is the probability of deleting an edge , reflecting the importance degree of the edge ; Calculate edge centrality: ; Among them, is the centrality of the edge , which is used to represent the influence of the importance between two connected nodes. , represent node centrality measures; Calculate the fusion probability of each edge according to the centrality of each edge : Set to mitigate the impact of dense nodes. After normalization, convert the value to a probability through the conversion formula as follows: ； Among them, is a hyperparameter, aiming to control the overall probability of deleting edges; is the maximum value of is the average value of , used to limit the maximum value of the probability; According to the fusion probability each edge is incorporated into the node in the order of size to obtain the fused semantic representation.

8. The multimodal text intelligent parsing system according to claim 7, wherein Learn the fused semantic representation through a graph convolutional neural network to obtain the learning result, which specifically includes the following processes: Construct the fused semantic representation into a graph structure according to the task requirements and data characteristics; Use the pre-trained word embedding and sentence embedding methods to initialize the representation of the nodes, and this representation serves as the input to the graph convolutional neural network; Perform iterative learning on the graph structure through a multi-layer graph convolutional neural network; In each layer, the graph convolutional neural network aggregates the information of the nodes and their neighbors, updates the representation of the nodes, captures the complex relationships and semantic information in the graph structure, and obtains the learning result.

Images