A demand semantic analysis method based on cross-modal fusion

Abstract

The present application relates to natural language processing and cross-modal semantic understanding technical field, especially relate to a kind of demand semantic analysis method based on cross-modal fusion, including the acquisition text, structured form image and voice instruction etc.cross-modal data;Respectively using ModernBERT, LayoutLMv3 extracts text semantic vector, layout structure vector and audio semantic vector;The heterogeneous graph structure of fusing three kinds of nodes and multiple edge relations is constructed;Adopt HGT graph neural network to carry out type perception semantic propagation, generate unified cross-modal joint representation vector;The present application fuses language, image and multi-source information of voice, realizes to complex user's cross-modal data sample accurate understanding, with stronger semantic consistency, generalization ability and robustness, applicable to task scheduling, service recommendation etc.Intelligent response scene.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and cross-modal semantic understanding, and in particular to a demand semantic analysis method based on cross-modal fusion. Background Technology

[0002] With the widespread deployment of artificial intelligence and intelligent service systems, the requests submitted by users during actual interactions are showing a trend towards cross-modality, typically including text descriptions, image screenshots, structured form data, and voice commands. However, most existing semantic analysis methods are still limited to processing single-modal information, lacking a unified modeling capability for semantic relationships between cross-modal data, making it difficult to accurately understand the intent behind complex cross-modal requests.

[0003] While some systems incorporate graph neural networks for relation modeling, they often employ isomorphic graph structures, failing to distinguish between node types and edge relationships. This results in insufficient structural information representation and difficulty in effectively capturing the heterogeneity and complementarity between text, layout, and speech semantics. Furthermore, most existing methods employ sequential processing flows, lacking fusion mechanisms between the processing results of different modalities. This leads to fragmented semantic understanding and weak context awareness, impacting the accuracy and real-time performance of downstream service scheduling and task matching.

[0004] Especially in real-world scenarios with noisy speech, complex screenshots and forms, and uneven keyword distribution, single-modal processing methods are prone to missing core intents, leading to service scheduling failures or distorted responses, severely restricting the overall performance and user experience of cross-modal intelligent service systems. Currently, there is a lack of a unified modeling, cross-modal fusion, and semantic linkage high-precision semantic analysis solution, which urgently requires technological breakthroughs.

[0005] Therefore, how to provide a demand semantic analysis method based on cross-modal fusion is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] This invention provides a demand semantic analysis method based on cross-modal fusion, constructing a unified semantic understanding process that integrates cross-modal data such as text, structured form screenshots, and voice commands. It utilizes ModernBERT for text semantic modeling, LayoutLMv3 to extract structured layout information, and a Samba-based speech recognition system to transcribe speech text and extract speech semantic vectors. Furthermore, it constructs a heterogeneous graph structure containing text nodes, structured nodes, and audio text nodes, and leverages HGT graph neural networks to achieve cross-modal semantic propagation and fusion updates between nodes, generating a unified demand semantic representation vector. This method possesses advantages such as strong heterogeneous data processing capabilities, high semantic fusion depth, and high demand understanding accuracy, significantly improving task scheduling efficiency and service response intelligence in complex interactive scenarios. It addresses the shortcomings of existing methods in cross-modal data processing, semantic consistency modeling, and fine-grained task matching.

[0007] To achieve the above objectives, the present invention employs the following technical solution: A demand semantic analysis method based on cross-modal fusion includes the following steps: S1. Collect cross-modal data of user input text information, structured forms or page screenshots, and voice commands, and generate a cross-modal data sample set; S2. Perform contextual semantic modeling on the text information using ModernBERT to generate text vectors; S3. Identify the character content and spatial coordinates in structured forms or screenshots, and input the LayoutLMv3 model to extract layout vectors that integrate structural and semantic information; S4. Transcribe the speech data into text content, and input the transcribed text into ModernBERT to extract audio semantic vectors; S5. Write the text vector, the layout vector of semantic information, and the audio semantic vector as text nodes, structured nodes, and audio text nodes into the heterogeneous graph structure to be constructed, and establish type distinction and edge connection rules to generate the heterogeneous graph structure. S6. Using HGT graph neural network, semantic propagation is performed on nodes of different types in heterogeneous graph. The attention mechanism of node type and edge type is used to perform feature interaction and generate cross-modal joint representation vector.

[0008] Furthermore, S1 specifically includes: S11. Receive the user's natural language requirement description through the text input box deployed on the terminal page, write the received string data into a text file in an encoded format, and save it by name and timestamp. S12. Receive image files submitted by users through the web upload interface, number each image according to the upload order and store it in the preset image data path, saving it as a naming format of image ID and upload time; S13. Perform image content filtering on the uploaded image file, match the preset form template area, extract the image area containing the table or UI control layout, and save it as form image data, named according to the image ID and area type. S14. Call the microphone through the embedded audio acquisition module, set the sampling rate, quantization bit depth, and single-channel PCM format, acquire continuous voice input from the user, and save the recording as an audio file named with the voice ID and acquisition time. S15. Create a cross-modal data sample set, using the sample ID as the primary key. Write the paths of the generated text files, image files, form area image fragments, and audio files into the sample field respectively, record them according to a unified structure, and generate a cross-modal data sample set.

[0009] Furthermore, S2 specifically includes: S21. Read the text file in the cross-modal data sample set, extract the text content, traverse each character in character order, convert the character into the corresponding encoding value, and replace the character with a space character when the character is a newline character, carriage return character, or control character in the ASCII range. Concatenate the encodings of all characters in the original text order to generate a string encoding sequence. S22. Divide the string encoding sequence into several segments according to the space character. For each segment, perform sub-word segmentation using the maximum matching strategy from left to right. Starting from the current character, search the vocabulary for consecutive substrings that start with the character and whose length is from longest to shortest to match the vocabulary initialized when the ModernBERT model is loaded. S23. If the match is successful, the substring is recorded as a subword and the matching continues from the next character after the end. If the match is unsuccessful, the character is replaced by a flag and the match is skipped. Each subword obtains the vocabulary number by looking up the hash table. If no match is found, it is mapped to a predefined unknown word flag number. All the numbers are concatenated in order to generate a subword ID array of length L. S24. Insert a predefined start marker number as the start marker at the beginning of the sub-word ID array, add a predefined end marker number at the end, and use the Padding marker, which represents no semantics, to fill the sequence to the specified maximum length, and construct an integer ID array. S25. Input the array of integer IDs into the ModernBERT model, look up the predefined vocabulary to map each integer ID to the corresponding embedding vector, form an embedding matrix, and input the embedding matrix into the 12-layer Transformer encoder in sequence. Perform linear transformation on each layer to generate query, key and value vectors. Calculate the self-attention weight matrix through dot product, and sum the value vectors according to the weights to generate attention output. S26. The attention output is added to the input vector to form a residual connection, which is then fed into a two-layer feedforward neural network. The first layer performs a linear transformation and then inputs the ReLU activation function. The second layer performs a linear mapping to restore the dimension. The feedforward network output is added to the residual result, and then the layer normalization operation is performed to output the context vector sequence. S27. Read the vector containing the position of the special start marker number from the last layer output of the Transformer, use the vector as the context semantic representation of the current text, and save it as a text file in array format.

[0010] Furthermore, S3 specifically includes: S31. Read the form image file in the cross-modal data sample set, convert the color image to grayscale image, call the text recognition engine to scan the image pixel array, perform image segmentation on the grayscale change area, locate the minimum bounding rectangle of the text area, recognize the character content in the rectangle area, calculate the image coordinates of the upper left and lower right corners of the area, combine the character content and coordinates into a quadruple, and write it into the recognition result list. S32. For each character quadruple in the recognition result list, read the characters in character order, convert them into codes and concatenate them into a string. Read the upper left and lower right coordinates, divide them by the image width and height respectively to normalize them, retain six decimal places, and combine the encoded text and normalized coordinates into an input pair and write it into the sub-word coordinate pair list. S33. For each item in the sub-word coordinate pair list, use the ModernBERT segmenter to segment the text fragment into sub-words according to the maximum matching strategy. Copy the four normalized coordinates corresponding to the sub-words. Concatenate all sub-words into the input sub-word sequence. Write the coordinates of the sub-words into a two-dimensional array in order. Read the image file, use the image reading library to convert it into an RGB tensor, normalize the pixel values, and generate three input items. S34. Input the encapsulated text sequence, coordinate array and image tensor into the LayoutLMv3 model. For each word in the input word sequence, use word embedding lookup table to map it into a word vector. Read the corresponding position code from the two-dimensional coordinate array and concatenate them to the end of the word vector in dimensional order. S36. Extract the local RGB tensor features corresponding to the coordinate regions in the image, extract the pixel blocks in the coordinate regions of the image, average the R, G, and B values ​​of each pixel in the region to generate a three-dimensional color vector, map the color vector to the image feature vector through a fully connected layer, and concatenate it to the end of the sub-word vector to generate a fused representation. S37. Take the fusion vector corresponding to each sub-word as input and pass it to the encoder of LayoutLMv3. Calculate the dot product of the query of the current sub-word and the keys of all sub-words, divide by the scaling factor and perform Softmax normalization to obtain the attention weight. Use the attention weight to perform a weighted sum of the value vectors of all sub-words as the new context representation of the current sub-word. S38. Read the image and text attention vector corresponding to the current sub-word, add it element by element along the same dimension to the context vector after text attention weighting, generate a fusion vector, add the fusion vector to the input vector element by element to form a residual connection, normalize the residual vector according to the dimension, repeat 12 layers, and output the context embedding vector of each sub-word. S39. For all word embedding vectors, sum all vectors element by element along the dimension and divide by the total number of words to generate a single layout vector, and save it as a text file in array format.

[0011] Furthermore, S4 specifically includes: S41. Read the audio file, extract the waveform array and the original sampling rate, reset the sampling rate to a uniform sampling frequency, convert the waveform to a uniform encoding format, cut a fixed-length sampling point frame segment every preset number of sampling points, multiply each frame by a Hamming window of equal length, perform FFT to obtain the spectrum, and take the square of the complex modulus to obtain the power spectrum. S42. Project the energy distribution in each frame's spectrum onto a fixed number of Mel filter banks with equal perceptual intervals according to the frequency coordinates. For each filter channel, multiply the power spectrum value in the corresponding frequency range by a preset weighting coefficient and sum them up to obtain an equal number of weighted energy values. Take the natural logarithm of each weighted energy value to form the equal-dimensional Mel spectrum feature vector of the current frame. S43. Input the Mel spectrum features of each frame into the encoder of the speech recognition system based on the Samba structure, perform matrix multiplication and ReLU activation to generate acoustic vectors, concatenate them into a two-dimensional tensor and send them into the beam search decoder, multiply each frame with the classification weight matrix to obtain the log probability, normalize it with Softmax to the character probability distribution, construct the path, select the path with the highest cumulative score, look up the table and concatenate the characters to output the transcribed text string. S44. Traverse each character in the transcribed text in character order, convert it to a string encoding, replace the escape character with a space, and then concatenate them into a string encoding sequence, which is used as the word segmentation input text for the ModernBERT model. S45. Use ModernBERT's maximum matching tokenizer to decompose the string into several sub-word fragments, perform a table lookup operation on each sub-word, and mark it as a predefined unknown word tag number if no match is found. Generate a sub-word ID sequence, insert the start tag number and the end tag number at the beginning and end, and use the Padding tag, which represents a placeholder without semantic meaning, to fill the length of the sequence. S46. Input the integer array of subword IDs into the ModernBERT embedding layer, map the embedding matrix, and then input the 12-layer Transformer structure in sequence. Perform self-attention calculation, feedforward network mapping, residual connection and layer normalization in each layer to generate the context embedding vector sequence corresponding to each subword position. S47. Extract the position vector of the starting marker number from the output of the last layer of Transformer, use it as the audio semantic vector corresponding to the audio, and save it as an array file.

[0012] Furthermore, S6 specifically includes: S61. Read the data file of the heterogeneous graph structure, parse the node records, extract the node number, type and attribute. If it is text, structured, or audio text, add the node to the text, structured, or audio text node set respectively. Read the edge connection record, extract the starting node number, target node number and edge type of the edge, and write it into the adjacency dictionary with the starting number to the target number as the key and the edge type as the value to construct the adjacency representation of the heterogeneous graph. S62. Perform graph neural network propagation operation, initialize independent parameter matrices for text, structured, and audio text nodes, traverse and read the corresponding text semantic vector, layout vector, or audio semantic vector, perform linear transformation on nodes of the same type using the parameter matrix, perform weighted summation and bias shift on each dimension of the vector, generate a unified dimension of in-type feature representation, and write the transformation results of all nodes into the in-type feature set. S63. Traverse each type of edge connection, load the attention parameters corresponding to the edge type, read the feature representations of the source node and the target node, multiply them by the query matrix weights to generate a query vector, multiply the target node feature vector by the key matrix weights to generate a key vector, and sum the key vectors after performing element-wise multiplication to obtain the scaled dot product as the attention score. Perform Softmax normalization on all edge scores to obtain the attention weight coefficients. S64. Multiply the attention weight by the value vector of the target node, perform element-wise multiplication and sum to obtain the feature vector propagating on the edge. Add the feature vector to the original feature representation of the source node element-wise to form the residual connection vector. Subtract the mean and divide by the standard deviation for each dimension of the residual connection vector, perform layer normalization operation, and output the fusion representation vector of the edge type. S65. When a node has multiple edge types, perform attention weight calculation, feature propagation, residual connection and layer normalization operations for each edge type to generate multiple information fusion representations and concatenate them in the dimensional direction. This serves as the multi-source fusion input for the node and is fed into a two-layer feedforward neural network. The first layer performs linear transformation and ReLU activation, and the second layer compresses the dimension to the target length through linear mapping, outputting the final feature representation. S66. Repeatedly execute the graph neural network propagation operation for a total of 12 layers. In each layer, update the feature vectors of all nodes with new representations. After completing the 12-layer operation, output the cross-modal joint representation vector set.

[0013] Compared with the prior art, the beneficial effects of the present invention are: This invention addresses the problems of ambiguous semantic understanding, fragmented modal information, and low task matching accuracy in existing cross-modal input scenarios. By combining ModernBERT, LayoutLMv3, and a Samba-based speech recognition system model, it extracts text semantics, structured layout information, and audio semantic vectors respectively, constructing a heterogeneous graph structure containing multiple types of nodes and edge relationships. HGT graph neural networks are then used to achieve semantic propagation and fusion expression between cross-modal nodes, generating a unified cross-modal semantic representation vector. This accurately expresses the contextual meaning of user needs and serves as the input basis for scheduling or matching tasks. This invention possesses advantages such as high modal fusion depth, strong semantic modeling capability, good adaptability to heterogeneous structures, and high accuracy of understanding results. It effectively solves the problems of fragmented understanding, inaccurate response, and scheduling failure in traditional methods when facing heterogeneous inputs such as text, screenshots, and speech, significantly improving the intelligence level of task response and the accuracy and stability of service matching. Attached Figure Description

[0014] Figure 1 This is a flowchart of the method described in this invention.

[0015] Figure 2 This is a flowchart of the heterogeneous graph structure construction and cross-modal node relationship establishment in this invention.

[0016] Figure 3 This is a flowchart of cross-modal semantic propagation and joint representation generation based on HGT graph neural network in this invention. Detailed Implementation

[0017] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings: See Figure 1 The present invention provides a demand semantic analysis method based on cross-modal fusion, comprising the following steps: S1. Collect cross-modal data such as user-input text information, structured forms or page screenshots, and voice commands through the terminal interface, and generate a cross-modal data sample set; S2. Perform contextual semantic modeling on the text information using ModernBERT to generate text vectors with semantic integrity; S3. Identify the character content and spatial coordinates in structured forms or screenshots, and input the LayoutLMv3 model to extract layout vectors that integrate structural and semantic information; S4. Transcribe the speech data into text content, and input the transcribed text into ModernBERT to extract audio semantic vectors; S5. Write the text vector, the layout vector of semantic information, and the audio semantic vector as text nodes, structured nodes, and audio text nodes into the heterogeneous graph structure to be constructed, respectively, and establish type distinction and edge connection rules to generate a heterogeneous graph structure containing multiple node types and multiple edge relationships. S6. Using HGT graph neural network, semantic propagation is performed on nodes of different types in heterogeneous graph. The attention mechanism of node type and edge type is used to perform feature interaction and generate cross-modal joint representation vector, which represents the overall semantic content of the current cross-modal requirement and is used for subsequent service scheduling or task matching.

[0018] This implementation method accurately constructs a computable semantic representation by fusing cross-modal features from text, image layout, and voice information, significantly improving the ability to understand complex user needs. It enhances the completeness and accuracy of language expression by modeling text context semantics using ModernBERT; it extracts visual structure and semantic content from structured forms using LayoutLMv3, achieving deep perception of nested layouts and key information areas; it uses a Samba-based speech recognition system to accurately transcribe voice content and extract semantics, opening up the voice input channel; it constructs a heterogeneous graph structure by fusing features from the three modalities, and completes semantic interaction and feature propagation between nodes based on the type-aware attention mechanism of the HGT graph neural network, ultimately outputting a unified cross-modal joint representation vector. This comprehensively expresses the semantics of current needs, supports downstream service matching, task scheduling, and intelligent recommendation, and exhibits stronger robustness, generalization ability, and semantic consistency in complex scenarios such as multi-source input, semantic ambiguity, and modal differences, providing a highly reliable semantic foundation for intelligent systems.

[0019] Step S1 specifically includes: S11. Receive the user's natural language requirement description through the text input box deployed on the terminal page, write the received string data into a text file in an encoded format, and save it by name and timestamp. S12. Receive image files submitted by users through the web upload interface, number each image according to the upload order and store it in the preset image data path, saving it as a naming format of image ID and upload time; S13. Perform image content filtering on the uploaded image file, match the preset form template area, extract the image area containing the table or UI control layout, and save it as form image data in .png format, named according to image ID and area type; S14. Call the microphone through the embedded audio acquisition module, set the sampling rate to 16kHz, the quantization bit depth to 16bit, and the single-channel PCM format, acquire the user's continuous voice input within 10 seconds, save the recording as a .wav file, and name it with the voice ID and acquisition time. S15. Create a cross-modal data sample set. Using the sample ID as the primary key, write the paths of the generated text files, .png image files, form area image fragments, and .wav audio files into the sample field respectively, and record them in a CSV table according to a unified structure. This will generate a cross-modal data sample set with four types of heterogeneous modal data: text, images, form structures, and audio. This set will serve as the input data sample source for subsequent feature extraction and graph structure construction.

[0020] This implementation method constructs a standardized cross-modal data acquisition process, generating a data sample set with four modalities: text, image, form structure, and audio. This significantly improves the consistency and data quality of cross-modal information processing. Text input is saved in a standardized encoding format, image data is accurately extracted from form regions through template matching, and voice acquisition uses a high-fidelity configuration to ensure speech clarity. All data is uniformly recorded in a structured CSV table using sample IDs, enabling automatic data archiving and rapid indexing. This effectively ensures the structural integrity and modal alignment of multi-source heterogeneous data, laying a high-quality, low-noise input foundation for subsequent semantic modeling and heterogeneous graph construction, and greatly improving the accuracy and stability in cross-modal requirement understanding tasks.

[0021] Step S2 specifically includes: S21. Read the text file in the cross-modal data sample set, extract the text content, traverse each character in character order, convert the character to the corresponding Unicode encoding value, and replace the character with a space character when the character is a newline character, carriage return character, or control character in the ASCII range. Concatenate the Unicode encodings of all characters in the original text order to generate a Unicode string encoding sequence of the same length, which is used as the original text input for the subsequent word segmenter. S22. Divide the Unicode string encoding sequence into several segments according to the space character. Each segment represents a continuous English word, number string or single Chinese character. For each segment, use the maximum matching strategy from left to right to perform sub-word segmentation. Starting from the current character, search the vocabulary in turn for a continuous substring that starts with the character and has the longest to shortest length in the vocabulary initialized when the ModernBERT model is loaded. S23. If a match is successful, the substring is recorded as a subword and the matching continues from the next character after the end. If a match is unsuccessful, the [UNK] flag is used to replace the character and the match is skipped. Each subword obtains its vocabulary number by hash lookup. If a match is not found, it is mapped to a predefined unknown word flag number. All numbers are concatenated in order to generate a subword ID array of length L, which serves as the input tensor for subsequent model encoding. S24. Insert a start marker with the number 101 at the beginning of the sub-word ID array, add a stop marker with the number 102 at the end, and use the placeholder marker with the number 0 to represent no semantics, Padding, to fill the sequence to the specified maximum length L=512, thus constructing an integer ID array of fixed length. S25. Input the integer ID array into the ModernBERT model, look up the predefined vocabulary to map each integer ID to the corresponding 768-dimensional embedding vector, form an embedding matrix of [512, 768], and input the embedding matrix into 12 layers of Transformer encoders in sequence. Perform linear transformation on each layer to generate query, key and value vectors, calculate the self-attention weight matrix through dot product, and sum the value vectors according to the weights to generate attention output. S26. The attention output is added to the input vector to form a residual connection, which is then fed into a two-layer feedforward neural network. The first layer performs a linear transformation and then inputs the ReLU activation function. The second layer performs a linear mapping to restore the dimension. The feedforward network output is added to the residual result, and then the layer normalization operation is performed to update the context semantic representation of each position. The final output is a context vector sequence of shape [512, 768]. S27. Read the vector at the position of the special start marker [CLS] with the corresponding number 101 at the start position from the last layer output of the Transformer, use the vector as the context semantic representation of the current text, and save it as a .npy file in NumPy array format.

[0022] This implementation uses the ModernBERT model to model the contextual semantics of natural language text, effectively extracting text vectors with semantic integrity. Character-level preprocessing ensures consistency of input text at the Unicode level, and a maximum matching strategy is used for refined word segmentation to enhance the model's ability to identify semantic boundaries. Special markers and a fixed-length padding mechanism are used to construct standardized input tensors, ensuring consistent processing of different text samples during the encoding stage. The embedding layer maps word numbers to high-dimensional vectors, and the Transformer structure performs self-attention calculation and feedforward network updates layer by layer. Combined with residual connections and layer normalization operations, the contextual dependencies between words are fully modeled. The extracted [CLS] vector serves as the overall semantic representation of the text, possessing semantic condensation and task adaptability capabilities. The final generated semantic vectors are saved in standard NumPy format, providing a high-quality semantic foundation for subsequent cross-modal semantic alignment and heterogeneous graph modeling, significantly improving the accuracy and consistency of cross-modal information understanding.

[0023] Specifically, S3 includes: S31. Read the form image file in the cross-modal data sample set, convert the color image to grayscale image, call the OCR text recognition engine to scan the image pixel array, perform image segmentation on the grayscale change area, locate the minimum bounding rectangle of the text area, recognize the character content in the rectangle area, calculate the image coordinates of the upper left and lower right corners of the area, form a quadruple with the character content and coordinates, and write it into the recognition result list. S32. For each character quadruple in the recognition result list, read the characters in character order, convert them to Unicode encoding and concatenate them into a string, read the upper left and lower right coordinates, divide them by the image width and height respectively to normalize them, retain six decimal places, combine the encoded text and normalized coordinates to form an input pair, and write it into the sub-word coordinate pair list. S33. For each item in the list of sub-word coordinate pairs, use the ModernBERT segmenter to segment the text fragment into sub-words according to the maximum matching strategy. Copy the four normalized coordinates corresponding to each sub-word, concatenate all sub-words into the input sub-word sequence, write the coordinates of the sub-words into a two-dimensional array in order, read the image file, use the image reading library to convert it into an RGB tensor, divide the pixel values ​​by 255 for normalization, and generate three input items to be passed into the LayoutLMv3 model. S34. Input the encapsulated text sequence, coordinate array and image tensor into the LayoutLMv3 model. For each word in the input word sequence, use word embedding lookup table to map it into a 768-dimensional word vector. Read the corresponding position code from the two-dimensional coordinate array and concatenate them to the end of the word vector in dimensional order. S36. Extract the local RGB tensor features corresponding to the coordinate regions in the image, extract the pixel blocks in the coordinate regions of the image, average the R, G, and B values ​​of each pixel in the region to generate a three-dimensional color vector, map the color vector to a 256-dimensional image feature vector through a fully connected layer, and concatenate it to the end of the sub-word vector to generate a fused representation. S37. Take the 1028-dimensional fusion vector corresponding to each sub-word as input and pass it to the encoder of LayoutLMv3. Calculate the dot product of the query of the current sub-word and the keys of all sub-words, and divide by the scaling factor. Then, Softmax normalization is performed to obtain attention weights. The attention weights are used to perform a weighted sum of the value vectors of all sub-words, which serves as the new context representation of the current token, thus completing the cross-sub-word information aggregation. S38. Read the image and text attention vector corresponding to the current sub-word, add it element by element along the same dimension to the context vector after text attention weighting, generate a fusion vector, add the fusion vector to the input vector element by element to form a residual connection, normalize the residual vector according to the dimension to a standard vector with a mean of 0 and a standard deviation of 1, repeat 12 layers, and output the 768-dimensional context embedding vector of each sub-word. S39. For all word embedding vectors, sum all vectors element by element along the dimensional direction and divide by the total number of words to generate a single 768-dimensional layout vector, and save it as a .npy file in Numpy array format.

[0024] This implementation uses the LayoutLMv3 model to deeply fuse and represent text and layout information in structured form images, effectively extracting semantic vectors with spatial structure awareness. An OCR text recognition engine is used to perform character recognition and coordinate localization on the image, generating precise character content and region coordinate quadruples. Normalization and Unicode encoding are then performed to map character fragments into word sequences, simultaneously processing spatial location and image features. A ModernBERT segmenter is used to segment the words, extracting RGB image features of the corresponding coordinate regions for each word, constructing an input vector that fuses text, location, and visual information. The encoding layer sequentially performs image-text cross-attention calculation and context modeling. Through multi-layer residual connections and normalization mechanisms, a semantic embedding representation for each word is generated. Finally, average pooling yields a single 768-dimensional layout vector with both semantic expression and spatial awareness capabilities. This layout vector provides a stable and highly expressive semantic foundation for structured graph construction and cross-modal requirement matching, significantly improving the accuracy of information extraction and structural understanding in image-text fusion scenarios.

[0025] Step S4 specifically includes: S41. Read the .wav audio file, extract the waveform array and the original sampling rate, reset the sampling rate to 16kHz, multiply the waveform by 32768 and take the integer part to convert it to 16-bit PCM format, extract 400 sampling point frames every 160 sampling points, multiply each frame by a 400-dimensional Hamming window, perform FFT to obtain the spectrum, and take the square of the complex modulus to obtain the power spectrum. S42. Project the energy distribution in each frame's spectrogram onto 40 equally spaced Mel filter banks according to frequency coordinates. For each filter channel, multiply the power spectral value within the corresponding frequency range by a preset weighting coefficient and sum them to obtain 40 weighted energy values. Take the natural logarithm of each weighted energy value to form a 40-dimensional Mel spectral feature vector for the current frame. S43. Input the Mel spectrum features of each frame into the encoder of the speech recognition system based on the Samba structure, perform matrix multiplication and ReLU activation to generate acoustic vectors, concatenate them into a two-dimensional tensor and send them into the beam search decoder, multiply each frame with the classification weight matrix to obtain the log probability, normalize it with Softmax to the character probability distribution, construct the path, select the path with the highest cumulative score, look up the table and concatenate the characters to output the transcribed text string. S44. Traverse each character in the transcribed text in character order, convert it to Unicode encoding, replace the escape characters with spaces, and then concatenate them into a Unicode string encoding sequence, which is used as the word segmentation input text for the ModernBERT model. S45. Use ModernBERT's maximum matching tokenizer to decompose the Unicode string into several sub-word fragments. Perform a lookup operation on each sub-word. If no match is found, use the [UNK] marker with number 100 to generate a sub-word ID sequence. Insert start and end markers with numbers 101 and 102 at the beginning and end, and use the Padding marker with number 0 to fill the sequence to a length of 512. S46. Input the integer array of subword IDs into the ModernBERT embedding layer and map it into a 512×768 embedding matrix. Then, input it into a 12-layer Transformer structure in sequence. Perform self-attention calculation, feedforward network mapping, residual connection and layer normalization in each layer to generate a 768-dimensional context embedding vector sequence corresponding to each subword position. S47. Extract the position vector numbered 101 from the output of the last layer of Transformer, use it as the audio semantic vector corresponding to the audio, and save it as an array file in .npy format.

[0026] This implementation combines a Samba-based speech recognition system with the ModernBERT model to achieve deep semantic extraction from .wav format audio data. The audio is resampled to 16kHz and converted to 16-bit PCM format. Frame-by-frame truncation and windowing are performed followed by FFT to extract the spectrogram of each frame. A 40-dimensional Mel-based spectral feature is calculated using a Mel filter bank to comprehensively capture semantic and phoneme variations in the speech. The Mel-based feature is input into the Samba-based speech recognition system model to generate acoustic vectors frame-by-frame. These vectors are then fed into a beam search decoder, outputting high-precision transcribed text with consistent temporal order. The ModernBERT model is then used to perform contextual modeling on the transcribed text, ultimately outputting a 768-dimensional audio semantic vector. This approach integrates acoustic structure and linguistic semantic information, possessing strong expressive power. As input to heterogeneous graphs for audio text nodes, it significantly improves the accuracy of speech information understanding and structural matching capabilities in cross-modal demand modeling.

[0027] See Figure 2 Step S5 specifically includes: S51. Traverse each text vector file output by ModernBERT, read the corresponding text node ID as the node primary key, read the semantic vector content as the attribute value, create a node record with the type marked as text, and write it into the graph structure node table to be built. S52. Traverse each structured layout vector file output by LayoutLMv3, read the structured node ID as the primary key, read the corresponding 768-dimensional vector as the node attribute, create a node record marked as structured, and add it to the graph structure node table. S53. Traverse each audio semantic vector file output by ModernBERT, read the frequency text node ID as the primary key, read the corresponding .npy format vector as the node attribute, create a node record with the type marked as audio text, and write it into the graph structure node table; S54. For each text node in the graph structure, search whether the keywords in the text content overlap with the text fragments of the structured node. If there is an intersection, add an edge connecting the text node and the matching structured node, and set the edge type to text and structure association. S55. For each audio text node, match the keywords in the transcribed text with the original content of the text node. If there is at least one matching segment, create an edge connecting the audio text node and the text node, and set the edge type to speech-text association. S56. For any two structured nodes in the graph structure, if the distance between their coordinate positions is less than a set spatial threshold, an edge connection is established, and the edge type is set to structural proximity. S57. Write all node records and edge connection records into the graph database modeling template, and save them as structured graph data files in the format of "node ID-node type-attribute value" and "starting point ID-ending point ID-edge type", and finally generate a heterogeneous graph structure containing three types of nodes and three types of edges.

[0028] This implementation constructs a heterogeneous graph structure with three types of nodes and three types of edge relationships to achieve a unified representation of cross-modal data. Text, structured layout, and audio semantic vectors output from ModernBERT and LayoutLMv3 models are read sequentially, and three types of nodes—text, structured layout, and audio text—are created respectively, retaining the semantic vectors as node attributes. Through keyword matching and spatial proximity analysis, three types of edge connections are automatically established: text-to-structure association, speech-to-text association, and structure proximity relationship, achieving semantic and spatial hierarchy fusion between different modalities. Finally, a structured graph data file is generated, serving as the foundation for subsequent graph neural network modeling and providing highly consistent and semantically aligned graph structure support for understanding cross-modal needs.

[0029] See Figure 3 Step S6 specifically includes: S61. Read the data file of the heterogeneous graph structure, parse the node records, extract the node number, type and attribute. If it is text, structured, or audio text, add the node to the text, structured, or audio text node set respectively. Read the edge connection record, extract the starting node number, target node number and edge type of the edge, and write it into the adjacency dictionary with the starting number to the target number as the key and the edge type as the value to construct the adjacency representation of the heterogeneous graph. S62. Perform graph neural network propagation operation, initialize independent parameter matrices for text, structured, and audio text nodes, traverse and read the corresponding text semantic vector, layout vector, or audio semantic vector, perform linear transformation on nodes of the same type using the parameter matrix, perform weighted summation and bias shift on each dimension of the vector, generate a unified dimension of in-type feature representation, and write the transformation results of all nodes into the in-type feature set. S63. Traverse each type of edge connection, load the attention parameters corresponding to the edge type, read the feature representations of the source node and the target node, multiply them by the query matrix weights to generate a query vector, multiply the target node feature vector by the key matrix weights to generate a key vector, and sum the key vectors after performing element-wise multiplication to obtain the scaled dot product as the attention score. Perform Softmax normalization on all edge scores to obtain the attention weight coefficients. S64. Multiply the attention weight by the value vector of the target node, perform element-wise multiplication and sum to obtain the feature vector propagating on the edge. Add the feature vector to the original feature representation of the source node element-wise to form the residual connection vector. Subtract the mean and divide by the standard deviation for each dimension of the residual connection vector, perform layer normalization operation, and output the fusion representation vector of the edge type. S65. When a node has multiple edge types, perform attention weight calculation, feature propagation, residual connection and layer normalization operations for each edge type to generate multiple information fusion representations and concatenate them in the dimensional direction. This serves as the multi-source fusion input for the node and is then fed into a two-layer feedforward neural network. The first layer performs linear transformation and ReLU activation, while the second layer compresses the dimension to the target length through linear mapping and outputs the final feature representation, thus completing the update propagation of the graph neural network. S66. Repeatedly execute the graph neural network propagation operation for a total of 12 layers. In each layer, update the feature vectors of all nodes with new representations. After completing the 12-layer operation, output a set of cross-modal joint representation vectors that integrate text semantics, layout structure and audio semantics. This set is used to achieve the required semantic understanding in subsequent task scheduling and semantic matching operations.

[0030] This implementation achieves deep fusion modeling of cross-modal semantics by constructing an adjacency representation of a heterogeneous graph and a multi-layer graph neural network propagation mechanism. It parses the graph structure file, classifies text, structured, and audio text nodes by type, and establishes an adjacency dictionary for each start-end-edge type. For different node types, it initializes a dedicated linear transformation parameter matrix, performs weighted summation and bias shift on the semantic vectors, and generates a unified-dimensional feature representation. Attention calculation, feature propagation, residual connections, and layer normalization are performed according to edge type to ensure accurate propagation of semantic relationships between different types. When a node has multiple edge types, the fused representation is concatenated and then introduced into a feedforward network to extract the final semantic features. The entire graph neural network structure propagates through 12 layers, aggregating multi-source information layer by layer, and finally outputs a cross-modal joint representation vector that fuses text, structured, and audio semantics. This provides a deep semantic expression capability that is semantically consistent, modally complete, and accurately aligned for subsequent task scheduling and semantic matching scenarios, significantly improving the system's understanding accuracy and response capability to complex cross-modal requirements.

[0031] The following embodiments are implemented based on the technical solution of the present invention, providing detailed implementation methods and specific operation processes. However, the scope of protection of the present invention is not limited to the following embodiments. Unless otherwise specified, the methods used in the following embodiments are conventional methods.

[0032] Example 1

[0033] To verify the feasibility and effectiveness of this invention in implementation, it was applied to the production collaborative scheduling platform of a large intelligent manufacturing enterprise in a certain province. The platform serves the task flow, equipment scheduling, and fault reporting between multiple workshops and production lines. The enterprise originally used keyword matching and manual classification to analyze and distribute the task requests submitted by employees, including cross-departmental task requests in different forms such as equipment malfunction reports, material replenishment requests, production schedule adjustments, and process switching instructions. However, due to the diverse user input methods, including voice commands, form screenshots, and semantic description text, in the original system, different modal inputs had to enter independent processing flows, resulting in problems such as fragmented data understanding, mismatched task scheduling, and cross-modal information silos, which seriously restricted response efficiency and the level of intelligence.

[0034] After deploying this invention to the platform, users can submit task requirements in any way, such as voice, screenshots, or natural language descriptions, on mobile terminals. The system automatically performs cross-modal data unified encoding on user input, extracts semantic representations through the text processing model ModernBERT, extracts text and spatial layout vectors from form screenshots through the LayoutLMv3 model, transcribes speech into text and obtains audio semantic vectors through a speech recognition system model based on the Samba structure, automatically constructs a heterogeneous graph structure, embeds all information into graph nodes, and then completes semantic fusion modeling and task category matching of cross-modal information through the HGT graph neural network.

[0035] During 20 consecutive days of operation, key indicators such as cross-modal input recognition accuracy, inter-modal semantic fusion success rate, and task understanding accuracy were collected and compared for three main task scenarios: equipment repair, process adjustment, and material scheduling. The results are as follows: Table 1. Comparison of Recognition and Fusion Performance of Different Methods in Cross-Modal Input Scenarios As shown in Table 1, the cross-modal fusion-based semantic analysis method proposed in this invention demonstrates significant advantages over traditional methods in multiple dimensions, including recognition rate, fusion efficiency, and response speed for cross-modal input tasks. Taking equipment repair tasks as an example, the original system, lacking the ability to collaboratively understand speech and screenshots, could only achieve a recognition accuracy of 72.6%, with a fusion success rate as low as 38.2%, easily leading to task classification errors and response mismatches. In contrast, this invention, by constructing cross-modal edge connections between speech text nodes and screenshot structured nodes, achieves a recognition rate as high as 96.3% and a semantic fusion rate of 94.8% under the propagation mechanism of HGT graph neural networks, with a collaborative error rate of only 2.1%, significantly improving the system's robustness.

[0036] In scenarios such as process adjustment and material scheduling, this invention also demonstrates strong semantic alignment capabilities. The original system relies on rule matching or manual annotation, which makes it difficult to understand the potential consistent semantic relationships in different modal inputs, resulting in a modal collaborative understanding error of generally more than 14%. However, this invention effectively bridges the modal gap between structured information, text semantics and voice commands by differentiating node types and embedding edge types in heterogeneous graphs, reducing the modal collaborative error to 1.6%-2.9% and achieving a task recognition accuracy of over 97%.

[0037] In terms of response speed, the original system took more than 40 seconds on average to process a single cross-modal task, and task scheduling heavily relied on manual intervention. However, thanks to the unified graph structure modeling and graph neural network inference mechanism, the system of this invention reduces the average response latency to less than 12 seconds. The material scheduling task has the lowest response latency, at only 11.2 seconds, demonstrating the significant improvement in scheduling efficiency brought about by cross-modal fusion.

[0038] In summary, the method of this invention not only improves the understanding depth and fusion efficiency of cross-modal information, but also constructs a unified semantic expression space through heterogeneous graph modeling and HGT graph neural network, breaking the limitations of fragmented processing of various modalities in traditional task systems. In industrial scheduling scenarios with high-frequency interweaving of multi-source inputs, it has strong generalization ability and practical deployment value, effectively supporting core links such as intelligent task parsing, semantic alignment and real-time response, and providing stable and efficient underlying semantic understanding support for intelligent manufacturing and human-machine collaborative systems.

Claims

1. A requirement semantic analysis method based on cross-modal fusion, characterized in that, Includes the following steps: S1. Collect cross-modal data of user input text information, structured forms or page screenshots, and voice commands, and generate a cross-modal data sample set; S2. Perform contextual semantic modeling on the text information using ModernBERT to generate text vectors; S3. Identify the character content and spatial coordinates in structured forms or screenshots, and input the LayoutLMv3 model to extract layout vectors that integrate structural and semantic information; S4. Transcribe the speech data into text content, and input the transcribed text into ModernBERT to extract audio semantic vectors; S5. Write the text vector, the layout vector of semantic information, and the audio semantic vector as text nodes, structured nodes, and audio text nodes into the heterogeneous graph structure to be constructed, and establish type distinction and edge connection rules to generate the heterogeneous graph structure. S6. Using HGT graph neural network, semantic propagation is performed on nodes of different types in heterogeneous graph. The attention mechanism of node type and edge type is used to perform feature interaction and generate cross-modal joint representation vector.

2. The requirement semantic analysis method based on cross-modal fusion according to claim 1, characterized in that, S1 specifically includes: S11. Receive the user's natural language requirement description through the text input box deployed on the terminal page, write the received string data into a text file in an encoded format, and save it by name and timestamp. S12. Receive image files submitted by users through the web upload interface, number each image according to the upload order and store it in the preset image data path, saving it as a naming format of image ID and upload time; S13. Perform image content filtering on the uploaded image file, match the preset form template area, extract the image area containing the table or UI control layout, and save it as form image data, named according to the image ID and area type. S14. Call the microphone through the embedded audio acquisition module, set the sampling rate, quantization bit depth, and single-channel PCM format, acquire continuous voice input from the user, and save the recording as an audio file named with the voice ID and acquisition time. S15. Create a cross-modal data sample set, using the sample ID as the primary key. Write the paths of the generated text files, image files, form area image fragments, and audio files into the sample field respectively, record them according to a unified structure, and generate a cross-modal data sample set.

3. The requirement semantic analysis method based on cross-modal fusion according to claim 1, characterized in that, S2 specifically includes: S21. Read the text file in the cross-modal data sample set, extract the text content, traverse each character in character order, convert the character into the corresponding encoding value, and replace the character with a space character when the character is a newline character, carriage return character, or control character in the ASCII range. Concatenate the encodings of all characters in the original text order to generate a string encoding sequence. S22. Divide the string encoding sequence into several segments according to the space character. For each segment, perform sub-word segmentation using the maximum matching strategy from left to right. Starting from the current character, search the vocabulary for consecutive substrings that start with the character and whose length is from longest to shortest to match the vocabulary initialized when the ModernBERT model is loaded. S23. When a match is successful, the substring is recorded as a subword and the matching continues from the next character after the end. When a match is unsuccessful, a flag is used to replace the character and the match is skipped. Each subword obtains its vocabulary number by looking up the hash table. If a match is not found, it is mapped to a predefined unknown word flag number. All numbers are concatenated in order to generate a subword ID array of length L. S24. Insert a predefined start marker number as the start marker at the beginning of the sub-word ID array, add a predefined end marker number at the end, and use the Padding marker, which represents no semantics, to fill the sequence to the specified maximum length, and construct an integer ID array. S25. Input the array of integer IDs into the ModernBERT model, look up the predefined vocabulary to map each integer ID to the corresponding embedding vector, form an embedding matrix, and input the embedding matrix into the 12-layer Transformer encoder in sequence. Perform linear transformation on each layer to generate query, key and value vectors. Calculate the self-attention weight matrix through dot product, and sum the value vectors according to the weights to generate attention output. S26. The attention output is added to the input vector to form a residual connection, which is then fed into a two-layer feedforward neural network. The first layer performs a linear transformation and then inputs the ReLU activation function. The second layer performs a linear mapping to restore the dimension. The feedforward network output is added to the residual result, and then the layer normalization operation is performed to output the context vector sequence. S27. Read the vector containing the position of the special start marker number from the last layer output of the Transformer, use the vector as the context semantic representation of the current text, and save it as a text file in array format.

4. The requirement semantic analysis method based on cross-modal fusion according to claim 1, characterized in that, S3 specifically includes: S31. Read the form image file in the cross-modal data sample set, convert the color image to grayscale image, call the text recognition engine to scan the image pixel array, perform image segmentation on the grayscale change area, locate the minimum bounding rectangle of the text area, recognize the character content in the rectangle area, calculate the image coordinates of the upper left and lower right corners of the area, combine the character content and coordinates into a quadruple, and write it into the recognition result list. S32. For each character quadruple in the recognition result list, read the characters in character order, convert them into codes and concatenate them into a string. Read the upper left and lower right coordinates, divide them by the image width and height respectively to normalize them, retain six decimal places, and combine the encoded text and normalized coordinates into an input pair and write it into the sub-word coordinate pair list. S33. For each item in the sub-word coordinate pair list, use the ModernBERT segmenter to segment the text fragment into sub-words according to the maximum matching strategy. Copy the four normalized coordinates corresponding to the sub-words. Concatenate all sub-words into the input sub-word sequence. Write the coordinates of the sub-words into a two-dimensional array in order. Read the image file, use the image reading library to convert it into an RGB tensor, normalize the pixel values, and generate three input items. S34. Input the encapsulated text sequence, coordinate array and image tensor into the LayoutLMv3 model. For each word in the input word sequence, use word embedding lookup table to map it into a word vector. Read the corresponding position code from the two-dimensional coordinate array and concatenate them to the end of the word vector in dimensional order. S36. Extract the local RGB tensor features corresponding to the coordinate regions in the image, extract the pixel blocks in the coordinate regions of the image, average the R, G, and B values ​​of each pixel in the region to generate a three-dimensional color vector, map the color vector to the image feature vector through a fully connected layer, and concatenate it to the end of the sub-word vector to generate a fused representation. S37. Take the fusion vector corresponding to each sub-word as input and pass it to the encoder of LayoutLMv3. Calculate the dot product of the query of the current sub-word and the keys of all sub-words, divide by the scaling factor and perform Softmax normalization to obtain the attention weight. Use the attention weight to perform a weighted sum of the value vectors of all sub-words as the new context representation of the current sub-word. S38. Read the image and text attention vector corresponding to the current sub-word, add it element by element along the same dimension to the context vector after text attention weighting, generate a fusion vector, add the fusion vector to the input vector element by element to form a residual connection, normalize the residual vector according to the dimension, repeat 12 layers, and output the context embedding vector of each sub-word. S39. For all word embedding vectors, sum all vectors element by element along the dimension and divide by the total number of words to generate a single layout vector, and save it as a text file in array format.

5. The requirement semantic analysis method based on cross-modal fusion according to claim 1, characterized in that, S4 specifically includes: S41. Read the audio file, extract the waveform array and the original sampling rate, reset the sampling rate to a uniform sampling frequency, convert the waveform to a uniform encoding format, cut a fixed-length sampling point frame segment every preset number of sampling points, multiply each frame by a Hamming window of equal length, perform FFT to obtain the spectrum, and take the square of the complex modulus to obtain the power spectrum. S42. Project the energy distribution in each frame's spectrum onto a fixed number of Mel filter banks with equal perceptual intervals according to the frequency coordinates. For each filter channel, multiply the power spectrum value in the corresponding frequency range by a preset weighting coefficient and sum them up to obtain an equal number of weighted energy values. Take the natural logarithm of each weighted energy value to form the equal-dimensional Mel spectrum feature vector of the current frame. S43. Input the Mel spectrum features of each frame into the encoder of the speech recognition system based on the Samba structure, perform matrix multiplication and ReLU activation to generate acoustic vectors, concatenate them into a two-dimensional tensor and send them into the beam search decoder, multiply each frame with the classification weight matrix to obtain the log probability, normalize it with Softmax to the character probability distribution, construct the path, select the path with the highest cumulative score, look up the table and concatenate the characters to output the transcribed text string. S44. Traverse each character in the transcribed text in character order, convert it to a string encoding, replace the escape character with a space, and then concatenate them into a string encoding sequence, which is used as the word segmentation input text for the ModernBERT model. S45. Use ModernBERT's maximum matching tokenizer to decompose the string into several sub-word fragments, perform a table lookup operation on each sub-word, and mark it as a predefined unknown word tag number if no match is found. Generate a sub-word ID sequence, insert the start tag number and the end tag number at the beginning and end, and use the Padding tag, which represents a placeholder without semantic meaning, to fill the length of the sequence. S46. Input the integer array of subword IDs into the ModernBERT embedding layer, map the embedding matrix, and then input the 12-layer Transformer structure in sequence. Perform self-attention calculation, feedforward network mapping, residual connection and layer normalization in each layer to generate the context embedding vector sequence corresponding to each subword position. S47. Extract the position vector of the starting marker number from the output of the last layer of Transformer, use it as the audio semantic vector corresponding to the audio, and save it as an array file.

6. The requirement semantic analysis method based on cross-modal fusion according to claim 1, characterized in that, S6 specifically includes: S61. Read the data file of the heterogeneous graph structure, parse the node records, extract the node number, type and attribute. If it is text, structured, or audio text, add the node to the text, structured, or audio text node set respectively. Read the edge connection record, extract the starting node number, target node number and edge type of the edge, and write it into the adjacency dictionary with the starting number to the target number as the key and the edge type as the value to construct the adjacency representation of the heterogeneous graph. S62. Perform graph neural network propagation operation, initialize independent parameter matrices for text, structured, and audio text nodes, traverse and read the corresponding text semantic vector, layout vector, or audio semantic vector, perform linear transformation on nodes of the same type using the parameter matrix, perform weighted summation and bias shift on each dimension of the vector, generate a unified dimension of in-type feature representation, and write the transformation results of all nodes into the in-type feature set. S63. Traverse each type of edge connection, load the attention parameters corresponding to the edge type, read the feature representations of the source node and the target node, multiply them by the query matrix weights to generate a query vector, multiply the target node feature vector by the key matrix weights to generate a key vector, and sum the key vectors after performing element-wise multiplication to obtain the scaled dot product as the attention score. Perform Softmax normalization on all edge scores to obtain the attention weight coefficients. S64. Multiply the attention weight by the value vector of the target node, perform element-wise multiplication and sum to obtain the feature vector propagating on the edge. Add the feature vector to the original feature representation of the source node element-wise to form the residual connection vector. Subtract the mean and divide by the standard deviation for each dimension of the residual connection vector, perform layer normalization operation, and output the fusion representation vector of the edge type. S65. When a node has multiple edge types, perform attention weight calculation, feature propagation, residual connection and layer normalization operations for each edge type to generate multiple information fusion representations and concatenate them in the dimensional direction. This serves as the multi-source fusion input for the node and is fed into a two-layer feedforward neural network. The first layer performs linear transformation and ReLU activation, and the second layer compresses the dimension to the target length through linear mapping, outputting the final feature representation. S66. Repeatedly execute the graph neural network propagation operation for a total of 12 layers. In each layer, update the feature vectors of all nodes with new representations. After completing the 12-layer operation, output the cross-modal joint representation vector set.

Images