A cross-modal signal data analysis method for natural language processing models

Abstract

The present invention relates to the field of natural language processing and cross-modal learning technology, and in particular to a cross-modal signal data analysis method for a natural language processing model, including improving the efficiency and accuracy of data processing by classifying and processing original data sets and assigning identifiers; enabling data of different modalities to be compared and analyzed on the same dimension through extracted features; providing a platform for the fusion of data of different modalities by establishing a shared representation space; better reflecting the essence and inherent laws of data through multi-modal feature information fusion, and providing a data basis for subsequent time dynamic information modeling and analysis; capturing the changing trend in data through modeling and analysis of time dynamic information; and taking timely measures to handle abnormal events through abnormal data identification and event warning. The present invention is used to solve the technical problems of cross-modal data fusion and accurate abnormal warning in existing solutions.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing and cross-modal learning, and in particular to a cross-modal signal data analysis method for a natural language processing model. Background Art

[0002] Natural language processing is an important branch of artificial intelligence, focusing on computer understanding and generation of human language, including tasks such as speech recognition, machine translation, and text summarization; cross-modal learning refers to a learning method that establishes connections between data in different modalities to achieve the transmission and fusion of cross-modal information.

[0003] With the continuous development of science and technology, the application of natural language processing and cross-modal learning technology is particularly extensive. However, there are some problems in the current cross-modal signal data analysis, including the following aspects: the limitation of feature extraction. The current feature extraction method can often only capture part of the data information and it is difficult to fully reflect the essential characteristics of the data; the lack of fusion strategy. Although there are some methods for fusing multimodal data, how to effectively select and apply these fusion strategies is still a problem; the challenge of time dynamic information processing. For cross-modal data containing time series information, how to effectively process and analyze time dynamic information is a challenge; the above are all aspects that need to be focused on in the current research and application of cross-modal signal data analysis. Summary of the invention

[0004] The purpose of the present invention is to solve the problems in the background technology and to propose a cross-modal signal data analysis method for a natural language processing model.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A cross-modal signal data analysis method for a natural language processing model, comprising:

[0007] Step 1: After collecting the original data set, classify it and assign a unique number identifier to each category of data set; each category of data set includes a text data set, an image data set, and an audio data set;

[0008] Step 2: Extract features from all numbered data sets;

[0009] Step 3: Establish a cross-modal shared representation space, unify the features of different modalities, and reconstruct the feature vector of the original modality;

[0010] Step 4: Based on the feature vector of the reconstructed original modality, the multimodal feature information is fused to generate a comprehensive multimodal data representation;

[0011] Step 5: Modeling and analyzing time dynamic information for multimodal representation data;

[0012] Step 6: Further analyze the modeling and analysis results of time dynamic information, identify abnormal data, and complete abnormal event warning.

[0013] It should be noted that the application object of the cross-modal signal data analysis method of a natural language processing model proposed in the present invention can be data analysis of multiple modalities such as text, image, audio, video, etc., and can be used to monitor and analyze whether the system operation, equipment status, user behavior, etc. are normal, including but not limited to equipment performance, network traffic, user interaction behavior, etc. Specifically, it can be through deep learning and natural language processing technology analysis, comprehensive monitoring and precise analysis of text content, image data, audio signals, etc., extract features and construct a cross-modal shared space, integrate multimodal information, and use time dynamic information analysis to realize abnormality identification and early warning, ensuring the real-time, accuracy and comprehensiveness of the data, which is conducive to trend prediction and decision support for complex application scenarios.

[0014] Furthermore, the process of assigning a unique number identifier to each category of the original data set includes:

[0015] The text dataset, image dataset, and audio dataset are marked as ;

[0016] in, Respectively represent single sample data in text, image and audio datasets, and m, n and p represent the total number of text, image and audio data, k, j and Indicates text, image, and audio data numbers.

[0017] Furthermore, the process of extracting features from all the numbered data sets includes:

[0018] For a text dataset T, traverse all texts in the entire text dataset T, summarize all words, remove duplicates, and sort them in descending order of word frequency to form a vocabulary V; where the vocabulary V contains the set of all unique words;

[0019] Determine the word set contained in the text data set T;

[0020] Based on the defined vocabulary for each text Construct a vector of the same length as V , where each position of the vector corresponds to a word in the vocabulary V; |V| represents the size of the vocabulary V, that is, the number of words in V; Words to express In text The frequency of occurrence in , where b represents the index of the word in the defined vocabulary, and b=1,2,…,|V|; is text The total number of words in the text Index of words in , Represents text The cth word in ; Indicates the text Every word in Perform summation; is the characteristic function, when When the text Vocabulary in Equal to a word in the vocabulary When , it returns 1, when When , it returns 0;

[0021] Will As input image, is the forward propagation function of the CNN model, L is the index of the second-to-last layer in the model, is the output feature map of this layer;

[0022] It is understandable that a pre-trained CNN model consists of multiple convolutional layers, pooling layers, fully connected layers and an output layer; when extracting feature vectors, the penultimate layer of the model, that is, before the last fully connected layer, is usually focused on because this layer can provide high-level feature representation of the image;

[0023] For each image , whose feature vector at layer L is Expressed as ;

[0024] If the output of the Lth layer is a d-dimensional vector, then , where d represents the dimension index of the image feature vector; Represents each image The eigenvalues ​​at level L;

[0025] For each audio sample Perform frame processing to obtain a windowed frame signal set ; In the formula, q represents the frame signal, and N represents the total number of frame signals;

[0026] For each frame signal Perform short-time Fourier transform to obtain the complex spectrum :

[0027] , where is the complex spectrum of frame z; z represents the number of the frame signal, z=1, 2, ..., N; f represents the frequency; Indicates time; is a complex exponential function, which is used to multiply the signal by sine waves at different frequencies to extract the spectral characteristics of the signal. represents an imaginary unit;

[0028] Extract all complex spectrum values ​​and calculate the spectrum centroid using the formula:

[0029] , where represents the centroid of the spectrum, that is, the center frequency of the spectrum; fmin and fmax are the minimum and maximum values ​​of the frequency range considered respectively; Represents the power spectrum at frequency f, the complex spectrum obtained by Fourier transform The square of is obtained;

[0030] Based on the spectrum centroid calculation results, obtain several spectrum centroid sequences , where R represents the total number of spectrum centroids;

[0031] Use the spectrum centroid sequence as the audio feature vector part of ;

[0032] It is understandable that from the audio samples The eigenvalues ​​containing the spectrum centroid information are extracted and used for the establishment of audio feature vectors and subsequent shared representation space.

[0033] Furthermore, the process of establishing a cross-modal shared representation space, unifying the features of different modalities, and reconstructing the feature vector of the original modality includes:

[0034] Define GX={g1, g2, …, gr} as a shared representation space; where GX represents a shared set, g represents a single data point in the shared set, and r is the dimension of the shared representation;

[0035] Feature vectors for text, images, and audio , use encoder E to map it into a shared representation space, that is, ;

[0036] The feature vector of the original modality is reconstructed from the shared representation space through the decoder D:

[0037] , where They are the reconstructed text, image and audio feature vectors respectively;

[0038] It can be understood that a shared representation space is established using a multimodal autoencoder in deep learning; the features of different modalities are mapped into the shared representation space, and then multiple decoders are used to reconstruct the shared representation back into the original modality; by encoding and decoding the feature vectors of text, images, and audio, not only meaningful features can be extracted from data of different modalities, but also these features can be mapped into the same representation space, laying a solid foundation for further cross-modal fusion analysis; in addition, the shared representation space can fuse feature information from different modalities (text, images, audio), capture the intrinsic connections and complementarity between them, and jointly construct a more complete representation space.

[0039] Furthermore, the process of fusing multimodal feature information based on the feature vector of the reconstructed original modality to generate a comprehensive multimodal data representation includes:

[0040] Set the multimodal feature information to , the attention matrix is ​​U;

[0041] Then the feature DF after fusion of different modalities is expressed as , where softmax is an activation function used to convert a set of arbitrary input values ​​into a probability distribution, that is, the output is a vector in which each element is a value between 0 and 1, and the sum of all elements is 1, thereby mapping each input value to a larger range, that is, it can enhance the difference between the maximum value and other values ​​in the input vector, so that the output corresponding to the maximum value is close to 1, while other values ​​are close to 0; H represents the transpose of the attention weight matrix; y represents the three types of text, image, and audio, and y=1, 2, 3; e is the base of the natural logarithm;

[0042] It is understandable that the introduction of weight parameters in the weighted mechanism can dynamically adjust the contribution of different modalities to the final representation, automatically adjust the focus of attention, and process multimodal information more flexibly; by fusing features of different modalities, it not only reflects the comprehensive information of multimodal data, but also provides an explanation of the decision-making process of the natural language processing model based on weight parameter analysis. Any modality contributes the most to the decision in a specific situation, thereby integrating and optimizing multi-source information.

[0043] Furthermore, the process of modeling and analyzing time dynamic information for multimodal representation data includes:

[0044] The formed multimodal representation DF is used as input data, that is, ;in, represents the fused multimodal feature vector at time point h, J represents the total number of time points, and h=1, 2, ..., J;

[0045] like Related to the multimodal feature representation of the previous o time points, a temporal dynamic information model is constructed:

[0046] , where B represents the constant term; Represents model parameters, each The model parameters reflect the previous time point For the current time point The degree of influence is the index subscript of the model parameters, that is ; o represents the order of the model; represents the multidimensional white noise error vector;

[0047] Combining time dynamic information model When performing trend analysis on the value, if If the value increases with time, it indicates that the multimodal representation is on an upward trend, revealing that the complexity of feature expression is high; if If the value decreases over time, it indicates that there is a downward trend in multimodal representation, revealing that the complexity of feature expression is low;

[0048] Specifically, 1. For text analysis, An increase in the value may indicate an increase in the intensity of sentiment in the text or an increase in the relevance of the topic. For example, in news text analysis, if the frequency of keywords for a specific topic increases, The value may increase accordingly; on the contrary, A decrease in the value may indicate a decrease in the intensity of sentiment in the text or a decrease in the relevance of the topic. For example, if a certain event is no longer mentioned in the news text, the frequency of keywords related to the event decreases. The value may decrease accordingly; 2. For audio analysis, The increase in value may indicate an increase in volume, a change in pitch, or an enhancement of a certain frequency component. For example, in music analysis, a sudden increase in volume may indicate that the music is entering a climax. A decrease in value may indicate a decrease in volume or a weakening of a certain frequency component. For example, in speech recognition, if the speaker lowers the volume or changes the tone of voice, The value may decrease accordingly; 3. For image analysis, An increase in the value may indicate an enhancement of a feature in the image, such as color saturation, edge sharpening, or the presence of a specific object. For example, in video analysis, if a fast-moving object appears, the characteristics of the object's motion in the image may be enhanced. The value increases accordingly; A decrease in value may indicate a weakening of a feature in the image, such as fading of colors, blurring of edges, or disappearance of a particular object. For example, in video surveillance, if the target object leaves the frame, features associated with the target object may weaken. The value decreases accordingly;

[0049] It is understandable that dynamic information modeling aims to learn and understand the laws of temporal evolution of multimodal representations in time series data, capture the interactions between modalities and the temporal dependencies within modalities, so as to enhance the representation ability and predictive performance of the model on time series tasks, and further establish dynamic associations between and within modalities that change over time.

[0050] Furthermore, the process of further analyzing the modeling analysis results of the time dynamic information, identifying abnormal data, and completing the abnormal event warning includes:

[0051] Preset the upper abnormal threshold G1 and the lower abnormal threshold G2;

[0052] Through piecewise function Perform comparative analysis:

[0053] ,Will Each value of is mapped to -1, 0 or 1. When the value is -1, it means that it is lower than the lower abnormal threshold G2 and is in the first abnormal state. When the value is 0, it means the normal state. When the value is 1, it means that it is higher than the upper abnormal threshold G1 and is in the second abnormal state.

[0054] Generate an abnormal event warning signal according to the first abnormal state and the second abnormal state, and feed it back to the technical staff for debugging and optimization;

[0055] Specifically, the first abnormal state and the second abnormal state include: 1. For text analysis, if If the value is higher than the upper abnormal threshold G1, it means that the intensity or frequency of a feature in the text is abnormal. For example, in sentiment analysis, if the sentiment intensity of the text at a certain time point h is abnormally high, it may indicate that extreme emotions appear in the text, such as extreme anger or extreme joy. In topic detection, if the frequency of mentions of a topic is abnormally high, it may indicate that the text is over-focused on a certain topic. If the value is lower than the lower abnormal threshold G2, it means that a certain feature in the text is missing or weakened. For example, in news reports, if the coverage of an event suddenly decreases and is lower than the normal level, it may mean that the media's attention to the event has decreased, or the development of the event itself has entered a trough period; 2. For audio analysis, if If the value is higher than the upper abnormal threshold G1, it indicates that the volume, pitch or specific frequency components are abnormally enhanced. For example, in music analysis, if the volume or the frequency components of a specific instrument suddenly increase, it may indicate that the music has entered a climax or a solo of a specific instrument; If the value is lower than the lower abnormal threshold G2, it indicates that the volume is weakened or a specific frequency component is missing. For example, in speech recognition, if the speaker's volume suddenly decreases, it may indicate a change in the speaker's mood or interference from environmental noise. 3. For image analysis, if If the value is higher than the upper abnormal threshold G1, it means that there may be high-contrast edges or high-saturation color areas in the image. These areas may be caused by the prominence of specific objects, sudden changes in lighting conditions, or abnormal output of image processing algorithms, or abnormal noise or texture in the image, such as failure of electronic equipment, interference in image transmission, or algorithm errors in image processing. Abnormal increase in values, or sudden appearance or movement of objects in surveillance video or continuous image streams may also cause If the value is abnormally high If the value is lower than the lower abnormal threshold G2, it means that the features in the image (such as edges, textures, or colors) may be weakened due to insufficient lighting, image blur, or disappearance of objects, resulting in Abnormally low values ​​can also result if the background or environment becomes very homogeneous with no distinct features or contrast. Abnormally low value.

[0056] Compared with the existing technology, the present invention provides a cross-modal signal data analysis method for a natural language processing model, which has the following advantages:

[0057] The present invention collects and classifies raw data sets, effectively integrates multi-source data, lays a solid data foundation for subsequent analysis, and assigns a unique number identifier to each category of data set, greatly facilitating data management, tracking and reference; by extracting features from all numbered data sets, the extracted features enable data from different modalities to be compared and analyzed on the same dimension;

[0058] The present invention breaks down the barriers between different modal data by establishing a cross-modal shared representation space, and realizes seamless fusion and interaction of data. The establishment of a shared representation space provides a platform for mutual understanding and fusion of different modal data; through multi-modal feature information fusion, information of multiple dimensions is integrated, so that data of different modalities can complement each other in the fusion process;

[0059] Through modeling and analysis of time dynamic information, the present invention can capture the changing trends and laws in the data, providing strong support for future predictions and analysis, so as to respond to data changes in real time; through abnormal data identification and event warning, potential problems can be discovered and handled in a timely manner, avoiding possible losses and consequences.

[0060] In summary, the present invention realizes the integration, processing, fusion and analysis of multi-source data, significantly improves the accuracy, depth and breadth of data analysis, can capture data changes in real time, improve the timeliness of data analysis, and enhance the accuracy of prediction of future data changes, ensuring the normal implementation of a subsequent cross-modal signal data analysis method of a natural language processing model. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a flow chart of a cross-modal signal data analysis method for a natural language processing model proposed in the present invention. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the implementation regulations described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] Reference Figure 1 , a cross-modal signal data analysis method for a natural language processing model, comprising:

[0064] Step 1: After collecting the original data set, classify it and assign a unique number identifier to each category of data set; each category of data set includes a text data set, an image data set, and an audio data set;

[0065] Step 2: Extract features from all numbered data sets;

[0066] Step 3: Establish a cross-modal shared representation space, unify the features of different modalities, and reconstruct the feature vector of the original modality;

[0067] Step 4: Based on the feature vector of the reconstructed original modality, the multimodal feature information is fused to generate a comprehensive multimodal data representation;

[0068] Step 5: Modeling and analyzing time dynamic information for multimodal representation data;

[0069] Step 6: Further analyze the modeling and analysis results of time dynamic information, identify abnormal data, and complete abnormal event warning.

[0070] It should be noted that the application object of the cross-modal signal data analysis method of a natural language processing model proposed in the present invention can be data analysis of multiple modalities such as text, image, audio, video, etc., and can be used to monitor and analyze whether the system operation, equipment status, user behavior, etc. are normal, including but not limited to equipment performance, network traffic, user interaction behavior, etc. Specifically, it can be through deep learning and natural language processing technology analysis, comprehensive monitoring and precise analysis of text content, image data, audio signals, etc., extract features and construct a cross-modal shared space, integrate multimodal information, and use time dynamic information analysis to realize abnormality identification and early warning, ensuring the real-time, accuracy and comprehensiveness of the data, which is conducive to trend prediction and decision support for complex application scenarios.

[0071] See also Figure 1 The present invention provides a cross-modal signal data analysis method for a natural language processing model. In step 1, the step of assigning a unique number identifier to each category of the original data set includes:

[0072] Step 101: Mark the text dataset, image dataset, and audio dataset as ;

[0073] In step 101, Respectively represent single sample data in text, image and audio datasets, and m, n and p represent the total number of text, image and audio data, k, j and Indicates text, image, and audio data numbers.

[0074] See also Figure 1 The present invention provides a cross-modal signal data analysis method for a natural language processing model. In the step 2, the step of extracting features from all numbered data sets includes:

[0075] Step 201: for a text data set T, traverse all texts in the entire text data set T, summarize all words, remove duplicates, and arrange them in descending order of word frequency to form a vocabulary V; wherein the vocabulary V contains a set of all unique words;

[0076] Step 202: Determine the word set included in the text data set T;

[0077] Step 203: Based on the defined vocabulary, Construct a vector of the same length as V , where each position of the vector corresponds to a word in the vocabulary V; |V| represents the size of the vocabulary V, that is, the number of words in V; Words to express In text The frequency of occurrence in , where b represents the index of the word in the defined vocabulary, and b=1,2,…,|V|; is text The total number of words in the text Index of words in , Represents text The cth word in ; Indicates the text Every word in Perform summation; is the characteristic function, when When the text Vocabulary in Equal to a word in the vocabulary When , it returns 1, when When , it returns 0;

[0078] Step 204: As input image, is the forward propagation function of the CNN model, L is the index of the second-to-last layer in the model, is the output feature map of this layer;

[0079] In step 204, a pre-trained CNN model is used, which consists of a plurality of convolutional layers, pooling layers, fully connected layers, and an output layer; when extracting feature vectors, the penultimate layer of the model, i.e., before the last fully connected layer, is usually focused on because this layer can provide high-level feature representation of the image;

[0080] Step 205: For each image , whose feature vector at layer L is Expressed as ;

[0081] Step 206: If the output of the Lth layer is a d-dimensional vector, then , where d represents the dimension index of the image feature vector; Represents each image The eigenvalues ​​at level L;

[0082] Step 207: For each audio sample Perform frame processing to obtain a windowed frame signal set ; In the formula, q represents the frame signal, and N represents the total number of frame signals;

[0083] Step 208: for each frame signal Perform short-time Fourier transform to obtain the complex spectrum :

[0084] , where is the complex spectrum of frame z; z represents the number of the frame signal, z=1, 2, ..., N; f represents the frequency; Indicates time; is a complex exponential function, which is used to multiply the signal by sine waves at different frequencies to extract the spectral characteristics of the signal. represents an imaginary unit;

[0085] Step 209: extract all complex spectrum values ​​and calculate the spectrum centroid using the formula:

[0086] , where represents the centroid of the spectrum, that is, the center frequency of the spectrum; fmin and fmax are the minimum and maximum values ​​of the frequency range considered respectively; Represents the power spectrum at frequency f, the complex spectrum obtained by Fourier transform The square of is obtained;

[0087] Step 210: Based on the spectrum centroid calculation result, obtain several spectrum centroid sequences , where R represents the total number of spectrum centroids;

[0088] Step 211: Use the spectrum centroid sequence as the audio feature vector part of ;

[0089] In step 211, from the audio sample The eigenvalues ​​containing the spectrum centroid information are extracted and used for the establishment of audio feature vectors and subsequent shared representation space.

[0090] See also Figure 1 The present invention provides a cross-modal signal data analysis method for a natural language processing model. In step three, the steps of establishing a cross-modal shared representation space, uniformly representing features of different modalities, and reconstructing feature vectors of the original modality include:

[0091] Step 301: define GX={g1, g2, ..., gr} as a shared representation space; wherein GX represents a shared set, g represents a single data point in the shared set, and r represents the dimension of the shared representation;

[0092] Step 302: Feature vectors for text, image, and audio , use encoder E to map it into a shared representation space, that is, ;

[0093] Step 303: Reconstruct the feature vector of the original modality from the shared representation space through the decoder D:

[0094] , where They are the reconstructed text, image and audio feature vectors respectively;

[0095] In steps 301-303, a shared representation space is established using a multimodal autoencoder in deep learning; features of different modalities are mapped to the shared representation space, and then multiple decoders are used to reconstruct the shared representation back to the original modality; by encoding and decoding the feature vectors of text, image, and audio, not only meaningful features can be extracted from data of different modalities, but also these features can be mapped to the same representation space, laying a solid foundation for further cross-modal fusion analysis; in addition, the shared representation space can fuse feature information from different modalities (text, image, audio), capture the intrinsic connection and complementarity between them, and jointly construct a more complete representation space.

[0096] See also Figure 1 The present invention provides a cross-modal signal data analysis method for a natural language processing model. In step 4, the step of fusing multi-modal feature information based on the feature vector of the reconstructed original modality to generate a comprehensive multi-modal data representation includes:

[0097] Step 401: Set the multimodal feature information to , the attention matrix is ​​U;

[0098] Step 402: The feature DF after fusion of different modalities is expressed as , where softmax is an activation function used to convert a set of arbitrary input values ​​into a probability distribution, that is, the output is a vector in which each element is a value between 0 and 1, and the sum of all elements is 1, thereby mapping each input value to a larger range, that is, it can enhance the difference between the maximum value and other values ​​in the input vector, so that the output corresponding to the maximum value is close to 1, while other values ​​are close to 0; H represents the transpose of the attention weight matrix; y represents the three types of text, image, and audio, and y=1, 2, 3; e is the base of the natural logarithm;

[0099] In steps 401-402, the weight parameters of the weighting mechanism are introduced, which can dynamically adjust the contribution of different modalities to the final representation, automatically adjust the focus of attention, and process multimodal information more flexibly; by fusing the features of different modalities, it not only reflects the comprehensive information of multimodal data, but also provides an explanation of the decision-making process of the natural language processing model based on the weight parameter analysis. Any modality contributes the most to the decision in a specific situation, thereby integrating and optimizing multi-source information.

[0100] See also Figure 1The present invention provides a cross-modal signal data analysis method for a natural language processing model. In step 5, the step of modeling and analyzing time dynamic information for multi-modal representation data includes:

[0101] Step 501: Use the formed multimodal representation DF as input data, that is, ;in, represents the fused multimodal feature vector at time point h, J represents the total number of time points, and h=1, 2, ..., J;

[0102] Step 502: If Related to the multimodal feature representation of the previous o time points, a temporal dynamic information model is constructed:

[0103] , where B represents the constant term; Represents model parameters, each The model parameters reflect the previous time point For the current time point The degree of influence is the index subscript of the model parameters, that is ; o represents the order of the model; represents the multidimensional white noise error vector;

[0104] Step 503: Combine the time dynamic information model When performing trend analysis on the value, if If the value increases with time, it indicates that the multimodal representation is on an upward trend, revealing that the complexity of feature expression is high; if If the value decreases over time, it indicates that there is a downward trend in multimodal representation, revealing that the complexity of feature expression is low;

[0105] In step 503, 1. for text analysis, An increase in the value may indicate an increase in the intensity of sentiment in the text or an increase in the relevance of the topic. For example, in news text analysis, if the frequency of keywords for a specific topic increases, The value may increase accordingly; on the contrary, A decrease in the value may indicate a decrease in the intensity of sentiment in the text or a decrease in the relevance of the topic. For example, if a certain event is no longer mentioned in the news text, the frequency of keywords related to the event decreases. The value may decrease accordingly; 2. For audio analysis, The increase in value may indicate an increase in volume, a change in pitch, or an enhancement of a certain frequency component. For example, in music analysis, a sudden increase in volume may indicate that the music is entering a climax. A decrease in value may indicate a decrease in volume or a weakening of a certain frequency component. For example, in speech recognition, if the speaker lowers the volume or changes the tone of voice, The value may decrease accordingly; 3. For image analysis, An increase in the value may indicate an enhancement of a feature in the image, such as color saturation, edge sharpening, or the presence of a specific object. For example, in video analysis, if a fast-moving object appears, the characteristics of the object's motion in the image may be enhanced. The value increases accordingly; A decrease in value may indicate a weakening of a feature in the image, such as fading of colors, blurring of edges, or disappearance of a particular object. For example, in video surveillance, if the target object leaves the frame, features associated with the target object may weaken. The value decreases accordingly;

[0106] In steps 501-503, dynamic information modeling aims to learn and understand the evolution of multimodal representations in time series data over time, capture the interactions between modalities and the temporal dependencies within modalities, so as to enhance the representation ability and predictive performance of the model on time series tasks, and further establish dynamic associations between modalities and within modalities that change over time.

[0107] See also Figure 1 The present invention provides a cross-modal signal data analysis method for a natural language processing model. In step six, the steps of further analyzing the modeling analysis results of the time dynamic information, identifying abnormal data, and completing the abnormal event warning include:

[0108] Step 601, presetting an upper abnormal threshold G1 and a lower abnormal threshold G2;

[0109] Step 602: Using piecewise function Perform comparative analysis:

[0110] ,Will Each value of is mapped to -1, 0 or 1. When the value is -1, it means that it is lower than the lower abnormal threshold G2 and is in the first abnormal state. When the value is 0, it means the normal state. When the value is 1, it means that it is higher than the upper abnormal threshold G1 and is in the second abnormal state.

[0111] Step 603: Generate an abnormal event warning signal according to the first abnormal state and the second abnormal state, and feed it back to the technical staff for debugging and optimization;

[0112] In steps 601-603, the specific first abnormal state and the second abnormal state are as follows: 1. For text analysis, if If the value is higher than the upper abnormal threshold G1, it means that the intensity or frequency of a feature in the text is abnormal. For example, in sentiment analysis, if the sentiment intensity of the text at a certain time point h is abnormally high, it may indicate that extreme emotions appear in the text, such as extreme anger or extreme joy. In topic detection, if the frequency of mentions of a topic is abnormally high, it may indicate that the text is over-focused on a certain topic. If the value is lower than the lower abnormal threshold G2, it means that a certain feature in the text is missing or weakened. For example, in news reports, if the coverage of an event suddenly decreases and is lower than the normal level, it may mean that the media's attention to the event has decreased, or the development of the event itself has entered a trough period; 2. For audio analysis, if If the value is higher than the upper abnormal threshold G1, it indicates that the volume, pitch or specific frequency components are abnormally enhanced. For example, in music analysis, if the volume or the frequency components of a specific instrument suddenly increase, it may indicate that the music has entered a climax or a solo of a specific instrument; If the value is lower than the lower abnormal threshold G2, it indicates that the volume is weakened or a specific frequency component is missing. For example, in speech recognition, if the speaker's volume suddenly decreases, it may indicate a change in the speaker's mood or interference from environmental noise. 3. For image analysis, if If the value is higher than the upper abnormal threshold G1, it means that there may be high-contrast edges or high-saturation color areas in the image. These areas may be caused by the prominence of specific objects, sudden changes in lighting conditions, or abnormal output of image processing algorithms, or abnormal noise or texture in the image, such as failure of electronic equipment, interference in image transmission, or algorithm errors in image processing. Abnormal increase in values, or sudden appearance or movement of objects in surveillance video or continuous image streams may also cause If the value is abnormally high If the value is lower than the lower abnormal threshold G2, it means that the features in the image (such as edges, textures, or colors) may be weakened due to insufficient lighting, image blur, or disappearance of objects, resulting in Abnormally low values ​​can also result if the background or environment becomes very homogeneous with no distinct features or contrast. Abnormally low value.

[0113] In the embodiment of the present invention, by classifying the original data set and assigning a unique number to each type of data set, it is convenient for subsequent feature extraction and model training, improves the efficiency and accuracy of data management, and is conducive to tracking and citation. By extracting features from all numbered data sets, data of different modalities can be compared and analyzed on the same dimension, which provides a basis for establishing a cross-modal shared representation space. By constructing a cross-modal shared representation space, the representation of data of different modalities in a unified space is realized, breaking the barriers between modalities, and providing the possibility for multimodal data fusion, so that data of different modalities can complement and enhance each other. By fusing feature information of different modalities, a more comprehensive and accurate data representation can be obtained, which improves the accuracy and depth of data analysis, and provides a richer and more accurate data foundation for subsequent time dynamic information modeling and analysis. Through time dynamic information modeling and analysis, the time change laws and trends in the data can be captured, which provides an important basis for anomaly detection and early warning. By identifying abnormal data and completing abnormal event early warning, abnormal situations in the data can be discovered and responded to in a timely manner, and measures can be taken in advance to prevent potential risks and losses. In summary, the examples of the present invention involve data processing, comprehensive analysis and decision-making for optimized operations, solving the technical problems of cross-modal data fusion and accurate abnormal warning in existing solutions. In actual situations, more data and contextual information may be needed to make specific decisions and optimization solutions.

[0114] In addition, the formulas involved in the above are all calculated by removing dimensions and taking their numerical values. They are a formula that is closest to the actual situation obtained by collecting a large amount of data and performing software simulation. The proportional coefficient in the formula and the various preset thresholds in the analysis process are set by technical personnel in this field according to actual conditions or obtained by simulating a large amount of data; the size of the proportional coefficient is to quantify each parameter to obtain a specific value for subsequent comparison. The size of the proportional coefficient depends on the amount of sample data and the preliminary setting of the corresponding processing coefficient for each group of sample data by technical personnel in this field; as long as it does not affect the proportional relationship between the parameter and the quantized value.

[0115] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically based on the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0116] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0117] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0118] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0119] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0120] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0121] Secondly: In the drawings of the embodiments disclosed in the present invention, only the structures related to the embodiments disclosed in the present invention are involved, and other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of the present invention can be combined with each other;

[0122] Finally: The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A cross-modal signal data analysis method for a natural language processing model, characterized by: Step 1: After collecting the original data set, classify it and assign a unique number identifier to each category of data set; each category of data set includes a text data set, an image data set, and an audio data set; Step 2: Extract features from all numbered data sets; The process of extracting features from all the numbered data sets in step 2 includes: For a text dataset T, traverse all texts in the entire text dataset T, summarize all words, remove duplicates, and sort them in descending order of word frequency to form a vocabulary V; where the vocabulary V contains the set of all unique words; Determine the word set contained in the text data set T; Based on the defined vocabulary for each text Construct a vector of the same length as V , where each position of the vector corresponds to a word in the vocabulary V; |V| represents the size of the vocabulary V, that is, the number of words in V; Words to express In text The frequency of occurrence in Right now , where b represents the index of the vocabulary in the defined vocabulary, and b=1,2,…,|V|; is text The total number of words in the text Index of words in , Represents text The cth word in ; Indicates the text Every word in Perform summation; is the characteristic function, when When the text Vocabulary in Equal to a word in the vocabulary When , it returns 1, when When , it returns 0; Will As input image, is the forward propagation function of the CNN model, L is the index of the second-to-last layer in the model, is the output feature map of this layer; For each image , whose feature vector at layer L is Expressed as ; If the output of the Lth layer is a d-dimensional vector, then , where d represents the dimension index of the image feature vector; Represents each image The eigenvalues ​​at level L; For each audio sample Perform frame processing to obtain a windowed frame signal set ; In the formula, q represents the frame signal, and N represents the total number of frame signals; For each frame signal Perform short-time Fourier transform to obtain the complex spectrum : , where is the complex spectrum of frame z; z represents the number of the frame signal, z=1, 2, ..., N; f represents the frequency; Indicates time; is a complex exponential function, which is used to multiply the signal by sine waves at different frequencies to extract the spectral characteristics of the signal. represents an imaginary unit; Extract all complex spectrum values ​​and calculate the spectrum centroid using the formula: , where represents the centroid of the spectrum, that is, the central frequency of the spectrum; fmin and fmax are the minimum and maximum values ​​of the frequency range considered respectively; Represents the power spectrum at frequency f, the complex spectrum obtained by Fourier transform The square of is obtained; Based on the spectrum centroid calculation results, obtain several spectrum centroid sequences , where R represents the total number of spectrum centroids; Use the spectrum centroid sequence as the audio feature vector part of ； Step 3: Establish a cross-modal shared representation space to unify the features of different modalities and reconstruct the feature vector of the original modality; the features of different modalities include text, image and audio features; The process of step 3, establishing a cross-modal shared representation space, unifying the features of different modalities, and reconstructing the feature vector of the original modality, includes: Define GX={g1, g2, …, gr} as a shared representation space; where GX represents a shared set, g represents a single data point in the shared set, and r is the dimension of the shared representation; Feature vectors for text, images, and audio , use encoder E to map it into a shared representation space, that is, ； The feature vector of the original modality is reconstructed from the shared representation space through the decoder D: , where They are the reconstructed text, image and audio feature vectors respectively; Step 4: Based on the feature vector of the reconstructed original modality, the multimodal feature information is fused to generate a comprehensive multimodal data representation; The process of step 4, fusing multimodal feature information based on the feature vector of the reconstructed original modality to generate a comprehensive multimodal data representation, includes: Set the multimodal feature information to , the attention matrix is ​​U; Then the feature DF after fusion of different modalities is expressed as , where softmax is the activation function; H represents the transpose of the attention weight matrix; y represents the three types of text, image, and audio, and y=1, 2, 3; e is the base of the natural logarithm; Step 5: Modeling and analyzing time dynamic information for multimodal representation data; The process of modeling and analyzing time dynamic information for multimodal representation data in step 5 includes: The formed multimodal representation DF is used as input data, that is, ;in, represents the fused multimodal feature vector at time point h, J represents the total number of time points, and h=1, 2, ..., J; like Related to the multimodal feature representation of the previous o time points, a temporal dynamic information model is constructed: , where B represents the constant term; Represents model parameters, each The model parameters reflect the previous time point For the current time point The degree of influence is the index subscript of the model parameters, that is ; o represents the order of the model; represents the multidimensional white noise error vector; Combining time dynamic information model When performing trend analysis on the value, if If the value increases with time, it indicates that the multimodal representation is on an upward trend, revealing that the complexity of feature expression is high; if If the value decreases over time, it indicates that there is a downward trend in multimodal representation, revealing that the complexity of feature expression is low; Step 6: Further analyze the modeling and analysis results of time dynamic information, identify abnormal data, and complete abnormal event warning.

2. The cross-modal signal data analysis method of a natural language processing model according to claim 1, characterized in that: In step 1, the process of assigning a unique number identifier to each category of the original data set includes: The text dataset, image dataset, and audio dataset are marked as ； in, Represents a single sample data in the text, image and audio data sets, respectively, and m, n and p represent the total number of text, image and audio data, k, j and Indicates text, image, and audio data numbers.

3. The cross-modal signal data analysis method of a natural language processing model according to claim 1, characterized in that: In step 6, the process of further analyzing the modeling analysis results of the time dynamic information, identifying abnormal data, and completing the abnormal event warning includes: Preset the upper abnormal threshold G1 and the lower abnormal threshold G2; Through piecewise function Perform comparative analysis: ,Will Each value of is mapped to -1, 0 or 1. When the value is -1, it means that it is lower than the lower abnormal threshold G2 and is in the first abnormal state. When the value is 0, it means the normal state. When the value is 1, it means that it is higher than the upper abnormal threshold G1 and is in the second abnormal state. An abnormal event warning signal is generated according to the first abnormal state and the second abnormal state, and is fed back to the technical staff for debugging and optimization.

Images