Cross-modal semantic analysis method

Abstract

The invention provides a cross-modal semantic analysis method, and relates to the technical field of data processing, and the method comprises the steps: calling a visual modal analysis strategy to analyze a source video, and obtaining visual analysis information; calling an auditory modal analysis strategy to analyze the source audio to obtain auditory analysis information; performing stage comparison analysis on the visual analysis information and the auditory analysis information; if the emotion comparison result reaches a predetermined comparison constraint, analyzing to obtain real-time modal weight distribution; and obtaining character analysis information, and fusing the character analysis information, the visual analysis information and the auditory analysis information in combination with real-time modal weight distribution to obtain a semantic analysis result. By means of the cross-modal semantic analysis method and device, the technical problem that in the prior art, the semantic analysis accuracy is low due to the fact that the information structures and expression modes of all the modes are large in difference is solved, and the accuracy and adaptability of cross-modal semantic analysis are improved through the semantic analysis method combining the visual mode, the auditory mode and the text mode.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a cross-modal semantic parsing method. Background Art

[0002] During the customer service interaction process, the user's intention is often expressed through multiple modalities, including text input, voice conversation, images (such as screenshots, photos), etc. There are significant differences in the expression methods and semantic contents of different modality (such as text, voice, image) data. Due to the differences in information structure and expression method between modalities, it is difficult to establish the semantic correspondence relationship between different modalities in cross-modal semantic parsing, resulting in the alignment problem of cross-modal data. In addition, due to the lack of the dynamic perception ability of the consistency of emotional expression and the semantic expression difference between modalities, the existing cross-modal semantic parsing performs poorly when dealing with modality conflicts, semantic ambiguities or modality information missing, ultimately resulting in low semantic parsing accuracy.

[0003] In summary, there is a technical problem in the prior art that due to the large differences in information structure and expression method between modalities, the accuracy of semantic parsing is low. Summary of the Invention

[0004] The purpose of this application is to provide a cross-modal semantic parsing method to solve the technical problem in the prior art that due to the large differences in information structure and expression method between modalities, the accuracy of semantic parsing is low.

[0005] In view of the above problems, this application provides a cross-modal semantic parsing method. Among them, the cross-modal semantic parsing method includes: retrieving a visual modality parsing strategy to parse a source video to obtain visual parsing information; retrieving an auditory modality parsing strategy to parse a source audio extracted from the source video to obtain auditory parsing information; performing a stage-by-stage comparative analysis on the visual parsing information and the auditory parsing information to obtain an emotional comparison result; if the emotional comparison result reaches a predetermined comparison constraint, analyzing a modality parsing database to obtain a real-time modality weight allocation; obtaining the text parsing information of the source text, and combining the real-time modality weight allocation to fuse the text parsing information, the visual parsing information and the auditory parsing information to obtain a semantic parsing result.

[0006] Optionally, perform dynamic image compression processing on the source video to obtain a compressed video; perform detection and analysis on the compressed video according to the shot detection mechanism in the visual modality parsing strategy to obtain a detection result; extract the first image sequence corresponding to the first shot in the detection result, and perform enhancement and fusion processing on the first image sequence to obtain a first target image; perform emotion parsing on the first target image according to the emotion parsing mechanism in the visual modality parsing strategy to obtain first emotion information; match the first video segment corresponding to the first shot in the source video, and form the visual parsing information based on the first mapping relationship between the first video segment and the first emotion information.

[0007] Optionally, obtain adjacent image groups of the compressed video; calculate the adjacent displacement value between the first image and the second image in the adjacent image groups according to the shot detection mechanism; if the adjacent displacement value does not meet the predetermined displacement limit, the first image and the second image are different shots, and the detection result is formed.

[0008] Optionally, collect the first feature parameter set of the first target image in multiple dimensions; extract the first emotion dimension from the predetermined emotion dimensions, where the predetermined emotion dimensions are stored in the emotion parsing mechanism; combine the first feature parameter set to obtain the first dimension coefficient of the first emotion dimension; form the first emotion information based on the first dimension coefficient.

[0009] Optionally, obtain the sound signal of the source audio, and analyze the sound signal to obtain the sound intensity time series; perform random segmentation on the sound intensity time series to obtain a segmentation result; collect the first time-domain feature parameter of the first time series in the segmentation result; collect the first frequency-domain feature parameter of the first frequency spectrum obtained by performing fast Fourier transform on the first time series; activate the sound emotion prediction model in the auditory modality parsing strategy to perform prediction and analysis on the first time-domain feature parameter and the first frequency-domain feature parameter to obtain first predicted emotion information; match the first audio segment corresponding to the first time series in the source audio, and form the auditory parsing information based on the second mapping relationship between the first audio segment and the first predicted emotion information.

[0010] Optionally, the sound emotion prediction model includes three prediction channels of the predetermined emotion dimensions, and the predetermined emotion dimensions include emotional valence, arousal, and dominance.

[0011] Optionally, construct a voice emotion dictionary, and form a first training data set of the first emotion dimension in the predetermined emotion dimension based on the voice emotion dictionary; perform supervised learning on the first training data set to obtain a first prediction channel, and form the voice emotion prediction model; wherein, the first training data set includes first training time domain feature parameters, first training frequency domain feature parameters, and first training emotion information.

[0012] Optionally, sample in the modality parsing database to obtain a modality parsing sample; perform manual emotion review on the modality parsing sample to obtain a review result; analyze the review result to obtain a modality parsing accuracy rate, where the modality parsing accuracy rate includes a visual parsing accuracy rate and an auditory parsing accuracy rate; perform variant weighted analysis on the visual parsing accuracy rate and the auditory parsing accuracy rate to obtain the real-time modality weight allocation.

[0013] Optionally, perform cleaning and word segmentation processing on the source text to obtain a processing result; perform feature analysis on the processing result based on the principle of the bag-of-words model to obtain a feature vector; perform emotion analysis on the feature vector in combination with a lexical emotion dictionary to obtain the text parsing information.

[0014] Optionally, extract any vocabulary in the feature vector; match any score of the any vocabulary in the lexical emotion dictionary, where the any score includes three scores of the predetermined emotion dimension; obtain the any frequency of the any vocabulary according to the feature vector, and combine the three scores to obtain the any emotion score of the any vocabulary; take the average of the any emotion scores to form the text parsing information.

[0015] The technical solutions provided in this application have at least the following beneficial effects: Parse the source video by invoking the visual modality parsing strategy to obtain visual parsing information; invoke the auditory modality parsing strategy to parse the source audio extracted from the source video to obtain auditory parsing information; perform a phased comparative analysis on the visual parsing information and the auditory parsing information to obtain an emotion comparison result; if the emotion comparison result meets the predetermined comparison constraint, analyze the modality parsing database to obtain real-time modality weight allocation; obtain the text parsing information of the source text, and combine the real-time modality weight allocation to fuse the text parsing information, the visual parsing information, and the auditory parsing information to obtain a semantic parsing result. That is to say, by parsing the visual content in the source video, visual parsing information is extracted; by parsing the source audio in the source video, auditory parsing information is obtained; the visual parsing information and the auditory parsing information are subjected to a phased comparative analysis to obtain an emotion comparison result; by analyzing the emotion comparison result, the weight of each modality is dynamically adjusted, and the text, visual, and auditory parsing information is fused according to the real-time modality weight allocation to obtain a semantic parsing result, improving the accuracy and adaptability of cross-modal semantic parsing.

[0016] The above description is only an overview of the technical solution of this application. In order to be able to understand the technical means of this application more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features, and advantages of this application more obvious and understandable, the following specifically illustrates the specific implementation manners of this application. It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of this application, nor is it used to limit the scope of this application. Other features of this application will become easily understood through the following description. Brief Description of the Drawings

[0017] In order to more clearly illustrate the technical solutions in this application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings described below are only exemplary, and for those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0018] Figure 1 It is a flowchart showing the process of a cross-modal semantic parsing method of this application.

[0019] Figure 2 It is a flowchart showing the process of obtaining visual parsing information in a cross-modal semantic parsing method of this application. Detailed Description of the Embodiments

[0020] The present application provides a cross-modal semantic parsing method, which solves the technical problem in the prior art that due to the large differences in information structures and expression modes between various modalities, the accuracy of semantic parsing is low. By parsing the visual content in the source video, visual parsing information is extracted; by parsing the source audio in the source video, auditory parsing information is obtained; the visual parsing information and the auditory parsing information are subjected to stage-by-stage comparative analysis to obtain an emotion comparison result; by analyzing the emotion comparison result, the weight of each modality is dynamically adjusted, and the parsing information of text, vision, and audition is fused according to the real-time modality weight distribution to obtain a semantic parsing result, improving the accuracy and adaptability of cross-modal semantic parsing.

[0021] Next, the technical solutions in the present application will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. It should be understood that the present application is not limited by the exemplary embodiments described herein. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the scope of protection of the present application. Additionally, it should be noted that for the sake of description, only the parts related to the present application are shown in the accompanying drawings rather than all of them.

[0022] Embodiment, please refer to the attached Figure 1 , the present application provides a cross-modal semantic parsing method, wherein, the cross-modal semantic parsing method specifically includes the following steps: S100: Invoke the visual modality parsing strategy to parse the source video to obtain visual parsing information.

[0023] Further, as shown in the attached Figure 2 figure, S100 of the present application includes: S110: Perform dynamic image compression processing on the source video to obtain a compressed video; S120: Detect and analyze the compressed video according to the shot detection mechanism in the visual modality parsing strategy to obtain a detection result; S130: Extract the first image sequence corresponding to the first shot in the detection result, and perform enhanced fusion processing on the first image sequence to obtain a first target image; S140: Perform emotion parsing on the first target image according to the emotion parsing mechanism in the visual modality parsing strategy to obtain first emotion information; S150: Match the first video segment corresponding to the first shot in the source video, and based on the first mapping relationship between the first video segment and the first emotion information, construct the visual parsing information.

[0024] Specifically, the source video refers to the original video file, which usually contains various audio-visual information, such as image frames, audio signals, and subtitles. The source video is processed by dynamic image compression to reduce the data size of the video and improve the efficiency of subsequent processing. Through compression algorithms such as H.264, HEVC, etc., efficient compression is achieved by reducing redundant data in the video. The specific process is as follows: Load the source video and apply the dynamic image compression algorithm for compression processing. During the compression process, mechanisms such as inter-frame prediction, motion estimation, and entropy coding are adopted to effectively eliminate the redundant information between adjacent frames in the video, thereby greatly reducing the data volume. For example, a voice video conversation collected by a customer service center, with the format of MP4, a resolution of 1920×1080 (Full HD), a frame rate of 30fps, a duration of 3 minutes, and an original size of 900MB. Using the HEVC (H.265) compression algorithm, the compression parameters are set as follows: The bitrate control is set to constant quality (CRF = 23); the encoding preset is set to slow. After compression, the output video size is 180MB, the compression ratio is approximately 80%, SSIM (Structural Similarity) = 0.96, PSNR (Peak Signal-to-Noise Ratio) = 40.2dB, indicating that the image quality is basically retained after compression.

[0025] Analyze the compressed video according to the shot detection mechanism in the visual modality parsing strategy to automatically identify different shots in the video. The shot detection mechanism is used to identify the scene change points in the video, that is, to determine whether the shot in the video has switched. A shot switch usually means a change in scene, person, perspective, or action. According to the shot detection results, extract the first image sequence corresponding to the first shot, which represents a segment of the scene in the video. The first image sequence is a series of consecutive image frames corresponding to the first image. Each image frame is a static image of the video, and the image sequence is the arrangement of these images in chronological order, reflecting the dynamic content in the video.

[0026] Perform enhancement and fusion processing on the extracted first image sequence to improve the quality of the images, including denoising, enhancing contrast, increasing image sharpness, etc. Through the enhancement and fusion processing, the details in the image sequence will be clearer, which is especially suitable for subsequent tasks such as emotion analysis and object recognition. The enhancement and fusion processing generate a more clear and detail-rich target image by improving the quality of the images (such as enhancing contrast, denoising, enhancing details, etc.). After the enhancement and fusion processing, the first target image is obtained, which is an optimized image and usually has higher clarity and more details. For example, if the source video is shot in a low-light environment, denoise and enhance the brightness of the image so that the facial expressions of the customer service staff are more clearly visible.

[0027] Through the emotion analysis mechanism in the visual modality parsing strategy, perform emotion analysis on the first target image to obtain the first emotion information, including emotion valence, arousal, and dominance. Match the first video segment corresponding to the first shot in the source video, and match the first emotion information with the corresponding first video segment in the source video to construct the first mapping relationship between the video and the emotion information. The mapping relationship describes the connection between the video content and its emotion characteristics. Repeat the above steps for other shots in the detection results, and finally obtain the mapping relationships between the video segments and emotion information corresponding to each shot, which together constitute the visual parsing information. The visual parsing information is the result obtained based on image or video analysis, and usually includes the feature information, emotion information of the image, and the relationship between them.

[0028] Through dynamic image compression processing, reduce the size of the video file, improve the processing efficiency, and reduce the requirements for storage space and transmission bandwidth; through the shot detection mechanism, automatically analyze and organize the video content to improve the efficiency of video processing; through the emotion analysis mechanism, provide an understanding of the emotional level of the video content, which helps to analyze the video content more deeply.

[0029] Furthermore, the present application further includes the following steps: S121: Obtain adjacent image groups of the compressed video; S122: Calculate the adjacent displacement value between the first image and the second image in the adjacent image groups according to the shot detection mechanism; S123: If the adjacent displacement value does not meet the predetermined displacement limit, then the first image and the second image are different shots, and the detection result is formed.

[0030] Specifically, in the compressed video, the frames of the video are usually divided into multiple image groups, and the adjacent image groups refer to two sets of image frames that are closely connected in the time series. The adjacent image groups refer to two or more consecutive frames in the video, and these images are in adjacent time periods, reflecting the continuous scenes or actions in the video. That is to say, in the video processing process, two or more consecutive frames in the video are organized into adjacent image groups, and these frames belong to consecutive time points. The shot detection mechanism is a technology used to analyze the changes of shots (i.e., scenes) in the video. By analyzing the changes between adjacent frames, it identifies the shot transitions in the video, and the shot transitions are usually the signs of scene changes.

[0031] Use the shot detection mechanism to perform motion estimation on two adjacent frames of images. By comparing the differences between adjacent frames or adjacent images, determine whether the shot has a displacement. Calculate the displacement between the images by comparing the pixel changes between two frames to obtain the adjacent displacement value between the first image and the second image. Through calculation, obtain the degree of content change between the first image and the second image, and this degree of change is quantified as the adjacent displacement value, which reflects whether the shot moves and whether there are sudden changes in the picture content (such as switching to another scene).

[0032] The adjacent displacement values are measured by calculating the motion amount of the image content, which can be accomplished through motion estimation techniques such as the optical flow method. The optical flow method estimates the motion information in the image based on the changes in image pixels. Preprocess two consecutive frames of images (such as the first image and the second image), and extract key feature points such as corners and edges. Using the optical flow method, by analyzing the brightness changes of pixels in adjacent frames, calculate the displacement vector of each pixel, including the displacement values in the horizontal and vertical directions. By integrating the optical flow vectors, calculate the overall displacement value between adjacent images, and calculate the average or maximum value of the displacements of all pixels. If the adjacent displacement value exceeds a preset threshold, it indicates that the two frames of images may belong to different shots. If a shot change occurs in the video (such as accidentally switching to another camera), the adjacent displacement value will increase significantly. By detecting this abnormal displacement, automatically mark these points to quickly locate the shot change points in the video. For example, extract consecutive frames from the video for calculation, such as forming adjacent image groups with the 1st frame and the 2nd frame, the 2nd frame and the 3rd frame, etc. In the 1st frame and the 2nd frame, the customer service staff is sitting steadily and the background has no change, and the displacement value is 5 pixels; in the 2nd frame and the 3rd frame, the shot switches to the customer side, with different backgrounds and expressions, and the displacement value is 40 pixels.

[0033] Compare the calculated displacement value with a preset displacement limit value (such as 20 pixels). If the displacement value is greater than the predetermined limit value, it indicates that the scene has changed and a shot switch has occurred; if the displacement value is small, it means that there is no obvious change in the picture and it still belongs to the same shot. Determine whether it is a shot switch according to the preset displacement limit value, and the detection result will mark the switching time in the source video. The detection result shows which frames in the video belong to the same shot and which frames belong to different shots. By calculating the adjacent displacement values, identify the shot change points in the video, accurately locate each shot, thereby improving the quality of video analysis.

[0034] Furthermore, the present application further includes the following steps: S141: Collect the first feature parameter set of the first target image in multiple dimensions; S142: Extract the first emotion dimension from the predetermined emotion dimensions, where the predetermined emotion dimensions are stored in the emotion analysis mechanism; S143: Obtain the first dimension coefficient of the first emotion dimension by combining the first feature parameter set; S144: Construct the first emotion information based on the first dimension coefficient.

[0035] Specifically, multi-dimensional collection is performed on the first target image to obtain a first set of feature parameters, including facial expressions, action units (such as raised eyebrows, raised corners of the mouth), color distribution, texture, etc. Multi-dimensional collection means extracting various different types of features from the first target image to comprehensively capture the emotional and state information of the image or scene. For example, specific facial expressions are extracted through facial action units, such as raised corners of the mouth and raised eyebrows; the positions of facial key points are detected through deep learning models (such as VGGFace, ResNet) to judge the opening and closing of the mouth, the degree of eye closure, etc.

[0036] From the collected multi-dimensional features, a first set of feature parameters is selected and sorted out, including various features related to sentiment analysis, such as facial muscle activities, eye states, expression changes, etc. For example: Facial expression: Raised eyebrows (AU1)=0.8, Raised corners of the mouth (AU12)=0.6; Hue saturation: 0.75 (a higher saturation indicates rich image colors); Eye opening degree: 0.85 (indicating that the eyes are fully open).

[0037] The predefined emotion dimensions refer to the predefined emotion standard frameworks in sentiment analysis, including emotional valence, arousal, and dominance. The predefined emotion dimensions are predefined and stored in the emotion analysis mechanism. When it is necessary to perform emotion analysis on the image sequences corresponding to each shot in the source video, these predefined emotion dimensions are first accessed. According to the set of feature parameters corresponding to the image sequence, the coefficients of each emotion dimension are calculated to quantitatively describe the emotional state in the image. The emotion analysis mechanism is used to analyze and understand the emotional content in images, videos, or other data. Emotional valence is an index in the emotion dimension, used to measure the polarity of emotions, that is, the positive and negative degrees of emotions. Usually, the emotional valence value is between -1 and +1, with negative values indicating negative emotions and positive values indicating positive emotions. Arousal measures the intensity of emotions, that is, whether the emotion is calm or excited, usually between -1 and +1. A higher arousal value indicates an excited or happy emotion, while a lower arousal value indicates a calm or peaceful emotion. Dominance is a dimension that measures the control in individual emotion perception, usually measuring the sense of control or being controlled felt by a person, usually between -1 and +1. Positive values indicate a controlling or dominant emotion, while negative values indicate a passive or restricted emotion.

[0038] Arbitrarily select one from the predetermined emotional dimensions as the first emotional dimension, and perform emotional analysis on the first set of feature parameters through the first emotional dimension to obtain the dimension coefficient corresponding to the emotional dimension (such as valence, arousal, or dominance). The first dimension coefficient refers to the value related to the first emotional dimension calculated according to the first set of feature parameters during the emotional analysis process. By processing multiple features extracted from the first set of feature parameters (such as facial expressions, voice frequencies, colors, etc.), a value related to the selected first emotional dimension (such as emotional valence) will be calculated. For example, by combining facial expression features (such as raised eyebrows and raised corners of the mouth) and audio features (such as intonation and speech rate) to infer the emotional valence of the image or audio, the first dimension coefficient of the emotional valence is obtained as 0.8, indicating that the emotion tends to be positive.

[0039] Combine the first dimension coefficient with the first emotional dimension and add it to the first emotional information. For the first target image, perform emotional analysis according to the other emotional dimensions in the predetermined emotional dimensions to obtain the corresponding dimension coefficients, which together constitute the first emotional information. The first emotional information is to calculate the corresponding three dimension coefficients respectively through the predetermined emotional dimensions for the feature parameters extracted from the first target image, so as to construct a complete three-dimensional emotional vector. For example, the emotional valence is 0.8 (positive emotion), the arousal is 0.61 (higher sense of excitement), and the dominance is 0.54 (moderate dominance). Through the extraction of multi-dimensional features and the calculation of emotional dimension coefficients, the emotional features in the video can be accurately reflected. Especially in interactive scenarios such as customer service, the emotional changes can be evaluated in real time. By calculating the emotional valence, arousal, and dominance simultaneously, the emotional state can be comprehensively captured, not only paying attention to the positive and negative polarity of the emotion, but also considering the intensity and sense of control of the emotion.

[0040] S200: Retrieve the auditory modality analysis strategy to analyze the source audio extracted from the source video to obtain the auditory analysis information.

[0041] Furthermore, S200 of this application includes: S210: Obtain the sound signal of the source audio and analyze the sound signal to obtain the sound intensity time series; S220: Randomly segment the sound intensity time series to obtain the segmentation result; S230: Collect the first time-domain feature parameters of the first time series in the segmentation result; S240: Collect the first frequency-domain feature parameters of the first frequency spectrum obtained by performing fast Fourier transform on the first time series; S250: Activate the sound emotion prediction model in the auditory modality analysis strategy to perform predictive analysis on the first time-domain feature parameters and the first frequency-domain feature parameters to obtain the first predicted emotional information; S260: Match the first audio segment corresponding to the first time series in the source audio, and based on the second mapping relationship between the first audio segment and the first predicted emotional information, form the auditory analysis information.

[0042] Specifically, the source audio in the source video is obtained, that is, the source audio extracted from the source video, which contains voice, background noise, and other sound information. The sound signal is extracted from the source audio by means of audio sampling. The audio signal is divided into multiple small time segments, and the amplitude and frequency information of each segment is extracted. The sound signal is usually sampled at a fixed sampling rate. The sampling rate is generally 16 kHz or 44.1 kHz, that is, 16,000 or 44,100 sampling points are collected per second. After sampling, the sound signal is represented as a set of digital amplitude data, and these amplitude data reflect the amplitude of the sound waveform at each sampling moment.

[0043] By calculating the amplitude (i.e., the amplitude) of the sound signal, the sound intensity time series is obtained. The larger the amplitude, the stronger the sound, and the smaller the amplitude, the weaker the sound. The amplitude of the sound signal usually fluctuates over time, and the sound intensity time series is the change of these amplitude values over time. For example, the audio signal is divided into multiple time windows, and the common window length is 20 - 50 milliseconds (ms). The frequency components and amplitudes of each time window are calculated to obtain the short-term sound intensity, thus forming the sound intensity time series. The sound intensity time series describes the change process of the audio signal over time, usually representing the change of sound intensity or amplitude over time, reflecting the change trend of the sound, such as the increase or decrease of volume, the rise and fall of audio, etc. For example, in the first 25 ms window of a customer service audio, the root mean square energy of the sampling point amplitude is 0.2 (normalized amplitude, range 0 - 1); the root mean square energy of the second window is 0.18, and the third window is 0.25, and so on; after 400 windows are calculated, a sound intensity time series with a length of 400 is obtained, reflecting the dynamic change of volume within these 10 seconds.

[0044] The sound intensity time series is randomly segmented, and the sound intensity time series data is randomly segmented into multiple time segments. The purpose is to perform local analysis on the sound signal, avoid making emotional predictions only from the entire time series, and thus improve the generalization ability of the model. The first time series is extracted from the segmentation result, and the first time domain feature parameters are collected, including average amplitude, maximum amplitude, variance, energy, zero crossing rate, etc., which are used to describe the properties of the sound signal in the time domain. For example, assume that the length of the overall sound intensity time series is 2800 points, and the length of the first time series after random segmentation is 450 points, forming the first time series. The corresponding first time domain feature parameters are collected to obtain an average amplitude of 0.22, a variance of 0.015, a maximum value of 0.55, a zero crossing rate of 0.08, and a short-term energy of 0.048.

[0045] Perform a fast Fourier transform on the first time-domain characteristic parameter. Take the time-window data (original sound waveform or amplitude sequence) of the sound signal corresponding to the first time sequence, apply the fast Fourier transform algorithm to this time series, convert the time-domain signal to the frequency domain to obtain the first frequency spectrum, and collect the first frequency-domain characteristic parameters of the first frequency spectrum, such as the main frequency, spectral centroid, bandwidth, frequency energy distribution, etc., to reflect the frequency characteristics of the signal. The frequency spectrum is an array of frequencies and corresponding amplitudes, representing the sound energy magnitude of each frequency band. For example, the first element in the frequency spectrum corresponds to 0 Hz (DC component), the 10th element corresponds to approximately 355 Hz (10 × 35.56 Hz), and so on.

[0046] Activate the sound emotion prediction model in the auditory modality parsing strategy, and use the first time-domain characteristic parameter and the first frequency-domain characteristic parameter as inputs to the trained sound emotion prediction model for prediction and analysis. Obtain the first predicted emotion information, that is, for each input audio segment, the model gives the predicted values of three emotion dimensions. The first predicted emotion information is the emotion prediction result generated by the sound emotion prediction model based on the input time-domain and frequency-domain characteristics, including the values of emotion valence, arousal, and dominance.

[0047] In the source audio, according to the time-window setting (such as every 2 seconds as a time sequence), match the first audio segment corresponding to the first time sequence, and establish the second mapping relationship between the first audio segment and the first predicted emotion information. Through the second mapping relationship, associate the predicted emotion information with the actual audio segment. For example, the emotion valence of the audio segment from 10 seconds to 12 seconds is 0.7, the arousal is 0.5, and the dominance is 0.3. For other time sequences in the source audio segmentation result, repeat the above steps to obtain the predicted emotion information corresponding to each audio segment and determine the mapping relationship. Combine the mapping relationships of all audio segments and their corresponding predicted emotion information to jointly form the auditory parsing information. The auditory parsing information is a set of information obtained by performing emotion prediction and parsing on the audio, including the emotion dimension values of each audio segment.

[0048] By randomly segmenting the sound intensity time sequence, the original data is segmented into multiple segments, which helps to extract and analyze the characteristics of the sound signal. By collecting the time-domain and frequency-domain characteristic parameters to determine the properties of the sound signal in the time domain and frequency domain, and using the sound emotion prediction model to predict the emotion state in the sound signal, it helps to analyze the audio content more deeply.

[0049] Furthermore, the present application further includes the following steps: The sound emotion prediction model includes three prediction channels for the predetermined emotion dimensions, and the predetermined emotion dimensions include emotion valence, arousal, and dominance.

[0050] Construct a voice emotion dictionary, and form a first training dataset for the first emotion dimension in the predetermined emotion dimension based on the voice emotion dictionary; perform supervised learning on the first training dataset to obtain a first prediction channel, and form the voice emotion prediction model; wherein, the first training dataset includes first training time-domain feature parameters, first training frequency-domain feature parameters, and first training emotion information.

[0051] Specifically, collect a large number of voice samples with emotion labels, including customer service recordings and public emotion speech databases, and perform manual or semi-automatic annotation on each voice sample to obtain its numerical labels in three dimensions of emotional valence, arousal, and dominance (usually continuous values or discrete levels between 0 and 1). The voice emotion dictionary simultaneously stores the time-domain and frequency-domain feature parameters corresponding to the samples to form complete training samples.

[0052] For the first emotion dimension in the predetermined emotion dimension, extract the time-domain features, frequency-domain features, and corresponding emotional valence values related to the first emotion dimension from the voice emotion dictionary to obtain the first training time-domain feature parameters, the first training frequency-domain feature parameters, and the first training emotion information, and constitute the first training dataset. Perform data preprocessing on the first training dataset, including data cleaning (removing missing or abnormal samples) and normalization (standardizing the time-domain and frequency-domain features to the range of 0~1 to reduce the influence of different dimensions). Select a suitable machine learning algorithm, such as a two-layer LSTM model. Construct the structure of the two-layer LSTM model. Input layer: 50-dimensional feature vector; first LSTM layer: 128 units, with Dropout(0.2); second LSTM layer: 64 units; fully connected layer: 1 neuron, outputting the prediction of the emotional valence value.

[0053] Use the first training dataset for model training, and use a loss function such as mean squared error (MSE) to measure the prediction error. Adjust the model parameters through the backpropagation algorithm to gradually reduce the loss. During training, all samples of the first training dataset are iterated in each training epoch. After each training epoch, evaluate the model performance on the validation set and adjust the hyperparameters. Stop training if the validation set loss does not improve for 5 consecutive training epochs.

[0054] For the other two prediction channels in the voice emotion prediction model, repeat the above steps to obtain the second prediction channel and the third prediction channel, which together form the voice emotion prediction model. The voice emotion prediction model can predict emotion dimensions (such as emotional valence, arousal, and dominance) based on the characteristics of audio signals. By constructing a voice emotion dictionary and forming a training data set related to emotions, it helps to train the voice emotion prediction model. Using the supervised learning method, the model is trained based on known input and output data so that it can predict unknown data. The voice emotion prediction model can provide an understanding of the emotional level of audio content, which helps to improve the accuracy and depth of multimodal semantic analysis.

[0055] S300: Conduct a phased comparative analysis of the visual parsing information and the auditory parsing information to obtain an emotion comparison result.

[0056] Specifically, conducting a phased comparative analysis of the visual parsing information and the auditory parsing information means comparing and analyzing the visual parsing information and the auditory parsing information at different time periods respectively to evaluate their emotional consistency or difference. The purpose of the comparative analysis is to detect whether the visual and auditory information is consistent, so as to enhance the accuracy of emotion recognition and avoid the deviation of a single modality.

[0057] Align the visual parsing information and the auditory parsing information in chronological order, and compare the emotional valence, arousal, and dominance at different time periods respectively. For each time period, use the Pearson correlation coefficient to calculate the similarity or difference of the emotional valence, arousal, and dominance between the visual and auditory information. If the Pearson correlation coefficient is high (close to 1), it indicates that the visual and auditory information is consistent during this time period; if the Pearson correlation coefficient is low (close to 0 or negative), it indicates that there may be conflicts between the visual and auditory information, and further adjustment or weighting processing is required.

[0058] After the phased comparative analysis, output the emotion comparison result for each time period. For example, in the time period from 10s to 12s, the visual valence = 0.7, the auditory valence = 0.68, showing high consistency; the visual arousal = 0.8, the auditory arousal = 0.75, showing high consistency; the visual dominance = 0.6, the auditory dominance = 0.65, showing high consistency. In the time period from 20s to 25s, the visual valence = -0.2, the auditory valence = -0.1, showing relatively low consistency; the visual arousal = 0.6, the auditory arousal = 0.3, showing medium consistency; the visual dominance = 0.5, the auditory dominance = 0.3, showing relatively low consistency.

[0059] Through stage-by-stage comparative analysis, an overall emotional comparison result is obtained. The emotional comparison result reveals the differences or consistencies in emotional expressions between video and audio. For example, if the emotions of vision and audition are inconsistent in certain time periods, it may indicate that the customer expresses emotions different from the speech in vision (such as masking real emotions through facial expressions, or the tone and facial expressions are inconsistent). According to the comparison result, corresponding adjustments are made, such as strengthening the processing of audio or video signals, weighted influence, etc. By comparing and analyzing the parsed information of vision and audition, the consistency of different modalities in the emotional dimension is judged to comprehensively understand the emotional state of the user.

[0060] S400: If the emotional comparison result meets the predetermined comparison constraint, analyze the modality parsing database to obtain real-time modality weight allocation.

[0061] Further, S400 of the present application includes: S410: Sample in the modality parsing database to obtain a modality parsing sample; S420: Conduct manual emotional review on the modality parsing sample to obtain a review result; S430: Analyze the review result to obtain the modality parsing accuracy rate, where the modality parsing accuracy rate includes visual parsing accuracy rate and auditory parsing accuracy rate; S440: Conduct variant weighted analysis on the visual parsing accuracy rate and the auditory parsing accuracy rate to obtain the real-time modality weight allocation.

[0062] Specifically, the predetermined comparison constraint is a set emotional consistency threshold or rule. Meeting the predetermined comparison constraint indicates that the emotional information of the two modalities is close enough to continue the analysis. The modality parsing database is a database storing multi-modal emotional parsing results, containing a large amount of historical sample data with visual parsing information, auditory parsing information, original segments, time tags, etc.

[0063] When the emotional comparison result between the visual and auditory modalities meets the predetermined comparison constraint, it indicates that the consistency of multi-modal emotional information has reached the trust standard. Sample from the modality parsing database to obtain a modality parsing sample. The modality parsing sample includes a visual parsing sample and an auditory parsing sample for manual emotional review. Conduct manual emotional review on the modality parsing sample. Subjectively evaluate and score the emotional parsing sample obtained by sampling through experts as a reference for the real emotional label. Use the mean square error (MSE) to analyze the review result to obtain the modality parsing accuracy rate. By analyzing the degree of consistency between the results of emotional analysis using the visual modality parsing strategy and the auditory modality parsing strategy and the manual review result, the modality parsing accuracy rate is obtained. The higher the accuracy rate, the closer the emotional prediction of the modality parsing strategy in this modality is to the real emotion manually labeled. The visual parsing accuracy rate is the accuracy of the emotional parsing result of the visual modality (such as facial expression, image analysis). The auditory parsing accuracy rate is the accuracy of the emotional parsing result of the auditory modality (such as speech analysis, audio feature extraction).

[0064] Calculate weights based on the fluctuations of the visual parsing accuracy and the auditory parsing accuracy, that is, use the standard deviation of the accuracy as the basis for weight adjustment. The modality with a smaller standard deviation (i.e., the modality with more stable accuracy) will be assigned a higher weight; the modality with a larger standard deviation (i.e., the modality with larger fluctuations in accuracy) will be assigned a lower weight. For example, the average error between the visual modality prediction and humans: valence MAE = 0.07, arousal MAE = 0.10, dominance MAE = 0.06. Calculate the visual MAE to be 0.0767, and the visual parsing accuracy is 1 - 0.0767 = 92.33%; the average error between the visual modality prediction and humans: valence MAE = 0.08, arousal MAE = 0.11, dominance MAE = 0.09. Calculate the auditory MAE to be 0.0933, and the auditory parsing accuracy is 90.67%. The visual standard deviation is 0.015, and the auditory standard deviation is 0.03. Calculate the stability index: the visual stability index is 61.55, and the auditory stability index is 30.22. Therefore, the real-time modality weight distribution obtained is: the visual modality weight is 67%; the auditory modality weight is 33%. Adaptive allocation of modality weights according to the current model accuracy and its stability can avoid overall misjudgment caused by excessive errors in a certain modality. The weight adaptive allocation mechanism avoids the fuzzy judgment caused by the average processing mode and has more judgment in multi-source information fusion.

[0065] S500: Obtain the text parsing information of the source text, and fuse the text parsing information, the visual parsing information, and the auditory parsing information in combination with the real-time modality weight distribution to obtain a semantic parsing result.

[0066] Furthermore, S500 of the present application includes: S510: Perform cleaning and word segmentation processing on the source text to obtain a processing result; S520: Perform feature analysis on the processing result based on the principle of the bag-of-words model to obtain a feature vector; S530: Perform sentiment analysis on the feature vector in combination with the lexical sentiment dictionary to obtain the text parsing information.

[0067] Furthermore, the present application also includes the following steps: S531: Extract any vocabulary in the feature vector; S532: Match any score of the any vocabulary in the lexical sentiment dictionary, where the any score includes three scores of the predetermined sentiment dimension; S533: Obtain the any frequency of the any vocabulary according to the feature vector, and combine the three scores to obtain the any sentiment score of the any vocabulary; S534: Take the mean of the any sentiment scores to form the text parsing information.

[0068] Specifically, clean and segment the source text, removing irrelevant characters, punctuation marks, and stop words, and splitting the text into words or lexical units to obtain the processing result. Preprocess the text to remove noise (such as punctuation marks, stop words, etc.). Word segmentation is to split the text into individual words. The source text is the original input text data, such as the conversation content between the customer and the customer service obtained from the customer service conversation, or the text extracted from the video subtitles. Cleaning includes removing all unnecessary symbols, punctuation, stop words (such as "de", "le"), etc. Use word segmentation algorithms to divide the text into independent words, including rule-based word segmentation (such as regular expressions) or statistics-based word segmentation (such as Jieba word segmentation, NLP toolkits, etc.).

[0069] The bag-of-words model is a text representation method that ignores the order of words and only considers the frequency of each word in the text. Feature analysis is to convert the processed text into a high-dimensional vector, where each dimension corresponds to a word and the value is the frequency of that word in the text. Based on the principle of the bag-of-words model, perform feature analysis on the processing result to obtain the feature vector. In this process, each word is a feature, and each dimension of the feature vector corresponds to a word, and its value represents the frequency of that word in the text. Specifically, collect all the words in the text and construct a vocabulary (dictionary) that contains all independent words. For each piece of text, count the number of times each word in the vocabulary appears. Convert the word frequency into a vector, and the length of the vector is equal to the size of the vocabulary. According to the word frequency statistics result, use the bag-of-words model to convert the text into a feature vector. The length of the feature vector is equal to the size of the vocabulary. The value at each position corresponds to the number of times the corresponding word in the vocabulary appears in the text.

[0070] Extract any word in the feature vector and match it in the lexical sentiment dictionary to obtain any score of any word, including three scores of the predetermined sentiment dimension, namely the sentiment valence score, the arousal score, and the dominance score. The sentiment score is the score of each word in the sentiment dictionary, usually including three sentiment dimensions: valence, arousal, and dominance. The lexical sentiment dictionary is a dictionary that contains words and their corresponding sentiment scores, and each word is associated with multiple sentiment dimensions, such as valence, arousal, and dominance.

[0071] Extract the arbitrary frequency of any word according to the feature vector, and calculate the arbitrary sentiment score of any word by combining the three scores. The arbitrary frequency is the number of times any word appears. The sentiment score of each word is the result calculated based on the frequency of that word in the text and its score in the sentiment dictionary. For example, the sentiment score of the customer service word is (valence 0.7, arousal 0.4, dominance 0.5), and the frequency is 1. According to the frequency and sentiment score of the customer service, calculate the sentiment score of this word as (0.7, 0.4, 0.5).

[0072] Calculate the mean of arbitrary sentiment scores to form the text analysis information. Repeat the above steps for each word in the feature vector to jointly form the text analysis information. According to the real-time modality weight assignment, fuse the text analysis information, visual analysis information, and auditory analysis information, that is, according to the analysis accuracy of different modalities (text, visual, auditory) at a certain moment, assign the weights of each modality in a weighted manner, and use the weighted average of the text analysis information, visual analysis information, and auditory analysis information after adjusting the weights to obtain the semantic analysis result. Through the real-time modality weight assignment, the contribution of each modality in semantic analysis can be dynamically adjusted according to the analysis accuracy of each modality, ensuring the reliability of the final result. Fuse the sentiment information of the text, visual, and auditory modalities to comprehensively reflect the sentiment state of the input data, making the semantic analysis more in-depth and diverse.

[0073] In summary, a cross-modal semantic analysis method provided by the present application has the following beneficial effects: Parse the source video by invoking the visual modality parsing strategy to obtain visual analysis information; invoke the auditory modality parsing strategy to parse the source audio extracted from the source video to obtain auditory analysis information; perform a phased comparative analysis on the visual analysis information and the auditory analysis information to obtain a sentiment comparison result; if the sentiment comparison result reaches a predetermined comparison constraint, analyze the modality parsing database to obtain a real-time modality weight assignment; obtain the text analysis information of the source text, and fuse the text analysis information, the visual analysis information, and the auditory analysis information in combination with the real-time modality weight assignment to obtain a semantic analysis result. That is to say, by parsing the visual content in the source video, extracting visual analysis information; parsing the source audio in the source video to obtain auditory analysis information; performing a phased comparative analysis on the visual analysis information and the auditory analysis information to obtain a sentiment comparison result; by analyzing the sentiment comparison result, dynamically adjusting the weight of each modality, and fusing the parsing information of text, visual, and auditory according to the real-time modality weight assignment to obtain a semantic analysis result, improving the accuracy and adaptability of cross-modal semantic analysis.

[0074] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0075] Obviously, for those skilled in the art, without departing from the principle of this application, several improvements and modifications can still be made to this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A cross-modal semantic parsing method, characterized in that, Including: Retrieve the visual modality parsing strategy to parse the source video and obtain visual parsing information; Retrieve the auditory modality parsing strategy to parse the source audio extracted from the source video and obtain auditory parsing information; Conduct a phased comparative analysis on the visual parsing information and the auditory parsing information to obtain an emotion comparison result; If the emotion comparison result meets the predetermined comparison constraint, analyze the modality parsing database to obtain real-time modality weight allocation; Obtain the text parsing information of the source text, and combine the real-time modality weight allocation to fuse the text parsing information, the visual parsing information, and the auditory parsing information to obtain a semantic parsing result.

2. The cross-modal semantic parsing method according to claim 1, wherein Retrieve the visual modality parsing strategy to parse the source video and obtain visual parsing information, including: Perform dynamic image compression processing on the source video to obtain a compressed video; Detect and analyze the compressed video according to the shot detection mechanism in the visual modality parsing strategy to obtain a detection result; Extract the first image sequence corresponding to the first shot in the detection result, and perform enhanced fusion processing on the first image sequence to obtain a first target image; Perform emotion parsing on the first target image according to the emotion parsing mechanism in the visual modality parsing strategy to obtain first emotion information; Match the first video segment corresponding to the first shot in the source video, and form the visual parsing information based on the first mapping relationship between the first video segment and the first emotion information.

3. The cross-modal semantic parsing method according to claim 2, wherein Detect and analyze the compressed video according to the shot detection mechanism in the visual modality parsing strategy to obtain a detection result, including: Obtain adjacent image groups of the compressed video; Calculate the adjacent displacement value between the first image and the second image in the adjacent image groups according to the shot detection mechanism; If the adjacent displacement value does not meet the predetermined displacement limit, the first image and the second image are different shots, and the detection result is formed.

4. The cross-modal semantic parsing method according to claim 2, wherein Perform emotion parsing on the first target image according to the emotion parsing mechanism in the visual modality parsing strategy to obtain first emotion information, including: Collect the first feature parameter set of the first target image in multiple dimensions; Extract the first emotion dimension in the predetermined emotion dimension, where the predetermined emotion dimension is stored in the emotion parsing mechanism; Combine the first feature parameter set to obtain the first dimension coefficient of the first emotion dimension; Form the first emotion information based on the first dimension coefficient.

5. The cross-modal semantic parsing method according to claim 4, wherein Retrieve the auditory modality parsing strategy to parse the source audio extracted from the source video and obtain auditory parsing information, including: Obtain the sound signal of the source audio, and analyze the sound signal to obtain the sound intensity time series; Randomly segment the sound intensity time series to obtain a segmentation result; Collect the first time-domain feature parameters of the first time series in the segmentation result; Collect the first frequency-domain feature parameters of the first frequency spectrum obtained by performing fast Fourier transform processing on the first time series; Activate the sound emotion prediction model in the auditory modality parsing strategy to perform prediction analysis on the first time-domain feature parameters and the first frequency-domain feature parameters to obtain first predicted emotion information; Match the first audio segment corresponding to the first time sequence in the source audio, and form the auditory analysis information based on the second mapping relationship between the first audio segment and the first predicted emotion information.

6. The cross-modal semantic parsing method according to claim 5, wherein The voice emotion prediction model includes three prediction channels for the predetermined emotion dimension, and the predetermined emotion dimension includes emotional valence, arousal, and dominance.

7. The cross-modal semantic parsing method according to claim 6, wherein Constructing the voice emotion prediction model includes: Construct a voice emotion dictionary, and form a first training data set for the first emotion dimension in the predetermined emotion dimension based on the voice emotion dictionary; Perform supervised learning on the first training data set to obtain a first prediction channel, and form the voice emotion prediction model; Among them, the first training data set includes first training time domain feature parameters, first training frequency domain feature parameters, and first training emotion information.

8. The cross-modal semantic parsing method according to claim 1, wherein, If the emotion comparison result reaches a predetermined comparison constraint, analyze the modal analysis database to obtain real-time modal weight allocation, including: Sample in the modal analysis database to obtain a modal analysis sample; Perform manual emotion review on the modal analysis sample to obtain a review result; Analyze the review result to obtain the modal analysis accuracy rate, where the modal analysis accuracy rate includes visual analysis accuracy rate and auditory analysis accuracy rate; Perform variant weighted analysis on the visual analysis accuracy rate and the auditory analysis accuracy rate to obtain the real-time modal weight allocation.

9. The cross-modal semantic parsing method according to claim 4, wherein, Obtain the text analysis information of the source text, including: Perform cleaning and word segmentation processing on the source text to obtain a processing result; Perform feature analysis on the processing result based on the principle of the bag-of-words model to obtain a feature vector; Combine the lexical emotion dictionary to perform emotion analysis on the feature vector to obtain the text analysis information.

10. The cross-modal semantic parsing method according to claim 9, wherein Combining the lexical emotion dictionary to perform emotion analysis on the feature vector to obtain the text analysis information includes: Extract any word in the feature vector; Match in the lexical emotion dictionary to obtain any score of the any word, where the any score includes three scores of the predetermined emotion dimension; Obtain the any frequency of the any word according to the feature vector, and combine the three scores to obtain the any emotion score of the any word; Take the average value of the any emotion scores to form the text analysis information.

Images