Social listening method and device based on voice recognition

Abstract

The invention provides a social listening method and device based on voice recognition, and belongs to the technical field of social media data processing and voice recognition crossing. The method comprises the following steps: acquiring voice content from a social media platform in real time, and performing format unified conversion, noise filtering and audio slice preprocessing on the acquired voice content to obtain standardized voice data; converting the standardized voice data into text data by using a deep learning voice recognition model and combining hierarchical decoding, noise suppression and context perception technologies; performing multi-modal emotion and emotion analysis and multi-language and dialect adaptation processing on the standardized voice data and the text data to obtain a multi-dimensional analysis result; and presenting an analysis result through an interactive billboard, monitoring real-time voice data, and triggering an alarm and generating a report when an abnormal emotion or a sensitive topic is found. According to the invention, the limitation that a traditional social listening system depends on text data is broken through, and the accuracy, real-time performance and multi-language adaptation capability of voice data processing are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of page interaction and content loading, and particularly relates to a social listening method and device based on voice recognition and electronic equipment. BACKGROUND

[0002] With the rapid development of social media platforms, user-generated content forms are increasingly diverse. In addition to text, podcast, voice live broadcast, short video audio track, and social chat voice account for a significant proportion of voice content. Under this background, social listening, as a core means for enterprises to obtain user feedback, analyze market trends, and respond to public opinion risks, has a growing demand for technical applications.

[0003] Currently, existing social listening systems mainly rely on text data analysis techniques such as natural language processing and keyword matching. However, there are obvious limitations in the mining capabilities of voice data, resulting in a large amount of valuable voice content not being effectively utilized, which in turn leads to multiple technical defects. Firstly, voice data processing is insufficient, and cannot analyze mainstream voice content such as podcasts and live audio. It can only process text information within the platform, resulting in incomplete data collection. Secondly, the emotional analysis dimension is single, and only the user's emotions are judged through text semantics, ignoring key emotional signals such as tone, speed, and energy changes in voice, resulting in inaccurate interpretation of user attitudes. Thirdly, real-time performance is poor. Traditional voice-to-text technology uses batch processing, which is difficult to quickly output analysis results in high-concurrency and low-latency social scenarios, and cannot meet the real-time monitoring needs of public opinion. Fourthly, the multi-language and dialect adaptation capability is weak. Traditional voice recognition models are mostly developed for single language or limited languages, and do not integrate multi-language processing technology, resulting in low recognition accuracy for local dialects, limiting their application range in global social platforms.

[0004] In summary, the technical defects of existing social listening systems in voice data processing, emotional analysis comprehensiveness, real-time performance, and multi-language adaptation have made it impossible to meet the current needs of enterprises for diversified social data analysis. There is an urgent need for a social listening technology solution that can efficiently process voice data to fill the gaps in existing technology. SUMMARY

[0005] Therefore, the embodiments of the present application provide a social listening method and device based on voice recognition and electronic equipment, at least partially solving the problems in the prior art.

[0006] In a first aspect, the embodiments of the present application provide a social listening method based on voice recognition, comprising the following steps:

[0007] Real-time collection of voice content from social media platforms, preprocessing of the collected voice content to obtain standardized voice data, the voice content including podcasts, live audio, short video audio tracks and social chat voice, the preprocessing including format conversion, noise filtering and audio slicing;

[0008] Processing the standardized voice data using a deep learning speech recognition model to accurately convert it into text data;

[0009] Multi-dimensional analysis of the standardized voice data and text data to obtain analysis results, the multi-dimensional analysis including sentiment and emotion analysis, multi-language and dialect adaptation processing;

[0010] Presenting the multi-dimensional analysis results through an interactive dashboard, while monitoring real-time voice data, triggering an alarm and generating an analysis report when abnormal emotions or sensitive topics are found.

[0011] According to a specific implementation of an embodiment of the present application, in the process of real-time collection of voice content from social media platforms, multi-source voice data access is achieved through dynamic interface adaptation technology, which can automatically adjust data request parameters and data parsing rules according to the API protocol differences of different social media platforms;

[0012] At the same time, through the flow dynamic allocation mechanism, data interruption caused by platform interface flow limiting in the collection process is avoided.

[0013] According to a specific implementation of an embodiment of the present application, in the step of preprocessing the collected voice content, the adaptive format parsing algorithm is used in the format conversion link, which can automatically identify the coding characteristics of multiple audio formats and generate adaptation conversion rules;

[0014] Audio slicing processing combines with speech semantic pause detection technology to determine the slicing position by recognizing the natural language break nodes in the voice.

[0015] According to a specific implementation of an embodiment of the present application, the deep learning speech recognition model uses an end-to-end speech recognition model based on the Transformer architecture, which inputs the spectral features of the standardized voice data into the model and models the voice time series information through the attention mechanism;

[0016] The model uses a hierarchical decoding strategy to segment long voice according to a preset time interval and transcribe each segment of voice separately;

[0017] Then, the context encoder integrates the transcription results of each segment to ensure semantic coherence;

[0018] At the same time, the model introduces a noise suppression model based on a variational autoencoder to extract clear voice signals in various complex background noise environments;

[0019] And configure the context-aware speech recognition module, in combination with the current interaction scene of the user, the timestamp and the related semantic context, improve the transcription accuracy of ambiguous vocabulary.

[0020] According to a specific implementation manner of the embodiment of the application, the sentiment and emotion analysis adopts a multi-modal sentiment analysis technology, wherein the sentiment analysis is realized through a sentiment classification model based on a Transformer;

[0021] The sentiment classification model supports positive, negative and neutral basic sentiment labels, and can also extend personalized sentiment labels of anger, support and satisfaction according to specific application scenarios;

[0022] And a sentiment feature embedding network is used to capture complex sentiment levels in user expressions, so as to realize multi-level sentiment extraction.

[0023] The emotion analysis is realized through a combined model of a convolutional neural network and a recurrent neural network, acoustic features of pitch, speech rate and energy of the speech are extracted through the convolutional neural network, and the time sequence correlation of the acoustic features is modeled through the recurrent neural network.

[0024] Finally, the text sentiment result obtained through the sentiment analysis and the acoustic feature analysis result obtained through the emotion analysis are combined, and emotion classification is realized through a multi-modal fusion network.

[0025] According to a specific implementation manner of the embodiment of the application, the multi-language and dialect adaptation processing is realized through a multi-language and dialect processing module, which adopts a multi-language speech recognition model based on multi-task learning.

[0026] The multi-language speech recognition model realizes unified processing of multi-language speech data through a shared bottom-layer feature extractor.

[0027] And a language-specific adaptive layer is introduced, so that the recognition strategy of the model can be dynamically adjusted when the system recognizes speech data of different language types.

[0028] Meanwhile, a cross-language data enhancement technology is used to expand the training data based on the similarity between languages, so as to improve the cross-language recognition effect.

[0029] For local dialect processing, a dialect adaptation model based on feature transfer learning is configured in the module, the model is pre-trained using standard language corpus, and then the pre-trained model is fine-tuned through a small amount of dialect corpus, so as to realize efficient dialect recognition.

[0030] According to a specific implementation manner of the embodiment of the present application, the interactive board adopts a dynamic visualization updating technology, and a hot topic graph is dynamically adjusted in association strength and position distribution of a topic node according to a newly acquired voice analysis result through a real-time correlation analysis algorithm.

[0031] The emotion trend analysis chart combines a time series prediction model to display a historical emotion change trend and predict and mark an emotion trend in a short future time.

[0032] According to a specific implementation manner of the embodiment of the present application, a multi-level threshold early warning mechanism is adopted for monitoring real-time voice data, and a basic emotion intensity threshold and an emotion diffusion speed threshold are set for abnormal emotions.

[0033] A topic mention frequency threshold and a topic propagation range threshold are set for sensitive topics.

[0034] When monitoring data triggers any threshold, the system first excludes false detection through an abnormal data verification algorithm.

[0035] Then, an alarm and an analysis report of a corresponding level are generated according to a threshold trigger level, and a high-level alarm is accompanied by real-time voice segment positioning and key emotion / topic marking.

[0036] In a second aspect, the embodiment of the present application further provides a social listening device based on voice recognition, comprising:

[0037] A collection preprocessing module is configured to collect voice content from a social media platform in real time, and preprocess the collected voice content to obtain standardized voice data, wherein the voice content includes podcasts, live audio, short video audio tracks, and social chat voice, and the preprocessing includes format uniform conversion, noise filtering, and audio slicing.

[0038] A voice-to-text module is configured to process the standardized voice data by using a deep learning voice recognition model to accurately convert the standardized voice data into text data.

[0039] A multi-dimensional analysis module is configured to perform multi-dimensional analysis on the standardized voice data and the text data to obtain analysis results, wherein the multi-dimensional analysis includes emotion and sentiment analysis, multi-language and dialect adaptation processing.

[0040] A visualization and monitoring alarm module is configured to present the multi-dimensional analysis results through an interactive board, and monitor real-time voice data, trigger an alarm and generate an analysis report when abnormal emotions or sensitive topics are found.

[0041] In a third aspect, the embodiment of the present application further provides an electronic device, which comprises:

[0042] at least one processor; and

[0043] a memory in communication with the at least one processor; wherein

[0044] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method in the first aspect or any implementation manner of the first aspect.

[0045] In a fourth aspect, the embodiments of the present application further provide a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method in the first aspect or any implementation manner of the first aspect.

[0046] In a fifth aspect, the embodiments of the present application further provide a computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions that, when executed by a computer, cause the computer to perform the method in the first aspect or any implementation manner of the first aspect.

[0047] The present application has the following beneficial effects:

[0048] Effectively break through the limitation of existing social listening systems relying on text data, realize comprehensive coverage of multiple types of voice content such as podcasts, live audio, short video audio tracks, social chat voice, etc. in social media platforms; through format unified conversion, noise filtering and audio slice preprocessing, combined with deep learning speech recognition model based on Transformer architecture, the integrity and standardization processing efficiency of voice data collection are greatly improved, and the problem of insufficient voice data mining capability of traditional systems is solved.

[0049] Multi-modal sentiment and emotion analysis technology is adopted, and text semantic features and voice acoustic features (intonation, speech rate, energy change, etc.) are comprehensively judged. Compared with single-dimensional analysis relying only on text, the accuracy of understanding user sentiment attitude is significantly improved, and analysis deviation caused by ignoring voice emotion signals is avoided.

[0050] Through the hierarchical decoding strategy, the dynamic context cache mechanism and the real-time monitoring algorithm, efficient processing of the whole process of speech data from collection, conversion to analysis and early warning is realized; in the high-concurrency social scene, the analysis result can be quickly output and the alarm can be triggered, meeting the real-time monitoring demand of enterprises on public opinion and improving the real-time poor defect of the traditional batch processing mode; relying on the multi-task learning multi-language speech recognition model and the dialect adaptation model of feature migration learning, not only unified processing of multiple languages is supported, but also efficient dialect recognition can be realized through a small amount of dialect corpus fine-tuning; the adaptation capability of the system in the global social platform is greatly improved, breaking the limitation of traditional models in single language or limited language processing and expanding the application range of the technology.

[0051] Through the interactive dashboard to dynamically present the hot topic graph and the sentiment trend, combined with the time series prediction function, the enterprise is provided with forward-looking decision reference; at the same time, the multi-level threshold early warning mechanism and the abnormal data verification algorithm can reduce the false alarm rate, and the positioning and key labeling of the voice segment attached to the high-level alarm can further improve the accuracy and efficiency of the enterprise public opinion response. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0053] Figure 1 A social listening method based on voice recognition provided by the embodiment of the present application is shown in the flowchart.

[0054] Figure 2 Another social listening method based on voice recognition provided by the embodiment of the present application is shown in the flowchart.

[0055] Figure 3 The structure diagram of a social listening device based on voice recognition provided by the embodiment of the present application is shown.

[0056] Figure 4 The schematic diagram of an electronic device provided by the embodiment of the present application is shown. DETAILED DESCRIPTION

[0057] The embodiments of the present application will be described in detail below with reference to the drawings.

[0058] Referring to Figure 1 and Figure 2 , the embodiment of the present application provides a social listening method based on voice recognition, including the following steps:

[0059] S101, real-time collection of voice content from social media platforms, preprocessing of the collected voice content to obtain standardized voice data, the voice content including podcasts, live audio, short video audio tracks and social chat voice, preprocessing including format conversion, noise filtering and audio slicing.

[0060] In the implementation of this step, first, the dynamic interface adaptation component is connected to at least 10 types of mainstream social media platform APIs, including but not limited to live streaming platforms, short video platforms, social chat tools and podcast platforms. The built-in API protocol analysis library in the component can automatically identify the interface specifications of different platforms (such as RESTful, WebSocket, etc.) and dynamically generate adapted data request parameters.

[0061] During the collection process, a distributed crawler cluster is used to realize multi-source concurrent collection, and a single node can process 50-100 voice stream data per second. For platform flow limiting mechanism, the system has a built-in flow monitoring module, which detects the response delay and return code of each platform interface in real time, and when it detects flow limiting risk, it automatically triggers the flow distribution algorithm to balance the request load to the standby node.

[0062] In the preprocessing stage, the format conversion module identifies more than 15 audio formats such as MP3, AAC, WAV through audio feature extraction algorithm, and calls the corresponding decoder to convert to 16kHz single-channel PCM format; noise filtering uses a combination of spectral subtraction and wavelet threshold denoising algorithm, which processes environmental noise (such as street noise, equipment noise) and sudden noise (such as screams, collision sounds) in layers, and the signal-to-noise ratio can be improved by 15-20dB; audio slicing uses voice activity detection (VAD) algorithm to identify natural language breaks, sets a dynamic slicing window of 5-15 seconds to ensure that each slice contains a complete semantic unit.

[0063] S102, using a deep learning speech recognition model to process the standardized voice data to accurately convert it into text data.

[0064] An end-to-end speech recognition system based on improved Transformer is used, and the model training data set contains more than 100,000 hours of multi-scene voice corpus (covering live streaming, social chat, podcast, etc. Scenes). The input layer converts the standardized voice data into mel-spectral features, and an 8-layer Transformer encoder is used for feature extraction, in which the cross-frame attention mechanism can capture voice correlation features within a 500ms time window.

[0065] The decoding stage adopts a three-level hierarchical strategy: first, long speeches longer than 10 minutes are segmented into 5-10 segments according to semantic paragraphs; each segment is transcribed by a 6-layer Transformer decoder to generate preliminary text; finally, the context encoder integrates the results of each segment, and the BERT pre-training model corrects the semantic discontinuity of the segmented transcription. The accuracy of long speech transcription can reach more than 95%.

[0066] For complex environments, the system integrates a noise suppression submodule based on a variational autoencoder, which learns noise distribution features through adversarial training, maintaining stable recognition performance in -5dB to 20dB SNR environments. The context-aware module associates user profiles (such as age, region, and commonly used language), interaction scenarios (such as live sales and daily conversations), and timestamp information in real time to construct a context feature vector, improving the recognition accuracy of ambiguous words (such as homophonic words and popular online slang) by more than 20%.

[0067] S103, multi-dimensional analysis of the standardized speech data and text data to obtain analysis results, the multi-dimensional analysis includes sentiment and emotion analysis, multi-language and dialect adaptation processing.

[0068] The sentiment analysis submodule adopts a dual-modal fusion architecture: the text analysis path inputs the transcribed text into a sentiment classification model based on RoBERTa. The pre-training corpus of this model contains 5 million annotated social texts, which can recognize basic sentiments such as positive, negative, and neutral, as well as 12 sub-sentiments such as anger, surprise, and disappointment. Through sentiment semantic mapping technology, the sentiment vocabulary and context semantics are associated with visualized weights, and the fine-grained classification accuracy reaches 88%.

[0069] The speech emotion analysis path processes standardized speech data through a CNN-RNN combined model: the CNN layer uses a 3-layer 1D convolution to extract 128-dimensional acoustic features (including fundamental frequency, speech rate, energy entropy, etc.); the BiLSTM layer models the feature sequence to capture dynamic changes in emotions; the output layer uses the CRF algorithm for emotion sequence labeling, with an accuracy of 85%.

[0070] Dual-modal features are deeply integrated through an attention-weighted fusion network. The dynamic weight coefficients of text features and speech features are automatically adjusted according to confidence (range 0.3-0.7), and the comprehensive sentiment analysis F1 value can reach 0.90.

[0071] In terms of multi-language and dialect adaptation, the system supports more than 20 mainstream languages and 30 local dialects. The multi-language model shares parameters through a shared 6-layer Transformer feature extractor, and the middle-layer language classifier uses a FastText model (accuracy 99%) to dynamically switch language adaptation parameters. Dialect processing is achieved through transfer learning, based on a model pre-trained on 100 hours of standard corpus, which only needs 5 hours of dialect corpus fine-tuning to achieve an accuracy of more than 80%.

[0072] S104, the multi-dimensional analysis result is presented through an interactive dashboard, and real-time voice data is monitored to trigger an alarm and generate an analysis report when an abnormal emotion or sensitive topic is found.

[0073] The interactive dashboard is designed using the B / S architecture, and the front end is based on ECharts for dynamic visualization. The hot topic map uses a force-directed layout algorithm, with node size mapping topic heat (mention frequency) and edge weight representing topic correlation strength. It is updated automatically every 30 seconds, supports zooming, dragging, and node detail viewing.

[0074] Sentiment trend analysis is displayed through a combination of time series heat maps and line charts. The horizontal axis represents time (minimum granularity 1 minute), and the vertical axis represents sentiment polarity value (-1 to 1). Combined with the ARIMA time series prediction model, the future 2-hour sentiment trend is predicted and labeled (error rate < 10%).

[0075] The real-time monitoring module uses a multi-level threshold mechanism. Abnormal emotion monitoring sets a basic threshold (e.g. negative emotion intensity > 0.8) and a diffusion speed threshold (e.g. 10 times growth in mention volume in 10 minutes). Sensitive topic monitoring sets a frequency threshold (e.g. > 50 mentions per minute) and a propagation range threshold (e.g. covers more than 3 platforms).

[0076] When the threshold is triggered, the system first excludes false positives using the Isolation Forest algorithm (accuracy 92%), and then generates red (emergency), yellow (warning), and blue (prompt) alerts according to severity. High-level alerts include voice segment positioning (accurate to the second) and key emotion / topic labeling, and automatically generate a PDF analysis report containing sentiment distribution, propagation path, and impact population profile. The response delay is less than 5 seconds.

[0077] According to a specific implementation manner of the embodiment of the application, in the process of collecting voice content from a social media platform in real time, multi-source voice data access is realized through a dynamic interface adaptation technology, which can automatically adjust data request parameters and data analysis rules according to API protocol differences of different social media platforms; at the same time, a flow dynamic allocation mechanism is used to avoid data interruption caused by platform interface flow limiting during the collection process.

[0078] Specifically, multi-source collection is performed on a live broadcast platform A (using a RESTful protocol), a short video platform B (using a WebSocket protocol) and a social chat platform C (using a GraphQL protocol). The dynamic interface adaptation technology internally includes a database of 200+ API protocol templates. When starting a collection task, the system first sends a detection request to obtain the response header information of each platform interface, and automatically matches the corresponding protocol template: for platform A, the system automatically generates the "Authorization: Bearer{token}" request header and the "timestamp={time}&sign={signVal}" parameter; for platform B, the system automatically establishes a long connection and sets the "heartbeat=30s" keep-alive mechanism; for platform C, the system automatically constructs the "query {voiceChats(userId:123, startTime:"2024-01-01"){content, timestamp}}" query statement. The flow allocation mechanism uses a sliding window algorithm to real-time statistics the request success rate of each platform interface. When the platform A interface returns "429 Too Many Requests", the system transfers 50% of the collection node load originally allocated to platform A to a backup node, and adjusts the request interval from 100ms to 300ms. The interface state is continuously monitored, and when the success rate is restored to more than 95%, the load proportion is gradually adjusted. The whole process has no data interruption, and the collection continuity reaches 99.8%.

[0079] According to a specific implementation manner of the embodiment of the application, in the step of preprocessing the collected voice content, the format uniform conversion link uses an adaptive format analysis algorithm, which can automatically identify the coding characteristics of multiple audio formats and generate adaptive conversion rules; the audio slicing processing combines with the voice semantic pause detection technology to determine the slicing position by identifying the natural language break nodes in the voice.

[0080] Specifically, in the format conversion stage, the adaptive format analysis algorithm first reads the first 100 bytes of the header information of the audio file, extracts the encoding type identifier (such as "ID3" for MP3, "ADTS" for AAC, and "RIFF" for WAV), and then analyzes the audio sampling rate, bit depth, and other parameters through Fourier transform. For the 1000 pieces of mixed format voice data collected (including 30% MP3, 40% AAC, and 30% WAV), the corresponding decoding rules are automatically generated, and the FFmpeg decoder is called to batch convert them into 16kHz, 16bit single-channel PCM format. The conversion time is 0.5 seconds per piece on average, and the format recognition accuracy is 100%. In the audio slicing stage, the voice semantic pause detection technology analyzes the voice energy and zero-crossing rate through the VAD algorithm. When the energy is lower than -40dB and the zero-crossing rate is less than 50 times per second within 200ms, it is determined as a semantic pause node. For a 15-minute live voice, the system identifies 28 natural semantic break nodes and generates 28 slices with a duration of 30-60 seconds. Each slice contains a complete sentence (such as "The core advantage of this product is its long battery life" and "Next, I will demonstrate the specific operation steps"), without semantic breakage, and the slice effectiveness reaches 100%.

[0081] According to a specific implementation manner of the embodiment of the application, the deep learning voice recognition model adopts an end-to-end voice recognition model based on a Transformer architecture. After the spectral features of the standardized voice data are input into the model, the voice time sequence information is modeled through an attention mechanism. The model adopts a hierarchical decoding strategy, segments the long voice according to a preset time interval, and transcribes each segment of voice separately. Then, the context encoder integrates the transcription results of each segment to ensure semantic coherence. Meanwhile, the model introduces a noise suppression model based on a variational autoencoder to extract clear voice signals in various complex background noise environments. In addition, the model is configured with a context-aware voice recognition module to improve the transcription accuracy of ambiguous words by combining the current interactive scene, timestamp, and related semantic context of the user.

[0082] Specifically, a 20-minute podcast voice (including street background noise, signal-to-noise ratio 8dB) can be processed. First, the standardized voice data is converted into 80-dimensional Mel spectrum features, and then input into an 8-layer Transformer encoder. Each layer captures the voice-related features (such as the phoneme association of continuous words such as "artificial intelligence" and "machine learning") within a 500ms time window through a multi-head attention mechanism (8 attention heads). During hierarchical decoding, a 180-second segmentation interval is preset, and the 20-minute voice is divided into 7 segments. Each segment is transcribed by a 6-layer Transformer decoder to generate a preliminary text (such as the third segment preliminary transcription "AI technology in the application of he therapy"). The context encoder is based on the BERT pre-training model to analyze the semantics of adjacent segments (such as the second segment containing "hospital data management" and the fourth segment containing "disease diagnosis assistance"), and determine that "he therapy" should be "medical treatment". After correction, the semantic coherence of the text is improved by 35%. The noise suppression model learns the noise distribution through a variational autoencoder. After processing the 8dB signal-to-noise ratio voice, the signal-to-noise ratio is improved to 22dB, and the voice clarity is significantly improved. The context-aware module combines the user scenario (the podcast theme is "medical technology") and the timestamp (corresponding to the "remote diagnosis" discussion session) to correct the ambiguous word "the system can realize x real-time monitoring" to "the system can realize real-time monitoring". The ambiguous word transcription accuracy is improved from 72% to 96%.

[0083] According to a specific implementation manner of the embodiment of the application, the sentiment and emotion analysis adopts a multi-modal sentiment analysis technology, wherein the sentiment analysis is realized through a Transformer-based sentiment classification model; the sentiment classification model supports basic sentiment labels such as positive, negative and neutral, and can also extend personalized sentiment labels such as anger, support and satisfaction according to specific application scenarios; and a sentiment feature embedding network is used to capture complex sentiment levels in user expressions, so as to realize multi-level sentiment extraction; the emotion analysis is realized through a combined model of a convolutional neural network and a recurrent neural network, acoustic features such as pitch, speech rate and energy of the voice are extracted through the convolutional neural network, and the time sequence correlation of the acoustic features is modeled through the recurrent neural network; finally, the text sentiment result obtained through the sentiment analysis and the acoustic feature analysis result obtained through the emotion analysis are combined, and emotion classification is realized through a multi-modal fusion network.

[0084] Specifically, a short video voice comment (text content: "This function is very useful, but the loading is a bit slow", voice with excited tone + slight complaint tone) can be analyzed. The sentiment classification model is based on RoBERTa pre-training. After inputting the text, the "useful" (positive weight 0.8) and "slow loading" (negative weight 0.6) features are extracted through the sentiment feature embedding network, and the "positive - slight negative" multi-level sentiment label is output, and the sentiment classification accuracy is 92%. In the emotion analysis stage, a 3-layer 1D convolutional neural network extracts acoustic features: average pitch 250Hz (excitement feature), speech rate 180 words per minute, and energy entropy 0.7 (complaint energy fluctuation). Then, the BiLSTM network models the feature sequence (excited tone lasts for 3 seconds, and then turns into slight complaint), and outputs the "excitement - dissatisfaction" emotion sequence. The multi-modal fusion network uses attention weighting algorithm, and the text feature weight is 0.6 and the acoustic feature weight is 0.4. The overall emotion is determined as "overall satisfaction but with slight dissatisfaction", which is consistent with the artificial annotation result, with a consistency of 90%. Compared with single text analysis (misjudged as "positive") or single voice analysis (misjudged as "contradictory"), the accuracy is significantly improved.

[0085] According to a specific implementation manner of the embodiment of the application, the multilingual and dialect adaptation processing is realized by a multilingual and dialect processing module, which adopts a multilingual speech recognition model based on multi-task learning; the multilingual speech recognition model realizes unified processing of multilingual speech data by sharing a bottom feature extractor; and a language-specific adaptive layer is introduced, which dynamically adjusts the recognition strategy of the model when the system recognizes speech data of different language types; meanwhile, a cross-language data enhancement technology is used to expand the training data based on the similarity between languages, so as to improve the cross-language recognition effect; for local dialect processing, a dialect adaptation model based on feature transfer learning is configured in the module, the model is pre-trained by using standard language corpus, and then the pre-trained model is fine-tuned by using a small amount of dialect corpus, so as to realize efficient dialect recognition.

[0086] Specifically, in terms of multi-language processing, mixed Chinese-English speech is processed ("The new function is very useful"), the multi-language model shares an 8-layer Transformer bottom-layer feature extractor to learn the general phoneme features of "function" (Chinese) and "new" (English); a language adaptation layer identifies the language type through a FastText model (training data contains 100,000 multi-language sentences), the Chinese part calls a Pinyin mapping module, the English part calls a letter pronunciation module, and the transcription result is "The new function is very useful", the mixed language recognition accuracy is 94%. Cross-language data augmentation is performed on Japanese and Korean (similar grammar structure), 50,000 Japanese speech is converted into Korean homologous speech through phoneme mapping, Korean training data is expanded by 30%, and Korean recognition accuracy is improved from 85% to 91%. Dialect processing is performed on Sichuan dialect, a model is pre-trained using 100 hours of Mandarin corpus, and then 5 hours of Sichuan dialect corpus (containing dialect words such as "Ba Sui" and "Yao De") is input for fine-tuning, after fine-tuning, the Sichuan dialect recognition accuracy is 88%, compared with direct training (requiring 50 hours of dialect corpus), the data amount requirement is reduced by 90%, and the training efficiency is improved by 8 times.

[0087] According to a specific implementation manner of the embodiment of the application, the interactive billboard adopts a dynamic visualization updating technology, a hot topic graph is dynamically adjusted in association strength and position distribution according to newly acquired speech analysis results through a real-time association analysis algorithm, and a sentiment trend analysis chart combines a time sequence prediction model to display a historical sentiment change trend and predict and mark a future short-time sentiment trend.

[0088] Specifically, the hot topic graph is directed at the live speech of a product launch, and the real-time correlation analysis algorithm counts the frequency of topic mentions (such as "battery life" 200 times, "fast charging" 150 times, and "price" 120 times) every 30 seconds. The size of the node is set in proportion to the frequency (the "battery life" node has a diameter of 20px, the "fast charging" node has a diameter of 15px, and the "price" node has a diameter of 12px). The edge weight is calculated according to the co-occurrence times (the "battery life" and "fast charging" co-occur 80 times, and the edge width is 8px). When 100 new speeches containing "heat dissipation" are collected, the system automatically adds a "heat dissipation" node (with a diameter of 10px) and connects the "battery life" node according to the co-occurrence relationship (with an edge width of 5px). The graph dynamically updates in response time < 1 second. The sentiment trend analysis is directed at user comments within 24 hours. Through the ARIMA time series prediction model, based on the sentiment trend in the past 6 hours (the positive proportion increased from 40% to 60%), it is predicted that the positive proportion in the next 2 hours will remain at 58%-62%. The prediction interval is marked with a dashed line in the line chart, and the prediction error rate is 8%, helping enterprises to predict the trend of user attitudes in advance.

[0089] According to a specific implementation manner of the embodiment of the application, a multi-level threshold early warning mechanism is adopted for monitoring real-time speech data, a basic emotion intensity threshold and an emotion diffusion speed threshold are set for abnormal emotions, a topic mention frequency threshold and a topic propagation range threshold are set for sensitive topics, when the monitoring data triggers any threshold, the system first excludes false detection through an abnormal data verification algorithm, and then generates a corresponding level of alarm and analysis report according to the threshold trigger level, wherein the high-level alarm is accompanied by real-time speech segment positioning and key emotion / topic labeling.

[0090] Specifically, the multi-level threshold is set as follows: abnormal emotion (negative intensity > 0.8 as a basic threshold, and 10 times growth of negative mention quantity within 10 minutes as a diffusion threshold), sensitive topic (frequency threshold of “quality defect” mention > 50 times per minute, and propagation threshold of covering more than 3 platforms). The system monitors the “product leakage” topic in a certain period of time, and the number of mentions increases from 10 times per minute to 60 times per minute within 10 minutes (triggering the frequency threshold), and covers live broadcast, short video and social chat three platforms (triggering the propagation threshold). The abnormal data verification algorithm uses an isolation forest model, inputs 200 voice data related to the topic, and detects 15 mislabeled data (such as “product charging” misjudged as “product leakage”). After excluding the misjudged data, it is confirmed that the effective trigger is triggered. The system generates a high-level red alert, with voice segment positioning (accurate to the 12th second “this product has a risk of leakage”), labels the key topic “product leakage” and the emotion “panic”, and generates an analysis report, including the topic propagation path (from the short video platform to the social chat platform), the user portrait (70% of users aged 25-35), the alert generation and report output time < 3 seconds, helping enterprises to start crisis response within 10 minutes.

[0091] Referring to Figure 3 The application further discloses a social listening device based on voice recognition, which comprises:

[0092] A collection and preprocessing module is configured to collect voice content from a social media platform in real time, and to preprocess the collected voice content to obtain standardized voice data. The voice content includes podcasts, live audio, short video audio tracks and social chat voice. The preprocessing includes format conversion, noise filtering and audio slicing.

[0093] A voice-to-text module is configured to process the standardized voice data by using a deep learning voice recognition model, and to accurately convert the standardized voice data into text data.

[0094] A multi-dimensional analysis module is configured to perform multi-dimensional analysis on the standardized voice data and the text data to obtain analysis results. The multi-dimensional analysis includes sentiment and emotion analysis, multi-language and dialect adaptation processing.

[0095] A visualization and monitoring alarm module is configured to present the multi-dimensional analysis results through an interactive dashboard, and to monitor real-time voice data. When an abnormal emotion or a sensitive topic is found, an alarm is triggered and an analysis report is generated.

[0096] Referring to Figure 4 The electronic device 60 comprises:

[0097] at least one processor; and

[0098] a memory in communication with the at least one processor; wherein

[0099] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the social listening method based on voice recognition in the foregoing method embodiments.

[0100] The embodiment of the present application also provides a non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method in the foregoing method embodiments.

[0101] The embodiment of the present application also provides a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, cause the computer to perform the social listening method based on voice recognition in the foregoing method embodiments.

[0102] As shown in Figure 4 The electronic device 60 can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601 that can perform various appropriate actions and processes in accordance with programs stored in a read-only memory (ROM) 602 or loaded into a random access memory (RAM) 603 from a storage device 608. Various programs and data required for the operation of the electronic device 60 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0103] In general, the following devices can be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, a gyro sensor, a geomagnetic sensor, and the like; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; storage devices 608 including, for example, a magnetic tape, a hard disk, and the like; and communication devices 609. The communication devices 609 can allow the electronic device 60 to perform wireless or wired communication with other devices to exchange data. Although Figure 4 The electronic device 60 is shown with various devices, but it should be understood that all of the shown devices need not be implemented or present. More or fewer devices can alternatively be implemented or present.

[0104] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program in accordance with embodiments of the application. For example, embodiments of the application include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the methods of the embodiments of the application are performed.

[0105] The above description is merely illustrative of the application and is not intended to limit the scope of the application. Any changes and modifications that can be made to the application in accordance with the principle of the application should be encompassed within the scope of the application. The scope of the application should be determined by the appended claims.

Claims

1. A voice recognition based social listening method, characterized by, Includes the following steps: Voice content is collected in real time from social media platforms. The collected voice content is preprocessed to obtain standardized voice data. The voice content includes podcasts, live audio, short video audio tracks and social chat voices. The preprocessing includes format unification conversion, noise filtering and audio slicing. The standardized speech data is processed using a deep learning speech recognition model to accurately convert it into text data. The standardized speech data and text data are subjected to multi-dimensional analysis to obtain analysis results. The multi-dimensional analysis includes sentiment and emotion analysis, multilingual and dialect adaptation processing. The multi-dimensional analysis results are presented through an interactive dashboard, while real-time voice data is monitored. When abnormal emotions or sensitive topics are detected, alarms are triggered and analysis reports are generated.

2. The social listening method based on speech recognition according to claim 1, characterized in that, In the process of collecting voice content in real time from social media platforms, multi-source voice data access is achieved through dynamic interface adaptation technology. This technology can automatically adjust data request parameters and data parsing rules according to the differences in API protocols of different social media platforms. At the same time, a dynamic traffic allocation mechanism is used to avoid data interruption caused by platform interface rate limiting during the collection process.

3. The social listening method based on speech recognition according to claim 1, characterized in that, In the preprocessing step of the acquired speech content, the format unification and conversion stage adopts an adaptive format parsing algorithm, which can automatically identify the encoding characteristics of various audio formats and generate adaptation conversion rules. Audio slicing processing, combined with speech semantic pause detection technology, determines the slice location by identifying natural speech break nodes in the speech.

4. The social listening method based on speech recognition according to claim 1, characterized in that, The deep learning speech recognition model adopts an end-to-end speech recognition model based on the Transformer architecture. After inputting the spectral features of standardized speech data into the model, the speech time series information is modeled through an attention mechanism. The model employs a hierarchical decoding strategy, dividing long speech into segments according to preset time intervals and transcribing each segment separately. Then, the transcription results from each segment are integrated through a context encoder to ensure semantic coherence; Meanwhile, the model introduces a noise suppression model based on variational autoencoder to extract clear speech signals in various complex background noise environments; Furthermore, it is equipped with a context-aware speech recognition module, which combines the user's current interaction scenario, timestamp, and relevant semantic context to improve the transcription accuracy of ambiguous words.

5. The social listening method based on speech recognition according to claim 1, characterized in that, The sentiment and emotion analysis employs multimodal sentiment analysis technology, wherein sentiment analysis is implemented through a Transformer-based sentiment classification model; This sentiment classification model supports basic sentiment labels of positive, negative, and neutral, and can also be extended to personalized sentiment labels of anger, support, and satisfaction according to specific application scenarios; Furthermore, it captures the complex emotional layers in user expressions through emotional feature embedding networks, achieving multi-level emotion extraction; Sentiment analysis is achieved through a combination model of convolutional neural networks and recurrent neural networks. Convolutional neural networks are used to extract the pitch, speech rate and energy acoustic features of speech, while recurrent neural networks are used to model the time-series correlation of acoustic features. Finally, by combining the text sentiment results obtained from sentiment analysis with the acoustic feature analysis results obtained from emotion analysis, emotion classification is achieved through a multimodal fusion network.

6. The social listening method based on speech recognition according to claim 1, characterized in that, The multilingual and dialect adaptation processing is implemented through a multilingual and dialect processing module, which adopts a multilingual speech recognition model based on multi-task learning. This multilingual speech recognition model achieves unified processing of multilingual speech data by sharing a low-level feature extractor; In addition, a language-specific adaptive layer is introduced to dynamically adjust the model's recognition strategy when the system recognizes speech data of different language types; At the same time, cross-language data augmentation technology is used to expand the training data based on the similarity between languages, thereby improving the cross-language recognition effect; For local dialect processing, the module is configured with a dialect adaptation model based on feature transfer learning. The model is first pre-trained using standard language corpus, and then fine-tuned using a small amount of dialect corpus to achieve efficient dialect recognition.

7. The social listening method based on speech recognition according to claim 1, characterized in that, The interactive dashboard uses dynamic visualization update technology, and the hot topic map dynamically adjusts the association strength and position distribution of topic nodes based on newly acquired voice analysis results through real-time correlation analysis algorithm. Sentiment trend analysis charts, combined with time series forecasting models, not only display historical sentiment trends but also predict and annotate sentiment trends in the near future.

8. The social listening method based on speech recognition according to claim 1, characterized in that, The monitoring of real-time voice data adopts a multi-level threshold early warning mechanism, setting a basic emotion intensity threshold and an emotion diffusion speed threshold for abnormal emotions; Set thresholds for the frequency of mention and the scope of dissemination of sensitive topics; When the monitored data triggers any threshold, the system first uses an abnormal data verification algorithm to eliminate false detections. Then, based on the threshold trigger level, corresponding alerts and analysis reports are generated, with high-level alerts accompanied by real-time voice segment localization and key emotion / topic annotations.

9. A social listening device based on speech recognition, characterized in that, include: The acquisition and preprocessing module is used to acquire voice content from social media platforms in real time, and preprocess the acquired voice content to obtain standardized voice data. The voice content includes podcasts, live audio, short video audio tracks and social chat voice. The preprocessing includes format unification conversion, noise filtering and audio slicing. The speech-to-text module is used to process the standardized speech data using a deep learning speech recognition model and accurately convert it into text data. The multi-dimensional analysis module is used to perform multi-dimensional analysis on the standardized speech data and text data to obtain analysis results. The multi-dimensional analysis includes sentiment and emotion analysis, multilingual and dialect adaptation processing. The visualization and monitoring alarm module is used to present the multi-dimensional analysis results through an interactive dashboard, while monitoring real-time voice data, triggering alarms and generating analysis reports when abnormal emotions or sensitive topics are detected.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

Images