Intelligent warning method, device, equipment and storage medium based on voice detection

Abstract

The present application provides an intelligent warning method, device, equipment and storage medium based on voice detection, the method comprising: processing voice data to obtain a voice spectrum graph and text data; inputting the voice spectrum graph and text data into a preset natural language model to obtain emotion description information, the emotion description information comprising: related objects, emotion nouns and intensity adverbs; determining a target object among the related objects, and setting the emotion noun corresponding to the target object as the first noun; setting the first noun successfully matched in a preset noun database as the second noun, and setting the intensity adverb corresponding to the second noun as the target adverb; generating an emotion scoring curve according to the second noun, the target adverb and a preset mapping rule, determining the emotion fluctuation amplitude and the emotion fluctuation frequency according to the emotion scoring curve; determining the emotion fluctuation score according to the emotion fluctuation amplitude and the emotion fluctuation frequency, and issuing a warning according to the emotion fluctuation score.

Description

Technical Field

[0001] The present application relates to the field of speech analysis technology, and in particular to an intelligent warning method, device, equipment and storage medium based on speech detection. Background Art

[0002] In recent years, with the rapid development of artificial intelligence (AI) technology, AI alarm systems based on voice detection, as an important innovation in the field of intelligent security, are gradually being applied to scenarios such as schools and public places, in order to detect and intervene in potential security threats, such as school bullying, as early as possible. These systems usually use advanced audio analysis technology and natural language processing algorithms to monitor environmental sounds in real time, identify specific keywords or intonation patterns, and automatically trigger alarm mechanisms once possible emergencies are detected. However, despite the great potential of AI alarm systems in improving public safety, they still face many challenges in practical applications. One of the main problems is the high frequency of false positives and false negatives. This is because most current systems rely too much on keyword recognition and lack a deep understanding of the context. In complex and changing real environments, simple keyword matching often cannot accurately distinguish between normal conversations and real crisis signals. For example, in a campus scenario, words such as "fighting" and "bullying" may appear in a variety of harmless situations, causing the system to mistakenly trigger an alarm. On the other hand, some threatening words or obscure expressions that are not preset as keywords may be ignored by the system, resulting in false negatives. Summary of the invention

[0003] The present application provides a method, device, equipment and storage medium for intelligent warning based on voice detection.

[0004] In a first aspect, an embodiment of the present application provides an intelligent warning method based on voice detection, the method comprising:

[0005] Performing spectral processing on the speech data in the detection scene to obtain a speech spectrogram, and performing text extraction on the speech data to obtain text data;

[0006] Inputting the speech spectrogram and the text data into a preset natural language model to obtain emotion description information, wherein the emotion description information includes: related objects, emotion nouns and intensity adverbs, wherein the intensity adverbs are used to indicate the emotion intensity of the emotion nouns;

[0007] Calculating the occurrence frequency of the related objects within a preset time, determining a target object from the related objects according to the occurrence frequency, and setting the emotion noun corresponding to the target object as the first noun;

[0008] According to the matching of the first noun in a preset noun database, the first noun that is successfully matched is set as a second noun, and the intensity adverb corresponding to the second noun is set as a target adverb;

[0009] Generate a sentiment scoring curve according to the second noun, the target adverb and a preset mapping rule, and determine the sentiment fluctuation amplitude and the sentiment fluctuation frequency according to the sentiment scoring curve;

[0010] An emotion fluctuation score is determined according to the emotion fluctuation amplitude and the emotion fluctuation frequency, and when it is detected that the emotion fluctuation score is greater than a preset score threshold, warning information is generated and output according to the target object, the second noun and the target adverb.

[0011] In a second aspect, an embodiment of the present application provides an intelligent warning device based on voice detection, the device comprising:

[0012] A data processing module is used to perform spectral processing on the voice data in the detection scene to obtain a voice spectrum graph, and to perform text extraction on the voice data to obtain text data;

[0013] A sentiment analysis module, used for inputting the speech spectrogram and the text data into a preset natural language model to obtain sentiment description information, wherein the sentiment description information includes: related objects, sentiment nouns and intensity adverbs, wherein the intensity adverbs are used to indicate the sentiment intensity of the sentiment nouns;

[0014] A word frequency analysis module, used to calculate the frequency of occurrence of the related objects within a preset time, determine the target object from the related objects according to the frequency of occurrence, and set the emotional noun corresponding to the target object as the first noun;

[0015] a word matching module, configured to match the first noun in a preset noun database, set the first noun that successfully matches as a second noun, and set the intensity adverb corresponding to the second noun as a target adverb;

[0016] A data conversion module, used to generate a sentiment scoring curve according to the second noun, the target adverb and a preset mapping rule, and determine the sentiment fluctuation amplitude and the sentiment fluctuation frequency according to the sentiment scoring curve;

[0017] A result output module is used to determine an emotion fluctuation score according to the emotion fluctuation amplitude and the emotion fluctuation frequency, and when it is detected that the emotion fluctuation score is greater than a preset score threshold, generate and output warning information according to the target object, the second noun and the target adverb.

[0018] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor;

[0019] The memory is used to store computer programs;

[0020] The processor is used to execute the computer program and implement the intelligent warning method based on voice detection as described in any one of the embodiments of the present application when executing the computer program.

[0021] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the intelligent warning method based on voice detection as described in any one of the embodiments of the present application.

[0022] The embodiment of the present application provides an intelligent warning method based on voice detection, the method comprising: performing spectral processing on voice data in a detection scene to obtain a voice spectrum graph, performing text extraction on the voice data to obtain text data; inputting the voice spectrum graph and the text data into a preset natural language model to obtain emotion description information, the emotion description information comprising: related objects, emotion nouns and intensity adverbs, wherein the intensity adverbs are used to indicate the emotion intensity of the emotion nouns; calculating the occurrence frequency of the related objects within a preset time, determining the target object in the related objects according to the occurrence frequency, and setting the emotion noun corresponding to the target object as the first noun; matching the first noun in a preset noun database, setting the successfully matched first noun as the second noun, and setting the intensity adverb corresponding to the second noun as the target adverb; generating an emotion scoring curve according to the second noun, the target adverb and a preset mapping rule, and determining the emotion fluctuation amplitude and the emotion fluctuation frequency according to the emotion scoring curve; determining the emotion fluctuation score according to the emotion fluctuation amplitude and the emotion fluctuation frequency, and generating and outputting warning information according to the target object, the second noun and the target adverb when it is detected that the emotion fluctuation score is greater than a preset score threshold. Through the above method, the speech spectrum graph and text data are simultaneously analyzed through the natural language model, the two data are mutually verified, and the emotional description information in the form of natural language is output, which can avoid the limitations of labeled description. Based on the emotional description information, it is detected whether there are specific target-related objects that appear frequently in the emotional description information, and the final emotional fluctuation score is determined according to the second noun and target adverb corresponding to the relevant object. If the emotional fluctuation score is greater than the preset score threshold, it means that there may be a language attack against the target-related object, and a warning message needs to be output to prompt relevant personnel to intervene and deal with it, so as to warn of possible risk events in advance, thereby improving the perception and intervention capabilities of risk events. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0024] Figure 1 A schematic flow chart of an intelligent warning method based on voice detection provided in an embodiment of the present application;

[0025] Figure 2 A schematic block diagram of an intelligent warning device based on voice detection provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0028] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0029] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0030] See also Figure 1 , Figure 1 The embodiment of the present application also provides a schematic flow chart of an intelligent warning method based on voice detection. Figure 1 As shown, the specific steps of the intelligent warning method based on voice detection include: S101-S106.

[0031] S101, performing spectral processing on the speech data in the detection scene to obtain a speech spectrum graph, and performing text extraction on the speech data to obtain text data.

[0032] Exemplarily, the voice data in the scene to be detected is raw data, and there is noise such as ambient sound, so denoising is required. Use window functions such as Hamming window to divide the denoised voice data into a series of overlapping short time frames. Apply fast Fourier transform (FFT) to the signal in each frame, convert it from time domain to frequency domain, and obtain the spectrum of the frame. Calculate the amplitude or power of the spectrum, and usually take the logarithm to enhance the contrast. Arrange these spectra to form a two-dimensional image, in which the horizontal axis represents time, the vertical axis represents frequency, and the color or grayscale represents intensity, so that a spectrogram is obtained. The voice data is transcribed through natural language models such as GPT to obtain text data.

[0033] S102. Input the speech spectrogram and text data into a preset natural language model to obtain emotion description information, which includes: related objects, emotion nouns and intensity adverbs, wherein the intensity adverbs are used to indicate the emotion intensity of the emotion nouns.

[0034] For example, conventional methods divide emotions into a series of discrete labels, such as joy, sarcasm, anger, frustration and anger, and then score them by intensity values. This usually cannot warn of hidden language threats, such as calmly expressing sarcasm and loudly expressing joy. These conflicting information often cause the model to misclassify emotions. Therefore, it is necessary to input emotion description information based on voice data, which provides broader emotion information and more reasonably judges the combination and intensity of emotions.

[0035] Related objects can often be nouns such as names of people and flowers. Emotional nouns include frustrated, sarcastic, indignant, frustrated, and angry. Intensity adverbs include: calmly, sometimes calm and sometimes angry, high-pitched, and nervously.

[0036] By utilizing the powerful language understanding capabilities of natural language models and combining them with speech spectrograms, we can accurately extract the speaker's emotional information and obtain the "implied meaning" in text data.

[0037] S103, calculating the occurrence frequency of related objects within a preset time, determining a target object among the related objects according to the occurrence frequency, and setting the emotion noun corresponding to the target object as the first noun.

[0038] Exemplarily, the hardware devices for acquiring voice data are installed in places such as toilets, corridors, and stairwells. For school bullying, these scenes are often blind spots for monitoring, or there is a need to avoid privacy. By detecting the voice data in these scenes, in addition to emergency situations, it is possible to directly alarm, for example, by detecting keywords such as "help" and "beating", or detecting slapping sounds, etc., it is also possible to prevent school bullying that is still brewing in advance, such as long-term verbal bullying. If verbal bullying occurs, there will be a specific related object, and this related object will appear in the discussion at a high frequency in a short period of time. Therefore, after the target object is detected at a high frequency, it is necessary to determine whether there is a language threat against the target object.

[0039] It should be noted that the installation and use of the above-mentioned hardware devices must be permitted by laws and regulations, and the security of personal data and personal privacy must be ensured during use.

[0040] S104, matching the first noun in a preset noun database, setting the successfully matched first noun as the second noun, and setting the intensity adverb corresponding to the second noun as the target adverb.

[0041] Exemplarily, the noun database contains a variety of noun combinations related to bullying. If the matching degree between the first noun of the target object and the above noun combinations reaches a preset matching threshold, for example, 80%, it is considered that the first noun is matched successfully.

[0042] S105 . Generate a sentiment scoring curve according to the second noun, the target adverb and a preset mapping rule, and determine the sentiment fluctuation amplitude and the sentiment fluctuation frequency according to the sentiment scoring curve.

[0043] Exemplarily, the preset mapping rule requires mapping the second noun mapping and the corresponding target adverb to coordinates on the coordinate system. During the mapping process, it is necessary to define a set of emotional nouns, including positive and negative words. For example, "happy" may have a higher positive intensity value, and "sad" may have a higher negative intensity value. During the mapping process, a set of emotional noun combinations can also be defined, and the positive words associated with high negative emotional nouns will also be mapped to negative intensity values. By arranging these scores in the order in which they appear in the text, an emotional score curve can be drawn. Then, by observing the highest and lowest points of the curve, the amplitude of emotional fluctuations can be calculated; by counting the number of transitions of the curve from high points to low points or from low points to high points, the frequency of emotional fluctuations can be determined.

[0044] S106, determining an emotion fluctuation score according to the emotion fluctuation amplitude and the emotion fluctuation frequency, and when detecting that the emotion fluctuation score is greater than a preset score threshold, generating and outputting a warning message according to the target object, the second noun and the target adverb.

[0045] For example, some offensive emotional nouns may cause the score curve to fluctuate widely, or cause emotional fluctuations to occur frequently, which may lead to an increase in the emotional fluctuation score. Therefore, when it is detected that the emotional fluctuation score is greater than the preset score threshold, a warning message needs to be output for the target object. The warning message needs to include the target object, as well as the overall language threat assessment information generated based on the second noun and the target adverb.

[0046] The embodiment of the present application provides an intelligent warning method based on voice detection, the method comprising: performing spectral processing on voice data in a detection scene to obtain a voice spectrum graph, performing text extraction on the voice data to obtain text data; inputting the voice spectrum graph and the text data into a preset natural language model to obtain emotion description information, the emotion description information comprising: related objects, emotion nouns and intensity adverbs, wherein the intensity adverbs are used to indicate the emotion intensity of the emotion nouns; calculating the occurrence frequency of the related objects within a preset time, determining the target object in the related objects according to the occurrence frequency, and setting the emotion noun corresponding to the target object as the first noun; matching the first noun in a preset noun database, setting the successfully matched first noun as the second noun, and setting the intensity adverb corresponding to the second noun as the target adverb; generating an emotion scoring curve according to the second noun, the target adverb and a preset mapping rule, and determining the emotion fluctuation amplitude and the emotion fluctuation frequency according to the emotion scoring curve; determining the emotion fluctuation score according to the emotion fluctuation amplitude and the emotion fluctuation frequency, and generating and outputting warning information according to the target object, the second noun and the target adverb when it is detected that the emotion fluctuation score is greater than a preset score threshold. Through the above method, the speech spectrum graph and text data are simultaneously analyzed through the natural language model, the two data are mutually verified, and the emotional description information in the form of natural language is output, which can avoid the limitations of labeled description. Based on the emotional description information, it is detected whether there are specific target-related objects that appear frequently in the emotional description information, and the final emotional fluctuation score is determined according to the second noun and target adverb corresponding to the relevant object. If the emotional fluctuation score is greater than the preset score threshold, it means that there may be a language attack against the target-related object, and a warning message needs to be output to prompt relevant personnel to intervene and deal with it, so as to warn of possible risk events in advance, thereby improving the perception and intervention capabilities of risk events.

[0047] In order to more clearly introduce the technical solution of the present application, the technical solution of the present application will be introduced through specific embodiments below. It should be noted that the specific embodiments are used to expand the technical solution of the present application, but are not intended to limit the present application.

[0048] In some embodiments, the preset natural language model includes an audio encoder, a feature extractor and a text decoder, and the feature extractor includes a self-attention layer, a cross-attention layer and a linear layer. The preset natural language model is trained.

[0049] When the speech spectrogram and text data are input into the preset natural language model through the preset natural language model to obtain the emotion description information, the package is specifically used to implement the following steps: S201-S207.

[0050] S201. Based on the audio encoder, extract features from the speech spectrogram to obtain a high-dimensional feature matrix.

[0051] Exemplarily, after the input speech spectrogram and the corresponding text data are processed, the speech spectrogram is preliminarily processed to extract the basic features of the original speech. This process uses the audio encoder part of the pre-trained speech feature extraction model to convert the input speech spectrogram into a frame-level feature representation. These frame-level features contain the frequency and time domain information of the speech signal as the basis for subsequent processing. Through preliminary processing, a preliminary high-dimensional feature matrix is ​​generated, which will serve as the input for the next self-attention layer processing.

[0052] S202: Input the high-dimensional feature matrix into the feature extractor, perform weighted summation on the high-dimensional feature matrix through the self-attention layer, and obtain a compressed feature matrix.

[0053] Exemplarily, after obtaining the preliminary high-dimensional feature matrix, the self-attention layer calculates the query, key, and value of the high-dimensional feature matrix, and uses Q-queries as learnable parameters to enhance the focus on emotion-related features. The self-attention layer generates an attention weight matrix by calculating the dot product of the query vector and the frame-level features, and performs weighted summation on the high-dimensional feature matrix according to the attention weight matrix to obtain a compressed feature matrix, thereby screening out the most important features. This process effectively compresses the input high-dimensional feature matrix, reduces redundant information, and generates a compressed feature vector matrix, ready to enter the cross-attention layer for further processing.

[0054] S203. Through the cross attention layer, according to the preset query vector, the compressed feature matrix and the high-dimensional feature matrix are cross-attention calculated to obtain an enhanced feature matrix.

[0055] After the self-attention layer generates a compressed feature matrix, the cross-attention layer uses the compressed feature matrix to further strengthen the features related to emotions. Specifically, by using the initial Q-queries as the query vector, the correlation between the high-dimensional feature matrix and the compressed feature matrix is ​​calculated, and the features most closely related to emotional expression are accurately extracted to obtain the enhanced feature matrix, and the representation of these features in the output matrix is ​​strengthened. Through the processing of the cross-attention layer, the system generates an emotionally enhanced feature vector matrix with higher emotional expression capabilities.

[0056] S204, projecting the enhanced feature matrix to a preset text representation space through a linear layer, and performing activation or function processing to obtain a projected feature matrix.

[0057] Exemplarily, the enhanced feature matrix after sentiment enhancement is passed to the linear layer for feature projection and transformation. The main task of the linear layer is to project the enhanced feature matrix into the target text representation space so that it can be accepted and processed by the text encoder. The linear layer first performs matrix multiplication and addition operations on the feature vector through a linear transformation function to compress its dimension to the size of the target space. Subsequently, the linear layer also activates the projected feature vector to enhance the expressiveness and discrimination of the feature. Finally, the linear layer generates a feature vector matrix projected into the text representation space to obtain a projected feature matrix, which will be optimized in subsequent steps.

[0058] S205, projecting the text data into a preset text representation space to obtain a text projection matrix, performing multimodal comparative learning on the text projection matrix and the projection feature matrix to obtain an optimized projection feature matrix.

[0059] After completing feature projection and conversion, the linear layer enters the feature optimization stage of multimodal contrastive learning. In this step, the linear layer optimizes the projected feature matrix by combining the input text data with the emotional description text using multimodal contrastive learning technology. Specifically, the linear layer optimizes the similarity between the emotional feature vector and the text data in the same representation space through contrastive learning methods to ensure that speech samples with similar emotions are closer in the feature space, while samples with large emotional differences are farther apart. Through this process, the system finally obtains an optimized projected feature matrix with higher consistency and accuracy in emotional expression.

[0060] S206. According to a preset loss function, the upper bound value of the mutual information between the optimized projection feature matrix and the text projection matrix is ​​calculated, and the optimized projection feature matrix whose upper bound value of the mutual information is lower than the preset upper bound threshold is set as the target projection feature matrix.

[0061] In order to strengthen the association between speech content and emotional features, the system introduces speech-text mutual information minimization processing in this step. Specifically, the system uses the input text data and the optimized emotional feature vector to evaluate the correlation between the two through the mutual information estimation method. To avoid directly calculating the mutual information of high-dimensional features, the system uses the vCLUB method to estimate the upper bound of the mutual information and optimizes it as a loss function. Through this process, the system gradually reduces the correlation between the emotional feature vector and the speech content, thereby obtaining a feature vector matrix that has been processed with mutual information maximization, strengthening the association between speech content and emotional features.

[0062] S207, generating a speech feature embedding vector according to the target projection feature, and inputting the speech feature embedding vector into a text decoder to obtain emotion description information.

[0063] After completing the mutual information minimization process, the system performs final processing on the optimized sentiment feature vector matrix to generate sentiment-related feature vectors for downstream text generation. This step ensures that the output sentiment feature vectors can adapt to the input requirements of the downstream text decoder through further post-processing operations such as normalization and feature enhancement, and provides a basis for generating accurate speech sentiment description text. These final generated sentiment feature vectors can be directly input into the text encoder to support the automatic speech sentiment description generation task.

[0064] In some embodiments, a sentiment scoring curve is generated according to the second noun, the target adverb and a preset scoring rule, and the sentiment fluctuation amplitude and the sentiment fluctuation frequency are determined according to the sentiment scoring curve, including: according to the preset mapping rule, sentiment intensity mapping processing is performed on the second noun and the target adverb to obtain an initial sentiment intensity value; according to the preset time weighting coefficient, the initial sentiment intensity value is time-weighted to obtain a weighted sentiment intensity value; the weighted sentiment intensity value is smoothed using a smoothing algorithm to obtain a smoothed sentiment intensity curve; according to the preset peak detection algorithm, the smoothed sentiment intensity curve is peak-detected to obtain a sentiment fluctuation peak; the sentiment fluctuation peak is amplitude-calculated to obtain the sentiment fluctuation amplitude; the time interval between the sentiment fluctuation amplitude and the sentiment fluctuation peak is frequency-calculated to obtain the sentiment fluctuation frequency.

[0065] The above method generates a sentiment scoring curve by mapping the sentiment intensity and performing time weighting processing on sentiment-related nouns and adverbs, and combines the smoothing algorithm and the peak detection algorithm to accurately capture the amplitude and frequency of sentiment fluctuations. This processing method can effectively eliminate noise, smooth sentiment data, and improve the detection accuracy of sentiment fluctuations, thereby providing more delicate sentiment analysis results. By calculating the amplitude and frequency of sentiment fluctuations, it can more accurately reflect the changing trend of sentiment, providing strong support for sentiment analysis and sentiment-related applications.

[0066] In some embodiments, an emotion fluctuation score is determined based on the emotion fluctuation amplitude and the emotion fluctuation frequency, including: normalizing the emotion fluctuation amplitude and the emotion fluctuation frequency respectively to obtain a normalized emotion fluctuation amplitude value and a normalized emotion fluctuation frequency value; weighting the normalized emotion fluctuation amplitude value and the normalized emotion fluctuation frequency value according to a preset weighting coefficient to obtain a weighted emotion fluctuation value; scoring the weighted emotion fluctuation value according to a preset scoring rule to obtain an emotion fluctuation score.

[0067] The above method normalizes the amplitude and frequency of emotional fluctuations to ensure that different emotional fluctuation characteristics are compared and analyzed on the same scale, and comprehensively considers the relative importance of the amplitude and frequency of emotional fluctuations through weighted processing to generate a weighted emotional fluctuation value. Subsequently, the weighted value is scored according to the preset scoring rules to generate the final emotional fluctuation score. This process enhances the representativeness and reliability of emotional fluctuation characteristics, makes the emotional fluctuation score more accurate and objective, helps to more accurately reflect the overall intensity of emotional changes, and provides a more scientific and effective basis for emotional evaluation and emotional analysis.

[0068] In some embodiments, the weighted emotional fluctuation value is scored according to a preset scoring rule to obtain an emotional fluctuation score, including: using a sliding window technology to divide the weighted emotional fluctuation value into multiple overlapping time windows, each time window including multiple weighted emotional fluctuation values; calculating the weighted emotional fluctuation value in each time window to obtain an average emotional fluctuation value corresponding to each time window; performing difference calculation on the average emotional fluctuation values ​​of two adjacent time windows to obtain an emotional fluctuation change value; performing nonlinear mapping processing on the average emotional fluctuation value to obtain an emotional fluctuation intensity score, and performing nonlinear mapping processing on the emotional fluctuation change value to obtain an emotional fluctuation change rate score; performing weighted averaging processing on the emotional fluctuation intensity score and the emotional fluctuation change rate score to obtain an emotional fluctuation score for each time window.

[0069] The above method divides the weighted emotional fluctuation value sequence into multiple overlapping time windows through the sliding window technology, and performs weighted processing and normalization calculation on the data in each window, effectively capturing the local dynamic changes of emotional fluctuations. By applying weighted functions and normalization processing, it ensures that the emotional fluctuation values ​​in different time windows are analyzed on the same scale, which improves the expression accuracy of emotional data. At the same time, the overlapping design of sliding windows enhances the sensitivity to continuous emotional changes, allowing the system to capture subtle changes in emotional fluctuations more finely, thereby providing a solid data foundation for subsequent emotional fluctuation analysis.

[0070] In some embodiments, warning information is generated and output based on the target object, the second noun and the target adverb, including: performing data combination processing on the target object, the second noun and the target adverb to obtain a warning information template; performing content filling processing on the warning information template to obtain warning information content; performing formatting processing on the warning information content to obtain warning information, and outputting the warning information through a preset warning device.

[0071] See also Figure 2 , Figure 2 The embodiment of the present application also provides a schematic block diagram of an intelligent warning device based on voice detection, and the intelligent warning device based on voice detection 200 is used to execute the aforementioned intelligent warning method based on voice detection. The intelligent warning device based on voice detection 200 can be configured in a server.

[0072] Among them, the server can be an independent server or a server cluster, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN) and big data and artificial intelligence platforms.

[0073] like Figure 2 As shown, the intelligent warning device 200 based on voice detection includes: a data processing module 201, a sentiment analysis module 202, a word frequency analysis module 203, a word matching module 204, a data conversion module 205 and a result output module 206.

[0074] The data processing module 201 is used to perform spectral processing on the speech data in the detection scene to obtain a speech spectrum graph, and perform text extraction on the speech data to obtain text data.

[0075] The sentiment analysis module 202 is used to input the speech spectrogram and text data into a preset natural language model to obtain sentiment description information, which includes: related objects, sentiment nouns and intensity adverbs, wherein the intensity adverbs are used to indicate the sentiment intensity of the sentiment nouns.

[0076] In some embodiments, the preset natural language model includes an audio encoder, a feature extractor and a text decoder, the feature extractor includes a self-attention layer, a cross-attention layer and a linear layer, and the sentiment analysis module 202 is used to input the speech spectrogram and text data into the preset natural language model to obtain sentiment description information, and is specifically used to implement: based on the audio encoder, feature extraction is performed on the speech spectrogram to obtain a high-dimensional feature matrix; the high-dimensional feature matrix is ​​input into the feature extractor, and the high-dimensional feature matrix is ​​weighted summed through the self-attention layer to obtain a compressed feature matrix; through the cross-attention layer, according to the preset query vector, the compressed feature matrix and the high-dimensional feature matrix are cross-attention calculated to obtain to the enhanced feature matrix; through the linear layer, the enhanced feature matrix is ​​projected to the preset text representation space, and activation or function processing is performed to obtain the projection feature matrix; the text data is projected to the preset text representation space to obtain the text projection matrix, and multimodal comparative learning is performed on the text projection matrix and the projection feature matrix to obtain the optimized projection feature matrix; according to the preset loss function, the upper bound value of the mutual information of the optimized projection feature matrix and the text projection matrix is ​​calculated, and the optimized projection feature matrix whose upper bound value of the mutual information is lower than the preset upper bound threshold is set as the target projection feature matrix; according to the target projection feature, a speech feature embedding vector is generated, and the speech feature embedding vector is input into the text decoder to obtain the emotional description information.

[0077] The word frequency analysis module 203 is used to calculate the occurrence frequency of related objects within a preset time, determine the target object among the related objects according to the occurrence frequency, and set the emotional noun corresponding to the target object as the first noun.

[0078] The word matching module 204 is used to match the first noun in a preset noun database, set the successfully matched first noun as the second noun, and set the intensity adverb corresponding to the second noun as the target adverb.

[0079] The data conversion module 205 is used to generate a sentiment scoring curve according to the second noun, the target adverb and the preset mapping rule, and determine the sentiment fluctuation amplitude and the sentiment fluctuation frequency according to the sentiment scoring curve.

[0080] In some embodiments, when the data conversion module 205 is used to generate a sentiment scoring curve according to the second noun, the target adverb and the preset scoring rules, and determine the sentiment fluctuation amplitude and the sentiment fluctuation frequency according to the sentiment scoring curve, it is specifically used to implement: according to the preset mapping rules, the second noun and the target adverb are subjected to sentiment intensity mapping processing to obtain an initial sentiment intensity value; according to the preset time weighting coefficient, the initial sentiment intensity value is subjected to time weighting processing to obtain a weighted sentiment intensity value; the weighted sentiment intensity value is smoothed using a smoothing algorithm to obtain a smoothed sentiment intensity curve; according to the preset peak detection algorithm, the smoothed sentiment intensity curve is subjected to peak detection processing to obtain a sentiment fluctuation peak value; the sentiment fluctuation peak value is subjected to amplitude calculation processing to obtain a sentiment fluctuation amplitude; the time interval between the sentiment fluctuation amplitude and the sentiment fluctuation peak value is subjected to frequency calculation processing to obtain a sentiment fluctuation frequency.

[0081] The result output module 206 is used to determine the emotion fluctuation score according to the emotion fluctuation amplitude and the emotion fluctuation frequency, and when it is detected that the emotion fluctuation score is greater than a preset score threshold, generate and output warning information according to the target object, the second noun and the target adverb.

[0082] In some embodiments, when the result output module 206 is used to determine the emotion fluctuation score based on the emotion fluctuation amplitude and the emotion fluctuation frequency, it is specifically used to achieve: normalizing the emotion fluctuation amplitude and the emotion fluctuation frequency respectively to obtain the normalized emotion fluctuation amplitude value and the normalized emotion fluctuation frequency value; weighting the normalized emotion fluctuation amplitude value and the normalized emotion fluctuation frequency value according to a preset weighting coefficient to obtain a weighted emotion fluctuation value; scoring the weighted emotion fluctuation value according to a preset scoring rule to obtain an emotion fluctuation score.

[0083] In some embodiments, when the result output module 206 is used to implement scoring processing of weighted emotional fluctuation values ​​according to preset scoring rules to obtain an emotional fluctuation score, it is specifically used to implement: using sliding window technology to divide the weighted emotional fluctuation values ​​into multiple overlapping time windows, each time window including multiple weighted emotional fluctuation values; calculating the weighted emotional fluctuation values ​​in each time window to obtain the average emotional fluctuation value corresponding to each time window; performing difference calculation on the average emotional fluctuation values ​​of two adjacent time windows to obtain an emotional fluctuation change value; performing nonlinear mapping processing on the average emotional fluctuation value to obtain an emotional fluctuation intensity score, and performing nonlinear mapping processing on the emotional fluctuation change value to obtain an emotional fluctuation change rate score; performing weighted average processing on the emotional fluctuation intensity score and the emotional fluctuation change rate score to obtain an emotional fluctuation score for each time window.

[0084] In some embodiments, when the result output module 206 is used to generate and output warning information based on the target object, the second noun and the target adverb, it is specifically used to implement: performing data combination processing on the target object, the second noun and the target adverb to obtain a warning information template; performing content filling processing on the warning information template to obtain the warning information content; performing formatting processing on the warning information content to obtain warning information, and outputting the warning information through a preset warning device.

[0085] An embodiment of the present application provides an electronic device, which includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement any intelligent warning method based on voice detection as described in the embodiments of the present application when executing the computer program.

[0086] An embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements any intelligent warning method based on voice detection as described in the embodiments of the present application.

[0087] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. An intelligent warning method based on voice detection, characterized in that: The method comprises: Performing spectral processing on the speech data in the detection scene to obtain a speech spectrogram, and performing text extraction on the speech data to obtain text data; Inputting the speech spectrogram and the text data into a preset natural language model to obtain emotion description information, wherein the emotion description information includes: related objects, emotion nouns and intensity adverbs, wherein the intensity adverbs are used to indicate the emotion intensity of the emotion nouns; Calculating the occurrence frequency of the related objects within a preset time, determining a target object from the related objects according to the occurrence frequency, and setting the emotion noun corresponding to the target object as the first noun; According to the matching of the first noun in a preset noun database, the first noun that is successfully matched is set as a second noun, and the intensity adverb corresponding to the second noun is set as a target adverb; Generate a sentiment scoring curve according to the second noun, the target adverb and a preset mapping rule, and determine the sentiment fluctuation amplitude and the sentiment fluctuation frequency according to the sentiment scoring curve; An emotion fluctuation score is determined according to the emotion fluctuation amplitude and the emotion fluctuation frequency, and when it is detected that the emotion fluctuation score is greater than a preset score threshold, warning information is generated and output according to the target object, the second noun and the target adverb.

2. The intelligent warning method based on voice detection according to claim 1, characterized in that: The preset natural language model includes an audio encoder, a feature extractor and a text decoder, wherein the feature extractor includes a self-attention layer, a cross-attention layer and a linear layer, and the speech spectrogram and the text data are input into the preset natural language model to obtain the emotion description information, including: Based on the audio encoder, feature extraction is performed on the speech spectrogram to obtain a high-dimensional feature matrix; Inputting the high-dimensional feature matrix into the feature extractor, performing weighted summation on the high-dimensional feature matrix through the self-attention layer to obtain a compressed feature matrix; Through the cross attention layer, according to a preset query vector, the compressed feature matrix and the high-dimensional feature matrix are cross-attended to obtain an enhanced feature matrix; Through the linear layer, the enhanced feature matrix is ​​projected to a preset text representation space, and activation or function processing is performed to obtain a projection feature matrix; Projecting the text data to a preset text representation space to obtain a text projection matrix, performing multimodal comparative learning on the text projection matrix and the projection feature matrix to obtain an optimized projection feature matrix; According to a preset loss function, the upper bound value of the mutual information between the optimized projection feature matrix and the text projection matrix is ​​calculated, and the optimized projection feature matrix whose upper bound value of the mutual information is lower than the preset upper bound threshold is set as the target projection feature matrix; A speech feature embedding vector is generated according to the target projection feature, and the speech feature embedding vector is input into the text decoder to obtain the emotion description information.

3. The intelligent warning method based on voice detection according to claim 1, characterized in that: The step of generating a sentiment scoring curve according to the second noun, the target adverb and a preset scoring rule, and determining the sentiment fluctuation amplitude and the sentiment fluctuation frequency according to the sentiment scoring curve comprises: According to a preset mapping rule, performing sentiment intensity mapping processing on the second noun and the target adverb to obtain an initial sentiment intensity value; According to a preset time weighting coefficient, the initial emotion intensity value is subjected to time weighting processing to obtain a weighted emotion intensity value; Using a smoothing algorithm to smooth the weighted emotion intensity value to obtain a smoothed emotion intensity curve; According to a preset peak detection algorithm, peak detection processing is performed on the smoothed emotion intensity curve to obtain an emotion fluctuation peak value; Performing amplitude calculation processing on the emotion fluctuation peak value to obtain the emotion fluctuation amplitude; The emotion fluctuation amplitude and the time interval of the emotion fluctuation peak are subjected to frequency calculation processing to obtain the emotion fluctuation frequency.

4. The intelligent warning method based on voice detection as claimed in claim 1, characterized in that: Determining the emotion fluctuation score according to the emotion fluctuation amplitude and the emotion fluctuation frequency includes: Normalizing the emotion fluctuation amplitude and the emotion fluctuation frequency respectively to obtain a normalized emotion fluctuation amplitude value and a normalized emotion fluctuation frequency value; According to a preset weighting coefficient, the normalized emotion fluctuation amplitude value and the normalized emotion fluctuation frequency value are weighted to obtain a weighted emotion fluctuation value; The weighted emotion fluctuation value is scored according to a preset scoring rule to obtain an emotion fluctuation score.

5. The intelligent warning method based on voice detection as claimed in claim 4, characterized in that: The step of scoring the weighted emotion fluctuation value according to a preset scoring rule to obtain an emotion fluctuation score includes: Using a sliding window technique, the weighted emotion fluctuation value is divided into a plurality of overlapping time windows, each of the time windows including a plurality of the weighted emotion fluctuation values; Calculating the weighted emotion fluctuation value in each time window to obtain an average emotion fluctuation value corresponding to each time window; Calculating the difference between the average emotion fluctuation values ​​of two adjacent time windows to obtain an emotion fluctuation change value; Performing nonlinear mapping processing on the average emotion fluctuation value to obtain an emotion fluctuation intensity score, and performing nonlinear mapping processing on the emotion fluctuation change value to obtain an emotion fluctuation change rate score; The emotion fluctuation intensity score and the emotion fluctuation change rate score are weighted averaged to obtain the emotion fluctuation score of each time window.

6. The intelligent warning method based on voice detection as claimed in claim 1, characterized in that: The generating and outputting warning information according to the target object, the second noun and the target adverb includes: Performing data combination processing on the target object, the second noun and the target adverb to obtain a warning information template; Filling the warning information template with content to obtain warning information content; The warning information content is formatted to obtain warning information, and the warning information is output through a preset warning device.

7. An intelligent warning device based on voice detection, characterized in that: The intelligent warning device based on voice detection includes: A data processing module is used to perform spectral processing on the voice data in the detection scene to obtain a voice spectrum graph, and to perform text extraction on the voice data to obtain text data; A sentiment analysis module, used for inputting the speech spectrogram and the text data into a preset natural language model to obtain sentiment description information, wherein the sentiment description information includes: related objects, sentiment nouns and intensity adverbs, wherein the intensity adverbs are used to indicate the sentiment intensity of the sentiment nouns; A word frequency analysis module, used to calculate the frequency of occurrence of the related objects within a preset time, determine the target object from the related objects according to the frequency of occurrence, and set the emotional noun corresponding to the target object as the first noun; a word matching module, configured to match the first noun in a preset noun database, set the first noun that successfully matches as a second noun, and set the intensity adverb corresponding to the second noun as a target adverb; A data conversion module, used to generate a sentiment scoring curve according to the second noun, the target adverb and a preset mapping rule, and determine the sentiment fluctuation amplitude and the sentiment fluctuation frequency according to the sentiment scoring curve; A result output module is used to determine an emotion fluctuation score according to the emotion fluctuation amplitude and the emotion fluctuation frequency, and when it is detected that the emotion fluctuation score is greater than a preset score threshold, generate and output warning information according to the target object, the second noun and the target adverb.

8. An electronic device, characterized in that: The electronic device comprises a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the intelligent warning method based on voice detection as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the intelligent warning method based on voice detection as described in any one of claims 1 to 6.

Images