Crank call identification method and device, electronic equipment and storage medium

Abstract

The invention discloses a crank call identification method and apparatus, an electronic device and a storage medium. The method comprises the steps of obtaining a target audio feature and a target text feature obtained by performing feature extraction on target audio data and target text data of a target call; the target text data is obtained by transferring the target audio data; respectively carrying out attention processing on the target audio feature and the target text feature to obtain a target audio attention feature corresponding to the target audio feature and a target text attention feature corresponding to the target text feature; performing weighting processing on the target audio attention feature and the target text attention feature by using respective corresponding adaptive weights to obtain a target fusion feature; and determining whether the target call is a crank call based on the target fusion feature. In this way, the crank call recognition accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of communication security technology, and in particular to a method and device for identifying harassing calls, an electronic device, and a storage medium. Background Art

[0002] With the rapid development of information technology and the increasing diversification of communication means, the telephone has become an indispensable communication tool in modern society. However, the problem of harassing calls has become increasingly serious, not only interfering with people's daily lives, but also posing a serious threat to users' mental health and property safety in some cases.

[0003] Nuisance calls are usually made without the recipient's consent, and their purpose is usually to promote sales, defraud or intimidate. The content of these calls varies, including advertising, impersonating customer service, and fake official organizations. These harassing behaviors are often covert and complex, and are extremely confusing and deceptive. Over time, the language design of nuisance calls has become more and more complex, and even began to include advanced techniques such as ambiguous language and emotional manipulation, which are difficult to accurately identify with traditional identification methods. Summary of the invention

[0004] The main technical problem solved by this application is to provide a multi-classification model verification method, electronic device and storage medium, which can improve the accuracy of harassing call identification.

[0005] To solve the above technical problems, the first aspect of the present application provides a method for identifying harassing calls, the method comprising: obtaining target audio features and target text features obtained by respectively extracting the target audio data and target text data of the target phone; the target text data is obtained by transcribing the target audio data; performing attention processing on the target audio features and the target text features respectively to obtain target audio attention features corresponding to the target audio features and target text attention features corresponding to the target text features; performing weighted processing on the target audio attention features and the target text attention features using their respective corresponding adaptive weights to obtain target fusion features; based on the target fusion features, determining whether the target phone is a harassing call.

[0006] To solve the above technical problems, the second aspect of the present application provides a device for identifying harassing calls, which includes: an acquisition module, an attention module, a fusion module and an identification module. The acquisition module is used to obtain target audio features and target text features obtained by respectively extracting target audio data and target text data of the target phone; the target text data is obtained by transcribing the target audio data; the attention module is used to perform attention processing on the target audio features and target text features respectively to obtain target audio attention features corresponding to the target audio features and target text attention features corresponding to the target text features; the fusion module is used to perform weighted summation of the target audio attention features and the target text attention features using their respective corresponding adaptive weights to obtain target fusion features; the identification module is used to determine whether the phone is a harassing call based on the target fusion features.

[0007] To solve the above technical problem, the third aspect of the present application provides an electronic device, which includes a processor and a memory, wherein program instructions are stored in the memory, and the processor is used to execute the program instructions to implement the method provided in the first aspect.

[0008] In order to solve the above technical problems, the fourth aspect of the present application provides a computer-readable storage medium, which stores program instructions, and the program instructions are used to implement the method provided by the first aspect above.

[0009] The beneficial effects of the present application are as follows: Different from the prior art, the present application transcribes the target audio data of the target phone to obtain the target text data, obtains the target audio features and target text features obtained by extracting the target audio data and target text data of the target phone respectively, and performs attention processing on the target audio features and target text features respectively to obtain the target audio attention features corresponding to the target audio features and the target text attention features corresponding to the target text features; weights the target audio attention features and the target text attention features using their respective corresponding adaptive weights to obtain the target fusion features; and finally, based on the target fusion features, determines whether the target phone is a harassing call. In the process of identifying harassing calls, the target audio data and the target text data are combined, and corresponding adaptive weights are set for the target audio data and the target text data to improve the accuracy of the target fusion features, thereby improving the accuracy of harassing call identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a flowchart of an implementation method of a harassing call identification method provided by this application;

[0011] Figure 2 It is a flowchart of another implementation method of the harassing call identification method provided by the present application;

[0012] Figure 3 It is a schematic diagram of the framework structure of an implementation method of a harassing call identification device provided by the present application;

[0013] Figure 4 It is a schematic diagram of the framework structure of an embodiment of an electronic device provided by the present application;

[0014] Figure 5 It is a schematic diagram of the framework structure of an implementation method of a computer-readable storage medium provided in this application. DETAILED DESCRIPTION

[0015] The following is a clear and complete description of the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0016] It should be noted that in the embodiments of the present application, there are descriptions involving "first", "second", etc., which are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features.

[0017] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0018] The inventors found in their long-term research that the current technology for identifying harassing calls mainly relies on automatic speech recognition technology, which transcribes the content of voice calls into text data, and then uses a pre-trained language model to classify these text data to identify call content with potential harassment. This text classification-based method performs well when dealing with standard Mandarin and clearly stated harassing calls, and can accurately identify malicious or bad intentions. However, in actual applications, the content of harassing calls is often more complex and diverse, and the method of relying solely on text content for classification exposes many limitations.

[0019] The diversity and complexity of the content of harassing calls make it difficult to deal with single-modal recognition methods. Many harassing calls confuse the caller through cleverly designed rhetoric, ambiguous language expressions, and even rhetorical questions and hints. After the speech is transcribed into text, such rhetoric often cannot fully reflect its potential malicious intentions, making it difficult for text-based classification methods to accurately identify harassing behavior. In addition, some harassing calls use technical means to change sound features (such as machine sounds), which causes errors in the transcription process of the speech content, further affecting the accuracy of recognition.

[0020] The impact of dialects and accents is also a major challenge facing current technology. China is a vast country with many dialects, and the accents of the same language vary significantly in different regions. Callers of harassing calls often speak in dialects or Mandarin with a strong local accent, and existing speech recognition systems have low accuracy when processing these dialects and accents, and are prone to recognition errors. These recognition errors directly affect the subsequent text classification process, causing the model to be unable to accurately determine the nature of the call. Due to the diversity of dialects and accents, the applicability of existing text-based harassing call recognition technology in different regions varies significantly, which limits the widespread application of the system.

[0021] In addition, the expression of tone and emotion plays a vital role in the identification of harassing calls. In many cases, the threatening or deceptive intentions of harassing calls are not expressed through clear language, but through changes in tone and emotion. For example, callers may exert pressure or express threats by raising their voices, speaking faster, or adopting special tones, and this information is often ignored or not correctly reflected in the traditional text transcription process. Existing text classification models cannot capture these emotional and tone changes, resulting in some potentially threatening or deceptive harassing calls that cannot be accurately identified. Some harassing calls use deliberate whispering, a gentle tone, and other methods to conceal their true intentions. These features are difficult to identify through traditional text analysis methods after text transcription.

[0022] Due to the limitations of the prior art, the harassment call identification method that relies on a single modality has been difficult to meet actual needs when facing complex and changeable real-world application scenarios. To solve this problem, the present application obtains the target audio features and target text features obtained by feature extraction of the target audio data and target text data of the target phone, and then performs attention processing on the target audio features and target text features respectively to obtain the target audio attention features corresponding to the target audio features and the target text attention features corresponding to the target text features; the target audio attention features and the target text attention features are weighted using their respective corresponding adaptive weights to obtain the target fusion features; finally, based on the target fusion features, it is determined whether the target phone is a harassment call. In the process of harassment call identification, audio data and text data are combined, and corresponding adaptive weights are set for the audio data and text data to improve the accuracy of the target fusion features, thereby improving the accuracy of harassment call identification.

[0023] See also Figure 1 , Figure 1 It is a flow chart of an implementation method of a harassing call identification method provided by the present application, and the method comprises:

[0024] S11: Obtain target audio features and target text features obtained by extracting features from target audio data and target text data of a target phone respectively.

[0025] In one embodiment, the target text data is obtained by transcribing the target audio data. The target audio data can be feature extracted using any existing speech feature extraction method, and the target text data can be feature extracted using any existing text feature extraction method. The speech feature extraction method includes but is not limited to Fourier transform, linear prediction analysis, etc. The text feature extraction method includes but is not limited to the use of feature extraction models, feature word frequency statistics, etc.

[0026] In a specific embodiment, a pre-trained speech model is used to process the speech signal for the target audio data. The speech model encodes the input speech signal and generates a speech vector with time series characteristics. The speech vector is then feature extracted to obtain a high-dimensional audio feature vector, that is, the target audio feature. The high-dimensional audio feature vector may include at least one of the frequency feature, speech rate feature, intonation feature, emotion feature and accent feature in the speech signal. Furthermore, the high-dimensional audio feature vector can also be mapped to a higher-dimensional emotion and accent feature space to capture subtle changes in speech rate, tone and pronunciation pattern. Through this process, the ability to characterize the emotion and accent features in the speech is enhanced, thereby improving the recognition accuracy and robustness of the model when processing complex speech signals.

[0027] For target text data, we can use the pre-trained language model to perform multi-layer encoding on the text sequence to obtain a text feature sequence containing rich contextual semantic information. This feature sequence not only contains the semantic information of the text content, but also can capture the emotional expression in the text, and further extracts high-level semantic features that can reflect the text intent through a multi-layer deep learning network.

[0028] S12: Perform attention processing on the target audio feature and the target text feature respectively to obtain a target audio attention feature corresponding to the target audio feature and a target text attention feature corresponding to the target text feature.

[0029] In one embodiment, attention processing may be performed on the target audio features and the target text features respectively, wherein the attention processing may be self-attention processing.

[0030] In a specific implementation, in order to better capture the correlation between the fused features, a multi-head attention mechanism is used to process the target audio features and the target text features. The multi-head attention mechanism aggregates and analyzes information in different subspaces through multiple parallel attention heads, so that the system can focus on features from multiple angles.

[0031] The calculation formula for multi-head attention is:

[0032]

[0033] Where Q, K and V represent the query vector, key vector and value vector respectively, d k is the feature dimension.

[0034] Specifically, based on the target audio features and the corresponding audio weights, a first query vector, a first key vector and a first value vector can be obtained, and multi-head attention processing can be performed using the first query vector, the first key vector and the first value vector to obtain a target audio attention feature corresponding to the target audio feature.

[0035] Among them, the first query vector can be the product of the first audio weight and the target audio feature, the first key vector can be the product of the second audio weight and the target audio feature, and the first value vector can be the product of the third audio weight and the target audio feature. The first audio weight, the second audio weight, and the third audio weight can be set as needed, or they can be weights trained by neural network back propagation in the model pre-training stage. In some embodiments, the first audio weight, the second audio weight, and the third audio weight may be the same or different.

[0036] Based on the target text features and the corresponding text weights, a second query vector, a second key vector and a second value vector can be obtained, and multi-head attention processing can be performed using the second query vector, the second key vector and the second value vector to obtain a target text attention feature corresponding to the target text feature.

[0037] Among them, the second query vector can be the product of the first text weight and the target text feature, the second key vector can be the product of the second text weight and the target text feature, and the second value vector can be the product of the third text weight and the target text feature. The determination of the first text weight, the second text weight and the third text weight is similar to the audio weight, and will not be repeated here.

[0038] S13: Perform weighted processing on the target audio attention feature and the target text attention feature using their respective corresponding adaptive weights to obtain a target fusion feature.

[0039] In one embodiment, corresponding adaptive weights may be set for the target audio features and the target text features, respectively. The adaptive weights may be set as needed. For example, the importance of the audio data and the text data may be manually analyzed to set the weights accordingly. Assuming that the accuracy of the target text data obtained by transcribing the target audio data is low, a lower adaptive weight may be set for the text data.

[0040] In different harassing call scenarios, the quality of text and voice features may vary significantly, and simply weighting the two together may not fully utilize the advantages of each modality. To this end, the present application proposes an innovative modal weight adaptive adjustment mechanism that can dynamically adjust the weight distribution in the fusion process according to the feature quality of the input voice and text. The system determines the weight during fusion by analyzing the credibility of each modal feature. The credibility of the text modality may be affected by the degree of structuring of the language and the complexity of the content, while the credibility of the voice modality depends on the sound quality, emotional intensity, and the influence of dialect or accent. For example, when the emotion and intonation in the voice features change significantly, the model will assign a higher weight to the voice modality, otherwise, it will rely more on the text modality for classification. Therefore, in another embodiment, the first credibility of the target text data is first obtained, and the second credibility of the target audio data is obtained; then based on the first credibility and the second credibility, the adaptive weights corresponding to the target audio features and the target text features are determined.

[0041] In a specific implementation, the sum of the first credibility and the second credibility is obtained; the ratio of the first credibility to the sum is used as the adaptive weight corresponding to the target text data, and the ratio of the second credibility to the sum is used as the adaptive weight corresponding to the target audio data.

[0042] Among them, the first credibility and the second credibility can be obtained by analyzing the target text features and the target audio features. Specifically, for the text modality, the main influencing factors of the first credibility include the degree of text structuring and the complexity of the content. The degree of text structuring refers to the grammatical correctness and logical coherence of the sentence. For example, for clear declarative sentences, the degree of structuring is higher and the credibility is correspondingly higher. On the other hand, the complexity of the content refers to whether the text contains ambiguous or obscure language expressions. For complex rhetoric, such as rhetorical questions, hints or metaphors, the credibility of the text may be lower.

[0043] In one embodiment, a text complexity score and a text structured score can be determined based on the target text features, and the text complexity score and the text structured score are weighted and summed to obtain a first credibility. The text complexity score and the text structured score can be obtained by a pre-trained language model. For example, by using the attention distribution of words in the language model, the sentence structure with ambiguity and complexity in the text can be analyzed. At the same time, the model generates corresponding complexity scores and structured scores through vocabulary scarcity, sentence length and grammatical analysis. The weights of the text complexity score and the text structured score can be set as needed.

[0044] For the speech modality, the second credibility is affected by the sound quality, emotional intensity, and dialect or accent. Sound quality can be measured by indicators such as signal-to-noise ratio and audio clarity. The higher the signal-to-noise ratio, the better the sound quality and the higher the credibility. Emotional intensity is one of the key factors in the identification of harassing calls. The model will classify the extracted emotional features and analyze the emotional features of the voice (such as anger, threats, etc.) through structures such as multi-layer perceptrons. Voices with large emotional fluctuations are usually more credible. Dialects and accents will affect the parsing quality of the speech modality. The speech credibility of dialects or accent areas with low recognition accuracy is relatively low.

[0045] In one embodiment, the sound quality score and the emotion intensity score of the audio data can be determined based on the target audio feature, and the sound quality score and the emotion intensity score are weighted and summed to obtain the second credibility. The weights corresponding to the sound quality score and the emotion intensity score can be set as needed and are not limited here.

[0046] After determining the adaptive weight corresponding to the target audio feature and the adaptive weight corresponding to the target text feature, the adaptive weight corresponding to the target audio feature is multiplied by the target audio attention feature to obtain a first product; the adaptive weight corresponding to the target text feature is multiplied by the target text attention feature to obtain a second product. The first product and the second product are summed to obtain the target fusion feature.

[0047] S14: Based on the target fusion features, determine whether the target phone is a nuisance call.

[0048] In one embodiment, a trained harassing call recognition model is used to determine whether a target phone is a harassing call based on target fusion features. The harassing call recognition model may be the same model as the language model and speech model mentioned above, or may not be the same model.

[0049] In the above method, the target audio data of the target phone is transcribed to obtain the target text data, the target audio features and target text features are obtained by extracting the target audio data and target text data of the target phone respectively, and the target audio features and target text features are respectively subjected to attention processing to obtain the target audio attention features corresponding to the target audio features and the target text attention features corresponding to the target text features; the target audio attention features and the target text attention features are weighted using their respective corresponding adaptive weights to obtain the target fusion features; finally, based on the target fusion features, it is determined whether the target phone is a harassing call. In the process of harassing call identification, audio data and text data are combined, and corresponding adaptive weights are set for the audio data and text data to improve the accuracy of the target fusion features, thereby improving the accuracy of harassing call identification.

[0050] The above steps S11-S14 can all be performed by the harassment call recognition model, or only step S14 can be performed by the harassment call recognition model. In one embodiment, the harassment call recognition model training step includes: obtaining sample audio features and sample text features obtained by extracting features from sample audio data and sample text data of the sample phone respectively; the sample text data is obtained by transcribing the sample audio data. Perform attention processing on the sample audio features and sample text features respectively to obtain sample audio attention features corresponding to the sample audio features and sample text attention features corresponding to the sample text features. Perform weighted sum processing on the sample audio attention features and the sample text attention features using their respective corresponding adaptive weights to obtain sample fusion features. Using the recognition model based on the sample fusion features, the sample phone is identified to obtain an identification result that characterizes whether the sample phone is a harassment call. Among them, the processing process of the sample audio data and the sample text data is the same as the processing process of the target audio data and the target text data, which will not be repeated here. Based on the recognition result and the labeling result of the sample phone, determine the loss; based on the loss, adjust the network parameters of the harassment call recognition model. Among them, the loss can be determined by the following loss function.

[0051]

[0052] P(y i |y ＜i ,X f )=softmax(W·h i +b)

[0053] Among them, h i represents the hidden state of the current time step, X f is the fusion feature, b is the bias term, θ is the model parameter, and N is the number of training samples.

[0054] In other implementations, other loss functions may also be used, for example, a cross entropy loss function may also be used.

[0055] See also Figure 2 , Figure 2 It is a flow chart of another implementation method of the harassing call identification method provided in this application.

[0056] S21: Obtain target audio features and target text features obtained by extracting features from target audio data and target text data of the target phone respectively.

[0057] The target text data is obtained by transcribing the target audio data.

[0058] S22: Align the target audio features and the target text features.

[0059] In one embodiment, the alignment process includes dimensional alignment and / or temporal alignment.

[0060] In a specific implementation, the alignment process includes dimensional alignment, and the target audio feature and the target text feature are processed by a one-dimensional convolutional neural network respectively to achieve dimensional alignment. Assume that the target text feature X t and target audio feature X a After being processed by 1D convolutional layers respectively, the aligned text feature vectors are obtained. and speech feature vector

[0061]

[0062] Among them, k t and k a Represent the convolution kernel sizes of text and speech modalities respectively. Through this alignment operation, the consistency of the target text features and the target audio features in dimension is ensured, which facilitates the subsequent fusion operation.

[0063] In another specific embodiment, the alignment process includes time alignment, and the dynamic time warping algorithm (Dynamic Time Warping, DTW) can be used to determine the optimal alignment path between the target audio features and the target text features based on the target audio features and the target text features; based on the optimal alignment path, the target audio features and the target text features are time-aligned. The dynamic time warping algorithm is used to align the text and audio features in the time dimension, thereby ensuring that the features of different modalities can be associated and analyzed on the same time scale. Assume that the text feature sequence is X t =[x t1 ,x t2 ,…,x tn ], the audio feature sequence is X a =[x a1 ,x a2 ,…,x am ], using the DTW algorithm to calculate the time alignment path of text and audio features:

[0064] D(i,j)=min{D(i-1,j),D(i,j-1),D(i-1,j-1)}+d(x ti ,x aj )

[0065] Among them, D(i,j) is the target text feature x ti With the target audio feature x aj The distance matrix between them can be used to calculate the optimal alignment path for each text and speech feature, thereby solving the time mismatch problem between modalities.

[0066] In other specific implementations, the alignment process includes dimensional alignment and temporal alignment.

[0067] S23: Perform attention processing on the target audio feature and the target text feature respectively to obtain a target audio attention feature corresponding to the target audio feature and a target text attention feature corresponding to the target text feature.

[0068] In this embodiment, attention processing is performed on the aligned target audio features and target text features to obtain target audio attention features corresponding to the target audio features and target text attention features corresponding to the target text features.

[0069] S24: weighting the target audio attention feature and the target text attention feature using their corresponding adaptive weights to obtain a target fusion feature;

[0070] S25: Based on the target fusion features, determine whether the target phone is a nuisance call.

[0071] Among them, the detailed implementation of steps S21, S23-S25 please refer to the aforementioned steps S11-S14, which will not be repeated here.

[0072] This implementation aligns features of different modalities, enabling better fusion of multimodal features in the subsequent process.

[0073] See also Figure 3 , Figure 3 It is a schematic diagram of the framework structure of an implementation method of a harassing call identification device provided in this application.

[0074] The harassing call identification device 30 includes an acquisition module 31, an attention module 32, a fusion module 33 and an identification module 34. The acquisition module 31 is used to obtain target audio features and target text features obtained by extracting the target audio data and target text data of the target phone respectively; the target text data is obtained by transcribing the target audio data; the attention module 32 is used to perform attention processing on the target audio features and the target text features respectively, and obtain the target audio attention features corresponding to the target audio features and the target text attention features corresponding to the target text features; the fusion module 33 is used to perform weighted summation of the target audio attention features and the target text attention features using their respective corresponding adaptive weights to obtain the target fusion features; the identification module 34 is used to determine whether the phone is a harassing call based on the target fusion features.

[0075] In one embodiment, the harassing call identification device 30 includes a weight module, which is used to obtain a first credibility of the target text data and a second credibility of the target audio data; based on the first credibility and the second credibility, determine the adaptive weights corresponding to the target audio features and the target text features respectively.

[0076] In one embodiment, the weight module determines the adaptive weights corresponding to the target audio features and the target text features based on the first credibility and the second credibility, including: obtaining the sum of the first credibility and the second credibility; using the ratio of the first credibility to the sum as the adaptive weight corresponding to the target text data; and, using the ratio of the second credibility to the sum as the adaptive weight corresponding to the target audio data.

[0077] In one embodiment, the weight module obtains the first credibility of the target text data, including: determining a text complexity score and a text structured score based on target text features; and performing weighted summation of the text complexity score and the text structured score to obtain the first credibility.

[0078] In one embodiment, the weight module obtains the second credibility of the target audio data, including: determining the sound quality score and the emotion intensity score of the audio data based on the target audio features; and performing weighted summation on the sound quality score and the emotion intensity score to obtain the second credibility.

[0079] In one embodiment, the attention module 32 performs attention processing on the target audio features and the target text features, including: obtaining a first query vector, a first key vector and a first value vector based on the target audio features and the corresponding audio weights; and obtaining a second query vector, a second key vector and a second value vector based on the target text features and the corresponding text weights; and performing multi-head attention processing using the first query vector, the first key vector and the first value vector, and the second query vector, the second key vector and the second value vector.

[0080] In one embodiment, the harassing call identification device 30 includes an alignment module, which is used to align the target audio features and the target text features before respectively performing attention processing on the target audio features and the target text features; the alignment processing includes dimensional alignment and / or temporal alignment.

[0081] In one embodiment, the alignment process includes dimensional alignment, and the alignment module performs alignment processing on the target audio features and the target text features, including: using a one-dimensional convolutional neural network to perform dimension alignment on the target audio features and the target text features.

[0082] In one embodiment, the alignment process includes timing alignment, and the alignment module performs alignment processing on the target audio features and the target text features, including: using a dynamic time warping algorithm to determine the optimal alignment path between the target audio features and the target text features based on the target audio features and the target text features; based on the optimal alignment path, performing timing alignment on the target audio features and the target text features.

[0083] In one embodiment, the target audio feature includes at least one of a frequency feature, a speech rate feature, a voice inflection feature, an emotion feature, and an accent feature.

[0084] See also Figure 4 , Figure 4 It is a schematic diagram of the framework structure of an embodiment of an electronic device provided in the present application.

[0085] The electronic device 40 includes a memory 41 and a processor 42 coupled to each other, the memory 41 stores program instructions, and the processor 42 is used to execute the program instructions stored in the memory 41 to implement the steps of any of the above method implementations. In a specific implementation scenario, the electronic device 40 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 40 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0086] Specifically, the processor 42 is used to control itself and the memory 41 to implement the steps of any of the above-mentioned method implementation methods. The processor 42 can also be called a CPU (Central Processing Unit). The processor 42 may be an integrated circuit chip with signal processing capabilities. The processor 42 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 42 can be implemented by an integrated circuit chip.

[0087] See also Figure 5 , Figure 5 It is a schematic diagram of the framework structure of an implementation method of a computer-readable storage medium provided in this application.

[0088] The computer-readable storage medium 50 stores program instructions 51 , and when the program instructions 51 are executed by a processor, they are used to implement the steps of any of the above method implementations.

[0089] The computer-readable storage medium 50 may specifically be a medium that can store computer programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or may be a server that stores the computer program. The server may send the stored computer program to other devices for execution, or may also execute the stored computer program itself.

[0090] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0091] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0092] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0093] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0094] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0095] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0096] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for identifying harassing calls, characterized in that: include: Obtain target audio features and target text features obtained by extracting features from target audio data and target text data of the target phone respectively; The target text data is obtained by transcribing the target audio data; Performing attention processing on the target audio feature and the target text feature respectively to obtain a target audio attention feature corresponding to the target audio feature and a target text attention feature corresponding to the target text feature; The target audio attention feature and the target text attention feature are weighted by using their corresponding adaptive weights to obtain a target fusion feature; Based on the target fusion feature, determine whether the target phone is a nuisance call.

2. The method according to claim 1, characterized in that The method further comprises: Acquiring a first credibility of the target text data, and acquiring a second credibility of the target audio data; Based on the first credibility and the second credibility, adaptive weights corresponding to the target audio feature and the target text feature are determined respectively.

3. The method according to claim 2, characterized in that The determining, based on the first credibility and the second credibility, the adaptive weights corresponding to the target audio feature and the target text feature, respectively, includes: Obtaining a sum of the first credibility and the second credibility; Using the ratio of the first credibility to the sum as the adaptive weight corresponding to the target text data; and The ratio of the second credibility to the sum is used as the adaptive weight corresponding to the target audio data.

4. The method according to claim 2, characterized in that: The obtaining of the first credibility of the target text data comprises: Based on the target text features, determining a text complexity score and a text structured score; Performing a weighted summation on the text complexity score and the text structured score to obtain the first credibility; And / or, obtaining the second credibility of the target audio data includes: Determining a sound quality score and an emotion intensity score of the audio data based on the target audio feature; The sound quality score and the emotion intensity score are weightedly summed to obtain the second credibility.

5. The method according to claim 1, characterized in that The performing attention processing on the target audio feature and the target text feature respectively includes: Based on the target audio features and the corresponding audio weights, a first query vector, a first key vector and a first value vector are obtained; and, Based on the target text features and the corresponding text weights, obtaining a second query vector, a second key vector, and a second value vector; Multi-head attention processing is performed using the first query vector, the first key vector and the first value vector, and the second query vector, the second key vector and the second value vector.

6. The method according to claim 1, characterized in that Before the attention processing is performed on the target audio feature and the target text feature respectively, the method further includes: The target audio feature and the target text feature are aligned; the alignment includes dimensional alignment and / or temporal alignment.

7. The method according to claim 6, characterized in that The alignment process includes dimensional alignment, and the alignment process of the target audio feature and the target text feature includes: Dimensionally aligning the target audio features and the target text features using a one-dimensional convolutional neural network; And / or, the alignment process includes time alignment, and the alignment process of the target audio feature and the target text feature includes: Determine an optimal alignment path between the target audio feature and the target text feature based on the target audio feature and the target text feature using a dynamic time warping algorithm; Based on the optimal alignment path, the target audio feature and the target text feature are temporally aligned.

8. The method according to claim 1, characterized in that The target audio feature includes at least one of a frequency feature, a speech speed feature, a voice intonation feature, an emotion feature, and an accent feature.

9. A device for identifying harassing calls, characterized in that: include: An acquisition module, used to acquire target audio features and target text features obtained by respectively extracting target audio data and target text data of a target phone; The target text data is obtained by transcribing the target audio data; An attention module is used to perform attention processing on the target audio feature and the target text feature respectively to obtain a target audio attention feature corresponding to the target audio feature and a target text attention feature corresponding to the target text feature; A fusion module, used to perform weighted summation on the target audio attention feature and the target text attention feature using their corresponding adaptive weights to obtain a target fusion feature; The identification module is used to determine whether the call is a nuisance call based on the target fusion feature.

10. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The storage medium stores program instructions, and the program instructions are used to implement the method according to any one of claims 1 to 8.

Images