A method and system for monitoring talkspurts

Abstract

The application relates to a call voice monitoring method and system, and belongs to the technical field of voice recognition. The method comprises the following steps: obtaining incoming call information data and call voice data of a to-be-monitored call; determining the call behavior of the to-be-monitored call according to the incoming call information data; when the to-be-monitored call is a suspicious call behavior, determining the behavior deviation score of the to-be-monitored call according to the incoming call information data, and updating the suspicious call behavior to a malicious call behavior in the case that the behavior deviation score is greater than or equal to a behavior deviation threshold; when the to-be-monitored call is a malicious call behavior, performing speaker separation on the call voice data to obtain call voice segments of multiple speakers, and converting the call voice segments into corresponding call voice texts; inputting the call voice segments and the call voice texts into a multi-branch call voice risk monitoring model to output the risk monitoring scores of the speakers; and outputting the monitoring result of the to-be-monitored call according to the risk monitoring scores of the speakers.

Description

Technical Field

[0001] This invention belongs to the field of speech recognition technology, and in particular relates to a method and system for monitoring voice during phone calls. Background Technology

[0002] With the widespread adoption of communication technologies and smart terminals, telephone calls have become an important means for users to communicate, conduct business, and receive information. At the same time, risky calls such as nuisance calls, fraudulent calls, and unusual marketing calls are increasing. How to monitor risky behaviors during calls in a timely and accurate manner has become a pressing issue in the field of communication security.

[0003] In existing technologies, call risk identification usually relies on call information rules such as number blacklists, call frequency, and abnormal location. This type of method is difficult to fully represent the complex relationship between different call characteristics in call behavior, resulting in low accuracy in identifying suspicious calls at the boundary and easy to make misjudgments or omissions.

[0004] In addition, existing call voice monitoring methods usually only make risk judgments based on the entire call voice or voice recognition text, which makes it difficult to distinguish the voice content of different speakers and the source of risk, and also makes it difficult to integrate voice acoustic risk and text semantic risk at the same time, resulting in insufficient accuracy and traceability of risk monitoring results. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the purpose of this invention is to provide a method and system for monitoring call voice. This method can determine the call behavior of the call to be monitored based on incoming call information data using a fuzzy C-means clustering algorithm. Based on the incoming call information data, it can determine the behavioral deviation score of the call to be monitored using a neural decomposition autoencoder. It can comprehensively judge the call information by combining rules such as number blacklists, call frequency, and abnormal location, and fully characterize the complex correlation between different incoming call characteristics in the call behavior, thereby improving the accuracy of identifying borderline suspicious calls and reducing the probability of misjudgment or omission. By separating the call voice data into speakers, multiple speaker voice segments are obtained, and these segments are converted into corresponding call voice text. The call voice segments and text are input into a multi-branch call voice risk monitoring model, which outputs a risk monitoring score for each speaker, representing the combined risk level of voice and text risks. This method can make risk judgments based on different speaker voice segments and corresponding voice recognition text, distinguishing the voice content and risk sources of different speakers, and integrating voice acoustic risk and text semantic risk, thereby improving the accuracy and traceability of risk monitoring results.

[0006] A first aspect of the present invention provides a method for monitoring voice during a phone call, the method comprising: S1, acquire incoming call information data and call voice data of the call to be monitored; S2: Based on the incoming call information data, determine the call behavior of the call to be monitored using the fuzzy C-means clustering algorithm; if the call to be monitored is a suspicious call behavior, proceed to S3; if the call to be monitored is a malicious call behavior, proceed to S4. S3, based on incoming call information data, determines the behavioral deviation score of the call to be monitored through a neural decomposition autoencoder, and updates the suspicious call behavior to malicious call behavior if the behavioral deviation score is greater than or equal to the behavioral deviation threshold. S4, perform speaker separation on the call voice data to obtain call voice segments from multiple speakers, and convert the call voice segments into corresponding call voice text; S5 inputs the call audio segments and call audio text into the multi-branch call audio risk monitoring model and outputs the speaker's risk monitoring score, which represents the combined risk level of audio risk and text risk. S6 determines the call risk score of the call to be monitored based on the risk monitoring scores of each speaker, and outputs the monitoring results of the call to be monitored based on the call risk scores.

[0007] Furthermore, the calls to be monitored specifically refer to calls between the caller and the called party that have been requested or have already been established; the incoming call information data specifically includes the caller's number, the called number, the call time, the number's location, the operator information, the device identifier, the call type, and historical call behavior data; the call voice data specifically refers to the voice stream data between the caller and the called party.

[0008] Furthermore, S2 specifically includes: S201: Obtain historical call information data of multiple normal call behaviors from cloud data, and extract features from the historical call information data to form multiple training feature sets; S202, set the search interval for the number of candidate clusters, and calculate the membership concentration index and membership uncertainty entropy of each candidate cluster number in the search interval based on each training feature set; S203, calculate the fuzzy deviation index and standardized membership entropy of each candidate cluster number based on the membership concentration index and membership uncertainty entropy; S204. Calculate the clustering effectiveness coefficient of each candidate cluster number based on the membership concentration index, membership uncertainty entropy, fuzzy deviation index, and standardized membership entropy. S205, the number of candidate clusters corresponding to the maximum clustering effectiveness coefficient is used as the target number of clusters for the fuzzy C-means clustering algorithm to determine multiple target cluster centers; S206, calculate the first Euclidean distance between each training feature set and the corresponding nearest target cluster center, sort the first Euclidean distances in ascending order, and calculate the normal call threshold and malicious call threshold based on the first ascending order sorting result; S207, extract features from incoming call information data to form a set of features of incoming call information to be monitored, and calculate the second Euclidean distance between the set of features of incoming call information to be monitored and the corresponding nearest target cluster center; S208, compare the second Euclidean distance with the normal call threshold and the malicious call threshold respectively to determine the call behavior of the call to be monitored.

[0009] Furthermore, the determination of multiple target cluster centers in S205 specifically includes: S2051, randomly generate the initial membership matrix, and calculate the current round cluster center matrix based on the initial membership matrix for the current iteration number; S2052, calculate the third Euclidean distance between each training feature set and each cluster center in the current round cluster center matrix, and sort all the third Euclidean distances corresponding to each training feature set in ascending order; S2053, update the membership matrix of the current round based on each third Euclidean distance, and calculate the temporary cluster center matrix based on the updated membership matrix; S2054, calculate the amount of cluster center movement between each cluster center in the current round of the cluster center matrix and each cluster center in the temporary cluster center matrix; S2055, based on the second ascending order sorting result and the maximum cluster center movement, select stable features from each training feature set to form a stable feature set; S2056, calculate the membership scaling parameter for each stable feature in the stable feature set, and use the membership scaling parameter to scale the membership of the updated membership matrix; S2057, Calculate the final cluster center matrix based on the membership matrix after scaling. S2058 Calculate the fourth Euclidean distance between the final cluster center matrix and the current round cluster center matrix, and compare the fourth Euclidean distance with the distance threshold to determine the target cluster center.

[0010] Furthermore, S3 specifically includes: S301, Search and embed the categorical and numerical features in the feature set of incoming call information to be monitored to obtain the embedding vector of each feature of incoming call information to be monitored in the feature set of incoming call information to be monitored; S302, Calculate the normalized attention weights for each incoming call information feature to be monitored; S303, based on the embedding vector and normalized attention weights, through the bidirectional interaction layer in the neural decomposition autoencoder, fuses first-order weighted features and second-order feature interactions to obtain a high-order feature interaction representation of the call to be monitored. S304, input the high-order feature interaction representation into the encoder in the neural decomposition autoencoder to obtain the low-dimensional call behavior representation; S305, calculate the fifth Euclidean distance between the low-dimensional call behavior representation and the low-dimensional call behavior representation corresponding to each normal call behavior in the cloud data, and determine the average of each fifth Euclidean distance as the behavior deviation score.

[0011] Furthermore, the multi-branch call voice risk monitoring model specifically includes: a voice segment risk monitoring branch, a voice text risk monitoring branch, and a fusion layer; the voice segment risk monitoring branch is specifically used to: perform risk monitoring on call voice segments and obtain a voice segment monitoring score; the voice text risk monitoring branch is specifically used to: perform risk monitoring on call voice text and obtain a voice text monitoring score; the fusion layer is specifically used to: perform weighted fusion of the voice segment monitoring score and the voice text monitoring score to obtain the speaker's risk monitoring score, which represents the comprehensive risk level of voice risk and text risk.

[0012] Furthermore, the voice segment risk monitoring branch is also specifically used for: According to the preset analysis window, the call voice segment is slid segmented to obtain multiple analysis voice windows, and acoustic features are extracted from each analysis voice window to obtain multiple acoustic risk feature vectors. Risk state mapping is performed on each acoustic risk feature vector to obtain multiple local risk states; By performing temporal correlation calculations on each local risk state, multiple global risk temporal states are obtained; Calculate the risk propagation intensity of each analysis voice window based on the global risk time sequence status; By performing attention-weighted aggregation on the intensity of each risk propagation, the risk aggregation intensity of the call audio segment is obtained; The risk aggregation intensity is converted into a speech segment monitoring score using the Sigmoid function.

[0013] Furthermore, the speech-text risk monitoring branch specifically refers to: a speech-text risk monitoring branch based on the Chinese BERT model; the speech-text risk monitoring branch is also specifically used for: Text preprocessing is performed on the voice text of the call to obtain a text word sequence. Text feature extraction, pinyin feature extraction and glyph feature extraction are performed on each text word in the text word sequence to obtain text embedding features, pinyin embedding features and glyph embedding features. Feature fusion is performed on text embedding features, pinyin embedding features, and glyph embedding features to obtain multimodal fusion features; The multimodal fusion features are input into the Transformer encoder of the ChineseBERT model, and the contextual semantic representation of each text word is output through a multi-layer hybrid self-attention mechanism. Pooling is performed on the semantic representations of each context to obtain the global semantic aggregation features of the call speech text. The global semantic aggregation features are then input into a multilayer perceptron to output the binary log probability of whether the call speech text is normal text or risky text. The binary log odds are converted into speech-text detection scores using the Softmax function.

[0014] Furthermore, S6 specifically refers to: The maximum risk monitoring score is determined as the call risk score of the call to be monitored. The call risk score is compared with the risk score threshold, and the monitoring results of the call to be monitored are output. The monitoring results include no-risk calls, low-risk calls, medium-risk calls, and high-risk calls.

[0015] A second aspect of the present invention provides a call voice monitoring system, comprising: a memory and a processor; The memory stores an application program adapted to be executed by the processor to implement the call voice monitoring method described in the first aspect.

[0016] The beneficial effects of this invention are as follows: In this embodiment of the invention, based on incoming call information data, the call behavior of the call to be monitored is determined by a fuzzy C-means clustering algorithm. Based on the incoming call information data, the behavioral deviation score of the call to be monitored is determined by a neural decomposition autoencoder. This can be combined with incoming call information rules such as number blacklists, call frequency, and abnormal location to make a comprehensive judgment, and fully characterize the complex correlation between different incoming call features in the call behavior, thereby improving the accuracy of identifying borderline suspicious calls and reducing the probability of misjudgment or omission. By separating the speaker from the call voice data, multiple speaker voice segments are obtained, and the call voice segments are converted into corresponding call voice text. The call voice segments and call voice text are input into a multi-branch call voice risk monitoring model, which outputs a speaker risk monitoring score to characterize the comprehensive risk level of voice risk and text risk. This can make risk judgments based on the call voice segments of different speakers and the corresponding voice recognition text, distinguish the voice content and risk sources of different speakers, and integrate voice acoustic risk and text semantic risk, thereby improving the accuracy and traceability of risk monitoring results. Attached Figure Description

[0017] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. It is obvious that the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings.

[0018] Figure 1 This is a flowchart illustrating a call voice monitoring method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a call voice monitoring system provided in an embodiment of the present invention. Detailed Implementation

[0019] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0020] Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts disclosed in this invention.

[0021] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The terms "installed," "connected," and "linked" should be interpreted broadly; for example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0022] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of methods and systems consistent with some aspects of the invention as detailed in the appended claims.

[0023] This invention proposes a method and system for monitoring call voice. It addresses the technical problems of traditional call risk identification methods, which typically rely on caller ID rules such as number blacklists, call frequency, and abnormal location information. These methods struggle to fully represent the complex relationships between different caller characteristics, leading to low accuracy in identifying suspicious calls at the boundary and a high risk of misjudgment or omission. Furthermore, existing call voice monitoring methods often rely solely on the entire call audio or speech recognition text for risk assessment, making it difficult to distinguish the voice content of different speakers and the source of risk. They also struggle to simultaneously integrate acoustic and semantic risks, resulting in insufficient accuracy and traceability of risk monitoring results.

[0024] Method Implementation Examples Reference Figure 1 The diagram shows a flowchart of a call voice monitoring method provided by an embodiment of the present invention.

[0025] This invention provides a method for monitoring voice during phone calls, the method comprising: Specifically, the method includes steps S1 to S6.

[0026] S1, acquire incoming call information data and call voice data of the call to be monitored.

[0027] In one possible implementation, the call to be monitored specifically refers to a call requested or already established between the calling party and the called party. Call information data specifically includes the calling number, called number, call time, number location, carrier information, device identifier, call type, and historical call behavior data. Call voice data specifically refers to the voice stream data between the calling party and the called party.

[0028] S2: Based on incoming call information data, the call behavior of the call to be monitored is determined using a fuzzy C-means clustering algorithm. If the call to be monitored is suspected, proceed to S3. If the call to be monitored is malicious, proceed to S4.

[0029] In one possible implementation, S2 specifically includes sub-steps S201 to S208: S201 retrieves historical call information data of multiple normal call behaviors from cloud data, and extracts features from the historical call information data to form multiple training feature sets.

[0030] S202, set the search interval for the number of candidate clusters, and calculate the membership concentration index and membership uncertainty entropy of each candidate cluster number in the search interval based on each training feature set.

[0031] It should be noted that the candidate cluster number search interval is [2,5]. Those skilled in the art can set the size of the candidate cluster number search interval according to actual needs, and this invention does not limit it.

[0032] S203, calculate the fuzzy deviation index and standardized membership entropy of each candidate cluster number based on the membership concentration index and membership uncertainty entropy.

[0033] S204, calculate the clustering effectiveness coefficient for each candidate cluster number based on the membership concentration index, membership uncertainty entropy, fuzzy deviation index, and standardized membership entropy:

[0034]

[0035]

[0036] Among them, CEC c represents the clustering effectiveness coefficient when the number of candidate clusters is c, and w1, w2, w3, and w4 represent the weight coefficients of the membership concentration index, membership uncertainty entropy, fuzzy deviation index, and standardized membership entropy, respectively. , Those skilled in the art can set the values ​​of each weight coefficient according to actual needs; this invention does not impose any limitations on this. (MCI) c The membership concentration index is represented when the number of candidate clusters is c, where n represents the number of training feature sets, c represents the number of candidate clusters (i.e., the number of candidate cluster centers), and u ij MUE represents the membership degree of the j-th training feature set to the i-th candidate cluster center. c FDI represents the membership uncertainty entropy when the number of candidate clusters is c, log represents the logarithmic function, and FDI is the derivation of the membership uncertainty entropy. c SME represents the fuzzy deviation index when the number of candidate clusters is c. cThis represents the standardized membership entropy when the number of candidate clusters is c. The membership concentration index characterizes the degree of membership concentration of the training feature set to each cluster center; a higher membership concentration index indicates a more definite sample affiliation. The membership uncertainty entropy characterizes the degree of uncertainty in the membership of the training feature set among multiple cluster centers; a lower membership uncertainty entropy indicates a more stable clustering result. The fuzzy deviation index characterizes the degree of deviation of the current clustering result from the ideal clustering state; a lower fuzzy deviation index indicates a clustering result closer to the ideal partitioning state. The standardized membership entropy is used to normalize the membership uncertainty entropy to improve the comparability of clustering results corresponding to different numbers of candidate clusters.

[0037] It should be noted that the above-mentioned clustering effectiveness coefficient is constructed based on the evaluation principles of the fuzzy C-means clustering algorithm: "the more concentrated the membership, the lower the membership uncertainty, the smaller the fuzzy deviation, and the more comparable the evaluation results under different numbers of clusters." Specifically, the membership concentration index is used to positively characterize the clarity of the cluster division, while the membership uncertainty entropy, fuzzy deviation index, and standardized membership entropy are used to negatively characterize the uncertainty, deviation, and scale difference of the clustering results. Therefore, this embodiment of the invention constructs the clustering effectiveness coefficient by positively weighting the membership concentration index and negatively weighting the membership uncertainty entropy, fuzzy deviation index, and standardized membership entropy, so as to select the target number of clusters with more stable clustering results and more reasonable division from multiple candidate cluster numbers.

[0038] S205, the number of candidate clusters corresponding to the maximum clustering effectiveness coefficient is used as the target number of clusters for the fuzzy C-means clustering algorithm to determine multiple target cluster centers.

[0039] In one possible implementation, determining multiple target cluster centers in S205 specifically includes sub-steps S2051 to S2058: S2051, randomly generate the initial membership matrix, and calculate the current round cluster center matrix based on the initial membership matrix for the current iteration number.

[0040] S2052, calculate the third Euclidean distance between each training feature set and each cluster center in the current round cluster center matrix, and sort all the third Euclidean distances corresponding to each training feature set in ascending order.

[0041] S2053, update the membership matrix of the current round based on each third Euclidean distance, and calculate the temporary cluster center matrix based on the updated membership matrix.

[0042] S2054, calculate the amount of cluster center movement between each cluster center in the current round of the cluster center matrix and each cluster center in the temporary cluster center matrix.

[0043] S2055. Based on the second ascending order sorting result and the maximum cluster center movement, stable features are selected from each training feature set to form a stable feature set.

[0044] S2056, calculate the membership scaling parameter for each stable feature in the stable feature set, and use the membership scaling parameter to scale the updated membership matrix:

[0045]

[0046]

[0047]

[0048]

[0049] in, This indicates that the j-th training feature set in the (t+1)-th iteration belongs to the first training feature set. The membership degree of each cluster center, M j β represents the theoretical upper bound of the membership degree of the stable feature corresponding to the j-th training feature set. j This represents the membership scaling parameter for the stable feature corresponding to the j-th training feature set. This indicates that the j-th training feature set in the t-th iteration belongs to the t-th iteration. The membership degree of each cluster center, Q t Let represent the set of stable features at iteration number t. Let arg min represent the nearest cluster center of the j-th training feature set, and let arg min represent taking the minimum value. This represents the training feature set of the j-th iteration and the j-th iteration. The third Euclidean distance between cluster centers Represents the target number of clusters. This represents the minimum third Euclidean distance from the j-th training feature set to all cluster centers. Let m represent the maximum third Euclidean distance from the j-th training feature set to all cluster centers, and m represent the fuzzy weighting exponent. Those skilled in the art can set the size of the fuzzy weighted index according to actual needs, and this invention does not limit it. This indicates that the j-th training feature set in the t-th iteration belongs to the nearest cluster center. membership degree δ represents the second smallest third Euclidean distance from the j-th training feature set to all cluster centers. max This represents the maximum cluster center movement. Represents the nearest cluster center corresponding to the j-th training feature set. The amount of cluster center movement, where max represents the maximum value. Indicates the first Cluster center movement of each cluster center This represents the position of the first cluster center in the temporary cluster center matrix at iteration number t+1. Cluster centers, This represents the first cluster center in the current round of the clustering center matrix at iteration number t. There are cluster centers, x j Let j represent the set of training features. Let represent the third Euclidean distance between the j-th training feature set and the k-th cluster center at iteration number t.

[0050] It should be noted that the above membership scaling method is constructed based on the stability of sample affiliation and the iterative movement of cluster centers. During the clustering iteration process, if a training feature set can still maintain its distance advantage to the nearest cluster center after considering the movement of cluster centers, it indicates that its cluster affiliation is relatively clear and can be treated as a stable feature. For such stable features, this embodiment of the invention enhances its membership degree to the nearest cluster center based on its relative distance relationship with the nearest cluster center and other cluster centers, and accordingly scales its membership relationship with other cluster centers to ensure that the scaled membership relationship still meets the normalization requirements. For boundary features that do not meet the stability condition, the original membership relationship remains unchanged to avoid premature division. Thus, the stability of cluster center iterative updates can be improved while maintaining the fuzzy C-means clustering constraint.

[0051] S2057, calculate the final cluster center matrix based on the membership matrix after scaling.

[0052] S2058 Calculate the fourth Euclidean distance between the final cluster center matrix and the current round cluster center matrix, and compare the fourth Euclidean distance with the distance threshold to determine the target cluster center.

[0053] Specifically, it is determined whether the fourth Euclidean distance is less than a distance threshold (those skilled in the art can set the size of the distance threshold according to actual needs, which is not limited in this invention). If so, the cluster centers in the final cluster center matrix are determined as the target cluster centers. Otherwise, the current iteration number is incremented by one, and the process returns to S2052.

[0054] It should be noted that the above-mentioned update calculation method for the membership matrix and cluster center matrix is ​​based on the original update formula of the fuzzy C-means clustering algorithm, which is a conventional calculation method and will not be elaborated further in this invention.

[0055] In this embodiment of the invention, by randomly generating an initial membership matrix and iteratively calculating cluster centers, each training feature set can gradually form a stable cluster structure under fuzzy membership relationships. By calculating the distance between the training feature set and each cluster center and updating the membership matrix, the correlation between samples and cluster centers can be dynamically adjusted. By introducing cluster center movement to filter stable features and scaling the membership of stable features, the contribution of samples with clear affiliations to their corresponding cluster centers can be strengthened, while avoiding premature fixation of boundary samples. By judging convergence by the distance between the final cluster center matrix and the current round of cluster center matrices, the stability and computational efficiency of the target cluster center determination process can be improved, thereby making the subsequent call behavior judgment based on the target cluster centers more accurate and reliable.

[0056] S206, calculate the first Euclidean distance between each training feature set and its corresponding nearest target cluster center, and sort the first Euclidean distances in ascending order. Based on the first ascending order sorting results, calculate the normal call threshold and the malicious call threshold:

[0057] Among them, T normal This represents the normal call threshold, Q1 and Q3 represent the first and third quartile values ​​of the Euclidean distance sequence, respectively, || represents taking the absolute value, and T risk This indicates the threshold for malicious calls.

[0058] It should be noted that the aforementioned normal call threshold and malicious call threshold are constructed based on the distance distribution of historical normal call behavior in the cluster space. Specifically, the distance between each training feature set and the nearest target cluster center can characterize its deviation from the normal behavior cluster center. After arranging the distance values ​​in ascending order, an Euclidean distance sequence representing the deviation distribution of historical normal call behavior can be formed. Since the quartile statistical method can reflect the concentrated interval and discrete range of the distance distribution and is not easily affected by a small number of abnormal samples or extreme distance values, this embodiment of the invention uses the Tukey quartile method to determine the first and third quartile values ​​based on the Euclidean distance sequence, and calculates the normal call threshold and malicious call threshold accordingly. Therefore, the threshold division result can be adapted to the actual distribution of historical normal call behavior, improving the accuracy of distinguishing between normal calls, suspicious calls, and malicious calls.

[0059] S207, extract features from incoming call information data to form a set of features of incoming call information to be monitored, and calculate the second Euclidean distance between the set of features of incoming call information to be monitored and the corresponding nearest target cluster center.

[0060] Optionally, the set of features for monitored incoming call information includes number attribute features, call time features, number location features, operator features, device identification features, call type features, and historical call behavior features. Among these, number attribute features characterize number type, number range, abnormal number associations, and the historical association between the calling and called numbers. Call time features characterize the time period of the call, date type, call frequency changes, and abnormal call times. Number location and operator features characterize the geographical distribution and operator association between the calling and called parties. Device identification features characterize the number usage of the same device, the device's historical call frequency, and device-related risks. Call type features characterize incoming, outgoing, transferred, or VoIP call types. Historical call behavior features characterize the number of calls, call intervals, connection rate, rejection rate, average call duration, number of short-term high-frequency calls, number of different called numbers, and number of historical risk markers within a preset historical time range. These features are obtained by combining data after field parsing, format standardization, category coding, statistical analysis, and normalization of the incoming call information data.

[0061] S208, compare the second Euclidean distance with the normal call threshold and the malicious call threshold respectively to determine the call behavior of the call to be monitored.

[0062] Specifically, when the second Euclidean distance is less than the normal call threshold, the call to be monitored is determined to be a normal call. When the second Euclidean distance is greater than or equal to the normal call threshold and less than the malicious call threshold, the call to be monitored is determined to be a suspicious call. When the second Euclidean distance is greater than or equal to the malicious call threshold, the call to be monitored is determined to be a malicious call.

[0063] In this embodiment of the invention, a clustering space representing normal call behavior can be formed by first constructing a training feature set based on historical normal call behavior data and then using a fuzzy C-means clustering algorithm to determine target cluster centers. Adaptively selecting the target number of clusters using a clustering effectiveness coefficient improves the stability and rationality of the clustering results. Calculating normal call thresholds and malicious call thresholds based on historical normal call distance distributions allows the threshold divisions to be adapted to the actual call behavior distribution. By comparing the distance between the features of the incoming call information to be monitored and the nearest target cluster center with the aforementioned thresholds, the call behavior of the call to be monitored can be quickly determined, thereby achieving graded screening of normal calls, suspicious calls, and malicious calls, providing an accurate entry point for subsequent behavior deviation review and voice risk monitoring.

[0064] S3, based on incoming call information data, uses a neural decomposition autoencoder to determine the behavioral deviation score of the call to be monitored, and updates the suspicious call behavior to malicious call behavior if the behavioral deviation score is greater than or equal to the behavioral deviation threshold.

[0065] In one possible implementation, S3 specifically includes sub-steps S301 to S305: S301, Search and embed the categorical and numerical features in the set of features to be monitored to obtain the embedding vector of each feature in the set of features to be monitored.

[0066] S302, calculate the normalized attention weights for each incoming call information feature to be monitored.

[0067] S303, based on the embedding vector and normalized attention weights, fuses first-order weighted features and second-order feature interactions through a bidirectional interaction layer in the neural decomposition autoencoder to obtain a high-order feature interaction representation of the call to be monitored:

[0068]

[0069]

[0070] in, Indicates the first High-order feature interaction representation at any given time. This represents the first-order weighted feature. This represents second-order feature interaction, where Z represents the number of incoming call information features to be monitored. Indicates the first The normalized attention weight of the z-th incoming call information feature to be monitored at time z. Indicates the first The embedding vector of the z-th incoming call information feature to be monitored at time z, where exp represents the exponential function. Indicates the first Learnable parameters of the z-th incoming call information feature to be monitored at time z ( (obtained by training a neural decomposition autoencoder), ψ z The embedding matrix (ψ) represents the feature of the z-th incoming call information to be monitored. z (Obtained by training a neural decomposition autoencoder). Indicates the first The original feature value of the z-th incoming call information feature to be monitored at time z. This represents a real matrix with d rows and 1 columns. Represents a d-dimensional real vector (optionally, Those skilled in the art can set the size of d according to actual needs; this invention does not impose any limitations on it.

[0071] It should be noted that the aforementioned high-order feature interaction representation is constructed based on the characteristic that various features in call information data have both independent influence and combined correlation. Specifically, the call information features to be monitored include categorical features such as number attributes, call time, location, operator, device identifier, and call type, as well as numerical features such as call frequency, call interval, connection rate, average call duration, and number of historical risk markers. Since different features contribute differently to call behavior risk, this embodiment of the invention maps different types of features to a unified vector space through embedding processing and highlights key feature information through attention weights. On this basis, first-order weighted features are used to characterize the independent contribution of a single call information feature, and second-order feature interaction is used to characterize the combined relationship between different call information features, thereby depicting abnormal call behavior patterns formed by multi-feature correlation.

[0072] S304, input the high-order feature interaction representation into the encoder in the neural decomposition autoencoder to obtain the low-dimensional call behavior representation.

[0073] S305, calculate the fifth Euclidean distance between the low-dimensional call behavior representation and the low-dimensional call behavior representation corresponding to each normal call behavior in the cloud data, and determine the average of each fifth Euclidean distance as the behavior deviation score.

[0074] Furthermore, if the behavior deviation score is greater than or equal to the behavior deviation threshold, the suspicious call behavior is updated to malicious call behavior. If the behavior deviation score is less than the behavior deviation threshold, the suspicious call behavior is updated to normal call behavior.

[0075] In this embodiment of the invention, by embedding categorical and numerical features from the set of incoming call information features to be monitored, incoming call information features of different data types can be uniformly converted into vector representations suitable for neural network processing. By calculating normalized attention weights, key features that contribute significantly to the judgment of call behavior deviation can be highlighted, reducing the interference of weakly correlated features on the judgment results. By fusing first-order weighted features and second-order feature interactions through a bidirectional interaction layer, the independent influence of a single incoming call information feature and the combined abnormal relationship between multiple incoming call information features can be simultaneously represented, thereby obtaining a more expressive high-order feature interaction representation. By mapping this high-order feature interaction representation to a low-dimensional call behavior representation through an encoder, and comparing the distance with the low-dimensional representation corresponding to normal call behavior, the degree of deviation of the monitored call from normal call behavior can be quantified. Thus, suspicious call behavior can be further verified, reducing boundary sample misjudgments caused by relying solely on clustering distance, and improving the accuracy of updating suspicious call behavior to normal or malicious call behavior.

[0076] S4. Speaker separation is performed on the call voice data to obtain call voice segments from multiple speakers, and the call voice segments are converted into corresponding call voice text.

[0077] Specifically, speech activity detection is performed on the speech stream data corresponding to the call speech data to identify speech intervals and non-speech intervals. Noise reduction, echo cancellation, and endpoint detection are then applied to the speech intervals to obtain multiple effective speech intervals. Each effective speech interval is segmented according to a preset time window to obtain multiple candidate speech frames. Acoustic features of each candidate speech frame are extracted, including at least one of Mel-spectrum features, speaker embedding features, fundamental frequency features, energy features, and speech rate features. Speaker similarity is calculated between different candidate speech frames based on their speaker embedding features. Based on speaker similarity, candidate speech frames are clustered, and those meeting preset similarity conditions are grouped into the same speaker category, resulting in speech frame sets corresponding to different speakers. Candidate speech frames belonging to the same speaker category and with a time interval less than a preset threshold are concatenated according to their temporal order in the speech stream data, and boundary smoothing is performed to obtain call speech segments carrying speaker identifiers, start timestamps, and end timestamps. Each call speech segment is input into a speech recognition model according to its start timestamp to obtain the corresponding call speech text.

[0078] In this embodiment of the invention, by performing voice activity detection, preprocessing, voiceprint similarity clustering, and voice frame splicing on the call voice data, the mixed voice stream can be accurately divided into call voice segments corresponding to different speakers, and interference from silence, noise, and invalid voice can be removed for subsequent recognition. By configuring speaker identifiers and timestamps for each call voice segment and converting them into call voice text, speaker-level voice content attribution can be achieved, providing an accurate data foundation for subsequently calculating risk monitoring scores for different speakers.

[0079] S5 inputs the call audio segments and call audio text into the multi-branch call audio risk monitoring model and outputs the speaker's risk monitoring score, which represents the combined risk level of audio and text risks.

[0080] In one possible implementation, the multi-branch call voice risk monitoring model specifically includes: a voice segment risk monitoring branch, a voice-text risk monitoring branch, and a fusion layer. The voice segment risk monitoring branch is specifically used to: perform risk monitoring on call voice segments and obtain a voice segment monitoring score. The voice-text risk monitoring branch is specifically used to: perform risk monitoring on call voice text and obtain a voice-text monitoring score. The fusion layer is specifically used to: perform weighted fusion of the voice segment monitoring score and the voice-text monitoring score to obtain a speaker risk monitoring score that characterizes the overall risk level of voice and text risks.

[0081] In this embodiment of the invention, by setting up a speech segment risk monitoring branch and a speech text risk monitoring branch, the speaker's risk level can be identified from both acoustic features and text semantics dimensions, avoiding missed or false judgments caused by relying solely on a single modality. By weightedly fusing the speech segment monitoring score and the speech text monitoring score through a fusion layer, a more comprehensive and stable speaker risk monitoring score can be obtained, thereby improving the accuracy of subsequent call risk score calculation and monitoring result determination.

[0082] In one possible implementation, the speech segment risk monitoring branch is further used for: According to the preset analysis window, the call audio segment is slidably segmented to obtain multiple analysis audio windows, and acoustic features are extracted from each analysis audio window to obtain multiple acoustic risk feature vectors.

[0083] Optionally, the preset analysis window can be set to a value range of [0.5s, 5s]. Those skilled in the art can set the size of the preset analysis window according to actual needs, but this invention does not limit it.

[0084] Specifically, overlapping sliding segments are performed on the time axis of the call speech segment according to a preset analysis window and a preset sliding step size. The speech data covered by each sliding segment is determined as an analysis speech window. The preset sliding step size is less than or equal to the preset analysis window size to maintain continuous coverage between adjacent analysis speech windows. Each analysis speech window undergoes pre-emphasis, framing, windowing, fast Fourier transform, and filtering. At least one of the following acoustic features is extracted: Mel spectral features, Mel frequency cepstral coefficient features, fundamental frequency features, energy features, speech rate features, pause features, spectral centroid features, spectral bandwidth features, zero-crossing rate features, and voiceprint embedding features. Statistical processing is performed on each type of acoustic feature to obtain the corresponding acoustic risk feature vector. Among them, Mel spectral features and Mel frequency cepstral coefficient features are used to characterize the speech spectrum distribution; fundamental frequency features and energy features are used to characterize speech prosody and intensity changes; speech rate features and pause features are used to characterize speaking rhythm; spectral centroid features, spectral bandwidth features, and zero-crossing rate features are used to characterize frequency domain and time domain changes; and voiceprint embedding features are used to characterize the speaker's acoustic characteristics.

[0085] Risk state mapping is performed on each acoustic risk feature vector to obtain multiple local risk states.

[0086] By performing time-series correlation calculations on each local risk state, multiple global risk time-series states are obtained.

[0087] Based on the global risk time sequence status, calculate the risk propagation intensity of each analysis voice window.

[0088] Attention-weighted aggregation of each risk propagation intensity yields the risk aggregation intensity of the call audio segment.

[0089] The risk aggregation intensity is converted into a speech segment monitoring score using the Sigmoid function:

[0090]

[0091]

[0092] Among them, S audio Indicates the score of the speech segment monitoring. R represents the intensity of risk aggregation, R represents the number of analysis speech windows, and α represents the risk aggregation intensity. r S represents the attention weight of the r-th speech analysis window. r Let q represent the risk propagation intensity of the r-th analysis speech window, q represent the global attention query vector, and the subscript T represent the transpose operation. r This represents the global risk time series status of the r-th analysis voice window. Indicates the first The global risk temporal state of the analysis speech window, where σ represents the Gaussian kernel bandwidth parameter (optionally, σ is 1), tanh represents the tanh activation function, and W... h h represents the current state weight matrix for time-series correlation calculation. r W represents the local risk state of the r-th analysis voice window. g b represents the historical state weight matrix for time-series correlation calculation. g φ represents the bias vector for time-series correlation computation, and φ represents the nonlinear activation function (optionally, the nonlinear activation function is the ReLU activation function). W f F represents the weight matrix for mapping risk states. r Let b represent the acoustic risk feature vector of the r-th analysis speech window. f The bias vector representing the risk state mapping should be noted. h W g b g W f and b f It was obtained through training the speech segment risk monitoring branch.

[0093] It should be noted that the above-mentioned calculation method for speech segment monitoring scores is based on the continuity, correlation, and cumulative nature of call speech risks over time. Specifically, the acoustic risk features of each analyzed speech window are used to characterize the local speech state. By mapping these features to local risk states and combining them with temporal relationships to obtain the global risk temporal state, the contextual risk meaning of each analyzed speech window within the entire call speech segment can be characterized. Based on this, by calculating the risk propagation intensity, the continuous occurrence and correlated diffusion of risky speech states among multiple analyzed speech windows can be reflected. Attention-weighted aggregation can highlight the contribution of key risk windows to the overall risk assessment. Converting the risk aggregation intensity into a speech segment monitoring score using the Sigmoid function yields quantitative speech risk results that are easy to integrate and compare with thresholds.

[0094] In this embodiment of the invention, by sliding segmenting the speech segments according to a preset analysis window, continuous speech can be converted into multiple analysis speech windows with a temporal order, facilitating fine-grained monitoring of speech risks. By extracting acoustic risk features such as Mel spectrum, fundamental frequency, energy, speech rate, pauses, spectral features, and voiceprint embedding, speech anomalies can be characterized from multiple dimensions, including timbre, prosody, rhythm, energy changes, and speaking state. By mapping acoustic risk features to local risk states and further performing temporal correlation calculations, continuous risks can be identified by combining contextual speech changes. By calculating the risk propagation intensity and performing attention-weighted aggregation, key speech windows with significant risk contributions can be highlighted, reducing the impact of local noise or transient anomalies on the overall judgment. By converting the risk aggregation intensity into speech segment monitoring scores, quantifiable and fusionable speech risk results can be obtained, thereby improving the accuracy and stability of speaker-level speech risk monitoring.

[0095] One possible implementation is a speech-text risk monitoring branch based on the Chinese BERT model.

[0096] The voice and text risk monitoring branch is also specifically used for: Text preprocessing is performed on the voice text of the call to obtain a text word sequence. Text feature extraction, pinyin feature extraction, and character shape feature extraction are then performed on each text word in the text word sequence to obtain text embedding features, pinyin embedding features, and character shape embedding features.

[0097] Specifically, the call audio text undergoes processes such as removing invalid symbols, converting between simplified and traditional Chinese characters, unifying capitalization, standardizing punctuation, and replacing special placeholders to obtain standardized call audio text. Following the word segmentation rules of the Chinese BERT model, the standardized call audio text is segmented into word units, and sentence-beginning and sentence-end markers are added to form a text word unit sequence. For each text word unit, text embedding features are obtained by querying a pre-trained text embedding matrix. A pinyin conversion dictionary maps the Chinese characters in the text word units to pinyin sequences, and the pinyin sequences are encoded and convolutional feature extracted to obtain pinyin embedding features. Finally, the Chinese characters in the text word units are converted into glyph images or glyph codes, and input into a glyph feature extraction network for stroke, structure, and contour feature extraction to obtain glyph embedding features.

[0098] Feature fusion is performed on text embedding features, pinyin embedding features, and character shape embedding features to obtain multimodal fusion features.

[0099] The multimodal fusion features are input into the Transformer encoder of the ChineseBERT model, and the contextual semantic representation of each text word is output through a multi-layer hybrid self-attention mechanism.

[0100] Pooling is performed on the semantic representations of each context to obtain the global semantic aggregation features of the call speech text. The global semantic aggregation features are then input into a multilayer perceptron to output the binary log probability of whether the call speech text is normal or risky text.

[0101] The binary log-odds ratio is converted into a speech-text detection score using the Softmax function.

[0102]

[0103] Among them, S text denoted by , e represents the base of the natural logarithm, p0 represents the log-odds probability that the spoken text is normal text, p1 represents the log-odds probability that the spoken text is risky text, p represents the binary classification log-odds vector, W2 represents the weight matrix of the second layer of the multilayer perceptron, ReLU represents the ReLU activation function, W1 represents the weight matrix of the first layer of the multilayer perceptron, h represents the global semantic aggregation feature, b1 represents the bias vector of the first layer of the multilayer perceptron, and b2 represents the bias vector of the second layer of the multilayer perceptron. It should be noted that W2, W1, b1, and b2 are obtained through training on the spoken text risk detection branch.

[0104] It should be noted that the above-mentioned speech-text monitoring score is obtained based on the semantic risk binary classification results of the call speech text. Specifically, after obtaining global semantic aggregation features, the speech-text risk monitoring branch outputs the discrimination result of whether the call speech text belongs to normal text or risk text through a multilayer perceptron. Since this discrimination result only represents the relative tendency of the category, it is difficult to use it directly as a risk score with a uniform scale. Therefore, this embodiment of the invention uses the Softmax function to perform normalization processing, converting the discrimination result corresponding to risk text into a speech-text monitoring score within a preset range.

[0105] In this embodiment of the invention, by constructing a speech-text risk monitoring branch based on the ChineseBERT model, the semantic, pinyin, and glyph information of Chinese text can be fully utilized to improve the ability to identify risk expressions in Chinese speech text. By standardizing and segmenting the speech text, the impact of invalid symbols, format differences, and text noise on the model's recognition results can be reduced. By extracting text embedding features, pinyin embedding features, and glyph embedding features respectively, and performing multimodal fusion, the model's adaptability to homophones, near-homophones, similar-looking characters, and text with speech recognition errors can be enhanced. By learning contextual semantic representations through the Transformer encoder, the system can determine whether the speech text contains risky semantics such as inducement, deception, threats, or abnormal prompts, based on the preceding and following context. By outputting speech-text monitoring scores through pooling, multilayer perceptrons, and the Softmax function, textual semantic risks can be converted into quantifiable and comparable risk results, facilitating fusion with speech segment monitoring scores and improving the accuracy of speaker risk monitoring scores and speech risk determination results.

[0106] S6 determines the call risk score of the call to be monitored based on the risk monitoring scores of each speaker, and outputs the monitoring results of the call to be monitored based on the call risk scores.

[0107] In one possible implementation, S6 specifically refers to: The maximum risk monitoring score is determined as the call risk score of the call to be monitored. The call risk score is compared with the risk score threshold, and the monitoring results of the call to be monitored are output. The monitoring results include no-risk calls, low-risk calls, medium-risk calls, and high-risk calls.

[0108] Optionally, the risk score threshold includes a first risk score threshold and a second risk score threshold, and the first risk score threshold is smaller than the second risk score threshold. Those skilled in the art can set the size of each risk score threshold according to actual needs, and the present invention does not limit this.

[0109] Specifically, if the monitored call is a normal call, its monitoring result is determined to be a risk-free call, and a green safety icon is displayed on the current call interface, maintaining the normal call status. When the call risk score is less than the first risk score threshold, its monitoring result is determined to be a low-risk call, a yellow risk warning icon is displayed, and the call risk score, the risk monitoring scores of each speaker, the call voice text, and caller information data are stored together. When the call risk score is greater than or equal to the first risk score threshold and less than the second risk score threshold, its monitoring result is determined to be a medium-risk call, an orange risk warning icon is displayed, and a medium-risk warning message is sent to the called party's terminal, the call monitoring platform, or the manual review terminal, while the relevant call data is marked and stored. When the call risk score is greater than or equal to the second risk score threshold, its monitoring result is determined to be a high-risk call, a red high-risk alarm icon is displayed, and a high-risk alarm message is sent to the called party's terminal, the call monitoring platform, or the risk handling terminal to perform at least one of the following risk handling operations: risk reminder, call restriction, call blocking, number marking, risk list update, and evidence retention.

[0110] In this embodiment of the invention, by using the maximum risk monitoring score of each speaker as the call risk score of the call to be monitored, high-risk situations caused by any speaker can be captured in a timely manner, avoiding the weakening of local high-risk signals by averaging the entire call. By comparing the call risk score with a risk score threshold and outputting monitoring results of different levels, hierarchical linkage of risk alerts, data storage, manual review, alarm reporting, and risk handling can be achieved, thereby improving the executability and efficiency of call risk monitoring results.

[0111] The beneficial effects of this invention are as follows: In this embodiment of the invention, based on incoming call information data, the call behavior of the call to be monitored is determined by a fuzzy C-means clustering algorithm. Based on the incoming call information data, the behavioral deviation score of the call to be monitored is determined by a neural decomposition autoencoder. This can be combined with incoming call information rules such as number blacklists, call frequency, and abnormal location to make a comprehensive judgment, and fully characterize the complex correlation between different incoming call features in the call behavior, thereby improving the accuracy of identifying borderline suspicious calls and reducing the probability of misjudgment or omission. By separating the speaker from the call voice data, multiple speaker voice segments are obtained, and the call voice segments are converted into corresponding call voice text. The call voice segments and call voice text are input into a multi-branch call voice risk monitoring model, which outputs a speaker risk monitoring score to characterize the comprehensive risk level of voice risk and text risk. This can make risk judgments based on the call voice segments of different speakers and the corresponding voice recognition text, distinguish the voice content and risk sources of different speakers, and integrate voice acoustic risk and text semantic risk, thereby improving the accuracy and traceability of risk monitoring results.

[0112] System Implementation Examples Reference manual attached Figure 2 The diagram shows a schematic representation of a call voice monitoring system provided in an embodiment of the present invention.

[0113] The present invention proposes a call voice monitoring system 30, comprising: a memory 303 and a processor 301.

[0114] The memory 303 stores an application program adapted to be executed by the processor 301 to implement the call voice monitoring method of the method embodiment.

[0115] The voice monitoring system 30 includes a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, for example, via a bus 302.

[0116] The structure of the call voice monitoring system 30 does not constitute a limitation on the embodiments of the present invention.

[0117] Processor 301 may be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in connection with this disclosure. Processor 301 may also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0118] Bus 302 may include a pathway for transmitting information between the aforementioned components. Bus 302 may be a PCI bus or an EISA bus, etc. Bus 302 may be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the figure, but this does not mean that there is only one bus or one type of bus.

[0119] The memory 303 may be a ROM or other type of static storage device capable of storing static information and instructions, RAM or other type of dynamic storage device capable of storing information and instructions, or it may be an EEPROM, CD-ROM or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the protection scope of the present invention.

Claims

1. A method for monitoring voice during a phone call, characterized in that, The call voice monitoring method includes: S1, acquire incoming call information data and call voice data of the call to be monitored; S2, Based on the incoming call information data, determine the call behavior of the call to be monitored using a fuzzy C-means clustering algorithm; when the call to be monitored is a suspicious call behavior, proceed to S3; when the call to be monitored is a malicious call behavior, proceed to S4; S3, based on the incoming call information data, determine the behavioral deviation score of the call to be monitored through a neural decomposition autoencoder, and update the suspicious call behavior to the malicious call behavior if the behavioral deviation score is greater than or equal to the behavioral deviation threshold. S4, perform speaker separation on the call voice data to obtain call voice segments of multiple speakers, and convert the call voice segments into corresponding call voice text; S5, input the call voice segment and the call voice text into the multi-branch call voice risk monitoring model, and output the speaker's risk monitoring score, which is used to characterize the comprehensive risk level of voice risk and text risk; S6. Based on the risk monitoring scores of each speaker, determine the call risk score of the call to be monitored, and output the monitoring result of the call to be monitored based on the call risk score.

2. The call voice monitoring method according to claim 1, characterized in that, The call to be monitored is specifically a call requested or already established between the caller and the called party; the incoming call information data specifically includes the caller's number, the called party's number, the call time, the number's location, the operator information, the device identifier, the call type, and historical call behavior data; the call voice data is specifically the voice stream data between the caller and the called party.

3. The call voice monitoring method according to claim 1, characterized in that, S2 specifically includes: S201, Obtain historical call information data of multiple normal call behaviors from cloud data, and extract features from the historical call information data to form multiple training feature sets; S202, set a search interval for the number of candidate clusters, and calculate the membership concentration index and membership uncertainty entropy of each candidate cluster number in the search interval based on each of the training feature sets. S203, calculate the fuzzy deviation index and standardized membership entropy of each candidate cluster number based on the membership concentration index and the membership uncertainty entropy; S204, calculate the clustering effectiveness coefficient of each of the candidate cluster numbers based on the membership concentration index, the membership uncertainty entropy, the fuzzy deviation index, and the standardized membership entropy; S205, the number of candidate clusters corresponding to the maximum clustering effectiveness coefficient is taken as the target number of clusters in the fuzzy C-means clustering algorithm, and multiple target cluster centers are determined; S206, calculate the first Euclidean distance between each training feature set and the corresponding nearest target cluster center, sort each first Euclidean distance in ascending order, and calculate the normal call threshold and the malicious call threshold based on the first ascending order sorting result; S207, extract features from the incoming call information data to form a set of features of incoming call information to be monitored, and calculate the second Euclidean distance between the set of features of incoming call information to be monitored and the corresponding nearest target cluster center; S208, compare the second Euclidean distance with the normal call threshold and the malicious call threshold respectively to determine the call behavior of the call to be monitored.

4. The call voice monitoring method according to claim 3, characterized in that, The determination of multiple target cluster centers in S205 specifically includes: S2051, Randomly generate an initial membership matrix, and calculate the current round cluster center matrix based on the initial membership matrix for the current iteration number; S2052, calculate the third Euclidean distance between each of the training feature sets and each cluster center in the current round cluster center matrix, and sort all the third Euclidean distances corresponding to each of the training feature sets in ascending order; S2053, Update the current round membership matrix according to each of the third Euclidean distances, and calculate the temporary cluster center matrix according to the updated membership matrix; S2054, Calculate the cluster center movement between each cluster center in the current round cluster center matrix and each cluster center in the temporary cluster center matrix; S2055, based on the second ascending sorting result and the maximum cluster center movement, select stable features from each of the training feature sets to form a stable feature set; S2056, Calculate the membership scaling parameter for each stable feature in the stable feature set, and use the membership scaling parameter to scale the membership of the updated membership matrix. S2057, Calculate the final cluster center matrix based on the membership matrix after scaling. S2058, calculate the fourth Euclidean distance between the final cluster center matrix and the current round cluster center matrix, and compare the fourth Euclidean distance with a distance threshold to determine the target cluster center.

5. The call voice monitoring method according to claim 1, characterized in that, S3 specifically includes: S301, Search and embed the categorical features and numerical features in the set of features to be monitored to obtain the embedding vector of each feature to be monitored in the set of features to be monitored. S302, calculate the normalized attention weights of each of the incoming call information features to be monitored; S303, based on the embedding vector and the normalized attention weight, the first-order weighted features and the second-order feature interaction are fused through the bidirectional interaction layer in the neural decomposition autoencoder to obtain the high-order feature interaction representation of the call to be monitored. S304, The high-order feature interaction representation is input into the encoder in the neural decomposition autoencoder to obtain a low-dimensional call behavior representation; S305, calculate the fifth Euclidean distance between the low-dimensional call behavior representation and the low-dimensional call behavior representation corresponding to each normal call behavior in the cloud data, and determine the average value of each fifth Euclidean distance as the behavior deviation score.

6. The call voice monitoring method according to claim 1, characterized in that, The multi-branch call voice risk monitoring model specifically includes: a voice segment risk monitoring branch, a voice text risk monitoring branch, and a fusion layer; the voice segment risk monitoring branch is specifically used to: perform risk monitoring on the call voice segment and obtain a voice segment monitoring score; the voice text risk monitoring branch is specifically used to: perform risk monitoring on the call voice text and obtain a voice text monitoring score; the fusion layer is specifically used to: perform weighted fusion of the voice segment monitoring score and the voice text monitoring score to obtain the speaker's risk monitoring score, which characterizes the comprehensive risk level of voice risk and text risk.

7. The call voice monitoring method according to claim 6, characterized in that, The voice segment risk monitoring branch is also specifically used for: According to the preset analysis window, the call voice segment is slidably segmented to obtain multiple analysis voice windows, and acoustic features are extracted from each of the analysis voice windows to obtain multiple acoustic risk feature vectors. Risk state mapping is performed on each of the acoustic risk feature vectors to obtain multiple local risk states; Perform temporal correlation calculations on each of the local risk states to obtain multiple global risk temporal states; Based on the global risk time sequence status, calculate the risk propagation intensity of each of the analysis voice windows; The risk aggregation intensity of the call audio segment is obtained by performing attention-weighted aggregation on each of the aforementioned risk propagation intensities. The risk aggregation intensity is converted into a monitoring score for the speech segment using the Sigmoid function.

8. The call voice monitoring method according to claim 6, characterized in that, The speech-text risk monitoring branch is specifically a speech-text risk monitoring branch based on the Chinese BERT model; the speech-text risk monitoring branch is also specifically used for: The voice text of the call is preprocessed to obtain a text word sequence, and text feature extraction, pinyin feature extraction and character shape feature extraction are performed on each text word in the text word sequence to obtain text embedding features, pinyin embedding features and character shape embedding features; The text embedding features, the pinyin embedding features, and the character shape embedding features are fused to obtain multimodal fusion features; The multimodal fusion features are input into the Transformer encoder of the ChineseBERT model, and the contextual semantic representation of each text word is output through a multi-layer hybrid self-attention mechanism. Pooling is performed on each of the context semantic representations to obtain the global semantic aggregation features of the call voice text. The global semantic aggregation features are then input into a multilayer perceptron to output the binary log probability of whether the call voice text is normal text or risky text. The binary log odds are converted into the speech-text monitoring score using the Softmax function.

9. The call voice monitoring method according to claim 1, characterized in that, Specifically, S6 is: The maximum risk monitoring score is determined as the call risk score of the call to be monitored, and the call risk score is compared with the risk score threshold to output the monitoring result of the call to be monitored. The monitoring result includes no-risk call, low-risk call, medium-risk call and high-risk call.

10. A call voice monitoring system, characterized in that, include: Memory and processor; The memory stores an application program adapted to be executed by the processor to implement the call voice monitoring method according to any one of claims 1 to 9.

Images