Multi-role collaborative customer service intelligent agent method

Abstract

The application relates to the technical field of artificial intelligence, and discloses a multi-role cooperative customer service intelligent agent method, which comprises the following steps: step S101, constructing a frame window mapping table; step S102, extracting features to obtain a video phase track; step S103, extracting features to obtain an audio phase track; step S104, obtaining linear scale parameters through linear scale mapping, and calculating global consistency scores; step S105, obtaining a video type through a phase perception model; and step S106, determining a breakpoint index in a suspicious window. Through linear scale mapping and global consistency scores, the application can accurately judge the cross-modal correlation degree of audio and video, accurately identify the video type by combining the phase perception model, position local abnormal breakpoints by using a sliding window, facilitate artificial rapid review, improve the efficiency of remote business handling in a customer service scene, and break through the bottleneck of low efficiency of traditional single-mode verification and artificial review.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, more specifically, it relates to a multi-role cooperative customer service intelligent agent method. BACKGROUND

[0002] With the popularization of artificial intelligence in the field of customer service, remote business handling usually relies on video interaction containing audio, such as online account opening, fault repair, etc., and video authenticity becomes the key to risk control; among them, the video that is forged, spliced or played back may lead to identity misjudgment, business fraud and other problems, therefore, the authenticity of the video needs to be strictly verified.

[0003] The existing video verification method of the customer service intelligent agent has the following obvious limitations:

[0004] 1. Usually relying on single modal data features, it is difficult to utilize the common physical correlation of audio and video, in actual scenes, lighting devices will produce periodic brightness fluctuations due to the influence of power grid frequency, and audio devices will also produce associated fluctuations due to the influence of power frequency radiation, the audio and video power frequency features of real videos should have a stable corresponding relationship, and splicing and playing back videos will destroy this correlation, making it difficult to accurately identify the abnormal type of the video;

[0005] 2. Lack of quality pre-control, when the video brightness fluctuation is weak and the audio noise is strong, the power frequency features are masked, and direct analysis may lead to misjudgment;

[0006] 3. Low efficiency of local anomaly positioning, even if the overall video anomaly is found, manual frame-by-frame inspection of splicing or editing breakpoints is still needed, and in the case of large amount of customer service scene videos, manual cost is high and time-consuming, which cannot quickly support business decision-making.

[0007] The above problems lead to insufficient accuracy and low efficiency of the customer service intelligent agent in video authenticity verification, which not only affects the business handling speed, but also makes it difficult to effectively prevent fraud risks, becoming a key bottleneck for the intelligent upgrading of customer service scenes. SUMMARY

[0008] The present application provides a multi-role cooperative customer service intelligent agent method to solve the technical problems in the background art.

[0009] The present application provides a multi-role cooperative customer service intelligent agent method, comprising the following steps:

[0010] Step S101, obtaining a target video containing audio and its metadata, and establishing a frame window set based on video frames, so that each video frame is associated with the audio samples in its time interval to construct a frame window mapping table;

[0011] The metadata includes: video frame rate, audio sampling rate, total number of video frames, frame-by-frame timestamp sequence and total number of audio samples;

[0012] Step S102, feature extraction is performed on each video frame to obtain a video phase trajectory;

[0013] Step S103, feature extraction is performed on the audio samples in the time interval of each video frame according to the frame window mapping table to obtain an audio phase trajectory, and a video side signal-to-noise ratio and an audio side signal-to-noise ratio are respectively calculated and obtained;

[0014] Step S104, linear scale mapping of the audio phase trajectory to the video phase trajectory is established to obtain linear scale parameters, and a global consistency score is calculated and obtained accordingly;

[0015] The linear scale parameters include a scale parameter and an offset parameter;

[0016] Step S105, a feature sequence is constructed based on the video phase trajectory and the audio phase trajectory, and is input into the phase-aware model after training, and a type of the target video is output;

[0017] The type of the target video includes normal, splicing, playback and screen shooting;

[0018] Step S106, a local consistency score is calculated through a sliding window, and if the local consistency score is less than a third threshold value, the local consistency score is marked as a suspicious window, and a breakpoint index in the suspicious window is determined according to a phase difference change rate.

[0019] Further, the time interval of each video frame is determined as a frame window, and the timestamp of each audio sample is determined, the frame index of each video frame is taken as a key, and the audio samples whose timestamps fall within the frame window corresponding to each video frame are taken as values to construct a frame window mapping table;

[0020] The time interval of the kth video frame is represented as: wherein 1≤k≤K, K represents the total number of video frames, represents the starting time of the kth video frame in the frame timestamp sequence, represents the video frame rate, wherein 1≤n≤N, N represents the total number of audio samples, represents the audio sampling rate.

[0021] Further, feature extraction is performed on each video frame to obtain a video phase trajectory, including the following steps:

[0022] Step S201, each video frame is subjected to grayscale processing to obtain a grayscale matrix;

[0023] Step S202, calculate the average value of all pixel gray values of each gray matrix as the brightness value, calculate the average value of the brightness values of all gray matrices as the global brightness value, and construct a brightness sequence by subtracting the global brightness value from the brightness value corresponding to each gray matrix;

[0024] Step S203, apply band-pass filtering to the brightness sequence to retain the frequency components related to the power frequency, and obtain a band-pass brightness sequence;

[0025] Step S204, convert the band-pass brightness sequence into an analytic signal by Hilbert transform, extract its principal value phase and eliminate periodic jumps, and obtain a video phase trajectory;

[0026] Each element value of the video phase trajectory is represented by a real scalar, corresponding to the video phase state of a video frame.

[0027] Further, apply band-pass filtering to the audio sample sequence in the time interval of each video frame, retain the frequency components related to the power frequency, and aggregate the filtered audio samples in each frame window by taking the average value to obtain a band-pass audio sequence. Convert the band-pass audio sequence into an analytic signal by Hilbert transform, extract its principal value phase and eliminate periodic jumps, and obtain an audio phase trajectory. The total number of elements of the audio phase trajectory is the same as that of the video phase trajectory. Each element value of the audio phase trajectory is represented by a real scalar, corresponding to the audio phase state of a video frame.

[0028] Perform Fourier transform on the band-pass brightness sequence and the band-pass audio sequence respectively to obtain the corresponding energy spectrum density, and perform integral calculation on the energy spectrum density of the main peak frequency point and its neighborhood to obtain the video side main peak power and the audio side main peak power. Then, perform integral calculation on the energy spectrum density corresponding to the band-pass brightness sequence and the band-pass audio sequence to obtain the video passband total power and the audio passband total power. Subtract the video side main peak power from the video passband total power to obtain the video side residual power, and subtract the audio side main peak power from the audio passband total power to obtain the audio side residual power. Then, calculate the video side signal-to-noise ratio and the audio side signal-to-noise ratio.

[0029] Further, take the minimum value of the video side signal-to-noise ratio and the audio side signal-to-noise ratio, and if the value is less than a first threshold value, enter the artificial review queue, wherein the first threshold value is a self-defined parameter.

[0030] Further, construct a linear equation by combining the video phase trajectory and the audio phase trajectory, and obtain the scale parameter and the offset parameter of the linear equation by least squares closed-form solution. The calculation formula of the linear equation is as follows:

[0031] where 1≤i≤L, L represents the total number of elements of the video phase trajectory or the audio phase trajectory, and respectively represent the i-th element value of the video phase trajectory and the audio phase trajectory, a represents a scale parameter, and b represents an offset parameter.

[0032] Further, a residual sequence is calculated according to the video phase trajectory, the audio phase trajectory, the scale parameter, and the offset parameter, and a global consistency score is calculated according to the residual sequence, and if the global consistency score is greater than or equal to a second threshold value, global consistency is output, otherwise global inconsistency is output, wherein the second threshold value is a self-defined parameter.

[0033] The i-th element value of the residual sequence is calculated according to the following formula:

[0034] wherein 1≤i≤L, L represents the total number of elements of the video phase trajectory or the audio phase trajectory, and respectively represent the i-th element value of the video phase trajectory and the audio phase trajectory, a represents a scale parameter, and b represents an offset parameter.

[0035] The global consistency score is calculated according to the following formula:

[0036] wherein represents the sample variance of the residual sequence, represents the sample variance of the video phase trajectory.

[0037] Further, the sequence unit number of the feature sequence is the same as the total number of elements of the video phase trajectory and the audio phase trajectory, and the sequence unit of the feature sequence is represented by a feature vector, and the feature vector is composed of the element value of the video phase trajectory, the element value of the audio phase trajectory, the circular domain phase difference, the video phase increment, the audio phase increment, the frequency lock error, the video phase encoding, and the audio phase encoding.

[0038] The circular domain phase difference of the i-th sequence unit corresponding to the feature vector is calculated according to the following formula:

[0039] wherein represents the i-th element value of the residual sequence, represents the floor function;

[0040] The video phase increment and the audio phase increment respectively represent the difference between the current element value and the previous element value of the video phase trajectory and the audio phase trajectory.

[0041] The frequency lock error is equal to the video phase increment minus the product of the scale parameter and the audio phase increment.

[0042] The video phase encoding of the feature vector corresponding to the i-th sequence unit The calculation formula is as follows:

[0043] , wherein represents a video signal-to-noise ratio, represents the i-th element value of the video phase trajectory, represents a video phase adjustment parameter, which is a self-defined parameter;

[0044] The audio phase encoding of the feature vector corresponding to the i-th sequence unit The calculation formula is as follows:

[0045] , wherein represents an audio signal-to-noise ratio, represents the i-th element value of the audio phase trajectory, represents an audio phase adjustment parameter, which is a self-defined parameter.

[0046] Further, the phase-aware model is constructed based on a gated neural network model, and the product of a weight parameter and a concentration degree is added in the calculation formula of the reset gate and the update gate;

[0047] The calculation formula of the concentration degree F is as follows:

[0048] , wherein represents a frequency lock error, and respectively represent a first weight parameter and a second weight parameter;

[0049] The sample label of the training sample for training the phase-aware model is obtained by manual labeling.

[0050] Further, the size of the sliding window is a self-defined parameter, the calculation formula of the local consistency score and the global consistency score is the same, the third threshold value is a self-defined parameter, and the frame index of the video frame corresponding to the largest change rate of the element value of the residual sequence corresponding to the sliding window is selected as the breakpoint index by calculating the change rate.

[0051] The application has the beneficial effects that: the application establishes audio-video time correlation based on video frames, extracts audio-video power frequency phase trajectory and calculates signal-to-noise ratio, first, low-quality data is shunted to manual review through quality gating to avoid misjudgment diffusion; then, the linear scale mapping and global consistency scoring are used to accurately judge the audio-video cross-modal correlation degree, and the phase perception model is used to realize accurate identification of video types, and the sliding window is used to locate the local abnormal breakpoint to facilitate manual rapid review; the application uses power frequency physical characteristics to ensure the accuracy of verification, reduces the artificial cost through quality pre-control, intelligent type identification and breakpoint positioning, and the consistent conclusion and abnormal evidence can support multi-role closed-loop cooperation, effectively prevent business fraud risk, improve the efficiency of remote business handling in customer service scenarios, and break through the bottleneck of traditional single modal verification and low artificial review efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0052] Fig. 1 is a flowchart of a multi-role cooperative customer service agent method of the application;

[0053] Fig. 2 is a flowchart of the application for feature extraction on each video frame to obtain video phase trajectory;

[0054] Fig. 3 is a test result graph of the phase perception model of the application. DETAILED DESCRIPTION

[0055] The subject matter described herein will now be discussed with reference to example implementations. It should be understood that discussions of these implementations are intended to serve as examples only and that alterations to the function and arrangement of elements discussed can be made without departing from the scope of the content of this specification. Various examples can omit, substitute, or add various procedures or components as appropriate, or in appropriate combination. Also, features described with respect to some examples can be combined in other examples.

[0056] It should be noted that the technical terms or scientific terms used in one or more embodiments of the present application should be understood as the general meaning understood by those skilled in the art of the field to which the present application belongs, unless otherwise defined. The terms "first", "second", and the like used in one or more embodiments of the present application do not represent any order, number, or importance, but are only used to distinguish different components. The terms "include" or "contain" and the like mean that the elements or objects appearing before the terms encompass the elements or objects listed after the terms and their equivalents, and do not exclude other elements or objects. The terms "connected" or "connected" and the like are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms "up", "down", "left", "right", and the like are only used to represent relative positional relationships, and when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0057] As shown in Figs. 1-3 A multi-role collaborative customer service agent method, comprising the following steps:

[0058] Step S101, obtaining a target video containing audio and its metadata, and establishing a frame window set based on video frames, so that each video frame is associated with the audio samples in its time interval to construct a frame window mapping table;

[0059] The metadata includes: video frame rate, audio sampling rate, total number of video frames, frame-by-frame timestamp sequence, and total number of audio samples;

[0060] Step S102, feature extraction is performed on each video frame to obtain a video phase trajectory;

[0061] Step S103, traversing the frame window mapping table, feature extraction is performed on the audio samples in the time interval of each video frame to obtain an audio phase trajectory, and video side signal-to-noise ratio and audio side signal-to-noise ratio are calculated and obtained respectively;

[0062] Step S104, linear scale mapping of the audio phase trajectory to the video phase trajectory is established to obtain linear scale parameters, and a global consistency score is calculated accordingly;

[0063] The linear scale parameters include: scale parameters and offset parameters;

[0064] Step S105, based on the video phase trajectory and the audio phase trajectory, a feature sequence is constructed and input into a phase-aware model after training, and the type of the target video is output;

[0065] The type of the target video includes: normal, splicing, playback, and screen capture;

[0066] In step S106, the local consistency score is calculated by a sliding window. If the local consistency score is less than a third threshold, the window is marked as a suspicious window, and a breakpoint index in the suspicious window is determined according to a phase difference change rate.

[0067] In an embodiment of the present application, if it is determined that the target video does not satisfy the monotonicity check and the full-time domain coverage check, it indicates that the target video is incorrect, and the user is prompted to upload the video again. The monotonicity check means that each timestamp in the sequence of frame-by-frame timestamps must be sequentially increased. The full-time domain coverage check means that the time interval of all video frames must completely cover the total duration of the target video.

[0068] It should be noted that the video frame rate represents the number of picture frames contained in a video per second. For example, if the video frame rate is 30 frames per second, it means that the video contains 30 consecutive pictures per second. The audio sampling rate represents the number of sampling points contained in the audio per second. The total number of video frames represents the total number of pictures contained in the video. For example, if the total duration of the video is 10 seconds and the video frame rate is as described above, the corresponding total number of video frames is 300 frames. The sequence of frame-by-frame timestamps represents the start time of each video frame. For example, the 0th frame is 0 seconds, the 1st frame is 1 / 30 seconds, and so on. The total number of audio samples is equal to the total duration of the video multiplied by the audio sampling rate. For example, if the audio sampling rate is 44100 samples per second (44.1 kHz) and the total duration of the video is as described above, the total number of audio samples is 441000.

[0069] In an embodiment of the present application, the time interval of each video frame is first determined as a frame window, and then the timestamp of each audio sample is determined. The frame index of each video frame is taken as a key, and the audio sample whose timestamp falls within the frame window corresponding to each video frame is taken as a value to construct a frame window mapping table.

[0070] The time interval of the kth video frame is represented as: wherein 1≤k≤K, K represents the total number of video frames, represents the start time of the kth video frame in the sequence of frame-by-frame timestamps, represents the video frame rate. The timestamp of the nth audio sample is represented as: wherein 1≤n≤N, N represents the total number of audio samples, represents the audio sampling rate.

[0071] It should be noted that determining the time interval of each video frame can ensure that the time intervals of adjacent frames do not overlap, avoid the same audio sample being repeatedly associated with multiple frames, and enable each video frame to uniquely correspond to the audio sample within its time interval, thereby providing a strict time reference for subsequent cross-modal analysis.

[0072] In an embodiment of the present application, as shown in Fig. 2 feature extraction is performed on each video frame to obtain a video phase trajectory, including the following steps:

[0073] Step S201, each video frame is grayed to obtain a gray matrix;

[0074] Step S202, the average value of all pixel gray values of each gray matrix is calculated as a brightness value, the average value of the brightness values of all gray matrices is calculated as a global brightness value, and each gray matrix corresponding brightness value is subtracted from the global brightness value to construct a brightness sequence;

[0075] Step S203, a band-pass filter is applied to the brightness sequence to retain the frequency components related to the power frequency to obtain a band-pass brightness sequence;

[0076] The band-pass filter can be a FIR (Finite Impulse Response) band-pass filter or an IIR (Infinite Impulse Response) band-pass filter, which will not be described here;

[0077] Step S204, the band-pass brightness sequence is converted into an analytic signal by Hilbert transform, the principal value phase is extracted and the periodic jump is eliminated to obtain a video phase trajectory;

[0078] Each element value of the video phase trajectory is represented by a real scalar, corresponding to the video phase state of a video frame.

[0079] It should be noted that the global brightness value is used to represent the average light and dark level of the target video, and the brightness value of each video frame is subtracted from the global brightness value to eliminate the influence of the overall illumination level on the analysis, thereby highlighting the fluctuations of each video frame brightness relative to the global average level. The power frequency refers to the standard frequency of the power system, which is usually 50Hz or 60Hz. Influenced by the power frequency, the lighting device will produce periodic brightness fluctuations related to the power frequency, and the frequency may be the power frequency itself or its harmonics such as 100Hz, 120Hz, etc. The band-pass filter retains the frequency components related to the power frequency, effectively filtering out low-frequency disturbances such as ambient light changes and high-frequency disturbances such as sensor noise, highlighting the periodic fluctuation characteristics caused by the power frequency in the brightness sequence. The real part of the analytic signal is the original band-pass brightness sequence, and the imaginary part is the orthogonal component obtained by Hilbert transform of the sequence. The principal value phase is the angle between the analytic signal and the positive direction of the real axis in the complex plane, and its value range is between -π and +π, which is used to represent the phase state of the signal at that time. When the signal phase continuously changes over time and exceeds +π, a periodic jump occurs, i.e. from +π to -π. By phase unwrapping processing, such jumps can be eliminated, i.e. when a jump is detected, the phase value is corrected by accumulating 2π, for example, corrected to +π+2π=3π, so that the phase changes continuously and smoothly over time.

[0080] In one embodiment of the present application, a band-pass filter is applied to the audio sample sequence in the time interval in which each video frame is located, the frequency components related to the power frequency are retained, and a band-pass audio sequence is obtained by taking the average value of the filtered audio samples in each frame window and aggregating the frame window. The band-pass audio sequence is converted into an analytic signal by Hilbert transform, the principal value phase is extracted and the periodic jump is eliminated to obtain an audio phase trajectory. The total number of elements of the audio phase trajectory and the total number of elements of the video phase trajectory are the same, and each element value of the audio phase trajectory is represented by a real scalar, which corresponds to the audio phase state of a video frame.

[0081] The Fourier transform is performed on the band-pass luminance sequence and the band-pass audio sequence respectively to obtain the corresponding energy spectrum density, and the energy spectrum density of the main peak frequency point and its neighborhood is integrated to obtain the video side main peak power and the audio side main peak power. The energy spectrum density corresponding to the band-pass luminance sequence and the band-pass audio sequence is integrated to obtain the video passband total power and the audio passband total power. The video passband total power is subtracted from the video side main peak power to obtain the video side residual power, and the audio passband total power is subtracted from the audio side main peak power to obtain the audio side residual power. The video side signal-to-noise ratio and the audio side signal-to-noise ratio are calculated respectively.

[0082] The video side signal-to-noise ratio The calculation formula is as follows:

[0083] , wherein and represent the video side main peak power and the video side residual power respectively.

[0084] The audio side signal-to-noise ratio The calculation formula is as follows:

[0085] , wherein and represent the audio side main peak power and the audio side residual power respectively.

[0086] It should be noted that the neighborhood width is a self-defined parameter, and preferably, the neighborhood width is set to ±5Hz; the audio sample represents the numerical record of the audio signal at the discrete time point, i.e. the instantaneous amplitude of the audio signal. According to the total number of video frames and the total number of audio samples, the number of audio samples contained in each frame window is 1470, and the total number of elements of the audio phase trajectory and the total number of elements of the video phase trajectory are both 300; the video side signal-to-noise ratio is used to measure the quality of the luminance signal related to the power frequency in the target video, and the audio side signal-to-noise ratio is used to measure the quality of the audio signal related to the power frequency in the target video. The larger the value of the signal-to-noise ratio, the less the interference.

[0087] In one embodiment of the present application, the minimum value of the video side signal-to-noise ratio and the audio side signal-to-noise ratio is taken, and if the value is less than a first threshold value, the value is entered into a manual review queue, wherein the first threshold value is a self-defined parameter, and preferably, the first threshold value is set to 10 dB.

[0088] It should be noted that when the smaller signal-to-noise ratio is less than the first threshold value, it indicates that the power frequency related signal in the video or audio is too weak and the noise is too much, and subsequent analysis is prone to error. Therefore, direct entry into the manual review queue is to avoid misjudgment through manual inspection to ensure that the final determination result is more real and reliable. The value range of the first threshold value is usually set to be between 8 and 15 dB. For example, if the signal-to-noise ratio is less than 8 dB, it indicates that the video or audio signal is almost submerged in noise and it is difficult to extract effective information. If the signal-to-noise ratio is higher than 15 dB, it indicates that the video or audio quality is good, but the requirement may be too strict, which will exclude some samples that can still be analyzed although there is slight noise. Therefore, the first threshold value is set to 10 dB to balance the signal quality and practicability.

[0089] In one embodiment of the present application, the video phase trajectory and the audio phase trajectory are combined to construct a linear equation, and the proportional parameter and the offset parameter of the linear equation are obtained by least square closed-form solution. The calculation formula of the linear equation is as follows:

[0090] , wherein 1≤i≤L, L represents the total number of elements of the video phase trajectory or the audio phase trajectory, i.e. L=K, K represents the total number of video frames, and respectively represent the i-th element value of the video phase trajectory and the audio phase trajectory, a represents the proportional parameter, and b represents the offset parameter.

[0091] The calculation formula of the proportional parameter a by least square closed-form solution is as follows:

[0092] , wherein represents the sample covariance between the audio phase trajectory and the video phase trajectory, represents the sample variance of the audio phase trajectory.

[0093] The calculation formula of the offset parameter b by least square closed-form solution is as follows:

[0094] , wherein and respectively represent the sample mean of the video phase trajectory and the audio phase trajectory.

[0095] It should be noted that the phase trajectories of both video and audio are affected by the power frequency, but due to the different acquisition devices (e.g., a camera for video and a microphone for audio), the two usually have a stable linear relationship. By combining the two to construct a linear equation, the influence of device differences can be eliminated, allowing the two to be compared under the same standard.

[0096] In one embodiment of the present invention, a residual sequence is calculated based on the video phase trajectory, the audio phase trajectory, the scaling parameter, and the offset parameter, and a global consistency score is calculated based on the residual sequence. If the global consistency score is greater than or equal to a second threshold value, then global consistency is output; otherwise, global inconsistency is output.

[0097] The value of the i-th element of the residual sequence The calculation formula is as follows:

[0098] Where 1≤i≤L, and L represents the total number of elements in the video phase trajectory or audio phase trajectory. and Let a and b represent the i-th element values ​​of the video phase trajectory and the audio phase trajectory, respectively, where a represents the scaling parameter and b represents the offset parameter.

[0099] Global Consistency Score The calculation formula is as follows:

[0100] ,in The sample variance of the residual sequence. The sample variance representing the video phase trajectory;

[0101] The second threshold value is a custom parameter; preferably, the second threshold value is set to 0.8.

[0102] It should be noted that the residual sequence is used to measure the fitting effect of the linear equation. The smaller the residual value, the better the linear equation reflects the correspondence between video and audio, that is, the phase changes of video and audio are consistent. The sample variance of the residual sequence represents the magnitude of the deviation in the fitting of the linear equation, and the sample variance of the video phase trajectory represents the total fluctuation of the video phase itself. The smaller the ratio of the two, the closer the global consistency score is to 1, indicating that the overall consistency of the video and audio phase trajectories is higher and the correspondence is stronger. In addition, edited and spliced ​​videos and audios have lower global consistency scores due to different signal sources, and the value of the second threshold is usually set between 0.7 and 0.95, which can effectively distinguish whether the video and audio are truly synchronized and whether there is human editing.

[0103] In one embodiment of the present invention, the number of sequence units of the feature sequence is the same as the total number of elements of the video phase trajectory and the audio phase trajectory. The sequence units of the feature sequence are represented by feature vectors. The feature vectors consist of element values ​​of the video phase trajectory, element values ​​of the audio phase trajectory, circular phase difference, video phase increment, audio phase increment, frequency locking error, video phase coding and audio phase coding, that is, the feature vectors have a total of 8 dimensions.

[0104] The circular phase difference of the eigenvector corresponding to the i-th sequence unit The calculation formula is as follows:

[0105] ,in This represents the value of the i-th element in the residual sequence. Indicates rounding down;

[0106] The video phase increment and the audio phase increment represent the difference between the current element value and the previous element value of the video phase trajectory and the audio phase trajectory, respectively.

[0107] The frequency locking error is equal to the video phase increment minus the product of the scaling parameter and the audio phase increment;

[0108] Video phase coding of the feature vector corresponding to the i-th sequence unit The calculation formula is as follows:

[0109] ,in Indicates the signal-to-noise ratio on the video side. This represents the value of the i-th element of the video phase trajectory. This represents the video phase adjustment parameter, which is a custom parameter. Preferably, the video phase adjustment parameter is set to 120 (1 / rad).

[0110] The audio phase encoding of the feature vector corresponding to the i-th sequence unit The calculation formula is as follows:

[0111] ,in Indicates the signal-to-noise ratio on the audio side. This represents the value of the i-th element of the audio phase trajectory. This represents the audio phase adjustment parameter, which is a custom parameter. Preferably, the audio phase adjustment parameter is set to 160 (1 / rad).

[0112] It should be noted that the original phase trajectory is dimensioned, which can retain the periodic structure and linear trend of the phase, avoid discontinuity caused by phase jump, and improve the expression ability of the phase-locked and frequency-locked mode, provide a learnable basis for the phase perception model, and enhance the robustness of the model; the video phase adjustment parameter and the audio phase adjustment parameter are used to adjust the dimension, and are also used to control the phase encoding output to be kept in a stable numerical range, that is, when the phase amplitude is small or the signal-to-noise ratio is high, the exponential term is prone to violent increase, and keeping the stable numerical range can help to accelerate the model convergence.

[0113] In an embodiment of the application, the phase perception model is constructed based on a gated recurrent unit (GRU) model, and the product of a weight parameter and a concentration degree is added to the calculation formula of the reset gate and the update gate.

[0114] The calculation formula of the concentration degree F is as follows:

[0115] , wherein denotes a frequency-locked error, and denote a first weight parameter and a second weight parameter, respectively;

[0116] The sample label of the training sample for training the phase perception model is obtained by manual labeling.

[0117] Specifically, the number of reset gates and update gates is the same as the number of sequence units of the feature sequence, and one-to-one correspondence exists.

[0118] Specifically, the calculation formula of the tth reset gate is as follows:

[0119] , wherein denotes the tth concentration degree, denotes a hidden vector (the number of dimensions is a self-defined parameter, for example, 64) output by the (t-1)th reset gate, is assigned as 0, denotes a feature vector corresponding to the tth sequence unit of the feature sequence input by the tth reset gate, and denote a first weight parameter and a second weight parameter of the tth reset gate, respectively, denotes a bias parameter of the tth reset gate, denotes a Sigmoid activation function;

[0120] Specifically, the calculation formula of the tth update gate is as follows:

[0121] , wherein and respectively represent the first weight parameter and the second weight parameter of the tth update gate, represents the bias parameter of the tth update gate.

[0122] It should be noted that the weight parameters and the bias parameters in the phase-aware model are all learnable parameters, a cross-entropy function is selected as a loss function, and the weight parameters and the bias parameters in the phase-aware model are updated reversely by a gradient optimizer (such as Adam, RMSProp, etc.) to minimize the loss value calculated by the loss function until a maximum iteration number is reached or the loss value calculated by the loss function is within a set range, and then the training of the model is completed; in addition, the phase-aware model can also be constructed based on an LSTM (Long Short-Term Memory recurrent neural network model) and a Transfomer (transformer) model, which will not be described here.

[0123] It should be noted that the phase concentration is introduced in the reset gate and the update gate, and the phase locking strength is directly converted into an adaptive weight. The high concentration indicates that the phase difference of the video frame is stable in the circular domain, the gate is more open, and the network retains and propagates such reliable information. The low concentration indicates that the phase locking is disturbed or there is an editing trace, and the gate tends to be closed, thereby inhibiting the interference of noise frames on the state, embedding the priori of physical quantities into the time sequence memory, which can significantly improve the gradient signal-to-noise ratio and the convergence speed, reduce the sensitivity of the model to hyperparameters, and thus enhance the robustness of the model.

[0124] In an embodiment of the present application, the size of the sliding window is a self-defined parameter, preferably, the size of the sliding window is set to 1 / 10 of the total number of video frames, the calculation formulas of the local consistency score and the global consistency score are the same, and the third threshold value is a self-defined parameter, preferably, the third threshold value is set to 0.75. By calculating the element value change rate of the residual sequence corresponding to the sliding window, the frame index of the video frame corresponding to the maximum change rate is selected as the breakpoint index.

[0125] It should be noted that the breakpoint index indicates that the corresponding video frame is likely to have splicing, playback, and screen shooting, and in addition, a plurality of frame indexes of video frames with large change rates can be selected for manual review.

[0126] In an embodiment of the present application, after the calculation of the phase trajectory and the signal-to-noise ratio is completed, the system enters a parallel branch: one calculates the global consistency score and outputs whether the second threshold value is met; the other reviews and identifies the type of the target video through the phase-aware model and outputs the video type; the third calculates the local consistency score through the sliding window and locates the breakpoint, and outputs the suspicious window and the breakpoint index; only when all the threshold values are met and the phase-aware model does not find any abnormalities, an in-situ consistent conclusion is given, and the case is transferred to technical support for further processing, thereby ensuring the closed-loop cooperation of multiple roles based on the same evidence.

[0127] As shown in Fig. 3 The phase-aware model after training is tested to obtain a confusion matrix, and the number of test samples of each video type is 100, wherein the test accuracy rates of the video types of normal, splicing, playback and screen capture are 96%, 96%, 95% and 97%, respectively.

[0128] It should be noted that the interval and the threshold size are set for ease of comparison, wherein the size of the threshold depends on the number of sample data and the base number set by the person skilled in the art for each group of sample data, as long as it does not affect the proportional relationship between the parameters and the quantized values. And the above formula is a dimensionless calculation of the value, and the formula is obtained by software simulation of a large amount of data to obtain a formula of the nearest real situation, and the preset parameters in the formula are set by the person skilled in the art according to the actual situation.

[0129] The above describes the embodiments of the present embodiment, but the present embodiment is not limited to the specific embodiments described above, and the specific embodiments described above are only illustrative but not limiting, and a person of ordinary skill in the art can make many forms under the inspiration of the present embodiment, which all belong to the protection of the present embodiment.

Claims

1. A multi-role collaborative customer service agent method, characterized in that, The method comprises the following steps: Step S101, obtaining a target video containing audio and metadata thereof, and establishing a frame window set based on video frames, so that each video frame is associated with an audio sample in a time interval thereof to construct a frame window mapping table; The metadata comprises a video frame rate, an audio sampling rate, a total number of video frames, a timestamp sequence of each frame, and a total number of audio samples; Step S102, performing feature extraction on each video frame to obtain a video phase trajectory; Step S103, traversing the frame window mapping table to perform feature extraction on the audio samples in the time interval of each video frame to obtain an audio phase trajectory, and respectively calculating a video side signal-to-noise ratio and an audio side signal-to-noise ratio; Step S104, establishing a linear scale mapping of the audio phase trajectory to the video phase trajectory to obtain a linear scale parameter, and calculating a global consistency score according to the linear scale parameter; The linear scale parameter comprises a scale parameter and an offset parameter; Step S105, constructing a feature sequence based on the video phase trajectory and the audio phase trajectory, inputting the feature sequence into a phase perception model trained to be completed, and outputting a type of the target video; The type of the target video comprises normal, splicing, playback, and screen capture; Step S106, calculating a local consistency score through a sliding window, determining whether the local consistency score is less than a third threshold value, and if so, marking the sliding window as a suspicious window, and determining a breakpoint index in the suspicious window according to a phase difference change rate. 2.The multi-role collaborative customer service agent method of claim 1, wherein, First, the time interval of each video frame is determined as a frame window, and then the timestamp of each audio sample is determined, each video frame index is taken as a key, and the audio samples whose timestamps fall within the frame window corresponding to each video frame are taken as values to construct a frame window mapping table; The time interval of the kth video frame is represented as: where 1≤k≤K, K represents the total number of video frames, represents the starting time of the kth video frame in the sequence of frame-by-frame timestamps, represents the video frame rate; the timestamp of the nth audio sample is represented as: where 1≤n≤N, N represents the total number of audio samples, represents the audio sampling rate.

3. The multi-role collaborative customer service agent method of claim 1, wherein, The feature extraction on each video frame to obtain a video phase trajectory comprises the following steps: Step S201, performing grayscale processing on each video frame to obtain a grayscale matrix; Step S202, calculating the average value of all pixel grayscale values of each grayscale matrix as a luminance value, calculating the average value of the luminance values of all grayscale matrices as a global luminance value, and constructing a luminance sequence by subtracting the global luminance value from the luminance value corresponding to each grayscale matrix; Step S203, applying a band-pass filter to the luminance sequence to retain frequency components related to the power frequency, and obtaining a band-pass luminance sequence; Step S204, converting the band-pass luminance sequence into an analytic signal through Hilbert transform, extracting the principal value phase thereof, and eliminating periodic jumps to obtain a video phase trajectory; Each element value of the video phase trajectory is represented by a real scalar, corresponding to the video phase state of a video frame.

4. The multi-role collaborative customer service agent method of claim 3, wherein, A band-pass filter is applied to the audio sample sequence in the time interval of each video frame to retain frequency components related to the power frequency, and a band-pass audio sequence is obtained by taking the average value of the filtered audio samples in each frame window for frame window aggregation, the band-pass audio sequence is converted into an analytic signal through Hilbert transform, the principal value phase thereof is extracted, and periodic jumps are eliminated to obtain an audio phase trajectory, the total number of element values of the audio phase trajectory is the same as that of the video phase trajectory, and each element value of the audio phase trajectory is represented by a real scalar, corresponding to the audio phase state of a video frame; The Fourier transform is performed on the band-pass luminance sequence and the band-pass audio sequence respectively to obtain corresponding energy spectrum densities, and the energy spectrum densities of the main peak frequency point and its neighborhood are integrated to obtain the video-side main peak power and the audio-side main peak power, respectively; the energy spectrum densities corresponding to the band-pass luminance sequence and the band-pass audio sequence are integrated to obtain the video passband total power and the audio passband total power, respectively; the video passband total power is subtracted by the video-side main peak power to obtain the video-side residual power, and the audio passband total power is subtracted by the audio-side main peak power to obtain the audio-side residual power; and the video-side signal-to-noise ratio and the audio-side signal-to-noise ratio are calculated respectively.

5. The multi-role collaborative customer service agent method of claim 1, wherein, The minimum value of the video-side signal-to-noise ratio and the audio-side signal-to-noise ratio is taken, and if the value is less than a first threshold value, it is determined that the value is less than the first threshold value, and then the artificial review queue is entered, wherein the first threshold value is a self-defined parameter.

6. The multi-role collaborative customer service agent method of claim 1, wherein, The video phase trajectory and the audio phase trajectory are combined to construct a linear equation, and the proportional parameter and the offset parameter of the linear equation are obtained by least square closed-form solution, and the calculation formula of the linear equation is as follows: where 1≤i≤L, L represents the total number of elements of the video phase trajectory or the audio phase trajectory, and respectively represent the i-th element value of the video phase trajectory and the audio phase trajectory, a represents a proportional parameter, and b represents an offset parameter.

7. The multi-role collaborative customer service agent method of claim 6, wherein, The residual sequence is calculated according to the video phase trajectory, the audio phase trajectory, the proportional parameter and the offset parameter, and the global consistency score is calculated according to the residual sequence, and if the global consistency score is greater than or equal to a second threshold value, it is determined that the global consistency score is greater than or equal to the second threshold value, and then the global consistency is output, otherwise the global inconsistency is output, wherein the second threshold value is a self-defined parameter; The value of the i-th element of the residual sequence The formula for calculating the value of the i-th element of the residual sequence is as follows: where 1≤i≤L, L represents the total number of elements of the video phase trajectory or the audio phase trajectory, and respectively represent the i-th element value of the video phase trajectory and the audio phase trajectory, a represents a proportional parameter, and b represents an offset parameter; Global consistency score The formula for calculating the global consistency score is as follows: wherein denotes the sample variance of the residual sequence, denotes the sample variance of the video phase trajectory. 8.The multi-role collaborative customer service agent method of claim 7, wherein, The number of sequence units of the feature sequence is the same as the total number of elements of the video phase trajectory and the audio phase trajectory, the sequence units of the feature sequence are represented by feature vectors, and the feature vectors are composed of the element value of the video phase trajectory, the element value of the audio phase trajectory, the circular domain phase difference, the video phase increment, the audio phase increment, the frequency lock error, the video phase encoding and the audio phase encoding; Phase difference of the circular domain corresponding to the feature vector of the i-th sequence unit The calculation formula is as follows: wherein denotes the value of the i-th element of the residual sequence, denotes the floor function; The video phase increment and the audio phase increment respectively represent the difference between the current element value and the previous element value of the video phase trajectory and the audio phase trajectory; The frequency lock error is equal to the video phase increment minus the product of the proportional parameter and the audio phase increment; Video phase encoding of the i-th sequence unit corresponding to the feature vector The calculation formula is as follows: wherein represents a video side signal-to-noise ratio, represents the i-th element value of a video phase trajectory, represents a video phase adjustment parameter, being a user-defined parameter; audio phase encoding of the i-th sequence unit corresponds to the feature vector The calculation formula is as follows: wherein represents the audio side signal-to-noise ratio, represents the i-th element value of the audio phase trajectory, represents the audio phase adjustment parameter, is a user-defined parameter. 9.The multi-role collaborative customer service agent method of claim 8, wherein, The phase perception model is constructed based on a gated neural network model, and the product of the weight parameter and the concentration is added to the calculation formula of the reset gate and the update gate; The calculation formula of the concentration F is as follows: wherein denotes a frequency lock error, and denotes a first weight parameter and a second weight parameter, respectively; The sample labels of the training samples used to train the phase perception model are obtained by manual annotation.

10. The multi-role collaborative customer service agent method of claim 7, wherein, The size of the sliding window is a self-defined parameter, the calculation formula of the local consistency score and the global consistency score is the same, the third threshold value is a self-defined parameter, the element value change rate of the residual sequence corresponding to the sliding window is calculated, and the frame index of the video frame corresponding to the maximum change rate is selected as the breakpoint index.

Images