Illegal behavior detection method applied to online review and related equipment

Abstract

The invention belongs to the technical field of audio processing, and relates to an illegal behavior detection method applied to online review and related equipment, and the method comprises the steps: carrying out the segmentation operation of to-be-detected video data, and carrying out the verbal skill recognition operation of a to-be-detected video segment according to an asr verbal skill model, and obtaining the segment verbal skill content; performing template matching operation on each segment verbal skill content in a verbal skill template database to obtain a matched video segment; performing voiceprint recognition operation on the matched video clip to obtain a matched voiceprint clip; performing similarity comparison on the matched voiceprint fragment and the voiceprint of the to-be-detected video fragment to obtain a similarity comparison result; and determining whether the video to be detected has an illegal behavior according to a similarity comparison result. According to the voiceprint-based illegal behavior early-warning framework, whether the early-warning video clip is similar to other video clips of the same video is compared, so that whether the video to be detected has the illegal behavior is determined, and the false alarm probability can be effectively reduced.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to a method and related equipment for detecting violations used in online audits. Background Art

[0002] Due to the convenience of the Internet, many face-to-face audit scenarios do not require the interviewer to be present on-site for the audit, but only require online audits. However, online audits have many security issues, such as customers answering questions on their behalf or receiving instructions from others. These cheating behaviors may cause great economic losses to the company's insurance and credit companies.

[0003] In order to eliminate these economic losses, it is now popular to use some artificial intelligence algorithms to detect these violations. For example, when the customer answers, detect whether the mouth moves, whether the customer's mouth movement and voice are in sync, etc. These algorithms greatly reduce the risk of customer cheating.

[0004] However, the applicant found that these algorithms also have some problems. To determine whether the mouth is moving, a mouth movement model is needed. To determine whether the customer's mouth and voice are in sync, a multimodal model of lip synchronization is needed. Due to poor video quality and short speech, these models often have many false positives in order to ensure the recall rate, and these false positives bring great manual review costs. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a method and related equipment for detecting violations in online audits, so as to solve the problem that false alarms often occur in traditional violation detection methods.

[0006] In order to solve the above technical problems, the present application embodiment provides a method for detecting illegal behaviors applied to online review, which adopts the following technical solutions:

[0007] Receiving a violation detection request sent by a user terminal, wherein the violation detection request carries video data to be detected;

[0008] Performing a segmentation operation on the video data to be detected to obtain video segments to be detected;

[0009] Perform speech recognition operation on the video clips to be detected according to the ASR speech model to obtain the speech content of each video clip to be detected;

[0010] Reading a speech template database, and performing a template matching operation on each of the speech content segments in the speech template database to obtain a matching video segment;

[0011] Performing a voiceprint recognition operation on the matching video segment to obtain a matching voiceprint segment;

[0012] Comparing the matching voiceprint segment with the voiceprint of the video segment to be detected respectively to obtain a similarity comparison result;

[0013] If the similarity comparison results all meet the preset similarity threshold, then confirming that the video data to be detected is normal behavior;

[0014] If the similarity comparison result does not meet the preset similarity threshold, the video data to be detected is confirmed to be a violation.

[0015] Furthermore, after the step of performing a segmentation operation on the video data to be detected to obtain the video segment to be detected and before the step of performing a voiceprint recognition operation on the matching video segment to obtain the matching voiceprint segment, the following step is also included:

[0016] Performing a proxy answering behavior detection operation on each video clip to be detected to obtain a proxy answering behavior detection result;

[0017] If the proxy answering behavior detection result is suspected proxy answering, the to-be-detected video segment suspected of proxy answering is determined as the matching video segment;

[0018] If the proxy answering behavior detection result is normal, it is determined that the video clip to be detected does not belong to the proxy answering behavior.

[0019] Furthermore, the step of performing the proxy answering behavior detection operation on each video clip to be detected to obtain the proxy answering behavior detection result specifically includes the following steps:

[0020] According to a pre-established answering behavior feature model, a plurality of video frame images of the video segment to be detected are obtained;

[0021] For each acquired video frame image, extract the human posture features and facial expression features;

[0022] Input the extracted human posture features and facial expression features into a pre-trained support vector machine classification model for classification judgment, and obtain a judgment result of whether there is a proxy answering behavior in each video frame;

[0023] According to the judgment result of each video frame, it is determined whether there is any answering behavior in the video segment where the video frame is located.

[0024] Furthermore, after the step of segmenting the video data to be detected to obtain the video segments to be detected, the following steps are also included:

[0025] The video clips to be detected are preprocessed respectively according to the speech enhancement technology based on deep learning to obtain high-quality video clips to be detected.

[0026] Furthermore, the step of performing a segmentation operation on the video data to be detected to obtain the video segments to be detected specifically includes the following steps:

[0027] Segmenting the video data to be detected according to preset time intervals or key frames to obtain a plurality of video segments;

[0028] Perform feature extraction on each video clip to obtain a feature vector for each video clip;

[0029] Performing classification operations on the feature vectors of each video clip according to a pre-built video content classification model to obtain a classification result;

[0030] Obtain labels corresponding to each classification result from a preset label library to obtain label information for each video clip;

[0031] The video segments with consistent label information are merged to obtain the video segment to be detected.

[0032] Furthermore, the step of comparing the similarity between the matching voiceprint segment and the voiceprint of the video segment to be detected to obtain a similarity comparison result specifically includes the following steps:

[0033] Performing a sound extraction operation on the video segment to be detected to obtain audio data of the video segment to be detected;

[0034] Preprocessing the audio data to obtain preprocessed audio data;

[0035] Extracting voiceprint features from the preprocessed audio data according to a speech feature extraction algorithm to obtain a voiceprint feature vector of the video clip to be detected;

[0036] The matching voiceprint feature vector of the matching voiceprint segment is obtained, and the similarity between the matching voiceprint feature vector and the voiceprint feature vector of the video segment to be detected is calculated according to a preset similarity algorithm to obtain the similarity comparison result.

[0037] In order to solve the above technical problems, the embodiment of the present application further provides a violation detection device for online review, which adopts the following technical solution:

[0038] A request receiving module, configured to receive a violation detection request sent by a user terminal, wherein the violation detection request carries video data to be detected;

[0039] A segmentation module, used for segmenting the video data to be detected to obtain video segments to be detected;

[0040] A speech recognition module is used to perform speech recognition operations on the video segments to be detected according to the ASR speech model to obtain the segment speech content corresponding to each video segment to be detected;

[0041] A template matching module is used to read a speech template database and perform a template matching operation on each speech content of the segment in the speech template database to obtain a matching video segment;

[0042] A voiceprint recognition module, configured to perform a voiceprint recognition operation on the matching video segment to obtain a matching voiceprint segment;

[0043] A similarity comparison module is used to compare the similarity between the matching voiceprint segment and the voiceprint of the video segment to be detected, and obtain a similarity comparison result;

[0044] A first comparison result module, configured to confirm that the video data to be detected is normal behavior if the similarity comparison results all meet a preset similarity threshold;

[0045] The second comparison result module is used to confirm that the video data to be detected is a violation if the similarity comparison result does not meet a preset similarity threshold.

[0046] Furthermore, the device also includes:

[0047] The proxy answering behavior detection module is used to perform a proxy answering behavior detection operation on each video clip to be detected to obtain a proxy answering behavior detection result;

[0048] A first proxy answering behavior detection result module is used to determine the suspected proxy answering to be detected video segment as the matching video segment if the proxy answering behavior detection result is suspected proxy answering;

[0049] The second proxy answering behavior detection result module is used to determine that the video clip to be detected does not belong to the proxy answering behavior if the proxy answering behavior detection result is normal.

[0050] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0051] It includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the above-mentioned method for detecting illegal behaviors applied to online review are implemented.

[0052] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0053] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the violation detection method applied to online review as described above are implemented.

[0054] The present application provides a violation detection method for online audit, including: receiving a violation detection request sent by a user terminal, wherein the violation detection request carries video data to be detected; performing a segmentation operation on the video data to be detected to obtain video segments to be detected; performing a speech recognition operation on the video segments to be detected according to an ASR speech model to obtain segment speech content corresponding to each video segment to be detected; reading a speech template database, and performing a template matching operation on each segment speech content in the speech template database to obtain a matching video segment; performing a voiceprint recognition operation on the matching video segment to obtain a matching voiceprint segment; performing a similarity comparison between the matching voiceprint segment and the voiceprint of the video segment to be detected to obtain a similarity comparison result; if the similarity comparison results all meet a preset similarity threshold, then the video data to be detected is confirmed to be normal behavior; if the similarity comparison results do not meet the preset similarity threshold, then the video data to be detected is confirmed to be a violation. Compared with the prior art, the present application uses a voiceprint-based violation warning framework to compare the warning video clip with other video clips of the same video to determine whether they are similar, thereby confirming whether there is any violation in the video to be detected, which can effectively reduce the probability of false alarms. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0057] Figure 2 It is a flowchart of the implementation of the violation detection method applied to online review provided by the embodiment of the present application;

[0058] Figure 3 It is a structural diagram of a violation detection device applied to online review provided by an embodiment of the present application;

[0059] Figure 4It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0061] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0062] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0063] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0064] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0065] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0066] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0067] It should be noted that the violation detection method for online review provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the violation detection device for online review is generally set in the server / terminal device.

[0068] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0069] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for detecting violations applied to online audit according to the present application. The method for detecting violations applied to online audit includes: step S201, step S202, step S203, step S204, step S205, step S206 and step S207.

[0070] In step S201, a violation detection request sent by a user terminal is received, wherein the violation detection request carries video data to be detected.

[0071] In the embodiments of the present application, the user terminal refers to a terminal device used to execute the image processing method for preventing document abuse provided by the present application. The user terminal can be a mobile terminal such as a mobile phone, a smart phone, a laptop computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a navigation device, etc., as well as a fixed terminal such as a digital TV, a desktop computer, etc. It should be understood that the examples of user terminals here are only for the convenience of understanding and are not used to limit the present application.

[0072] In an embodiment of the present application, the user inputs a behavior detection request through their terminal device (such as a mobile phone, computer, etc.), and this input is received by the system. The video data to be detected is the specific content that the user wants the system to detect. Specifically, the video data to be detected can be "transaction video data or payment video data or business video data or purchase video data" related to a financial institution (such as a bank, etc.). The video data to be detected can also be medical data related to medical scenarios, such as personal health records, prescriptions, examination reports and other video data. It should be understood that the examples of video data to be detected here are only for the convenience of understanding and are not used to limit this application.

[0073] In step S202, a segmentation operation is performed on the video data to be detected to obtain video segments to be detected.

[0074] In an embodiment of the present application, a complete video is divided into many segments based on the client's speaking time.

[0075] In step S203, a speech recognition operation is performed on the video segments to be detected according to the ASR speech model to obtain the segment speech content corresponding to each video segment to be detected.

[0076] In an embodiment of the present application, an ASR model is used to identify the speech content of each video clip.

[0077] In step S204, the speech template database is read, and a template matching operation is performed on each speech content segment in the speech template database to obtain a matching video segment.

[0078] In the embodiment of the present application, because most lip synchronization only performs template matching on specified speech, it is necessary to perform template matching on specified speech.

[0079] In step S205, a voiceprint recognition operation is performed on the matching video segment to obtain a matching voiceprint segment.

[0080] In the embodiment of the present application, if a match is found, the voiceprint of the current segment is extracted and stored in the voiceprint library. The operation is repeated until the voiceprints of all video segments are stored in the voiceprint library.

[0081] In step S206, the matching voiceprint segment is compared with the voiceprint of the video segment to be detected for similarity, and a similarity comparison result is obtained.

[0082] In an embodiment of the present application, the voiceprint of the video is compared with the voiceprints of other clips of the video for similarity.

[0083] In step S207 , if the similarity comparison results all meet the preset similarity threshold, it is determined that the video data to be detected is normal behavior.

[0084] In step S208 , if the similarity comparison result does not meet the preset similarity threshold, the video data to be detected is confirmed to be a violation.

[0085] In the embodiment of the present application, assuming that the voiceprint of the matching video clip and the voiceprint of other videos are higher than the threshold α, the video is considered to have similar voiceprints to the other videos. If the voiceprints of all other clips of the same video are similar, it means that all the answers are given by the same person, and the behavior of this video is considered to be normal behavior.

[0086] In an embodiment of the present application, a violation detection method for online audit is provided, including: receiving a violation detection request sent by a user terminal, wherein the violation detection request carries video data to be detected; performing a segmentation operation on the video data to be detected to obtain a video segment to be detected; performing a speech recognition operation on the video segment to be detected according to an ASR speech model to obtain the segment speech content corresponding to each video segment to be detected; reading a speech template database, and performing a template matching operation on each segment speech content in the speech template database to obtain a matching video segment; performing a voiceprint recognition operation on the matching video segment to obtain a matching voiceprint segment; respectively comparing the matching voiceprint segment with the voiceprint of the video segment to be detected to obtain a similarity comparison result; if the similarity comparison results all meet the preset similarity threshold, then the video data to be detected is confirmed to be a normal behavior; if the similarity comparison results do not meet the preset similarity threshold, then the video data to be detected is confirmed to be a violation. Compared with the prior art, the present application is based on a voiceprint-based violation warning framework, which uses the comparison of warning video segments with other video segments of the same video to determine whether they are similar, thereby confirming whether the video to be detected has a violation, which can effectively reduce the probability of false alarms.

[0087] In some optional implementations of the embodiments of the present application, after the step of performing a segmentation operation on the video data to be detected to obtain the video segment to be detected, and before the step of performing a voiceprint recognition operation on the matching video segment to obtain the matching voiceprint segment, the following steps are also included:

[0088] Performing a proxy answering behavior detection operation on each video clip to be detected to obtain a proxy answering behavior detection result;

[0089] If the result of the proxy answering behavior detection is suspected proxy answering, the to-be-detected video segment suspected of proxy answering is determined as a matching video segment;

[0090] If the result of the proxy answering behavior detection is normal, it is determined that the video clip to be detected does not belong to the proxy answering behavior.

[0091] In the embodiments of the present application, answering on behalf of another person refers to a situation in which a participant does not answer a question in person, but someone else answers the question on his or her behalf or uses a pre-recorded answer, such as in a video call, online exam, or other scenario requiring real-time or pre-recorded answers.

[0092] In the embodiment of the present application, first, we use a specific technology or algorithm to perform a detailed analysis on each video clip to be detected. This process is the answering behavior detection operation. This operation may involve a variety of technical means, including but not limited to:

[0093] ① Facial recognition and comparison: Analyze facial features in the video and compare them with known databases or previously submitted photos to confirm whether it is the person himself;

[0094] ② Voice analysis: Perform voiceprint recognition on the voice in the video, compare the voiceprint features with known voice samples, and detect whether there is voice replacement;

[0095] ③Behavior analysis: Analyze the actions and expressions in the video to detect whether there are unnatural or mechanical behaviors, which may be related to proxy answering behavior;

[0096] ④Text comparison: If the video contains text or subtitles, you can compare it with the known answer or standard answer to detect any inconsistencies.

[0097] In the embodiment of the present application, after the above detection operation, we will obtain the answering behavior detection result of each video clip to be detected. This result may indicate whether there is suspected answering behavior in the video clip, specifically:

[0098] ① Suspected proxy answering: If the detection result is suspected proxy answering, it means that certain features or evidence that may be related to proxy answering behavior are found in the video clip. In this case, we will determine this video clip as a matching video clip, that is, a video clip that needs further review or confirmation to determine whether there is proxy answering behavior.

[0099] ② Normal: If the test result is normal, it means that no features or evidence related to proxy answering behavior were found in the video clip. Therefore, we can determine that this video clip does not belong to proxy answering behavior and no further review is required.

[0100] In the embodiment of the present application, video clips that may contain answering behavior can be effectively screened out, and corresponding measures can be taken to review or process them.

[0101] In some optional implementations of the embodiments of the present application, the above-mentioned step of performing the proxy answering behavior detection operation on each video clip to be detected and obtaining the proxy answering behavior detection result specifically includes the following steps:

[0102] According to the pre-established answering behavior feature model, multiple video frame images of the video segment to be detected are obtained;

[0103] For each acquired video frame image, extract the human posture features and facial expression features;

[0104] The extracted human posture features and facial expression features are input into the pre-trained support vector machine classification model for classification judgment, and the judgment result of whether there is answering behavior on behalf of others is obtained in each video frame;

[0105] According to the judgment result of each video frame, it is determined whether there is any proxy answering behavior in the video segment where the video frame is located.

[0106] In an embodiment of the present application, first, according to a pre-established feature model of answering behavior on behalf of another, multiple video frame images of a video clip to be detected are obtained; for each video frame image obtained, the coordinates of key human points therein are extracted as posture features using a human posture estimation algorithm such as OpenPose, and a facial expression recognition algorithm such as FER is used to extract facial expression feature vectors; the extracted human posture features and facial expression features are spliced ​​into a high-dimensional feature vector, which is input into a pre-trained support vector machine SVM classification model for classification judgment; the SVM model is trained on a large number of positive and negative samples to segment samples of different categories with a hyperplane, thereby obtaining a judgment result on whether there is answering behavior on behalf of another in each video frame; if there is at least one video frame in the video clip that is judged to have answering behavior on behalf of another, then it is determined that the video clip has answering behavior on behalf of another.

[0107] In some optional implementations of the embodiments of the present application, after the step of segmenting the video data to be detected to obtain the video segments to be detected, the following steps are also included:

[0108] According to the deep learning-based speech enhancement technology, the video clips to be detected are preprocessed respectively to obtain high-quality video clips to be detected.

[0109] In the embodiment of the present application, in order to ensure the accuracy and reliability of the subsequent analysis, it is often necessary to pre-process these video clips. In particular, when the video contains voice content, the clarity and quality of the voice are crucial for the subsequent analysis.

[0110] In the embodiment of the present application, the preprocessing process can be: first, we need to extract the voice signal from the video clip, which involves converting the video file into an audio file, or using a specific technology to extract audio data from the video frame; after extracting the voice signal, we will use the deep learning model to perform noise analysis on the voice signal, identify and understand the noise components in the voice signal, including background noise, echo, distortion, etc.; based on the results of the noise analysis, the deep learning model will enhance the voice signal. This processing process will involve a variety of technologies such as noise suppression, echo cancellation, and voice clarity improvement. Through these technologies, the noise components in the voice signal can be significantly reduced while retaining the original characteristics and clarity of the voice. After completing the voice enhancement processing, the enhanced voice signal needs to be resynthesized into the video clip and ensure the synchronization of the voice and the video screen. This step requires the use of specific audio and video processing technologies to ensure high-quality synchronization effects.

[0111] In the embodiment of the present application, after the above preprocessing process, high-quality video clips to be detected can be obtained, and the speech signals in these video clips have been enhanced by the deep learning model and have higher clarity and quality. This helps to more accurately identify and understand the speech content in subsequent analysis, thereby improving the accuracy and reliability of the analysis.

[0112] In some optional implementations of the embodiments of the present application, the above step of performing a segmentation operation on the video data to be detected to obtain the video segments to be detected specifically includes the following steps:

[0113] Segment the video data to be detected according to preset time intervals or key frames to obtain a number of video clips;

[0114] Perform feature extraction on each video clip to obtain a feature vector for each video clip;

[0115] Performing classification operations on the feature vectors of each video clip according to a pre-built video content classification model to obtain a classification result;

[0116] Obtain labels corresponding to each classification result from a preset label library to obtain label information for each video clip;

[0117] The video segments with consistent label information are merged to obtain the video segment to be detected.

[0118] In an embodiment of the present application, when processing the video data to be detected, in order to analyze and process more efficiently, the video data will be segmented according to preset time intervals or key frames. This means that a continuous video data can be segmented according to a fixed time length (such as every 5 seconds, 10 seconds, etc.) or a specific key frame (such as scene switching, character action changes, etc.), thereby obtaining several smaller video clips. Such a segmentation operation helps to analyze the video content more carefully, and can also reduce the complexity of a single processing.

[0119] In the embodiment of the present application, after obtaining a number of video clips, the next step is to perform a feature extraction operation on each video clip. Feature extraction is an important step in video analysis, which involves extracting key information that can characterize the video content from video data. These features may include color, texture, shape, motion trajectory, etc. Through a specific algorithm or model, these features can be converted into a feature vector, that is, a vector containing multiple values, for subsequent classification and recognition operations.

[0120] In the embodiment of the present application, after extracting the feature vectors of each video clip, a pre-built video content classification model can be used to perform classification operations on these feature vectors. The video content classification model is a trained machine learning model that can determine the category to which the video clip belongs based on the input feature vectors. These categories include people, animals, scenes, events, etc. Through the classification operation, one or more category labels can be assigned to each video clip.

[0121] In the embodiment of the present application, after obtaining the classification results, it is necessary to obtain the labels corresponding to the classification results from the preset label library. The label library is a collection of multiple labels, each of which corresponds to a specific category or attribute. By searching the label library for a label that matches the classification result, a label information containing the category or attribute to which the video clip belongs can be generated for each video clip.

[0122] In the embodiment of the present application, video clips with consistent tag information are merged. This means that if multiple video clips are classified into the same category or have the same attributes (i.e., their tag information is the same), these clips can be merged into a larger video clip. Such a merging operation helps to more intuitively understand the overall structure and trend of the video content, and can also reduce the complexity of subsequent analysis.

[0123] In the embodiment of the present application, the merged video segments can be used in subsequent higher-level analysis or processing tasks.

[0124] In some optional implementations of the embodiments of the present application, the above-mentioned step of comparing the similarity of the matching voiceprint segment with the voiceprint of the video segment to be detected to obtain the similarity comparison result specifically includes the following steps:

[0125] Performing a sound extraction operation on the video segment to be detected to obtain audio data of the video segment to be detected;

[0126] Preprocessing the audio data to obtain preprocessed audio data;

[0127] According to the speech feature extraction algorithm, voiceprint features are extracted from the preprocessed audio data to obtain the voiceprint feature vector of the video clip to be detected;

[0128] The matching voiceprint feature vector of the matching voiceprint segment is obtained, and the similarity between the matching voiceprint feature vector and the voiceprint feature vector of the video segment to be detected is calculated according to a preset similarity algorithm to obtain a similarity comparison result.

[0129] In the embodiment of the present application, a video clip to be detected is obtained from a video source, and an audio and video processing library such as FFmpeg is used to decode the video clip to extract the audio data therein. The extracted audio data may contain background noise and needs to be preprocessed. De-noising algorithms such as spectral subtraction and wavelet analysis can be used to remove the background noise in the audio, and then the audio amplitude range is scaled to between [-1,1] through normalization processing to obtain preprocessed audio data.

[0130] In the embodiment of the present application, the Mel Frequency Cepstral Coefficient (MFCC) feature extraction algorithm is used to extract features from the preprocessed audio data. The MFCC algorithm converts the audio signal into the Mel frequency domain by simulating the auditory characteristics of the human ear, and then performs cepstral analysis to obtain the MFCC feature vector that reflects the spectral envelope characteristics of the audio signal, which is usually 12-20 dimensions.

[0131] In an embodiment of the present application, the extracted MFCC feature vector of the video segment to be detected is similar to the matching voiceprint feature vector of the matching voiceprint segment, wherein the similarity calculation can adopt algorithms such as Euclidean distance and cosine similarity, wherein the Euclidean distance calculates the sum of the squares of the differences between the corresponding elements of the two feature vectors, and the cosine similarity calculates the cosine value of the angle between the two feature vectors, and the obtained similarity score range is between [0,1].

[0132] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0133] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0134] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0135] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0136] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a violation detection device for online review. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0137] like Figure 3 As shown, the violation detection device 200 applied to online audit in the embodiment of the present application includes:

[0138] The request receiving module 210 is used to receive a violation detection request sent by a user terminal, wherein the violation detection request carries video data to be detected;

[0139] A segmentation module 220, configured to segment the video data to be detected to obtain a video segment to be detected;

[0140] The speech recognition module 230 is used to perform speech recognition operations on the video segments to be detected according to the ASR speech model to obtain the segment speech content corresponding to each video segment to be detected;

[0141] The template matching module 240 is used to read the speech template database and perform a template matching operation on each segment speech content in the speech template database to obtain a matching video segment;

[0142] A voiceprint recognition module 250 is used to perform a voiceprint recognition operation on the matching video segment to obtain a matching voiceprint segment;

[0143] A similarity comparison module 260 is used to compare the similarity between the matching voiceprint segment and the voiceprint of the video segment to be detected, and obtain a similarity comparison result;

[0144] A first comparison result module 270 is used to confirm that the video data to be detected is normal behavior if the similarity comparison results all meet a preset similarity threshold;

[0145] The second comparison result module 280 is used to confirm that the video data to be detected is an illegal behavior if the similarity comparison result does not meet a preset similarity threshold.

[0146] In an embodiment of the present application, a violation detection device 200 for online review is provided, including: a request receiving module 210, for receiving a violation detection request sent by a user terminal, wherein the violation detection request carries video data to be detected; a segmentation module 220, for segmenting the video data to be detected to obtain video segments to be detected; a speech recognition module 230, for performing speech recognition operations on the video segments to be detected according to an ASR speech model to obtain the segment speech content corresponding to each video segment to be detected; a template matching module 240, for reading a speech template database, and matching the speech templates in the speech template database. A template matching operation is performed on the speech content of each segment to obtain a matching video segment; a voiceprint recognition module 250 is used to perform a voiceprint recognition operation on the matching video segment to obtain a matching voiceprint segment; a similarity comparison module 260 is used to compare the matching voiceprint segment with the voiceprint of the video segment to be detected to obtain a similarity comparison result; a first comparison result module 270 is used to confirm that the video data to be detected is a normal behavior if the similarity comparison results all meet the preset similarity threshold; a second comparison result module 280 is used to confirm that the video data to be detected is a violation if the similarity comparison results do not meet the preset similarity threshold. Compared with the prior art, the violation warning framework based on voiceprints in this application is used to compare the warning video segment with other video segments of the same video to see if they are similar, so as to confirm whether the video to be detected has a violation, which can effectively reduce the probability of false alarms.

[0147] In some optional implementations of the embodiments of the present application, the above-mentioned violation detection device 200 applied to online review further includes:

[0148] The proxy answering behavior detection module is used to perform a proxy answering behavior detection operation on each video clip to be detected to obtain a proxy answering behavior detection result;

[0149] The first proxy answering behavior detection result module is used to determine the suspected proxy answering video segment to be detected as a matching video segment if the proxy answering behavior detection result is suspected proxy answering;

[0150] The second answering behavior detection result module is used to determine that the video clip to be detected does not belong to answering behavior if the answering behavior detection result is normal.

[0151] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of a computer device according to an embodiment of the present application.

[0152] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 300 having components 310-330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0153] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0154] The memory 310 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 can be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 can also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Of course, the memory 310 can also include both the internal storage unit of the computer device 300 and its external storage device. In the embodiment of the present application, the memory 310 is generally used to store the operating system and various application software installed on the computer device 300, such as computer-readable instructions for the violation detection method used for online review, etc. In addition, the memory 310 can also be used to temporarily store various data that have been output or are to be output.

[0155] The processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 320 is generally used to control the overall operation of the computer device 300. In the embodiment of the present application, the processor 320 is used to run the computer-readable instructions or process data stored in the memory 310, such as running the computer-readable instructions of the violation detection method applied to the online audit.

[0156] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.

[0157] The computer device provided in this application is based on a voiceprint violation warning framework, which compares the warning video clip with other video clips of the same video to see if they are similar, thereby confirming whether there is any violation in the video to be detected, which can effectively reduce the probability of false alarms.

[0158] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the violation detection method applied to online review as described above.

[0159] The computer-readable storage medium provided in the present application is a violation warning framework based on voiceprints. It compares the warning video clip with other video clips of the same video to determine whether they are similar, thereby confirming whether there is any violation in the video to be detected, which can effectively reduce the probability of false alarms.

[0160] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0161] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A method for detecting violations in online audits, characterized in that: The steps include: Receiving a violation detection request sent by a user terminal, wherein the violation detection request carries video data to be detected; Performing a segmentation operation on the video data to be detected to obtain video segments to be detected; Perform speech recognition operation on the video clips to be detected according to the ASR speech model to obtain the speech content of each video clip to be detected; Reading a speech template database, and performing a template matching operation on each of the speech content segments in the speech template database to obtain a matching video segment; Performing a voiceprint recognition operation on the matching video segment to obtain a matching voiceprint segment; Comparing the matching voiceprint segment with the voiceprint of the video segment to be detected respectively to obtain a similarity comparison result; If the similarity comparison results all meet the preset similarity threshold, then confirming that the video data to be detected is normal behavior; If the similarity comparison result does not meet the preset similarity threshold, the video data to be detected is confirmed to be a violation.

2. The method for detecting violations in online audit according to claim 1 is characterized in that: After the step of performing a segmentation operation on the video data to be detected to obtain the video segment to be detected and before the step of performing a voiceprint recognition operation on the matching video segment to obtain the matching voiceprint segment, the following steps are also included: Performing a proxy answering behavior detection operation on each video clip to be detected to obtain a proxy answering behavior detection result; If the proxy answering behavior detection result is suspected proxy answering, the to-be-detected video segment suspected of proxy answering is determined as the matching video segment; If the proxy answering behavior detection result is normal, it is determined that the video clip to be detected does not belong to the proxy answering behavior.

3. The method for detecting violations in online audit according to claim 2 is characterized in that: The step of performing the proxy answering behavior detection operation on each video clip to be detected to obtain the proxy answering behavior detection result specifically includes the following steps: According to a pre-established answering behavior feature model, a plurality of video frame images of the video segment to be detected are obtained; For each acquired video frame image, extract the human posture features and facial expression features; Input the extracted human posture features and facial expression features into a pre-trained support vector machine classification model for classification judgment, and obtain a judgment result of whether there is a proxy answering behavior in each video frame; According to the judgment result of each video frame, it is determined whether there is any answering behavior in the video segment where the video frame is located.

4. The method for detecting violations in online audit according to claim 1 is characterized in that: After the step of segmenting the video data to be detected to obtain the video segments to be detected, the following steps are also included: The video clips to be detected are preprocessed respectively according to the speech enhancement technology based on deep learning to obtain high-quality video clips to be detected.

5. The method for detecting violations in online audit according to claim 1 is characterized in that: The step of performing a segmentation operation on the video data to be detected to obtain the video segments to be detected specifically comprises the following steps: Segmenting the video data to be detected according to preset time intervals or key frames to obtain a plurality of video segments; Perform feature extraction on each video clip to obtain a feature vector for each video clip; Performing classification operations on the feature vectors of each video clip according to a pre-built video content classification model to obtain a classification result; Obtain labels corresponding to each classification result from a preset label library to obtain label information for each video clip; The video segments with consistent label information are merged to obtain the video segment to be detected.

6. The method for detecting violations in online audit according to claim 1, characterized in that: The step of respectively comparing the matching voiceprint segment with the voiceprint of the video segment to be detected to obtain a similarity comparison result specifically includes the following steps: Performing a sound extraction operation on the video segment to be detected to obtain audio data of the video segment to be detected; Preprocessing the audio data to obtain preprocessed audio data; Extracting voiceprint features from the preprocessed audio data according to a speech feature extraction algorithm to obtain a voiceprint feature vector of the video clip to be detected; The matching voiceprint feature vector of the matching voiceprint segment is obtained, and the similarity between the matching voiceprint feature vector and the voiceprint feature vector of the video segment to be detected is calculated according to a preset similarity algorithm to obtain the similarity comparison result.

7. A violation detection device for online audit, characterized in that: include: A request receiving module, configured to receive a violation detection request sent by a user terminal, wherein the violation detection request carries video data to be detected; A segmentation module, used for segmenting the video data to be detected to obtain video segments to be detected; A speech recognition module is used to perform speech recognition operations on the video segments to be detected according to the ASR speech model to obtain the segment speech content corresponding to each video segment to be detected; A template matching module is used to read a speech template database and perform a template matching operation on each speech content of the segment in the speech template database to obtain a matching video segment; A voiceprint recognition module, configured to perform a voiceprint recognition operation on the matching video segment to obtain a matching voiceprint segment; A similarity comparison module is used to compare the similarity between the matching voiceprint segment and the voiceprint of the video segment to be detected, and obtain a similarity comparison result; A first comparison result module, configured to confirm that the video data to be detected is normal behavior if the similarity comparison results all meet a preset similarity threshold; The second comparison result module is used to confirm that the video data to be detected is a violation if the similarity comparison result does not meet a preset similarity threshold.

8. The device for detecting violations for online auditing according to claim 7 is characterized in that: The device also includes: The proxy answering behavior detection module is used to perform a proxy answering behavior detection operation on each video clip to be detected to obtain a proxy answering behavior detection result; A first proxy answering behavior detection result module is used to determine the suspected proxy answering to be detected video segment as the matching video segment if the proxy answering behavior detection result is suspected proxy answering; The second proxy answering behavior detection result module is used to determine that the video clip to be detected does not belong to the proxy answering behavior if the proxy answering behavior detection result is normal.

9. A computer device comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the violation detection method applied to online review as described in any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the method for detecting violations applied to online review as claimed in any one of claims 1 to 6.

Images