Over-person answering-on-behalf detection method based on face-to-face video and related equipment

Abstract

The embodiment of the invention belongs to the field of video processing and the field of financial science and technology, and relates to a face audit video-based other person pickup detection method, which comprises the following steps of: obtaining audio information corresponding to a face audit video of a target object, and identifying voice information from the audio information; filtering the voice information according to preset verbal skill configuration information to obtain filtered voice segments; acquiring a video clip corresponding to the voice clip, and determining whether the target object enters a mirror or not according to the video clip; if it is determined that the target object enters the mirror, performing scene classification on the video clip to obtain a scene classification result; according to the scene classification result, performing audio and lip synchronous identification on the video clip to obtain an audio and lip synchronous identification result; and according to the audio-lip synchronization identification result, generating a others-on-behalf answering detection result corresponding to the face-to-face video. According to the method and the device, the detection accuracy of other people on-behalf answers is improved, and the detection robustness of other people on-behalf answers is enhanced.

Description

Technical Field

[0001] The present application relates to the fields of video processing and financial technology, and is applied to the scenario of detecting bystander answering questions based on face-to-face review videos, and in particular, to a bystander answering question detection method based on face-to-face review videos and related equipment. Background Art

[0002] With the rapid development of financial technology, its application in various business scenarios in the insurance industry is becoming more and more extensive, one of which is online face-to-face interview. As online face-to-face interviews have become popular in the insurance industry, anti-fraud and risk prevention and control for online face-to-face interviews should also be taken seriously. If violations in online face-to-face interviews cannot be identified in a timely manner, it may cause greater economic losses.

[0003] In online interviews, one of the common violations is proxy answering, which means that other people answer questions on behalf of the interviewer. Some existing proxy answering detection technologies can only handle some cases of proxy answering, but do not have perfect business logic and are not thorough in many aspects, resulting in low accuracy and weak robustness in proxy answering detection. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a method and related equipment for detecting bystander answering based on face-to-face video, so as to solve the technical problem that the business logic is not perfect when detecting bystander answering based on face-to-face video.

[0005] In order to solve the above technical problems, the present application embodiment provides a method for detecting bystander answering based on face-to-face video, which adopts the following technical solutions:

[0006] A method for detecting bystander answering based on face-to-face video, comprising the following steps:

[0007] Acquire audio information corresponding to the face-to-face interview video of the target object, and recognize voice information from the audio information;

[0008] Filtering the voice information according to preset speech configuration information to obtain filtered voice segments;

[0009] Obtaining a video clip corresponding to the voice clip, and determining whether the target object is in the frame according to the video clip;

[0010] If it is determined that the target object is in the frame, performing scene classification on the video clip to obtain a scene classification result;

[0011] According to the scene classification result, lip synchronization recognition is performed on the video clip to obtain a lip synchronization recognition result;

[0012] According to the lip synchronization recognition result, a bystander answering detection result corresponding to the face-to-face review video is generated.

[0013] Furthermore, if it is determined that the target object is in the frame, the step of performing scene classification on the video clip to obtain a scene classification result specifically includes:

[0014] If it is determined that the target object is in the frame, feature extraction is performed based on the video clip to obtain a scene image sample set;

[0015] Inputting the scene image sample set into a pre-trained scene classification model to obtain a classification result for each image in the scene image sample set;

[0016] According to the classification result of each image, it is determined whether the video clip belongs to a single-person scene or a multi-person scene as the scene classification result.

[0017] Furthermore, the step of performing lip synchronization recognition on the video clip according to the scene classification result to obtain the lip synchronization recognition result specifically includes:

[0018] If the video clip belongs to a single-person scene, performing lip synchronization recognition on the video clip to determine whether the mouth movement interval of the target object coincides with the video clip, as the lip synchronization recognition result;

[0019] If the video clip belongs to a multi-person scene, lip synchronization recognition is performed on the video clip to determine whether the mouth movement interval of the target object is greater than the mouth movement interval of other objects as the lip synchronization recognition result.

[0020] Furthermore, the step of generating a bystander answering detection result corresponding to the face-to-face review video according to the lip synchronization recognition result specifically includes:

[0021] When the video clip belongs to a single-person scene, if it is determined that the mouth movement interval of the target object coincides with the video clip, it is determined that the face-to-face review video passes the detection; otherwise, the face-to-face review video is marked as questionable and used as the bystander answering detection result;

[0022] When the video clip belongs to a multi-person scene, if it is determined that the mouth movement interval of the target object is greater than the mouth movement interval of other objects, the face review video is determined to have passed the detection; otherwise, the face review video is marked as questionable and used as the bystander answering detection result.

[0023] Furthermore, the speech configuration information includes a speech template and a specified speech list, and the step of filtering the voice information according to the preset speech configuration information to obtain the filtered voice segment specifically includes:

[0024] Comparing the voice information with the speech template to determine a first segment that does not conform to the speech template;

[0025] The first segment is screened according to the specified speech list to obtain a second segment matching the specified speech list as the voice segment.

[0026] Furthermore, the step of obtaining a video segment corresponding to the voice segment and determining whether the target object is in the frame according to the video segment specifically includes:

[0027] Performing target detection on the video clip according to a pre-trained target detection model to obtain a target detection result;

[0028] Determine whether the target object is in the frame according to the target detection result.

[0029] Furthermore, after the step of obtaining the video segment corresponding to the voice segment and determining whether the target object is in the frame according to the video segment, the method further includes:

[0030] If it is determined that the target object is not in the frame, it is determined that the face review video has failed the detection.

[0031] Furthermore, the step of obtaining audio information corresponding to the face-to-face interview video of the target object and identifying voice information from the audio information specifically includes:

[0032] Extracting the audio information from the face-to-face interview video;

[0033] Voice activity detection is performed on the audio information to identify and obtain the voice information.

[0034] In order to solve the above technical problems, the embodiment of the present application further provides a device for detecting bystander answering based on face-to-face video, which adopts the following technical solution:

[0035] A device for detecting bystander answering based on face-to-face video, comprising:

[0036] An acquisition module is used to acquire audio information corresponding to the face-to-face interview video of the target object, and recognize voice information from the audio information;

[0037] A filtering module, used to filter the voice information according to preset speech configuration information to obtain filtered voice segments;

[0038] A determination module, used to obtain a video segment corresponding to the voice segment, and determine whether the target object is in the frame according to the video segment;

[0039] A classification module, configured to classify the video clip into scenes if it is determined that the target object is in the frame, and obtain a scene classification result;

[0040] A recognition module, configured to perform lip synchronization recognition on the video clip according to the scene classification result to obtain a lip synchronization recognition result;

[0041] A generation module is used to generate a bystander answering detection result corresponding to the face-to-face review video according to the lip synchronization recognition result.

[0042] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0043] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the method for detecting bystander answering based on face-to-face video are implemented as described above.

[0044] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0045] The method for detecting bystander answering based on face-to-face review video disclosed in the present application obtains the audio information corresponding to the face-to-face review video of the target object, identifies the voice information from the audio information; and filters the voice information according to the preset speech configuration information to obtain the filtered voice segment; then, obtains the video segment corresponding to the voice segment, and determines whether the target object is in the frame according to the video segment; if it is determined that the target object is in the frame, the video segment is subjected to scene classification to obtain the scene classification result; then, according to the scene classification result, the video segment is subjected to lip synchronization recognition to obtain the lip synchronization recognition result; thus, according to the lip synchronization recognition result, the bystander answering detection result corresponding to the face-to-face review video is generated. The present application realizes bystander answering detection based on face-to-face review video through the coordination of multiple processes, so that when performing bystander answering detection on the face-to-face review video, a complete business logic chain is provided, the accuracy of bystander answering detection is improved, and the robustness of bystander answering detection is enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0048] Figure 2 is a flowchart of an embodiment of a method for detecting bystander answering based on face-to-face video according to the present application;

[0049] Figure 3 1 is a schematic diagram of the structure of an embodiment of a device for detecting a proxy answering question based on a face-to-face video according to the present application;

[0050] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0052] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0053] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0054] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0055] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0056] The terminal devices 101, 102, and 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, and the like.

[0057] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0058] It should be noted that the bystander answering detection method based on face-to-face video provided in the embodiment of the present application is generally executed by a server, and accordingly, the bystander answering detection device based on face-to-face video is generally set in the server.

[0059] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0060] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for detecting bystander answering based on face-to-face video according to the present application. The method for detecting bystander answering based on face-to-face video comprises the following steps:

[0061] Step S201, obtaining audio information corresponding to the face-to-face interview video of the target object, and recognizing voice information from the audio information.

[0062] In this embodiment, the electronic device (for example, Figure 1 The server shown in the figure) can send or receive data through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, Wi-Fi connection, Bluetooth connection, Wi MAX connection, Zigbee connection, UWB (Ultra Wi-Fi Band) connection, and other wireless connection methods currently known or to be developed in the future.

[0063] It should be noted that insurance is an important part of the financial industry, and financial technology has a significant impact on the insurance industry. With the development of financial technology, insurance companies' innovations in financial technology include channel innovation, product innovation, and service innovation based on the Internet. For insurance companies, whether it is products, marketing and underwriting, or claims, services and anti-fraud, they all need to combine innovations in financial technology on the basis of business scenarios in order to achieve better results.

[0064] In this embodiment, in order to achieve anti-fraud in the online face-to-face interview business, it is necessary to perform bystander answering detection on the face-to-face interview video to avoid economic losses caused by violations. First, the audio information corresponding to the face-to-face interview video of the target object can be obtained, and the voice information can be identified from the audio information. Specifically, when a customer has different face-to-face interview videos corresponding to the online face-to-face interview, when it is necessary to perform bystander answering detection on the face-to-face interview video of a certain customer, the customer can be used as the target object, and the corresponding audio information can be extracted from the face-to-face interview video of the target object by means of audio and video separation, and then voice activity detection is performed on the audio information to identify the voice information. For example, the part of the audio information that contains voice can be first identified by the VAD algorithm. This algorithm plays a very important role in voice signal processing and can identify voice segments and non-voice segments from a section of audio signal. After the part that contains voice is identified by the VAD algorithm, the voice separation method can be used to obtain the voice information.

[0065] Step S202, filtering the voice information according to preset speech configuration information to obtain filtered voice segments.

[0066] In this embodiment, after the voice information is identified from the audio information corresponding to the face-to-face video of the target object, the voice information can be filtered according to the preset speech configuration information to obtain the filtered voice segment. Specifically, the speech configuration information includes a speech template and a specified speech list, and the voice information is obtained by voice separation of the part of the audio information where the voice exists. The voice information can be filtered in sequence according to the preset speech template and the specified speech list. In the first step, the voice information can be compared with the speech template to determine the first segment that does not conform to the speech template. The speech template belongs to the fixed template of the face-to-face machine. Through this step, the sound of the face-to-face machine mixed in the voice information can be removed. In the second step, the first segment can be screened according to the specified speech list to obtain the second segment that matches the specified speech list as a voice segment. The specified speech list contains the specified speech of the business. The target object needs to use the specified speech to answer during the face-to-face interview. This step can remove the segment of the non-specified speech.

[0067] Step S203: obtaining a video segment corresponding to the voice segment, and determining whether the target object is in the frame according to the video segment.

[0068] In this embodiment, after filtering the voice information and obtaining the filtered voice segment, the video segment corresponding to the voice segment can be obtained, and whether the target object is in the frame can be determined based on the video segment. Specifically, a pre-trained target detection model can be used to perform target detection on the video segment to obtain a target detection result, and then determine whether the target object is in the frame based on the target detection result. For example, the video segment is divided into multiple video frames, and features are extracted for each video frame to obtain a character feature sample set. The character feature sample set is input into the target detection model, and the target detection model is used to determine whether the target object appears in the image of each video frame. If the target object appears in the image of each video frame, it is determined that the target object is in the frame; if the target object does not appear in the image of each video frame, it is determined that the target object is not in the frame, and the corresponding process can be executed later based on whether the target object is in the frame.

[0069] Step S204: if it is determined that the target object is in the frame, scene classification is performed on the video clip to obtain a scene classification result.

[0070] In this embodiment, when determining whether the target object is in the frame according to the video clip, if it is determined that the target object is in the frame, the video clip is subjected to scene classification to obtain a scene classification result. Specifically, if it is determined that the target object is in the frame, feature extraction can be performed according to the video clip to obtain a scene image sample set, and then the scene image sample set is input into a pre-trained scene classification model to obtain a classification result for each image in the scene image sample set, and then according to the classification result of each image, it is determined whether the video clip belongs to a single-person scene or a multi-person scene as the scene classification result, and subsequently, according to different scene classification results, lip synchronization recognition can be performed on the video clip to obtain a lip synchronization recognition result.

[0071] It is understandable that since the target subject needs to answer the questions during the face-to-face review, when the target subject is not in the frame, that is, the target subject does not appear in the video clip, it is impossible to determine the movement of his mouth, and it is also impossible to determine whether the answers are given by someone else. At this time, it can be directly determined that the face-to-face review video of the target subject has not passed the test.

[0072] Step S205: performing lip synchronization recognition on the video clip according to the scene classification result to obtain a lip synchronization recognition result.

[0073] In this embodiment, after the scene classification result is obtained, lip synchronization recognition can be performed on the video clip according to the scene classification result to obtain the lip synchronization recognition result. Specifically, according to the scene classification result, it can be determined whether the video clip belongs to a single-person scene or a multi-person scene. The judgment conditions for performing lip synchronization recognition are different. If the video clip belongs to a single-person scene, lip synchronization recognition is performed on the video clip to determine whether the mouth movement interval of the target object coincides with the video clip, which is used as the lip synchronization recognition result; if the video clip belongs to a multi-person scene, lip synchronization recognition is performed on the video clip to determine whether the mouth movement interval of the target object is greater than the mouth movement interval of other objects, which is used as the lip synchronization recognition result. For example, feature extraction is performed on the video clip to obtain the mouth features of the target object in a single-person scenario, and the mouth features of the target object and other objects in a multi-person scenario. The obtained mouth features are then input into a pre-trained mouth movement model, and the mouth movement model outputs the duration of the mouth movement of each object as the mouth movement interval. In the single-person scenario, it is determined whether the mouth movement interval of the target object coincides with the video clip, which is used as the lip synchronization recognition result; in the multi-person scenario, it is determined whether the mouth movement interval of the target object is greater than the mouth movement interval of other objects, which is used as the lip synchronization recognition result.

[0074] Step S206: Generate a bystander answering detection result corresponding to the face-to-face review video according to the lip synchronization recognition result.

[0075] In this embodiment, after obtaining the lip-sync recognition result, the bystander answering detection result corresponding to the face-to-face review video can be generated according to the lip-sync recognition result. Specifically, when the video clip belongs to a single-person scene, if it is judged that the mouth movement interval of the target object coincides with the video clip, it can be determined that the face-to-face review video of the target object has passed the detection, otherwise, that is, when it is judged that the mouth movement interval of the target object does not coincide with the video clip, the face-to-face review video is marked as doubtful as the bystander answering detection result corresponding to the face-to-face review video; when the video clip belongs to a multi-person scene, if it is judged that the mouth movement interval of the target object is greater than the mouth movement interval of other objects, it can be determined that the face-to-face review video of the target object has passed the detection, otherwise, that is, when it is judged that the mouth movement interval of the target object is less than or equal to the mouth movement interval of other objects, the face-to-face review video is marked as doubtful as the bystander answering detection result corresponding to the face-to-face review video.

[0076] It is understandable that when the face-to-face review video is marked as questionable, an early warning message needs to be sent to the administrator user end so that manual intervention can be performed to review whether there is a situation where someone else answers the question on behalf of the interview video.

[0077] This application realizes bystander answering detection based on face-to-face review videos through the coordination of multiple processes, so that when performing bystander answering detection on face-to-face review videos, there is a complete business logic chain, which improves the accuracy of bystander answering detection and enhances the robustness of bystander answering detection.

[0078] In some optional implementations of this embodiment, if it is determined that the target object is in the frame, the step of performing scene classification on the video clip to obtain the scene classification result includes:

[0079] If it is determined that the target object is in the frame, feature extraction is performed based on the video clip to obtain a scene image sample set;

[0080] Inputting the scene image sample set into a pre-trained scene classification model to obtain a classification result for each image in the scene image sample set;

[0081] According to the classification result of each image, it is determined whether the video clip belongs to a single-person scene or a multi-person scene as the scene classification result.

[0082] In this embodiment, after determining that the target object is in the frame, it is necessary to determine whether the video clip belongs to a single-person scene or a multi-person scene. Therefore, feature extraction can be performed based on the video clip to obtain a scene image sample set, and then the scene image sample set is input into a pre-trained scene classification model to obtain a classification result of each image in the scene image sample set, thereby determining whether the video clip belongs to a single-person scene or a multi-person scene based on the classification result of each image. Specifically, each video frame of the video clip can be firstly extracted, and feature extraction can be performed on the image of each video frame to obtain a scene image sample set, and then the scene image sample set can be input into a pre-trained scene classification model, and the scene classification model outputs the confidence of each image in the scene image sample set belonging to a single-person scene and a multi-person scene, and each image is judged to belong to a single-person scene or a multi-person scene based on the highest confidence, and finally a judgment is made. If there is an image of a video frame in the video clip that belongs to a multi-person scene, then the video clip is determined to belong to a multi-person scene; if the image of each video frame in the video clip belongs to a single-person scene, then the video clip is determined to belong to a single-person scene.

[0083] This application uses a scene classification model to determine whether a video clip belongs to a single-person scene or a multi-person scene. As one of the processes of bystander answering detection based on face-to-face review videos, it improves the business logic chain of bystander answering detection for face-to-face review videos, and ultimately improves the accuracy of bystander answering detection.

[0084] In some optional implementations of this embodiment, the step of performing lip synchronization recognition on the video clip to obtain a lip synchronization recognition result includes:

[0085] If the video clip belongs to a single-person scene, performing lip synchronization recognition on the video clip to determine whether the mouth movement interval of the target object coincides with the video clip, as the lip synchronization recognition result;

[0086] If the video clip belongs to a multi-person scene, lip synchronization recognition is performed on the video clip to determine whether the mouth movement interval of the target object is greater than the mouth movement interval of other objects as the lip synchronization recognition result.

[0087] In this embodiment, after obtaining the scene classification result, it can be determined whether the video clip belongs to a single-person scene or a multi-person scene according to the scene classification result. If the video clip belongs to a single-person scene, lip synchronization recognition is performed on the video clip to determine whether the mouth movement interval of the target object coincides with the video clip, which is used as the lip synchronization recognition result; if the video clip belongs to a multi-person scene, lip synchronization recognition is performed on the video clip to determine whether the mouth movement interval of the target object is larger than the mouth movement intervals of other objects, which is used as the lip synchronization recognition result. Specifically, after obtaining the scene classification result, it is necessary to perform lip synchronization recognition on the video clip, and the judgment condition of lip synchronization recognition depends on whether the video clip belongs to a single-person scene or a multi-person scene. Taking the single-person scene as an example, each video frame of the video clip can be extracted, and the image of each video frame can be feature extracted to obtain the mouth features of the target object, and then the mouth features are input into the pre-trained mouth movement model. The mouth movement model can determine the mouth movement amplitude value according to the mouth features, and identify whether the mouth moves according to the mouth movement amplitude value, thereby outputting the duration of the mouth movement as the mouth movement interval. By judging whether the mouth movement interval of the target object coincides with the video clip, it can be explained whether the mouth movement of the target object is synchronized with the video clip. Similarly, for multi-person scenes, it is necessary to extract the mouth features of the target object and other objects, and then output the mouth movement intervals of the target object and other objects according to the mouth movement model. By judging whether the mouth movement interval of the target object is greater than the mouth movement intervals of other objects, it can be explained whether the target object is answering.

[0088] This application combines scene classification with lip synchronization recognition to determine whether the target person answered the questions in the face-to-face review video. This is part of the process of bystander answering detection based on the face-to-face review video. It improves the business logic chain of bystander answering detection for the face-to-face review video and ultimately improves the accuracy of bystander answering detection.

[0089] In some optional implementations of this embodiment, the step of generating a bystander answering detection result corresponding to the face-to-face review video according to the lip synchronization recognition result includes:

[0090] When the video clip belongs to a single-person scene, if it is determined that the mouth movement interval of the target object coincides with the video clip, it is determined that the face-to-face review video passes the detection; otherwise, the face-to-face review video is marked as questionable and used as the bystander answering detection result;

[0091] When the video clip belongs to a multi-person scene, if it is determined that the mouth movement interval of the target object is greater than the mouth movement interval of other objects, the face review video is determined to have passed the detection; otherwise, the face review video is marked as questionable and used as the bystander answering detection result.

[0092] In this embodiment, since the scene classification result will affect the judgment condition of lip synchronization recognition, after the lip synchronization recognition result of the video clip is obtained, the corresponding process is executed according to the lip synchronization recognition result. When the video clip belongs to a single-person scene, if it is judged that the mouth movement interval of the target object coincides with the video clip, then the face review video of the target object is determined to have passed the detection; otherwise, the face review video is marked as doubtful and used as the bystander answering detection result corresponding to the face review video; when the video clip belongs to a multi-person scene, if it is judged that the mouth movement interval of the target object is larger than the mouth movement interval of other objects, then the face review video of the target object is determined to have passed the detection; otherwise, the face review video is marked as doubtful and used as the bystander answering detection result corresponding to the face review video. Specifically, when the video clip belongs to a single-person scene, there is only a target object in it. At this time, it is only necessary to determine whether the mouth movement interval of the target object coincides with the video clip, which can indicate whether its mouth movement is synchronized with the video clip. When it is determined that the mouth movement interval of the target object coincides with the video clip, it is determined that the face review video of the target object has passed the detection. Alternatively, when it is determined that the mouth movement interval of the target object does not coincide with the video clip, the face review video is marked as doubtful. In either case, it is used as the detection result of the bystander answering corresponding to the face review video; when the video clip belongs to a multi-person scene, there are other objects in addition to the target object. At this time, it is only necessary to determine whether the mouth movement interval of the target object is greater than the mouth movement intervals of other objects, which can indicate whether the target object is answering (speaking for the longest time). When it is determined that the mouth movement interval of the target object is greater than the mouth movement intervals of other objects, it is determined that the face review video of the target object has passed the detection. Alternatively, when it is determined that the mouth movement interval of the target object is less than or equal to the mouth movement intervals of other objects, the face review video is marked as doubtful. In either case, it is used as the detection result of the bystander answering corresponding to the face review video.

[0093] This application generates a bystander answering detection result corresponding to the face-to-face review video of the target object by integrating the scene classification results and the lip synchronization recognition results, improves the business logic chain of bystander answering detection for the face-to-face review video, improves the accuracy of bystander answering detection, and enhances the robustness of bystander answering detection.

[0094] In some optional implementations of this embodiment, the above-mentioned speech configuration information includes a speech template and a specified speech list, and the above-mentioned step of filtering the voice information according to the preset speech configuration information to obtain the filtered voice segment includes:

[0095] Comparing the voice information with the speech template to determine a first segment that does not conform to the speech template;

[0096] The first segment is screened according to the specified speech list to obtain a second segment matching the specified speech list as the voice segment.

[0097] In this embodiment, after the voice information is identified from the audio information corresponding to the face-to-face video of the target object, the voice information can be compared with the speech template to determine the first segment that does not conform to the speech template, and then the first segment is screened according to the specified speech list to obtain the second segment that matches the specified speech list as the filtered voice segment. Specifically, the preset speech template belongs to the fixed template of the face-to-face machine, and the preset specified speech list contains the specified speech of the target object in the business. After the voice information is obtained by voice separation from the part of the audio information where there is voice, the voice information needs to be filtered, and only the part of the target object's answer that belongs to the specified speech needs to be saved as the filtered voice segment. First, the voice information can be compared with the speech template to determine the first segment that does not conform to the speech template, and the voice of the face-to-face machine mixed in the voice information is eliminated. Then, the first segment is screened according to the specified speech list to obtain the second segment that matches the specified speech list as the voice segment, and the part of the target object's answer that only belongs to the specified speech is retained.

[0098] This application retains only the part of the target object's answer that belongs to the specified script through script filtering, avoids the interference of the sound of the face-to-face review machine and the voice of the non-specified script part on the detection of bystander answering, improves the business logic chain of bystander answering detection for face-to-face review videos, improves the accuracy of bystander answering detection, and enhances the robustness of bystander answering detection.

[0099] In some optional implementations of this embodiment, the step of obtaining a video segment corresponding to the voice segment and determining whether the target object is in the frame according to the video segment includes:

[0100] Performing target detection on the video clip according to a pre-trained target detection model to obtain a target detection result;

[0101] Determine whether the target object is in the frame according to the target detection result.

[0102] In this embodiment, after filtering the voice information and obtaining the filtered voice segment, target detection can be performed on the video segment according to the pre-trained target detection model to obtain the target detection result, and then determine whether the target object is in the frame according to the target detection result. Specifically, the video segment can be divided into multiple video frames, and facial features or hand features can be obtained from each video frame through feature extraction to form a character feature sample set. The character feature sample set is input into the pre-trained target detection model, and the target detection model can output the confidence level of the target object appearing in the image of each video frame, and compare it with the preset confidence threshold. If the confidence level of the target object appearing in the image of each video frame is all greater than the confidence threshold, it is determined that the target object is in the frame, otherwise, it is determined that the target object is not in the frame.

[0103] It can be understood that the target detection model can be obtained by training the training set enhanced by the MASO IC data, which can greatly enrich the detection data set and make it more robust.

[0104] This application uses a target detection model to determine whether the target object in a video clip is in frame. As one of the processes of bystander answering detection based on face-to-face review videos, it improves the business logic chain of bystander answering detection for face-to-face review videos, and ultimately improves the accuracy of bystander answering detection.

[0105] In some optional implementations of this embodiment, after the step of obtaining the video segment corresponding to the voice segment and determining whether the target object is in the frame according to the video segment, the method further includes:

[0106] If it is determined that the target object is not in the frame, it is determined that the face review video has failed the detection.

[0107] In this embodiment, when determining whether the target object is in the frame according to the video clip corresponding to the voice clip, if it is determined that the target object is not in the frame, then it is determined that the face-to-face review video has failed the test. Specifically, the face-to-face review needs to be answered by a specific person. For the face-to-face review video of the target object, if it is determined that the target object is not in the frame according to the video clip, it is impossible to determine whether there is someone else answering on its behalf. At this time, there is no need to continue the subsequent process, and the face-to-face review video is directly determined to have failed the test.

[0108] When judging that the target object is not in the frame, this application directly determines that the face review video has failed the detection. As one of the processes of bystander answering detection based on the face review video, it improves the business logic chain of bystander answering detection for the face review video and improves the efficiency of bystander answering detection.

[0109] In some optional implementations of this embodiment, the step of obtaining audio information corresponding to the face-to-face interview video of the target object and identifying voice information from the audio information includes:

[0110] Extracting the audio information from the face-to-face interview video;

[0111] Voice activity detection is performed on the audio information to identify and obtain the voice information.

[0112] In this embodiment, when performing bystander answering detection based on the face-to-face video, the audio information corresponding to the face-to-face video can be first extracted from the face-to-face video of the target object, and then voice activity detection can be performed on the audio information to identify the voice information. Specifically, in order to improve the business logic chain for bystander answering detection for face-to-face videos, the detection process can be planned from both the auditory and visual aspects. Taking hearing as an example, it is necessary to extract the corresponding audio information from the face-to-face video of the target object by means of audio and video separation, and then obtain the voice information by performing voice activity detection on the audio information. Generally speaking, the part of the audio information that contains voice can be identified by the VAD algorithm, and after feature extraction of the audio information, a feature data set is obtained. The feature data set is input into a pre-trained VAD model, and the VAD model identifies the part of the audio information that contains voice, and then voice separation is used to obtain the voice information.

[0113] This application extracts voice information from the audio information of the face-to-face review video through voice recognition detection, which serves as the basis for realizing bystander answering detection based on the face-to-face review video. It improves the business logic chain of bystander answering detection for the face-to-face review video and improves the efficiency of bystander answering detection.

[0114] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0115] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0116] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0117] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0118] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a device for detecting bystander answering based on face-to-face video. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0119] like Figure 3 As shown, the apparatus 300 for detecting bystander answering based on face-to-face video in this embodiment includes: an acquisition module 301, a filtering module 302, a determination module 303, a classification module 304, a recognition module 305 and a generation module 306. Among them:

[0120] An acquisition module 301 is used to acquire audio information corresponding to the face-to-face interview video of the target object, and recognize voice information from the audio information;

[0121] A filtering module 302, configured to filter the voice information according to preset speech configuration information to obtain filtered voice segments;

[0122] A determination module 303 is used to obtain a video segment corresponding to the voice segment, and determine whether the target object is in the frame according to the video segment;

[0123] The classification module 304 is used to classify the video clip into scenes if it is determined that the target object is in the frame, and obtain a scene classification result;

[0124] An identification module 305 is used to perform lip synchronization identification on the video clip according to the scene classification result to obtain a lip synchronization identification result;

[0125] The generation module 306 is used to generate a bystander answering detection result corresponding to the face-to-face review video according to the lip synchronization recognition result.

[0126] The bystander answering detection device based on face-to-face review video provided in the present application realizes bystander answering detection based on face-to-face review video through the coordination of multiple processes, so that when performing bystander answering detection on face-to-face review video, a complete business logic chain is provided, the accuracy of bystander answering detection is improved, and the robustness of bystander answering detection is enhanced.

[0127] In some optional implementations of this embodiment, the classification module 304 is further configured to:

[0128] If it is determined that the target object is in the frame, feature extraction is performed based on the video clip to obtain a scene image sample set;

[0129] Inputting the scene image sample set into a pre-trained scene classification model to obtain a classification result for each image in the scene image sample set;

[0130] According to the classification result of each image, it is determined whether the video clip belongs to a single-person scene or a multi-person scene as the scene classification result.

[0131] The device for detecting bystander answering based on face-to-face review videos provided in the present application uses a scene classification model to determine whether a video clip belongs to a single-person scene or a multi-person scene. As one of the processes of bystander answering detection based on face-to-face review videos, it improves the business logic chain of bystander answering detection for face-to-face review videos, and ultimately improves the accuracy of bystander answering detection.

[0132] In some optional implementations of this embodiment, the identification module 305 is further configured to:

[0133] If the video clip belongs to a single-person scene, performing lip synchronization recognition on the video clip to determine whether the mouth movement interval of the target object coincides with the video clip, as the lip synchronization recognition result;

[0134] If the video clip belongs to a multi-person scene, lip synchronization recognition is performed on the video clip to determine whether the mouth movement interval of the target object is greater than the mouth movement interval of other objects as the lip synchronization recognition result.

[0135] The device for detecting bystander answering based on face-to-face review videos provided in the present application, by combining lip synchronization recognition with scene classification to review whether the face-to-face review video is answered by the target object himself, is used as a part of the process of detecting bystander answering based on face-to-face review videos. It improves the business logic chain of detecting bystander answering for face-to-face review videos, and ultimately improves the accuracy of detecting bystander answering.

[0136] In some optional implementations of this embodiment, the generating module 306 is further configured to:

[0137] When the video clip belongs to a single-person scene, if it is determined that the mouth movement interval of the target object coincides with the video clip, it is determined that the face-to-face review video passes the detection; otherwise, the face-to-face review video is marked as questionable and used as the bystander answering detection result;

[0138] When the video clip belongs to a multi-person scene, if it is determined that the mouth movement interval of the target object is greater than the mouth movement interval of other objects, the face review video is determined to have passed the detection; otherwise, the face review video is marked as questionable and used as the bystander answering detection result.

[0139] The bystander answering detection device based on face-to-face review video provided in the present application generates a bystander answering detection result corresponding to the face-to-face review video of the target object by integrating the scene classification result and the lip synchronization recognition result, thereby improving the business logic chain of bystander answering detection for the face-to-face review video, improving the accuracy of bystander answering detection, and enhancing the robustness of bystander answering detection.

[0140] In some optional implementations of this embodiment, the above-mentioned speech configuration information includes a speech template and a specified speech list, and the filtering module 302 is further used to:

[0141] Comparing the voice information with the speech template to determine a first segment that does not conform to the speech template;

[0142] The first segment is screened according to the specified speech list to obtain a second segment matching the specified speech list as the voice segment.

[0143] The device for detecting bystander answering based on face-to-face review videos provided in the present application retains only the part of the target object's answer that belongs to the designated speech through speech filtering, thereby avoiding the interference of the sound of the face-to-face review machine and the voice of the non-designated speech part in the bystander answering detection, improving the business logic chain for bystander answering detection on the face-to-face review video, improving the accuracy of bystander answering detection, and enhancing the robustness of bystander answering detection.

[0144] In some optional implementations of this embodiment, the determining module 303 is further configured to:

[0145] Performing target detection on the video clip according to a pre-trained target detection model to obtain a target detection result;

[0146] Determine whether the target object is in the frame according to the target detection result.

[0147] The device for detecting bystander answering based on face-to-face review videos provided in the present application determines whether the target object in the video clip is in the frame through a target detection model. As one of the processes of bystander answering detection based on face-to-face review videos, it improves the business logic chain of bystander answering detection for face-to-face review videos, and ultimately improves the accuracy of bystander answering detection.

[0148] In some optional implementations of this embodiment, the apparatus 300 for detecting bystander answering based on face-to-face video is further used for:

[0149] If it is determined that the target object is not in the frame, it is determined that the face review video has failed the detection.

[0150] The bystander answering detection device based on the face review video provided in the present application directly determines that the face review video has not passed the detection when it determines that the target object is not in the frame. As one of the processes of bystander answering detection based on the face review video, it improves the business logic chain of bystander answering detection for the face review video and improves the efficiency of bystander answering detection.

[0151] In some optional implementations of this embodiment, the acquisition module 301 is further used to:

[0152] Extracting the audio information from the face-to-face interview video;

[0153] Voice activity detection is performed on the audio information to identify and obtain the voice information.

[0154] The device for detecting bystander answering based on face-to-face review video provided in the present application extracts voice information from the audio information of the face-to-face review video through voice recognition detection, which serves as the basis for realizing bystander answering detection based on face-to-face review video, improves the business logic chain for bystander answering detection for face-to-face review video, and improves the efficiency of bystander answering detection.

[0155] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0156] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital processors (DSPs), embedded devices, etc.

[0157] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0158] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (Smart Med ia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store an operating system and various application software installed on the computer device 4, such as computer-readable instructions of a method for detecting a bystander answering based on a face-to-face video. In addition, the memory 41 can also be used to temporarily store various data that have been output or are to be output.

[0159] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the computer-readable instructions or process data stored in the memory 41, such as running the computer-readable instructions of the bystander answering detection method based on the face-to-face video.

[0160] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0161] The computer device provided in this application realizes bystander answering detection based on face-to-face review videos through the coordination of multiple processes, so that when performing bystander answering detection on face-to-face review videos, a complete business logic chain is provided, the accuracy of bystander answering detection is improved, and the robustness of bystander answering detection is enhanced.

[0162] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned method for detecting bystander answering based on face-to-face review video.

[0163] The computer-readable storage medium provided in the present application realizes bystander answering detection based on face-to-face review videos through the coordination of multiple processes, so that when performing bystander answering detection on face-to-face review videos, a complete business logic chain is provided, the accuracy of bystander answering detection is improved, and the robustness of bystander answering detection is enhanced.

[0164] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0165] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A method for detecting bystander answering based on face-to-face video, characterized in that: The steps include: Acquire audio information corresponding to the face-to-face interview video of the target object, and recognize voice information from the audio information; Filtering the voice information according to preset speech configuration information to obtain filtered voice segments; Obtaining a video clip corresponding to the voice clip, and determining whether the target object is in the frame according to the video clip; If it is determined that the target object is in the frame, performing scene classification on the video clip to obtain a scene classification result; According to the scene classification result, lip synchronization recognition is performed on the video clip to obtain a lip synchronization recognition result; According to the lip synchronization recognition result, a bystander answering detection result corresponding to the face-to-face review video is generated.

2. The method for detecting bystander answering based on face-to-face video according to claim 1, characterized in that: If it is determined that the target object is in the frame, the step of performing scene classification on the video clip to obtain a scene classification result specifically includes: If it is determined that the target object is in the frame, feature extraction is performed based on the video clip to obtain a scene image sample set; Inputting the scene image sample set into a pre-trained scene classification model to obtain a classification result for each image in the scene image sample set; According to the classification result of each image, it is determined whether the video clip belongs to a single-person scene or a multi-person scene as the scene classification result.

3. The method for detecting bystander answering based on face-to-face video according to claim 2, characterized in that: The step of performing lip synchronization recognition on the video clip according to the scene classification result to obtain the lip synchronization recognition result specifically includes: If the video clip belongs to a single-person scene, performing lip synchronization recognition on the video clip to determine whether the mouth movement interval of the target object coincides with the video clip, as the lip synchronization recognition result; If the video clip belongs to a multi-person scene, lip synchronization recognition is performed on the video clip to determine whether the mouth movement interval of the target object is greater than the mouth movement interval of other objects as the lip synchronization recognition result.

4. The method for detecting bystander answering based on face-to-face video according to claim 3, characterized in that: The step of generating a bystander answering detection result corresponding to the face-to-face review video according to the lip synchronization recognition result specifically includes: When the video clip belongs to a single-person scene, if it is determined that the mouth movement interval of the target object coincides with the video clip, it is determined that the face-to-face review video passes the detection; otherwise, the face-to-face review video is marked as questionable and used as the bystander answering detection result; When the video clip belongs to a multi-person scene, if it is determined that the mouth movement interval of the target object is greater than the mouth movement interval of other objects, the face review video is determined to have passed the detection; otherwise, the face review video is marked as questionable and used as the bystander answering detection result.

5. The method for detecting bystander answering based on face-to-face video according to claim 1, characterized in that: The speech configuration information includes a speech template and a specified speech list, and the step of filtering the voice information according to the preset speech configuration information to obtain the filtered voice segment specifically includes: Comparing the voice information with the speech template to determine a first segment that does not conform to the speech template; The first segment is screened according to the specified speech list to obtain a second segment matching the specified speech list as the voice segment.

6. The method for detecting bystander answering based on face-to-face video according to claim 1, characterized in that: The step of obtaining a video segment corresponding to the voice segment and determining whether the target object is in the frame according to the video segment specifically includes: Performing target detection on the video clip according to a pre-trained target detection model to obtain a target detection result; Determine whether the target object is in the frame according to the target detection result.

7. The method for detecting bystander answering based on face-to-face video according to claim 1, characterized in that: After the step of obtaining the video segment corresponding to the voice segment and determining whether the target object is in the frame according to the video segment, the method further includes: If it is determined that the target object is not in the frame, it is determined that the face review video has failed the detection.

8. The method for detecting bystander answering based on face-to-face video according to any one of claims 1 to 7, characterized in that: The step of obtaining audio information corresponding to the face-to-face interview video of the target object and identifying voice information from the audio information specifically includes: Extracting the audio information from the face-to-face interview video; Voice activity detection is performed on the audio information to identify and obtain the voice information.

9. A device for detecting bystander answering based on face-to-face video, characterized in that: include: An acquisition module is used to acquire audio information corresponding to the face-to-face interview video of the target object, and recognize voice information from the audio information; A filtering module, used to filter the voice information according to preset speech configuration information to obtain filtered voice segments; A determination module, used to obtain a video segment corresponding to the voice segment, and determine whether the target object is in the frame according to the video segment; A classification module, configured to classify the video clip into scenes if it is determined that the target object is in the frame, and obtain a scene classification result; A recognition module, configured to perform lip synchronization recognition on the video clip according to the scene classification result to obtain a lip synchronization recognition result; A generation module is used to generate a bystander answering detection result corresponding to the face-to-face review video according to the lip synchronization recognition result.

10. A computer device, characterized in that: It includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the steps of the method for detecting bystander answering based on face-to-face video as described in any one of claims 1 to 8.

Images