Behavior recognition method and device for video face review, electronic equipment and medium

Abstract

The embodiment of the invention provides a behavior recognition method and device for video face review, electronic equipment and a medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring reference video data of a sample surface review object in a surface review state; performing image selection based on the normal behavior image to obtain a candidate behavior image, and performing audio selection based on the normal voice sample to obtain a candidate voice sample; and performing feature fusion according to the normal behavior image, the candidate behavior image, the normal voice sample and the candidate voice sample to obtain typical behavior video data and guidance behavior video data. And training a preset initial behavior recognition model based on the reference video data, the typical behavior video data and the guidance behavior video data to obtain a target behavior recognition model, and performing behavior recognition on the target video data according to the target behavior recognition model to obtain a behavior type. According to the embodiment of the invention, the auditing efficiency of the face-auditing video can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a behavior recognition method and device, electronic device and storage medium for video interview. Background Art

[0002] In the field of finance and insurance, online interviews are a type of remote interview or review conducted over the Internet. They are usually used in scenarios such as claims processing, customer identity verification, policy signing, and risk assessment. For example, during the claims process, insurance companies need to verify the authenticity of the identity information of the interviewee through online interviews, as well as determine the interviewee's true intentions, in order to prevent fraud. In related technologies, it is necessary to rely on manual experience to determine whether the interviewee has abnormal behavior. However, the audit efficiency of this method is low. Therefore, how to improve the audit efficiency of the interview video has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a behavior recognition method and device, electronic device and storage medium for video audit, aiming to improve the review efficiency of audit videos.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a behavior recognition method for video interview, the method comprising:

[0005] Acquire reference video data of a sample face-to-face interview subject in a face-to-face interview state; wherein the reference video data includes a normal behavior image and a normal voice sample of the sample face-to-face interview subject, and the behavior of the sample face-to-face interview subject in the face-to-face interview state in the reference video data is normal behavior;

[0006] Performing image selection based on the normal behavior image to obtain a candidate behavior image, and performing audio selection based on the normal voice sample to obtain a candidate voice sample;

[0007] Perform feature fusion according to the normal behavior image, the candidate behavior image, the normal voice sample and the candidate voice sample to obtain typical behavior video data; wherein the behavior of the sample face-to-face interview object in the face-to-face interview state in the typical behavior video data is normal behavior;

[0008] Perform feature fusion according to the normal behavior image, the normal voice sample and the candidate voice sample to obtain guidance behavior video data; wherein the behavior of the sample face-to-face review object in the face-to-face review state in the guidance behavior video data is abnormal behavior;

[0009] Training a preset initial behavior recognition model based on the reference video data, the typical behavior video data, and the guidance behavior video data to obtain a target behavior recognition model;

[0010] The target video data of the target review object in the review state is obtained, and behavior recognition is performed on the target video data according to the target behavior recognition model to obtain the behavior type of the target review object, wherein the behavior type is used to characterize whether the behavior of the target review object in the review state is normal behavior or abnormal behavior.

[0011] In some embodiments, the performing feature fusion according to the normal behavior image, the candidate behavior image, the normal voice sample, and the candidate voice sample to obtain typical behavior video data includes:

[0012] Performing image feature splicing on the normal behavior image and the candidate behavior image to obtain typical behavior image features;

[0013] Performing audio feature concatenation on the normal speech sample and the candidate speech sample to obtain typical behavior audio features;

[0014] The typical behavior image features and the typical behavior audio features are fused to obtain the typical behavior video data.

[0015] In some embodiments, the step of performing feature fusion according to the normal behavior image, the normal voice sample, and the candidate voice sample to obtain the guiding behavior video data includes:

[0016] Performing audio feature concatenation on the normal speech sample and the candidate speech sample to obtain a guiding audio feature;

[0017] The normal behavior image and the guidance audio features are subjected to feature fusion to obtain the guidance behavior video data.

[0018] In some embodiments, the training of a preset initial behavior recognition model based on the reference video data, the typical behavior video data, and the guidance behavior video data to obtain a target behavior recognition model includes:

[0019] Selecting sample behavior video data from the reference video data, the typical behavior video data, and the guiding behavior video data;

[0020] Extracting audio features from the sample behavior video data to obtain sample audio features;

[0021] Extracting picture features from the sample behavior video data to obtain sample picture features;

[0022] Performing feature fusion on the sample audio feature and the sample picture feature to obtain a sample fusion feature;

[0023] Performing classification prediction on the sample fusion features according to the initial behavior recognition model to obtain classification probability data;

[0024] Calculate the loss value according to the classification probability data, the preset label vector and the number of sample behavior video data to obtain behavior classification loss data;

[0025] The parameters of the initial behavior recognition model are adjusted according to the behavior classification loss data to obtain the target behavior recognition model.

[0026] In some embodiments, acquiring target video data of the target review object in the review state, and performing behavior recognition on the target video data according to the target behavior recognition model to obtain the behavior type of the target review object includes:

[0027] Extracting audio features from the target video data to obtain target audio features of the target audit object;

[0028] Extracting picture features from the target video data to obtain target picture features of the target face-to-face object;

[0029] Performing feature fusion on the target audio feature and the target picture feature to obtain a target fusion feature;

[0030] The target fusion feature is subjected to behavior type recognition to obtain the behavior type.

[0031] In some embodiments, extracting picture features from the target video data to obtain target picture features of the target face-to-face object includes:

[0032] Performing video frame interception on the target video data to obtain at least two video frames;

[0033] Performing facial feature recognition on each of the video frames to obtain target facial feature points of the target face review object;

[0034] Performing region extraction on the video frame according to the target pixel coordinates of the target facial feature points to obtain a candidate region image;

[0035] The candidate region image is subjected to pixel coordinate transformation according to the target pixel coordinates to obtain the target region image, and the target region image is subjected to screen feature extraction to obtain the target screen feature.

[0036] In some embodiments, after acquiring the target video data of the target audit object in the audit state and performing behavior recognition on the target video data according to the target behavior recognition model to obtain the behavior type of the target audit object, the method further includes:

[0037] If the behavior type is abnormal behavior, performing speech recognition on the target video data to obtain target text information;

[0038] Dividing the target text information into sentences to obtain language block information;

[0039] If the language block information meets the preset abnormal notification triggering condition, the abnormal behavior notification operation is performed; wherein the abnormal notification triggering condition includes that the meanings represented by adjacent language block information are different.

[0040] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a behavior recognition device for video interview, the device comprising:

[0041] A video acquisition module is used to acquire reference video data of a sample face-to-face interview subject in a face-to-face interview state; wherein the reference video data includes a normal behavior image and a normal voice sample of the sample face-to-face interview subject, and the behavior of the sample face-to-face interview subject in the face-to-face interview state in the reference video data is normal behavior;

[0042] A sample feature selection module, configured to select images based on the normal behavior images to obtain candidate behavior images, and to select audio based on the normal voice samples to obtain candidate voice samples;

[0043] A first feature fusion module is used to perform feature fusion according to the normal behavior image, the candidate behavior image, the normal voice sample and the candidate voice sample to obtain typical behavior video data; wherein the behavior of the sample face-to-face review object in the face-to-face review state in the typical behavior video data is normal behavior;

[0044] A second feature fusion module is used to perform feature fusion according to the normal behavior image, the normal voice sample and the candidate voice sample to obtain guidance behavior video data; wherein the behavior of the sample face-to-face review object in the face-to-face review state in the guidance behavior video data is an abnormal behavior;

[0045] A model training module, used to train a preset initial behavior recognition model based on the reference video data, the typical behavior video data and the guidance behavior video data to obtain a target behavior recognition model;

[0046] The behavior recognition module is used to obtain the target video data of the target review object in the review state, and perform behavior recognition on the target video data according to the target behavior recognition model to obtain the behavior type of the target review object, and the behavior type is used to characterize whether the behavior of the target review object in the review state is normal behavior or abnormal behavior.

[0047] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0048] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0049] The behavior recognition method and device, electronic device and storage medium for video audit proposed in the present application ensure that the standard and undisturbed behavior of the audit object can be identified by obtaining reference video data and images and audio samples of normal behavior. Then, by selecting candidate behavior images and candidate audio samples and performing different feature fusion operations to synthesize video data of typical behaviors and guiding behaviors, a training set containing normal and potentially abnormal behaviors is established. Then, the preset initial behavior recognition model is trained to obtain the target behavior recognition model, and the target behavior recognition model is used to determine the behavior type of the target audit object in the target audit video. Compared with the traditional solution that relies on manual experience to confirm the audit behavior, the method provided in the embodiment of the present application not only accelerates the recognition process of the audit video, but also greatly improves the accuracy of identifying abnormal behaviors through precise model training, reduces the dependence on manual operation, and thus solves the problem of low efficiency of manual review of audit videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flow chart of a behavior recognition method for video interview provided in an embodiment of the present application;

[0051] Figure 2 yes Figure 1 Flow chart of step S103 in FIG.

[0052] Figure 3 yes Figure 1 Flow chart of step S104 in FIG.

[0053] Figure 4 yes Figure 1 Flow chart of step S105 in FIG.

[0054] Figure 5 yes Figure 1 Flow chart of step S106 in FIG.

[0055] Figure 6 yes Figure 5 Flow chart of step S502 in FIG.

[0056] Figure 7is another flow chart of the behavior recognition method for video interview provided by an embodiment of the present application;

[0057] Figure 8 It is a structural schematic diagram of a behavior recognition device for video interview provided in an embodiment of the present application;

[0058] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0062] First, some nouns involved in this application are analyzed:

[0063] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0064] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages ​​(such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0065] Information Extraction: A text processing technology that extracts specified types of entity, relationship, event and other factual information from natural language text and forms structured data output. Information extraction is a technology that extracts specific information from text data. Text data is composed of some specific units, such as sentences, paragraphs, and chapters. Text information is composed of some small specific units, such as characters, words, phrases, sentences, paragraphs, or a combination of these specific units. Extracting noun phrases, names, place names, etc. from text data are all text information extraction. Of course, the information extracted by text information extraction technology can be various types of information.

[0066] In the field of finance and insurance, online interview is a remote interview or review method conducted through the Internet. It is usually used in scenarios such as claims process, customer identity verification, policy signing, and risk assessment. For example, in the claims process, insurance companies may need to verify the authenticity of the identity information of the interviewee through online interviews, and determine the true intention of the interviewee to prevent fraud. However, in online interviews, the interviewee often behaves abnormally. For example, during the online interview, a third person outside the camera screen answers, and the interviewee answers under the guidance of the third person. A typical guidance situation is that there is a person next to him saying "clear", and the interviewee then follows and says "clear". Such guidance behavior makes the result of the interview unable to truly reflect the intention of the interviewee himself, resulting in invalid interview. In related technologies, it still relies on manual observation of the behavior of the interviewee in the video to determine whether the above abnormal behavior occurs. This method is inefficient and its result page is not accurate enough. Therefore, how to improve the review efficiency of the interview video has become a technical problem that needs to be solved urgently.

[0067] Based on this, the embodiments of the present application provide a behavior recognition method and device, an electronic device and a storage medium for video review, aiming to improve the review efficiency of video review.

[0068] The behavior recognition method and device, electronic device and storage medium for video audit provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the behavior recognition method for video audit in the embodiments of the present application is described.

[0069] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0070] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0071] The behavior recognition method for video audit provided in the embodiment of the present application relates to the field of artificial intelligence technology. The behavior recognition method for video audit provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the behavior recognition method for video audit, etc., but is not limited to the above forms.

[0072] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0073] Figure 1 is an optional flow chart of a behavior recognition method for video interview provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.

[0074] Step S101, obtaining reference video data of a sample face-to-face interview subject in a face-to-face interview state.

[0075] Step S102: performing image selection based on the normal behavior image to obtain a candidate behavior image, and performing audio selection based on the normal voice sample to obtain a candidate voice sample.

[0076] Step S103, performing feature fusion according to the normal behavior image, the candidate behavior image, the normal voice sample and the candidate voice sample to obtain typical behavior video data.

[0077] Step S104, performing feature fusion according to the normal behavior image, the normal voice sample and the candidate voice sample to obtain the guiding behavior video data.

[0078] Step S105 , training a preset initial behavior recognition model based on the reference video data, the typical behavior video data, and the guidance behavior video data to obtain a target behavior recognition model.

[0079] Step S106, obtaining target video data of the target audit object in the audit state, and performing behavior recognition on the target video data according to the target behavior recognition model to obtain the behavior type of the target audit object.

[0080] Steps S101 to S106 shown in the embodiment of the present application ensure that the standard, undisturbed behavior of the audit object can be identified by obtaining reference video data and images and audio samples of normal behavior. Then, by selecting candidate behavior images and candidate audio samples and performing different feature fusion operations to synthesize video data of typical behaviors and guiding behaviors, a training set containing normal and potentially abnormal behaviors is established. Then, the preset initial behavior recognition model is trained to obtain a target behavior recognition model, and the target behavior recognition model is used to determine the behavior type of the target audit object in the target audit video. Compared with the traditional solution that relies on manual experience to confirm audit behavior, the method provided in the embodiment of the present application not only accelerates the recognition process of the audit video, but also greatly improves the accuracy of identifying abnormal behavior through precise model training, reduces the dependence on manual operation, and thus solves the problem of low efficiency of manual review of audit videos.

[0081] In step S101 of some embodiments, it should be noted that the interview process is usually conducted in the form of questions and answers. The party asking the questions is usually a staff member of the financial institution, and the party answering the questions is usually the interviewee. The state in which the interviewee is undergoing the interview process is the interview state. If the following scenarios occur: the interviewee does not appear in the video screen of the interview video, or the interviewee adjusts the camera position, or the interviewee does not respond to the staff's voice and actions, then it can be determined that the interviewee is not in the interview state.

[0082] The sample interview object is the interview object in the reference video data. The reference video data is the video data recording the normal interview process of the sample interview object, including the normal behavior image and normal voice sample of the sample interview object. The behavior of the sample interview object in the interview state in the reference video data is normal behavior. The normal behavior image is the action picture of the interview object performing normal behavior, which can be an image reflecting facial movements. For example, in the scenario of insurance claims, when the interview process is carried out, the staff explains the location of the accident to the interview object and asks the interview object whether the location is accurate. The interview object quickly and clearly responds "yes" or "no". At this time, the behavior of the interview object is normal behavior. The action clip of the interview object saying "yes" in the video is a normal behavior image, and the audio of the interview object saying "yes" is a normal voice sample.

[0083] In step S102 of some embodiments, it should be noted that the embodiments of the present application ultimately perform behavior recognition through a behavior recognition model. However, in reality, the number of reference video data is very large, while the training video data that includes guiding behaviors is actually very small. In order to ensure the accuracy of the behavior recognition model, it is necessary to use the reference video data for video synthesis to increase the training data of other behavior categories.

[0084] The candidate behavior image refers to a fragment image of the sample in the reference video data where the subject of the interview gives a normal answer, and the candidate voice sample refers to a fragment audio of the sample in the reference video data where the subject of the interview gives a normal answer. For example, in the reference video data, the sample subject of the interview answered "clear" to the first question and "no" to the second question. The candidate behavior image can be an action image of the subject of the interview saying "clear" or an action image of the subject of the interview saying "no". The candidate voice sample can be an audio of the subject of the interview saying "clear" or an audio of the subject of the interview saying "no".

[0085] In step S103 of some embodiments, the behavior of the sample interview object in the interview state in the typical behavior video data is normal behavior. It should be noted that in some embodiments, although the answer of the interview object is not like the answer standard in the above reference video data, it can also express the interview object's own wishes and is a normal behavior. For example, the continuous repetition of the answer "yes, yes", the expression of mixed degree in the answer, saying "agree, I completely agree", the same meaning of words using two different words, such as "clearly understand", or the answer contains modal particles, such as: "um, um, okay". Since this type of behavior is easily confused with some guiding behaviors (a third person outside the video screen says "yes", and the interview object then repeats "yes"), it is necessary to synthesize such non-standard but normal behavior video data for training to avoid the subsequent behavior recognition model from misjudging such non-standard but normal behavior. In the following embodiments, the above non-standard but normal behavior is defined as a typical behavior, and the video reflecting the typical behavior of the interview object is the typical behavior video data.

[0086] See also Figure 2 In some embodiments, step S103 may include but is not limited to steps S201 to S203:

[0087] Step S201, performing image feature splicing on the normal behavior image and the candidate behavior image to obtain typical behavior image features.

[0088] Step S202: concatenate the audio features of the normal speech sample and the candidate speech sample to obtain typical behavior audio features.

[0089] Step S203: fusing the typical behavior image features and the typical behavior audio features to obtain typical behavior video data.

[0090] In step S201 of some embodiments, the typical behavior image feature is an action image of the sample interview subject performing a typical behavior. Specifically, taking repeated answers in typical behaviors as an example, a series of continuous images of the sample interview subject saying a specific answer sentence is selected from the normal behavior image as a candidate behavior image, and then the candidate behavior image is inserted before the normal behavior image corresponding to the specific answer sentence, or inserted after the normal behavior image corresponding to the specific answer sentence. The typical image feature finally spliced ​​can reflect that the sample interview subject has done the same action twice. Assuming that the specific answer sentence is "yes", a video clip of the sample interview subject saying "yes" is selected from the normal behavior image, and the video clip is copied to obtain a candidate behavior image, and then the candidate behavior image is inserted before the video clip of the sample interview subject saying "yes", and finally an image of the sample interview subject saying "yes, yes" continuously is obtained.

[0091] In step S202 of some embodiments, the typical behavior audio feature is the audio of the sample interview subject performing a typical behavior. Specifically, taking repeated answers in typical behavior as an example, the audio of the sample interview subject saying a specific answer sentence is selected from the normal voice sample as a candidate voice sample, and then the audio corresponding to the specific answer sentence and the candidate voice sample are spliced ​​to obtain the typical behavior audio feature. Assuming that the specific answer sentence is "yes", an audio segment of the sample interview subject saying "yes" is selected from the normal voice sample, the audio segment is copied to obtain a candidate voice sample, and then the audio segment and the candidate voice sample are spliced, and finally the audio of the sample interview subject saying "yes, yes" continuously is obtained.

[0092] In step S203 of some embodiments, the time length of the typical behavior image feature and the time length of the typical behavior audio feature can be kept consistent during the execution of steps S201 and S202, and the starting timestamp of the typical behavior image feature in the video file is obtained, and the starting timestamp is set as the starting timestamp of the typical behavior audio feature, so that the typical behavior image feature and the typical behavior audio feature are aligned to ensure that the action in the video and the sound in the audio are synchronized in time. For example, if the starting timestamp of the typical behavior image feature in the entire video data is the 2nd second, the corresponding typical behavior audio should also start from the 2nd second. Then, the aligned typical behavior image features and the typical behavior images are feature fused to obtain typical behavior video data.

[0093] Steps S201 to S203 shown in the embodiment of the present application generate typical behavior images and typical audio samples by performing feature splicing of normal behavior images with candidate behavior images, and performing audio feature splicing of normal speech samples with candidate speech samples. Finally, the typical behavior video data obtained through feature fusion can simulate typical behaviors such as repeated answers that may occur in real face-to-face interview scenarios, increase the number of videos in the training data set, and further improve the performance of the behavior recognition model.

[0094] In step S104 of some embodiments, the behavior of the sample interview subject in the interview state in the guidance behavior video data is abnormal behavior, that is, the guidance behavior described in the above embodiments, including: a third person answers outside the screen, and the interview subject repeats the same answer. Although this scenario also has audio with repeated content, the difference from the above typical behavior is that in the video screen, the interview subject only made a behavioral action once. For example, there is an audio of "clear, clear" in the video, but the interview subject only has a corresponding facial movement when the second "clear" is said, so it is determined that the first "clear" is said by the third person outside the screen. At this time, it can be determined that the interview subject made the answer under the guidance of others.

[0095] See also Figure 3 In the face-to-face interview scenario where the interviewee repeats the answer of a third person, step S104 may include but is not limited to steps S301 to S302:

[0096] Step S301, concatenating the audio features of the normal speech sample and the candidate speech sample to obtain the guiding audio features.

[0097] Step S302: Fusing the normal behavior image and the guidance audio features to obtain guidance behavior video data.

[0098] In step S301 of some embodiments, the guidance audio feature is the audio of the sample interview object performing guidance behavior. The audio feature splicing in this embodiment is consistent with that in step S202. After the candidate voice sample is copied, it is spliced ​​with the normal voice sample to obtain the guidance audio feature, which will not be described in detail here.

[0099] In step S302 of some embodiments, the start timestamp of the normal behavior image is aligned with the audio splicing point in the guidance audio feature, and the guidance behavior video data finally obtained can reflect that the sample interview object did not make any action when the specific sentence audio appeared for the first time, but made corresponding mouth movements when the specific sentence audio appeared for the second time. Then, the aligned normal behavior image and the guidance audio feature are feature fused to obtain the guidance behavior video data.

[0100] In steps S301 to S302 shown in the embodiment of the present application, by splicing candidate voice samples and normal voice samples and performing feature fusion with normal behavior images, the guidance behavior data finally obtained effectively simulates the guidance behavior that may occur in a real face-to-face interview scenario, increases the number of videos in the training data set, and further improves the performance of the behavior recognition model.

[0101] In other embodiments, the guiding behavior can also be judged by identifying the facial expression of the interviewee. For example, during the interview, before each answer, the interviewee will first shift his gaze from the screen to the side and then answer. In this case, it can be inferred that a third person outside the screen is guiding the interviewee to answer through actions. Therefore, a video clip corresponding to the action can be selected from the normal behavior image and feature fused with the normal voice sample, so that the synthesized new video can reflect the scene in which the sample interviewee first shifts his gaze and then speaks out the answer.

[0102] See also Figure 4 In some embodiments, step S105 may include but is not limited to steps S401 to S407:

[0103] Step S401 , selecting sample behavior video data from reference video data, typical behavior video data, and instructional behavior video data.

[0104] Step S402: extract audio features from the sample behavior video data to obtain sample audio features.

[0105] Step S403: extracting picture features from the sample behavior video data to obtain sample picture features.

[0106] Step S404: Fusing the sample audio features and the sample picture features to obtain sample fusion features.

[0107] Step S405 , classify and predict the sample fusion features according to the initial behavior recognition model to obtain classification probability data.

[0108] Step S406, calculating the loss value according to the classification probability data, the preset label vector and the number of sample behavior video data to obtain behavior classification loss data.

[0109] Step S407, adjusting the parameters of the initial behavior recognition model according to the behavior classification loss data to obtain a target behavior recognition model.

[0110] In step S401 of some embodiments, each video data is assigned a unique digital number, and then selected in sequence in the order of the numbers to obtain sample behavior video data, or proportionally extracted from each category of reference video data, typical behavior video data, and guidance behavior video data until the training set contains a sufficient number of sample behavior video data for training.

[0111] In step S402 of some embodiments, the sample audio feature is an audio feature in the sample behavior video data, which may be in the form of an audio waveform, or in the form of Mel Frequency Cepstral Coefficients (MFCC), Spectral Roll-off, and Zero Crossing Rate (ZCR), etc. In some embodiments, the audio feature extraction of the sample behavior video data may be performed by a neural network, where the neural network may be a one-dimensional residual network (1D Residual Network, resnet1D), a one-dimensional convolutional neural network (1D-CNN), a recurrent neural network, etc., but is not limited thereto.

[0112] In step S403 of some embodiments, the sample screen features are visual information extracted from the video data in the sample behavior video data. This information is used to describe the scene, sample review object, action and expression of the sample review object and other visual elements in the sample behavior video. The screen features can be extracted through a neural network. In this embodiment, the sample behavior video data will first be disassembled into multiple continuous video frames. In order to utilize the time series information in the video frames, the neural network can be a 3D convolutional network (3D Convolutional Networks). In some embodiments, in order to reduce the overall amount of calculation, 3D convolutional layers can be used in the first several layers of the network to capture time series information, and then 2D convolutional networks are used in the subsequent parts of the network to process spatial information. This layout can effectively compress and integrate the initial time dimension information, so that the subsequent layers only need to focus on spatial features, thereby reducing the calculation requirements.

[0113] It should be noted that, in some embodiments, the sample screen features tend to focus more on the mouth movements of the sample review object. Since the typical behavior video data and the guidance behavior video data in the training set are very likely to be synthesized with other video data, the video screen will be incoherent in some synthesized places, thus affecting the training effect. In this regard, only the mouth image of the sample review object can be cut out and the mouth can be aligned. During the training process, only the mouth of the sample review object can be focused on, and unnecessary image information interference can be filtered out, thereby improving the accuracy of the subsequent behavior recognition model. The specific process of achieving mouth alignment will also be carried out during the use of the target behavior recognition model, and this process will be described in detail in the subsequent embodiments of steps S601 to S604.

[0114] In step S404 of some embodiments, the sample fusion feature is a fusion feature of the sample audio feature and the sample picture feature. Feature fusion can be performed through a cross-attention module and a self-attention module. The cross-attention mechanism focuses on how to map information from one modality to another modality, enhancing the model's understanding of the interaction between audio and video. The self-attention mechanism is responsible for comparing and fusing features within a single modality.

[0115] In step S405 of some embodiments, it should be noted that the initial behavior recognition model includes the neural network for extracting audio features, the neural network for extracting picture features, the neural network for feature fusion and a classification head (Classification Head) mentioned in the above embodiments. The classification head can be a fully connected layer. After the sample fusion feature is input into the classification head, the probability that the sample fusion feature belongs to each type will be obtained, and this probability is the classification probability data.

[0116] In step S406 of some embodiments, the preset label vector explicitly indicates the behavior category corresponding to each training sample. This label vector usually adopts a one-hot encoding format. In this embodiment, the label vector (1, 0, 0) corresponds to the standard answer behavior in the reference video data, the label vector corresponding to the typical behavior is (0, 1, 0), and the label vector corresponding to the guiding behavior is (0, 0, 1).

[0117] The behavior classification loss data can be calculated according to the following analytical formula (1):

[0118]

[0119] The behavior classification loss data is represented as loss, the number of sample behavior video data is represented as n, the sequence number of the video is represented as i, which represents the i-th sample behavior video. The category sequence is represented as j, which represents the j-th category. The number of categories is represented as k, and k is 3 in this embodiment. The label vector is represented as l ij , the classification probability data is represented as y ij .

[0120] In step S407 of some embodiments, the gradient descent algorithm can be used to adjust and optimize the parameters of the initial behavior recognition model (such as weight parameters and number of structural layers, etc.) to continuously reduce the value of the behavior classification loss data until the classification loss data no longer decreases significantly, or the number of iterations reaches a preset number, then the training can be stopped to obtain the final target behavior recognition model.

[0121] Steps S401 to S407 shown in the embodiment of the present application provide a rich variety of training samples by extracting samples from different categories of video data, ensuring that the behavior recognition model can handle a variety of face-to-face behavior scenarios, and by fusing audio features and picture features, ensuring that the model can integrate auditory and visual clues to improve recognition accuracy. Finally, the cross entropy loss function and gradient descent algorithm are used to adjust the model parameters during the training process, and the final target behavior recognition model achieves a high classification accuracy.

[0122] In step S106 of some embodiments, target video data of the target audit object in the audit state is obtained, and behavior recognition is performed on the target video data according to the target behavior recognition model to obtain the behavior type of the target audit object. The target audit object is an object that needs to be audited in the online audit scenario, and the target video data is a video of the target audit object being audited. The behavior type is used to characterize whether the behavior of the target audit object in the audit state is normal behavior or abnormal behavior. It can be understood that the target behavior recognition model includes a neural network for extracting audio features, a neural network for extracting picture features, a neural network for feature fusion, and a classification head.

[0123] See also Figure 5 In some embodiments, step S106 may include but is not limited to steps S501 to S504:

[0124] Step S501, extracting audio features from target video data to obtain target audio features of the target audit object.

[0125] Step S502, extracting picture features from the target video data to obtain target picture features of the target face-to-face object.

[0126] Step S503: Fusing the target audio feature and the target picture feature to obtain a target fused feature.

[0127] Step S504: identify the behavior type of the target fusion feature to obtain the behavior type.

[0128] In step S501 of some embodiments, the original audio file of the target video data can be extracted using multimedia tool software such as Fast Forward Moving Picture Experts Group (FFmpeg). Then, the audio feature extraction network in the target behavior recognition model is used to extract the audio features, and finally the audio features of the target audit object are obtained as the target audio features.

[0129] See also Figure 6 In some embodiments, step S502 includes but is not limited to steps S601 to S604:

[0130] Step S601: intercepting video frames of target video data to obtain at least two video frames.

[0131] Step S602, performing facial feature recognition on each video frame to obtain target facial feature points of the target face review object.

[0132] Step S603, performing region extraction on the video frame according to the target pixel coordinates of the target facial feature points to obtain a candidate region image.

[0133] Step S604, performing pixel coordinate conversion on the candidate region image according to the target pixel coordinates to obtain the target region image, and performing screen feature extraction on the target region image to obtain the target screen feature.

[0134] In step S601 of some embodiments, multimedia tool software may be used to decode the video into a plurality of continuous video frames.

[0135] In step S602 of some embodiments, facial feature recognition can be performed using the Haar cascade classifier provided in the OpenCV library or the Dlib library to obtain facial feature points. In this embodiment, the target facial feature points are mouth feature points that characterize the target facial object.

[0136] In step S603 of some embodiments, the candidate region image is a mouth region image of the target face review object in the video frame. The target pixel coordinates are the pixel coordinates of the target facial feature points. After the pixel coordinates of each target facial feature point are determined, the minimum bounding rectangle including these target pixel coordinates is selected, and the current video frame is cropped according to the minimum bounding rectangle to obtain the candidate region image.

[0137] In step S604 of some embodiments, after executing step S603 on multiple video frames, multiple candidate area images are obtained correspondingly, and first a candidate area image is selected as a reference image. Then, for each candidate area image, an affine transformation matrix is ​​calculated, and this matrix is ​​used to convert the coordinates of the target facial feature points in the image to the coordinates of the corresponding target facial feature points in the reference image. Thereby, the mouth areas in multiple consecutive video frames are aligned, and the aligned images are used as the target area images. Then, the picture feature extraction network in the target behavior recognition model is used to extract the picture features to obtain the target picture features.

[0138] In the steps S601 to S604 shown in the embodiment of the present application, facial features of the target face-to-face object are extracted and analyzed from the video, and facial feature points are identified. Then, the consistency of the image data is enhanced by cropping and aligning the target facial area, providing standardized input for the final picture feature extraction, thereby improving the accuracy and efficiency of the behavior recognition model.

[0139] In step S503 of some embodiments, a feature fusion network in the target behavior recognition model is used to perform feature fusion on the target audio feature and the target picture feature, and the obtained fusion feature is the target fusion feature.

[0140] In step S504 of some embodiments, the target fusion feature is identified and classified using the classification head in the target behavior recognition model, and the classification head outputs a vector representing the probability, and the specific behavior type is determined according to the maximum value and the label vector in the vector. Exemplarily, the classification head outputs a vector (0.2, 0.7, 0.1) for the target fusion feature, and the label vector (1, 0, 0) is pre-set to correspond to the standard answer behavior in the reference video data, the label vector corresponding to the typical behavior is (0, 1, 0), and the label vector corresponding to the guiding behavior is (0, 0, 1). Then the vector (0.2, 0.7, 0.1) indicates that the probability that the behavior of the target face-to-face object is the standard answer behavior is 0.2, the probability that the behavior of the target face-to-face object is the typical behavior is 0.7, and the probability that the behavior of the target face-to-face object is the guiding behavior is 0.1, so it can be determined that the behavior of the target face-to-face object belongs to the typical behavior, and the behavior type is further determined to be normal behavior. If the vector output by the classification head is (0.3, 0.2, 0.5), it can be determined that the behavior of the target interview object is a guided behavior, and thus the behavior type is determined to be an abnormal behavior.

[0141] Steps S501 to S504 shown in the embodiment of the present application improve the accuracy and efficiency of behavior classification by combining the key features of audio and video data. In the behavior type recognition stage, the fused features are used to predict the specific behavior category, so that the recognition process is not limited to the surface information of a single modality, but is based on more comprehensive data analysis, thereby providing more accurate and reliable results.

[0142] It should be noted that typical behaviors include continuous repeated answers, expressions of adverbs of degree or modal particles, and the use of different words to express the same meaning. Although these typical behaviors can be identified by training the behavior recognition model, in the preparation of the training set, only the scene of continuous repeated answers is easy to synthesize. If you want to ensure that the behavior recognition model can accurately identify other typical behaviors, you need to collect a large amount of material, and then synthesize video samples that can reflect other typical behaviors of the sample interview object, which undoubtedly increases the workload and training costs. Therefore, in some embodiments, only reference video data, typical behavior video data of repeated behaviors, and guiding behavior video data are trained during the training process. Usually, after the target behavior recognition model recognizes that the behavior of the target object is a guiding behavior, it will notify the staff. However, in this embodiment, the target interview video needs to be checked again. If it is the typical behavior of the above-mentioned non-repeated answer, no notification will be made, thereby avoiding the scenario of false alarm.

[0143] See also Figure 7 After step S106, the behavior recognition method for video interview provided in the embodiment of the present application may also include but is not limited to steps S701 to S703:

[0144] Step S701: If the behavior type is abnormal behavior, speech recognition is performed on the target video data to obtain target text information.

[0145] Step S702, dividing the target text information into sentences to obtain language block information.

[0146] Step S703: If the language block information meets the preset abnormal notification triggering condition, the abnormal behavior notification operation is performed.

[0147] In step S701 of some embodiments, the audio in the target video data may be converted into text using a speech recognition model (Automatic Speech Recognition, ASR), and the obtained text is the target text information.

[0148] In step S702 of some embodiments, the continuous target text information is segmented into words and sentences using natural language processing technology, which can break the target text information into multiple semantic phrase units. Based on the sentence segmentation, information extraction technology can be further applied, such as keyword extraction to mark specific words and phrases. In the embodiment of the present application, the language block information is these marked phrase units or phrases.

[0149] In step S703 of some embodiments, the abnormal notification triggering condition is usually based on predefined rules or model analysis to detect abnormal behavior. The abnormal notification triggering condition includes different meanings represented by adjacent language block information, such as "yes, no" and "well, OK". The abnormal notification triggering condition can be adaptively set according to actual needs and different scenarios of typical behavior, so as to avoid the situation where typical behavior is misjudged as abnormal behavior.

[0150] In some embodiments, the abnormal behavior notification operation is to notify the relevant staff of the financial institution that the current audit object has abnormal behavior during the audit process. The implementation method can be to display the notification text in a pop-up window on the interactive interface playing the target audit video data, indicating that the abnormal behavior of the target audit object has been identified. For example, the notification text can be "the audit object has abnormal behavior". It can also be that the background automatically records the abnormal behavior and generates a corresponding log file, or it can be fed back to the relevant staff through emails, text messages, mobile phone application push notifications, etc.

[0151] If the behavior type is normal behavior, or the chunk information does not meet the preset abnormal notification triggering conditions, then abnormal behavior notification is not required.

[0152] Steps S701 to S703 shown in the embodiment of the present application, by combining speech recognition and natural language processing technology to identify and respond to abnormal behaviors, can more accurately analyze and identify semantic units and speech patterns in the text, thereby distinguishing between typical behaviors and guiding behaviors, improving the recognition rate of behavior recognition, and thereby reducing false alarms.

[0153] See also Figure 8 The embodiment of the present application also provides a behavior recognition device for video interview, which can implement the above-mentioned behavior recognition method for video interview, and the device includes:

[0154] A video acquisition module is used to acquire reference video data of the sample face-to-face review object in the face-to-face review state; wherein the reference video data includes a normal behavior image and a normal voice sample of the sample face-to-face review object, and the behavior of the sample face-to-face review object in the face-to-face review state in the reference video data is normal behavior;

[0155] A sample feature selection module is used to select images based on normal behavior images to obtain candidate behavior images, and to select audio based on normal voice samples to obtain candidate voice samples;

[0156] The first feature fusion module is used to perform feature fusion based on the normal behavior image, the candidate behavior image, the normal voice sample and the candidate voice sample to obtain typical behavior video data; wherein the behavior of the sample face-to-face review object in the face-to-face review state in the typical behavior video data is normal behavior;

[0157] The second feature fusion module is used to perform feature fusion according to the normal behavior image, the normal voice sample and the candidate voice sample to obtain the guidance behavior video data; wherein the behavior of the sample face-to-face review object in the face-to-face review state in the guidance behavior video data is an abnormal behavior;

[0158] A model training module is used to train a preset initial behavior recognition model based on reference video data, typical behavior video data, and guidance behavior video data to obtain a target behavior recognition model;

[0159] The behavior recognition module is used to obtain the target video data of the target audit object in the audit state, and perform behavior recognition on the target video data according to the target behavior recognition model to obtain the behavior type of the target audit object. The behavior type is used to characterize whether the behavior of the target audit object in the audit state is normal behavior or abnormal behavior.

[0160] The specific implementation of the behavior recognition device for video interview is basically the same as the specific implementation of the behavior recognition method for video interview mentioned above, and will not be repeated here.

[0161] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned behavior recognition method for video face-to-face interview when executing the computer program. The electronic device can be any smart terminal including a tablet computer, a car computer, etc.

[0162] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0163] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0164] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the behavior recognition method for video face-to-face interview in the embodiment of this application;

[0165] Input / output interface 903, used to implement information input and output;

[0166] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0167] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0168] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0169] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned behavior recognition method for video interview is implemented.

[0170] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0171] The behavior recognition method for video audit, the behavior recognition device for video audit, the electronic device and the storage medium provided in the embodiment of the present application ensure that the standard and undisturbed behavior of the audit object can be recognized by obtaining reference video data and images and audio samples of normal behavior. Then, by selecting candidate behavior images and candidate audio samples and performing different feature fusion operations to synthesize video data of typical behaviors and guiding behaviors, a training set containing normal and potentially abnormal behaviors is established. Then, the preset initial behavior recognition model is trained to obtain the target behavior recognition model, and the target behavior recognition model is used to determine the behavior type of the target audit object in the target audit video. Compared with the traditional solution that relies on manual experience to confirm the audit behavior, the method provided in the embodiment of the present application not only accelerates the recognition process of the audit video, but also greatly improves the accuracy of identifying abnormal behaviors through precise model training, reduces the dependence on manual operation, and thus solves the problem of low efficiency of manual review of audit videos.

[0172] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0173] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0174] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0175] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0176] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0177] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0178] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0179] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0180] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0181] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0182] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A behavior recognition method for video interview, characterized in that: The method comprises: Acquire reference video data of a sample face-to-face interview subject in a face-to-face interview state; wherein the reference video data includes a normal behavior image and a normal voice sample of the sample face-to-face interview subject, and the behavior of the sample face-to-face interview subject in the face-to-face interview state in the reference video data is normal behavior; Performing image selection based on the normal behavior image to obtain a candidate behavior image, and performing audio selection based on the normal voice sample to obtain a candidate voice sample; Perform feature fusion according to the normal behavior image, the candidate behavior image, the normal voice sample and the candidate voice sample to obtain typical behavior video data; wherein the behavior of the sample face-to-face interview object in the face-to-face interview state in the typical behavior video data is normal behavior; Perform feature fusion according to the normal behavior image, the normal voice sample and the candidate voice sample to obtain guidance behavior video data; wherein the behavior of the sample face-to-face review object in the face-to-face review state in the guidance behavior video data is abnormal behavior; Training a preset initial behavior recognition model based on the reference video data, the typical behavior video data, and the guidance behavior video data to obtain a target behavior recognition model; The target video data of the target review object in the review state is obtained, and behavior recognition is performed on the target video data according to the target behavior recognition model to obtain the behavior type of the target review object, wherein the behavior type is used to characterize whether the behavior of the target review object in the review state is normal behavior or abnormal behavior.

2. The method according to claim 1, characterized in that The step of performing feature fusion according to the normal behavior image, the candidate behavior image, the normal voice sample and the candidate voice sample to obtain typical behavior video data includes: Performing image feature splicing on the normal behavior image and the candidate behavior image to obtain typical behavior image features; Performing audio feature concatenation on the normal speech sample and the candidate speech sample to obtain typical behavior audio features; The typical behavior image features and the typical behavior audio features are fused to obtain the typical behavior video data.

3. The method according to claim 1, characterized in that The step of performing feature fusion according to the normal behavior image, the normal voice sample and the candidate voice sample to obtain the guiding behavior video data includes: Performing audio feature concatenation on the normal speech sample and the candidate speech sample to obtain a guiding audio feature; The normal behavior image and the guidance audio features are subjected to feature fusion to obtain the guidance behavior video data.

4. The method according to claim 1, characterized in that: The training of a preset initial behavior recognition model based on the reference video data, the typical behavior video data, and the guiding behavior video data to obtain a target behavior recognition model includes: Selecting sample behavior video data from the reference video data, the typical behavior video data, and the guiding behavior video data; Extracting audio features from the sample behavior video data to obtain sample audio features; Extracting picture features from the sample behavior video data to obtain sample picture features; Performing feature fusion on the sample audio feature and the sample picture feature to obtain a sample fusion feature; Performing classification prediction on the sample fusion features according to the initial behavior recognition model to obtain classification probability data; Calculate the loss value according to the classification probability data, the preset label vector and the number of sample behavior video data to obtain behavior classification loss data; The parameters of the initial behavior recognition model are adjusted according to the behavior classification loss data to obtain the target behavior recognition model.

5. The method according to any one of claims 1 to 4, characterized in that: The step of acquiring target video data of the target face-to-face object in the face-to-face state, and performing behavior recognition on the target video data according to the target behavior recognition model to obtain the behavior type of the target face-to-face object includes: Extracting audio features from the target video data to obtain target audio features of the target audit object; Extracting picture features from the target video data to obtain target picture features of the target face-to-face object; Performing feature fusion on the target audio feature and the target picture feature to obtain a target fusion feature; The target fusion feature is subjected to behavior type recognition to obtain the behavior type.

6. The method according to claim 5, characterized in that The step of extracting picture features from the target video data to obtain target picture features of the target face-to-face object includes: Performing video frame interception on the target video data to obtain at least two video frames; Performing facial feature recognition on each of the video frames to obtain target facial feature points of the target face review object; Performing region extraction on the video frame according to the target pixel coordinates of the target facial feature points to obtain a candidate region image; The candidate region image is subjected to pixel coordinate transformation according to the target pixel coordinates to obtain the target region image, and the target region image is subjected to screen feature extraction to obtain the target screen feature.

7. The method according to claim 5, characterized in that After acquiring the target video data of the target audit object in the audit state and performing behavior recognition on the target video data according to the target behavior recognition model to obtain the behavior type of the target audit object, the method further includes: If the behavior type is abnormal behavior, performing speech recognition on the target video data to obtain target text information; Dividing the target text information into sentences to obtain language block information; If the language block information meets the preset abnormal notification triggering condition, the abnormal behavior notification operation is performed; wherein the abnormal notification triggering condition includes that the meanings represented by adjacent language block information are different.

8. A behavior recognition device for video interview, characterized in that: The device comprises: A video acquisition module is used to acquire reference video data of a sample face-to-face interview subject in a face-to-face interview state; wherein the reference video data includes a normal behavior image and a normal voice sample of the sample face-to-face interview subject, and the behavior of the sample face-to-face interview subject in the face-to-face interview state in the reference video data is normal behavior; A sample feature selection module, configured to select images based on the normal behavior images to obtain candidate behavior images, and to select audio based on the normal voice samples to obtain candidate voice samples; A first feature fusion module is used to perform feature fusion according to the normal behavior image, the candidate behavior image, the normal voice sample and the candidate voice sample to obtain typical behavior video data; wherein the behavior of the sample face-to-face review object in the face-to-face review state in the typical behavior video data is normal behavior; A second feature fusion module is used to perform feature fusion according to the normal behavior image, the normal voice sample and the candidate voice sample to obtain guidance behavior video data; wherein the behavior of the sample face-to-face review object in the face-to-face review state in the guidance behavior video data is an abnormal behavior; A model training module, used to train a preset initial behavior recognition model based on the reference video data, the typical behavior video data and the guidance behavior video data to obtain a target behavior recognition model; The behavior recognition module is used to obtain the target video data of the target review object in the review state, and perform behavior recognition on the target video data according to the target behavior recognition model to obtain the behavior type of the target review object, and the behavior type is used to characterize whether the behavior of the target review object in the review state is normal behavior or abnormal behavior.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images