Risk assessment method and device for video review, equipment and storage medium
Through image and voice acquisition, identity detection and abnormal analysis, combined with deep learning technology, the problem of low security in video surface review is solved, and efficient risk assessment and security improvement is achieved.
Patent Information
- Application Number
- CN202411192470.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-12-03
AI Technical Summary
It is difficult to prevent video review from being replaced by third parties through various means, resulting in poor safety. The existing risk assessment methods require a long time and are costly to manpower and material resources.
Through image acquisition, voice acquisition, face identity detection, voiceprint authenticity detection and facial review abnormality detection, video facial review subjects are subject to multiple risk assessments, combined with deep learning algorithms to analyze image and voice data, and identify abnormal behaviors and emotions. , integrate data for risk assessment.
It realizes efficient risk assessment, improves the security of video review, reduces the need for manual intervention and complex evaluation models, and reduces the evaluation cost.
Smart Images

Figure CN119068532A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the fields of artificial intelligence technology and financial technology, and in particular to a risk assessment method and apparatus, device, and storage medium for video interviews. Background Art
[0002] Video interview refers to the interview process and review process conducted through video technology. In this form of interview, the interviewer and the candidate do not need to communicate face to face, but communicate through online video calls. For example, in the field of financial technology, customers can apply for credit cards, savings cards and other bank cards through video interviews.
[0003] At present, video interviews are difficult to prevent third parties (i.e. people other than interviewers and candidates) from conducting alternative audits (such as applying for bank cards on behalf of others) through various means, such as providing false certificates, providing false voices, impersonating candidates to answer, etc., resulting in poor security. Although there are related technologies to conduct risk assessments on video interviews, it usually takes a long time to complete and has high manpower and material costs. Summary of the invention
[0004] The main purpose of the embodiments of the present application is to propose a risk assessment method and device, equipment, and storage medium for video interviews, which can efficiently perform risk assessments and improve the security of video interviews.
[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a risk assessment method for video interview, the method comprising:
[0006] Capturing an image of a target audit object in a video audit state to obtain an image of the target object;
[0007] Collecting the voice of the target interview subject who is in the video interview state to obtain the target subject's voice;
[0008] Performing face identity detection according to the target object image to obtain object identity information; wherein the object identity information is used to characterize whether the target face object is the target object or not;
[0009] Perform voiceprint authenticity detection according to the target object's voice to obtain the object's voiceprint information; wherein the object's voiceprint information is used to indicate whether the target object is the target object or not;
[0010] Performing face-to-face abnormality detection on the target face-to-face object according to the target face-to-face status data to obtain face-to-face abnormality information; wherein the target face-to-face status data includes at least one of the target object image and the target object voice, and the face-to-face abnormality information is used to characterize whether the face-to-face of the target face-to-face object is normal or abnormal;
[0011] A risk assessment is performed based on the object identity information, the object voiceprint information and the face-to-face audit abnormality information.
[0012] In some embodiments, performing face-to-face interview anomaly detection on the target face-to-face interview object according to the target face-to-face interview status data to obtain face-to-face interview anomaly information includes:
[0013] Performing behavior analysis based on the target face-to-face interview status data to obtain abnormal behavior analysis data of the target object;
[0014] Performing speech analysis based on the speech of the target object to obtain speech intervention analysis data of other objects;
[0015] Perform sentiment analysis based on the target interview status data to obtain target object sentiment analysis data;
[0016] Performing face-to-face review status analysis based on the target object image to obtain face-to-face review status analysis data of other objects;
[0017] The face-to-face review abnormality information is obtained by performing data fusion based on the target object's abnormal behavior analysis data, the other object's voice intervention analysis data, the target object's emotion analysis data and the other object's face-to-face review status analysis data.
[0018] In some embodiments, the performing behavior analysis based on the target interview status data to obtain abnormal behavior analysis data of the target object includes:
[0019] Performing face offset analysis based on the target object image and a preset reference face angle range to obtain face offset data;
[0020] Comparing the language features of the target object's speech with the preset reference language features to obtain language feature difference data;
[0021] Abnormal behavior is identified based on the face offset data and the language feature difference data to obtain abnormal behavior analysis data of the target object.
[0022] In some embodiments, performing speech analysis based on the speech of the target object to obtain speech intervention analysis data of other objects includes:
[0023] Performing speech feature recognition on the target object's speech to obtain target speech features;
[0024] Calculate similarity based on the target speech feature and the reference speech feature provided by the target object to obtain the object speech similarity;
[0025] Voice intervention discrimination is performed based on the object voice similarity to determine the voice intervention analysis data of the other objects.
[0026] In some embodiments, performing sentiment analysis based on the target face-to-face interview status data to obtain object sentiment analysis data includes:
[0027] Extracting a first emotion feature from the target object image to obtain a facial emotion feature;
[0028] Extracting a second emotion feature from the target object's speech to obtain a speech emotion feature;
[0029] Performing emotion classification according to the facial emotion features and the voice emotion features to obtain an object emotion category;
[0030] Determine the target object sentiment analysis data according to the object sentiment category.
[0031] In some embodiments, performing face-to-face interview status analysis based on the target object image to obtain face-to-face interview status analysis data of other objects includes:
[0032] Performing other object recognition on the target object image through a pre-trained face review state analysis model to obtain other object recognition results; wherein the other object recognition results include a first result, and the first result indicates that there are other objects entering the video face review state;
[0033] In response to the first result, the number of times is accumulated to obtain a target accumulated number of entries, and the duration is accumulated according to the first result to obtain a target accumulated duration of entry;
[0034] According to at least one of the target cumulative entry times and the target cumulative entry duration, the analysis data of the other objects entering the face-to-face review status is determined.
[0035] In some embodiments, the analysis data of other objects entering the face-to-face review state includes the cumulative number of target entries and the cumulative duration of target entry, wherein the cumulative number of target entries refers to the number of times other objects enter the face-to-face review state, and the cumulative duration of target entry refers to the duration of time other objects enter the face-to-face review state;
[0036] The data fusion is performed based on the abnormal behavior analysis data of the target object, the voice intervention analysis data of other objects, the emotion analysis data of the target object and the face-to-face interview status analysis data of other objects to obtain the face-to-face interview abnormality information, including:
[0037] If the target cumulative entry times is zero, the presence times of other objects are counted based on the target object abnormal behavior analysis data, the other object voice intervention analysis data and the target object emotion analysis data to obtain the presence times of other objects, and the face-to-face abnormality information is determined based on the presence times of other objects;
[0038] If the cumulative number of entries of the target is greater than zero, a score is determined based on the abnormal behavior analysis data of the target object, the voice intervention analysis data of other objects and the emotion analysis data of the target object to obtain the abnormal behavior score of the target object, the voice intervention score of other objects and the emotion score of the target object, and a score conversion is performed based on the cumulative number of entries of the target and the cumulative entry duration of the target to obtain the face-to-face review score of other objects. The score fusion is performed based on the abnormal behavior score of the target object, the voice intervention score of other objects, the emotion score of the target object and the face-to-face review score of other objects to obtain the face-to-face review abnormal information.
[0039] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a risk assessment device for video interview, the device comprising:
[0040] An image acquisition module is used to acquire images of a target subject in a video face-to-face review state to obtain an image of the target subject;
[0041] A voice collection module is used to collect the voice of the target interview subject in the video interview state to obtain the target subject's voice;
[0042] An identity detection module is used to perform face identity detection based on the target object image to obtain object identity information; wherein the object identity information is used to characterize whether the target face object is the target object or not;
[0043] A voiceprint detection module, used to detect the authenticity of the voiceprint according to the voice of the target object, and obtain the object voiceprint information; wherein the object voiceprint information is used to indicate whether the target object is the target object or not;
[0044] An anomaly detection module, used to perform an anomaly detection on the target face-to-face object according to the target face-to-face state data to obtain anomaly information; wherein the target face-to-face state data includes at least one of the target object image and the target object voice, and the anomaly information is used to characterize whether the face-to-face inspection of the target face-to-face object is normal or abnormal;
[0045] The risk assessment module is used to perform risk assessment based on the object identity information, the object voiceprint information and the face-to-face audit abnormality information.
[0046] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the risk assessment method for video interview described in the first aspect when executing the computer program.
[0047] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the risk assessment method for video interview described in the first aspect above.
[0048] The risk assessment method, device, equipment, and storage medium for video audit proposed in this application perform multi-faceted risk assessment on the target audit object in the video audit state through face identity detection, voiceprint authenticity detection, and audit anomaly detection. There is no need for manual assessment or the use of complex assessment models or algorithms. The embodiments of this application can efficiently perform risk assessment and improve the security of video audits.
[0049] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 is a flow chart of a risk assessment method for video interview provided in an embodiment of the present application;
[0051] Figure 2 yes Figure 1 Flowchart of step 105 in;
[0052] Figure 3 yes Figure 2 Flow chart of step 201 in;
[0053] Figure 4 yes Figure 2 Flowchart of step 202 in FIG.
[0054] Figure 5 yes Figure 2 Flow chart of step 203 in;
[0055] Figure 6 yes Figure 2 Flowchart of step 204 in;
[0056] Figure 7 It is a module structure block diagram of a risk assessment device for video interview provided in an embodiment of the present application;
[0057] Figure 8It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION
[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0059] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0061] First, some nouns involved in this application are analyzed:
[0062] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0063] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information image processing, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.
[0064] With the popularization of the Internet and the rapid development of e-commerce, savings cards, credit cards and other bank cards have become a convenient and fast payment method. However, the online credit card approval process is prone to abnormalities and requires risk control. In the online credit card approval process, traditional risk assessment solutions have the following problems:
[0065] (1) Single data source: Traditional risk control systems mainly rely on structured data within the bank for analysis and evaluation. This may result in a limited data source that cannot cover all risk-related information. The lack of diversified data sources may also limit the ability to fully evaluate and predict risks.
[0066] (2) Inaccurate risk assessment: Since traditional risk assessment mainly relies on structured data, it can only obtain limited information and cannot capture important information contained in unstructured or semi-structured data. This leads to a decline in the ability to assess risks and an inability to comprehensively and accurately analyze actual situations and predict potential risks.
[0067] (3) High cost of manual intervention: Traditional fraud monitoring and risk control identification procedures require a lot of manual intervention and review, which usually takes a lot of time and manpower costs.
[0068] (4) Lack of real-time performance: In traditional risk control systems, data is generally updated and recorded periodically. However, risk control requires real-time data for monitoring and decision-making. Relying solely on static data cannot capture the occurrence and changes of risk events in a timely manner. There is a certain lag, which affects the agility and real-time performance of risk control.
[0069] Based on the above problems, the embodiments of the present application provide a risk assessment method and apparatus, equipment, and storage medium for video interviews, that is, a technology based on multimodal signal recognition of faces and voiceprints and third-party detection is provided for online video interviews. The embodiments of the present application aim to combine voiceprint recognition technology and visual analysis technology, that is, by collecting the applicant's voice and image data, and using deep learning algorithms for analysis and comparison, to ensure the applicant's true identity, and detect whether there are other objects (also known as third parties) involved, to achieve identity recognition and risk prevention and control for online interviews. The embodiments of the present application can efficiently perform risk assessments and improve the security of video interviews.
[0070] The risk assessment method for video face-to-face interview provided in the embodiment of the present application can be applied to the terminal and the server side, and can also be software running on the server side. The server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the risk assessment method for video face-to-face interview, etc., but is not limited to the above forms.
[0071] The present application can be used in many general or special computer system environments or configurations. For example: server computers, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.
[0072] The embodiments of the present application provide a risk assessment method for video interviews, a risk assessment device, an electronic device, and a computer-readable storage medium for video interviews, which are specifically illustrated by the following embodiments. First, the risk assessment method for video interviews in the embodiments of the present application is described.
[0073] It should be noted that in each specific implementation method of the present application, when it comes to processing data related to the user's identity or characteristics such as the user's behavioral data, the user's permission or consent will be obtained first, and the collection, use and processing of such data will comply with relevant laws, regulations and standards.
[0074] Reference Figure 1 , Figure 1 It is an optional flowchart of a risk assessment method for video interview provided in an embodiment of the present application, which may include but is not limited to steps 101 to 106.
[0075] Step 101, collecting images of a target subject in a video face-to-face ...
[0076] Step 102, collecting the voice of the target person in the video interview state to obtain the voice of the target person;
[0077] Step 103, performing face identity detection based on the target object image to obtain object identity information; wherein the object identity information is used to characterize whether the target face object is the target object or not;
[0078] Step 104, performing voiceprint authenticity detection according to the target object's voice to obtain the target voiceprint information; wherein the target voiceprint information is used to indicate whether the target face-to-face object is the target object or not;
[0079] Step 105, performing face-to-face review anomaly detection on the target face-to-face review object according to the target face-to-face review status data to obtain face-to-face review anomaly information; wherein the target face-to-face review status data includes at least one of the target object image and the target object voice, and the face-to-face review anomaly information is used to characterize whether the face-to-face review of the target face-to-face review object is normal or abnormal;
[0080] Step 106, perform risk assessment based on the subject's identity information, the subject's voiceprint information, and face-to-face abnormality information.
[0081] In steps 101 to 106 shown in the embodiment of the present application, the present application performs multi-faceted risk assessment on the target audit object in the video audit state through face identity detection, voiceprint authenticity detection, and audit anomaly detection. There is no need for manual assessment or the use of complex assessment models or algorithms. The embodiment of the present application can efficiently perform risk assessment and improve the security of video audits.
[0082] In step 101 of some embodiments, image acquisition is performed on the target audit object in the video audit state to obtain a target object image. The video audit state refers to the object undergoing a video audit. In one example, audit object A wishes to apply for a credit card through a video audit, and then the video audit start button on the terminal (mobile phone, computer, etc.) can be triggered. First, personal information is filled in and traditional identity verification methods such as entering an ID card or passport through a mobile phone are used. Then, audit object A enters the video audit state, and then image acquisition is started for audit object A through an image acquisition unit (such as a camera) pre-deployed on the terminal to obtain a target object image corresponding to audit object A.
[0083] It is understandable that the target object image includes a face area and a background area. The face area contains facial information of the target object for face review. The background area contains information about the environment in which the target object for face review is located.
[0084] In step 102 of some embodiments, voice collection is performed on the target audit object in the video audit state to obtain the target object's voice. The video audit state has been explained above and will not be repeated here. In one example, in combination with the above example, after audit object A triggers the video audit start button, voice collection can be started for audit object A through a voice collection unit (such as a microphone) pre-deployed on the terminal to obtain the target object's voice corresponding to audit object A.
[0085] It is understandable that the target object speech includes the face-to-face object speech and background speech. The face-to-face object speech contains the speech information of the target face-to-face object. The background speech contains the speech information of the environment where the target face-to-face object is located, and may also contain the speech information of other objects.
[0086] In step 103 of some embodiments, face identity detection is performed based on the target object image to obtain object identity information. The object identity information is used to characterize whether the target review object is the target object or not. In conjunction with the above, the target review object is an object that undergoes video review with a certain identity identifier (ID number, passport number), also known as a candidate. The target object refers to an object with an identity identifier. In theory, the target review object should be consistent with the target object, but if other objects use the target object's identity identifier for video review, the target review object is inconsistent with the target object.
[0087] In one example, the object identity information is an object identity category. The object identity category includes a first identity category and a second identity category. The first identity category indicates that the target face-to-face object is a target object. The second identity category indicates that the target face-to-face object is not a target object. For example, if the first identity category is 1, the second identity category is 0.
[0088] In another example, the object identity information is the object identity probability. The larger the object identity probability, the more it indicates that the target face-to-face object is the target object. The smaller the object identity probability, the more it indicates that the target face-to-face object is not the target object.
[0089] In one embodiment, step 103 may include: firstly extracting facial features from the face area using a deep learning algorithm (such as a convolutional neural network) and converting them into facial feature vectors using an encoder, matching the extracted facial feature vectors with the suspicious face database in the background, and searching for suspicious faces. If a suspicious face is found, the object identity information is determined to be the second category, or the object identity probability is determined to be the first probability. If no suspicious face is found, a combination of liveness detection technology such as infrared light, eyeball activity, lip movement and other biometric features, as well as machine learning and deep learning-based methods are used to identify whether the target face-to-face object actually exists, to avoid attacks by static photos, AI people or videos. If it is determined that the target face-to-face object does not exist, the object identity information is determined to be the second category, or the object identity probability is determined to be the second probability. If it is determined that the target face-to-face object actually exists, the object identity information is determined to be the first category, or the object identity probability is determined to be the third probability. Among them, the first probability is less than the second probability, and the second probability is less than the third probability.
[0090] The above embodiment improves the accuracy of face identity authentication through suspicious face matching and liveness detection.
[0091] In step 104 of some embodiments, the target object's voice is subjected to a voiceprint authenticity check to obtain the target object's voiceprint information, wherein the target object's voiceprint information is used to indicate whether the target object is the target object or not.
[0092] In one example, the object voiceprint information is the object voiceprint category. The object voiceprint category includes a first voiceprint category and a second voiceprint category. The first voiceprint category indicates that the target face-to-face object is the target object. The second voiceprint category indicates that the target face-to-face object is not the target object. For example, if the first voiceprint category is 1, the second voiceprint category is 0.
[0093] In another example, the object voiceprint information is the object voiceprint probability. The larger the object voiceprint probability, the more it indicates that the target face-to-face object is the target object. The smaller the object voiceprint probability, the more it indicates that the target face-to-face object is not the target object.
[0094] Specifically, the target audit object may use recording or provide false voice to simulate the target object for audit during the video audit, which may cause the video audit to be too risky. In the video audit, this embodiment extracts the voiceprint features of the target audit object (such as channel features, spectrum features, time domain features such as Mel frequency cepstrum coefficients, etc.), and uses an encoder to convert the voiceprint features into a vector form that can be used for comparison, thereby obtaining the target voiceprint feature vector. Finally, the authenticity of the customer is evaluated based on machine learning and deep learning methods, such as matching the target voiceprint feature vector with the established suspicious voiceprint feature vector, and determining the object voiceprint information based on the matching degree of the voiceprint features, thereby evaluating the authenticity of the target audit object.
[0095] In step 105 of some embodiments, the target face-to-face review object is subjected to face-to-face review anomaly detection according to the target face-to-face review status data to obtain face-to-face review anomaly information, wherein the target face-to-face review status data includes at least one of the target object image and the target object voice, and the face-to-face review anomaly information is used to characterize whether the face-to-face review of the target face-to-face review object is normal or abnormal.
[0096] In one embodiment, referring to Figure 2 , step 105 may include:
[0097] Step 201, performing behavior analysis based on the target face-to-face interview status data to obtain abnormal behavior analysis data of the target object;
[0098] Step 202, performing speech analysis based on the speech of the target object to obtain speech intervention analysis data of other objects;
[0099] Step 203, performing sentiment analysis based on the target face-to-face interview status data to obtain target object sentiment analysis data;
[0100] Step 204, performing face-to-face review status analysis based on the target object image, and obtaining face-to-face review status analysis data of other objects;
[0101] Step 205, based on the target object abnormal behavior analysis data, other objects voice intervention analysis data, target object emotion analysis data and other objects entering the face-to-face review status analysis data, data fusion is performed to obtain face-to-face review abnormality information.
[0102] The benefit of the embodiment illustrated in the above steps 201 to 205 is that through behavioral analysis, voice analysis, sentiment analysis and face-to-face status analysis, face-to-face anomaly detection is achieved in a total of four dimensions, thereby improving detection accuracy.
[0103] In step 201, pattern recognition and machine learning algorithms can be used to identify abnormal behaviors. Abnormal behaviors include: facial deviation, number of words, intonation, language habits, etc. For example, if the target interviewee suddenly pauses for a long time and then answers quickly during the normal interview process, or looks around, it can be regarded as the intervention of other subjects.
[0104] In one embodiment, step 201 may include: performing face offset analysis based on the target object image and a preset reference face angle range to obtain face offset data; and determining abnormal behavior analysis data of the target object based on the face offset data.
[0105] The face offset data may be a face offset category. The face offset category includes a first offset category and a second offset category. The first face offset category indicates that face offset occurs in the target object image. The second face offset category indicates that face offset does not occur in the target object image. The face offset data may also be a face offset probability. The greater the face offset probability, the more it indicates that face offset occurs in the target object image. The smaller the face offset probability, the more it indicates that face offset does not occur in the target object image.
[0106] For example, the reference face angle range is [-10, 10]. If the angle of the face offset baseline obtained from the target object image recognition is 15 degrees, since 15 is not within [-10, 10], the face offset data is determined to be the first face offset category. If the angle of the face offset baseline obtained from the target object image recognition is 5 degrees, since 5 is within [-10, 10], the face offset data is determined to be the second face offset category. After obtaining the face offset data, the target object abnormal behavior analysis data can be determined based on a preset mapping relationship table or mapping formula, or the face offset data can be directly used as the target object abnormal behavior analysis data.
[0107] In another embodiment, step 201 may include: performing a language feature comparison between the target object's speech and a preset reference language feature to obtain language feature difference data; and determining abnormal behavior analysis data of the target object based on the language feature difference data.
[0108] The language feature difference data may be a language feature difference category. The language feature difference category includes a first language category and a second language category. The first language category indicates that there is other object voice intervention. The second language category indicates that there is no other object voice intervention. The language feature difference data may also be a language difference probability. The greater the voice difference probability, the greater the probability that there is no other object voice intervention. The smaller the voice difference probability, the greater the probability that there is other object voice intervention.
[0109] In one embodiment, referring to Figure 3 , step 201 may include:
[0110] Step 301, performing face offset analysis based on the target object image and a preset reference face angle range to obtain face offset data;
[0111] Step 302, performing language feature comparison between the target object speech and the preset reference language features to obtain language feature difference data;
[0112] Step 303, abnormal behavior identification is performed based on the face offset data and the language feature difference data to obtain abnormal behavior analysis data of the target object.
[0113] Step 301 and step 302 have been explained in the previous text and will not be repeated here.
[0114] In one embodiment, step 303 may include:
[0115] If the face offset data is of the first offset category and / or the language feature difference data is of the first language category, determining that the target object abnormal behavior analysis data is of the first abnormal behavior category;
[0116] If the face offset data is of the second offset category and the language feature difference data is of the second language category, the target object abnormal behavior analysis data is determined to be of the second abnormal behavior category, wherein the first abnormal behavior category indicates that the target object image has abnormal behavior, and the second abnormal behavior category indicates that the target object image has no abnormal behavior.
[0117] The above embodiment can analyze abnormal behavior from two aspects of face deviation and language feature difference, with higher accuracy.
[0118] It should be noted that, when the face offset data is the face offset probability and the language feature difference data is the language difference probability, the process of determining the abnormal behavior analysis data of the target object is similar to the above embodiment and will not be repeated here.
[0119] In one embodiment, referring to Figure 4 , step 202 may include:
[0120] Step 401, performing speech feature recognition on the target object speech to obtain target speech features;
[0121] Step 402, performing similarity calculation based on the target speech feature and the reference speech feature provided by the target object to obtain the object speech similarity;
[0122] Step 403: perform voice intervention discrimination based on the object voice similarity to determine the voice intervention analysis data of other objects.
[0123] Specifically, the target subject's speech is subjected to speech feature recognition, and the speech similarity is obtained by comparing and calculating the similarity between the target speech feature and the reference speech feature provided by the target subject. According to the speech similarity, it can be detected whether there is any intervention or interference by other subjects other than the target subject. The similarity calculation can be performed by using a calculation method such as cosine similarity or Euclidean distance.
[0124] The other object voice intervention analysis data includes voice intervention categories. The voice intervention categories include a first voice intervention category and a second voice intervention category. The first voice intervention category indicates that there is voice intervention from other objects. The second voice intervention category indicates that there is voice intervention from other objects. The other object voice intervention analysis data may also be voice intervention probability. The greater the voice intervention probability, the more it indicates that there is voice intervention from other objects. The smaller the voice intervention probability, the more it indicates that there is no voice intervention from other objects.
[0125] The benefit of the embodiment illustrated in the above steps 401 to 403 is that it can perform voice intervention analysis based on similarity, which helps to improve the accuracy of face-to-face anomaly detection.
[0126] In one embodiment, referring to Figure 5 , step 203 may include:
[0127] Step 501, extracting a first emotion feature from the target object image to obtain a facial emotion feature;
[0128] Step 502, extracting a second emotion feature from the target object's speech to obtain a speech emotion feature;
[0129] Step 503, performing emotion classification according to facial emotion features and voice emotion features to obtain the object emotion category;
[0130] Step 504: Determine the target object sentiment analysis data according to the object sentiment category.
[0131] Specifically, by using the first and second emotional features extracted from the target object's voice and face area, such as the intensity, frequency, number of times emotional words are used, facial micro-expressions, etc. of the voice, the constructed emotion classification model is then used to perform emotion classification on the facial emotion features and the voice emotion features to identify whether the target object is too nervous or has other abnormal emotional changes, so as to identify the honesty and reliability of the target object.
[0132] The target object sentiment analysis data includes sentiment abnormality categories. The sentiment abnormality categories include a first sentiment abnormality category and a second sentiment abnormality category. The first sentiment abnormality category indicates that the target audit object has abnormal emotions. The second sentiment abnormality category indicates that the target audit object does not have abnormal emotions. The target object sentiment analysis data can also be sentiment abnormality probability. The greater the sentiment abnormality probability, the more it indicates that the target audit object has abnormal emotions. The smaller the sentiment abnormality probability, the more it indicates that the target audit object does not have abnormal emotions.
[0133] The benefit of the embodiment illustrated in the above steps 501 to 503 is that it can perform sentiment anomaly analysis based on sentiment classification, which helps to improve the accuracy of face-to-face anomaly detection.
[0134] In one embodiment, referring to Figure 6 , step 204 may include:
[0135] Step 601, performing other object recognition on the target object image through the pre-trained face review state analysis model to obtain other object recognition results; wherein the other object recognition results include a first result, and the first result indicates that there are other objects entering the video face review state;
[0136] Step 602, accumulating the number of times in response to the first result to obtain a target cumulative number of entries, and accumulating the duration according to the first result to obtain a target cumulative entry duration;
[0137] Step 603, determining the status analysis data of other objects entering the face-to-face review according to at least one of the target cumulative number of entries and the target cumulative entry duration.
[0138] Specifically, a deep learning model is trained using an image data set with the presence and absence of a third party annotated to obtain a face-to-face interview status analysis model. The face-to-face interview status analysis model is used to identify whether there are other objects entering the face-to-face interview scene during the face-to-face interview process. If so, the number and time are accumulated. The target cumulative number of entries and the target cumulative entry duration are obtained. The target cumulative number of entries refers to the number of times other objects enter the face-to-face interview state. The target cumulative entry duration refers to the duration for which other objects enter the face-to-face interview state. In one example, if the number threshold and duration threshold are exceeded, a warning is issued to the target face-to-face interview object.
[0139] The benefit of the embodiment illustrated in steps 601 to 603 above is that by recording the number of times and duration that other objects enter the face-to-face review state, it is possible to detect other objects entering the face-to-face review state, which helps to improve the accuracy of video face-to-face review state detection.
[0140] In one embodiment, the analysis data of other objects entering the face-to-face interview state includes the cumulative number of target entries and the cumulative duration of target entry. Step 205 may include:
[0141] If the cumulative number of target entries is zero, the number of other objects' existences is counted based on the target object's abnormal behavior analysis data, other objects' voice intervention analysis data, and target object's emotion analysis data to obtain the number of other objects' existences;
[0142] Determine the abnormal information of the face-to-face review based on the number of times other objects exist.
[0143] For example, if the target object abnormal behavior analysis data is the first abnormal behavior category, the other object existence count is added by 1; if the target object abnormal behavior analysis data is the second abnormal behavior category, the other object existence count is added by 0; if the other object voice intervention analysis data is the first voice intervention category, the other object existence count is added by 1; if the other object voice intervention analysis data is the second voice intervention category, the other object existence count is added by 0; if the target object emotion analysis data is the first abnormal emotion category, the other object existence count is added by 1; if the target object emotion analysis data is the second abnormal emotion category, the other object existence count is added by 0. The other object existence count can be directly used as the face-to-face abnormal information, or it can be searched or calculated according to the other object existence count and the preset count abnormal mapping table and count abnormal formula to obtain the face-to-face abnormal information.
[0144] In another embodiment, step 205 further includes:
[0145] If the cumulative number of target entries is greater than zero, the target object abnormal behavior analysis data, other object voice intervention analysis data and target object emotion analysis data are scored and determined to obtain the target object abnormal behavior score, other object voice intervention score and target object emotion score;
[0146] The scores of other subjects entering the interview are converted according to the target's cumulative number of entries and the target's cumulative entry time, and the scores are obtained;
[0147] The abnormal behavior score of the target object, the voice intervention score of other objects, the emotion score of the target object and the face-to-face review score of other objects are integrated to obtain the face-to-face review abnormality information.
[0148] For example, if the target object abnormal behavior analysis data is the first abnormal behavior category, the target object abnormal behavior score is 80%; if the target object abnormal behavior analysis data is the second abnormal behavior category, the target object abnormal behavior score is 20%; if the other object voice intervention analysis data is the first voice intervention category, the other object voice intervention score is 70%; if the other object voice intervention analysis data is the second voice intervention category, the other object voice intervention score is 30%; if the target object emotion analysis data is the first abnormal emotion category, the target object emotion score is 90%; if the target object emotion analysis data is the second abnormal emotion category, the target object emotion score is 30%. The more the target cumulative entry times, the higher the other object entry face-to-face review score. The longer the target cumulative entry time, the higher the other object entry face-to-face review score. By taking the weighted sum of all the scores obtained, the face-to-face review abnormality information can be obtained.
[0149] The advantage of the above-mentioned embodiment of determining abnormal information of the face-to-face review is that different methods of determining abnormal information of the face-to-face review can be adopted based on whether the target cumulative number of entries is zero or non-zero, and it has high flexibility and applicability.
[0150] In step 106 of some embodiments, risk assessment is performed based on the object identity information, the object voiceprint information and the face-to-face review abnormality information. Corresponding threshold standards can be formulated according to the business needs of different application scenarios. For example, in the bank card face-to-face review scenario, the business party requires that the object voiceprint authenticity in the object voiceprint information corresponding to the voiceprint recognition must be higher than 75%, and the object identity authenticity matching degree of the object identity information corresponding to the face recognition must exceed 80%. The score of the face-to-face review abnormality information corresponding to the face-to-face review status detection must be higher than 80%, etc. When the risk assessment result of the target face-to-face review object is abnormal, a secondary verification is performed through manual intervention or other means. In one example, the target object image, target object voice and risk assessment results are entered into a database map each time to establish a suspicious voiceprint library and a suspicious face library and assist in the subsequent model parameter tuning process.
[0151] In summary of the above embodiments, this application can verify the identity of the target audit object, identify and warn other objects of illegal operations on behalf of others, and improve the accuracy of the audit by applying multimodal data recognition to the online video audit process. It can also reduce misjudgments due to manual negligence or subjective judgment errors, and thus timely warn of financial violation risks to protect the property safety of financial institutions and customers.
[0152] See also Figure 7 The embodiment of the present application also provides a risk assessment device for video interview, which can implement the risk assessment method for video interview. Figure 7This is a block diagram of the module structure of a risk assessment device for video interview provided in an embodiment of the present application, the device comprising: an image acquisition module 701, a voice acquisition module 702, an identity detection module 703, a voiceprint detection module 704, an anomaly detection module 705 and a risk assessment module 706. Among them, the image acquisition module 701 is used to acquire images of the target review object in the video review state to obtain the target object image; the voice acquisition module 702 is used to acquire voice of the target review object in the video review state to obtain the target object voice; the identity detection module 703 is used to perform face identity detection based on the target object image to obtain object identity information; wherein the object identity information is used to characterize whether the target review object is the target object or not; the voiceprint detection module 704 is used to perform voiceprint authenticity detection based on the target object voice to obtain object voiceprint information; wherein the object voiceprint information is used to characterize whether the target review object is the target object or not; the anomaly detection module 705 is used to perform anomaly detection on the target review object based on the target review state data to obtain anomaly information; wherein the target review state data includes at least one of the target object image and the target object voice, and the anomaly information is used to characterize whether the target review object is normal or abnormal; the risk assessment module 706 is used to perform risk assessment based on the object identity information, the object voiceprint information and the anomaly information.
[0153] It should be noted that the specific implementation of the risk assessment device for video interview is basically the same as the specific implementation of the risk assessment method for video interview mentioned above, and will not be repeated here.
[0154] The embodiment of the present application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and when the program is executed by the processor, the above-mentioned risk assessment method for video face-to-face interview is realized. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.
[0155] See also Figure 8 , Figure 8 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:
[0156] The processor 801 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.
[0157] The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 802, and the processor 801 calls and executes the risk assessment method for video face-to-face interview in the embodiment of this application;
[0158] Input / output interface 803, used to implement information input and output;
[0159] The communication interface 804 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WI FI, Bluetooth, etc.);
[0160] A bus 805 that transmits information between the various components of the device (e.g., the processor 801, the memory 802, the input / output interface 803, and the communication interface 804);
[0161] The processor 801 , the memory 802 , the input / output interface 803 and the communication interface 804 are connected to each other in communication within the device via a bus 805 .
[0162] An embodiment of the present application also provides a computer-readable storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned risk assessment method for video interview.
[0163] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0164] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0165] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.
[0166] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0167] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.
[0168] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0169] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0170] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0171] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0172] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0173] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.
[0174] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.
Claims
1. A risk assessment method for video interview, characterized in that: The method comprises: Capturing an image of a target audit object in a video audit state to obtain an image of the target object; Collecting the voice of the target interview subject who is in the video interview state to obtain the target subject's voice; Performing face identity detection according to the target object image to obtain object identity information; wherein the object identity information is used to characterize whether the target face object is the target object or not; Perform voiceprint authenticity detection according to the target object's voice to obtain the object's voiceprint information; wherein the object's voiceprint information is used to indicate whether the target object is the target object or not; Performing face-to-face abnormality detection on the target face-to-face object according to the target face-to-face status data to obtain face-to-face abnormality information; wherein the target face-to-face status data includes at least one of the target object image and the target object voice, and the face-to-face abnormality information is used to characterize whether the face-to-face of the target face-to-face object is normal or abnormal; A risk assessment is performed based on the object identity information, the object voiceprint information and the face-to-face audit abnormality information.
2. The method according to claim 1, characterized in that: The step of performing face-to-face review anomaly detection on the target face-to-face review object according to the target face-to-face review status data to obtain face-to-face review anomaly information includes: Performing behavior analysis based on the target face-to-face interview status data to obtain abnormal behavior analysis data of the target object; Performing speech analysis based on the speech of the target object to obtain speech intervention analysis data of other objects; Perform sentiment analysis based on the target interview status data to obtain target object sentiment analysis data; Performing face-to-face review status analysis based on the target object image to obtain face-to-face review status analysis data of other objects; The face-to-face review abnormality information is obtained by performing data fusion based on the target object's abnormal behavior analysis data, the other object's voice intervention analysis data, the target object's emotion analysis data and the other object's face-to-face review status analysis data.
3. The method according to claim 2, characterized in that The performing behavior analysis according to the target face-to-face interview status data to obtain abnormal behavior analysis data of the target object includes: Performing face offset analysis based on the target object image and a preset reference face angle range to obtain face offset data; Comparing the language features of the target object's speech with the preset reference language features to obtain language feature difference data; Abnormal behavior is identified based on the face offset data and the language feature difference data to obtain abnormal behavior analysis data of the target object.
4. The method according to claim 2, characterized in that: The performing speech analysis according to the speech of the target object to obtain speech intervention analysis data of other objects includes: Performing speech feature recognition on the target object's speech to obtain target speech features; Calculate similarity based on the target speech feature and the reference speech feature provided by the target object to obtain the object speech similarity; Voice intervention discrimination is performed based on the object voice similarity to determine the voice intervention analysis data of the other objects.
5. The method according to claim 2, characterized in that: The performing of sentiment analysis according to the target face-to-face interview status data to obtain the target sentiment analysis data includes: Extracting a first emotion feature from the target object image to obtain a facial emotion feature; Extracting a second emotion feature from the target object's speech to obtain a speech emotion feature; Performing emotion classification according to the facial emotion features and the voice emotion features to obtain an object emotion category; Determine the target object sentiment analysis data according to the object sentiment category.
6. The method according to claim 2, characterized in that The performing face-to-face review status analysis according to the target object image to obtain face-to-face review status analysis data of other objects includes: Performing other object recognition on the target object image through a pre-trained face review state analysis model to obtain other object recognition results; wherein the other object recognition results include a first result, and the first result indicates that there are other objects entering the video face review state; In response to the first result, the number of entries is accumulated to obtain a target accumulated number of entries, and the duration is accumulated according to the first result to obtain a target accumulated duration of entry; According to at least one of the target cumulative entry times and the target cumulative entry duration, the analysis data of the other objects entering the face-to-face review status is determined.
7. The method according to any one of claims 2 to 6, characterized in that: The analysis data of other objects entering the face-to-face review state includes the cumulative number of target entries and the cumulative duration of target entry, wherein the cumulative number of target entries refers to the number of times other objects enter the face-to-face review state, and the cumulative duration of target entry refers to the duration of time other objects enter the face-to-face review state; The data fusion is performed based on the abnormal behavior analysis data of the target object, the voice intervention analysis data of other objects, the emotion analysis data of the target object and the face-to-face interview status analysis data of other objects to obtain the face-to-face interview abnormality information, including: If the target cumulative entry times is zero, the presence times of other objects are counted based on the target object abnormal behavior analysis data, the other object voice intervention analysis data and the target object emotion analysis data to obtain the presence times of other objects, and the face-to-face abnormality information is determined based on the presence times of other objects; If the cumulative number of entries of the target is greater than zero, a score is determined based on the abnormal behavior analysis data of the target object, the voice intervention analysis data of other objects and the emotion analysis data of the target object to obtain the abnormal behavior score of the target object, the voice intervention score of other objects and the emotion score of the target object, and a score conversion is performed based on the cumulative number of entries of the target and the cumulative entry duration of the target to obtain the face-to-face review score of other objects. The score fusion is performed based on the abnormal behavior score of the target object, the voice intervention score of other objects, the emotion score of the target object and the face-to-face review score of other objects to obtain the face-to-face review abnormal information.
8. A risk assessment device for video interview, characterized in that: The device comprises: An image acquisition module is used to acquire images of a target subject in a video face-to-face review state to obtain an image of the target subject; A voice collection module is used to collect the voice of the target interview subject in the video interview state to obtain the target subject's voice; An identity detection module is used to perform face identity detection based on the target object image to obtain object identity information; wherein the object identity information is used to characterize whether the target face object is the target object or not; A voiceprint detection module, used to detect the authenticity of the voiceprint according to the voice of the target object, and obtain the object voiceprint information; wherein the object voiceprint information is used to indicate whether the target object is the target object or not; An anomaly detection module, used to perform an anomaly detection on the target face-to-face object according to the target face-to-face state data to obtain anomaly information; wherein the target face-to-face state data includes at least one of the target object image and the target object voice, and the anomaly information is used to characterize whether the face-to-face inspection of the target face-to-face object is normal or abnormal; The risk assessment module is used to perform risk assessment based on the object identity information, the object voiceprint information and the face-to-face audit abnormality information.
9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Face examination video data processing method and device, computer equipment and storage medium
CN110378228A
Facial examination monitoring method and device, equipment and storage medium
CN115424311A
Online review processing method and device, storage medium and equipment
CN116364094A
