Multi-person face-to-face audit audio and video recognition processing method and device, equipment and storage medium
By marking text content timestamps and counting the duration of mouth movement on audio and video files, combining voiceprints and large language models to judge, the problem of false alarms when the agent does not fully guide the insured's speech is solved, and the face-to-face review process is improved accuracy and efficiency.
Patent Information
- Application Number
- CN202411065847.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-11-26
AI Technical Summary
During the insurance industry's face-to-face review process, when the agent speaks to the insured not completely instructs the speech, the existing sound and lip synchronization model is inaccurate, resulting in frequent false alarms, which increases the review cost.
By obtaining the text content of audio and video files and marking them in timestamps, positioning target speech and corresponding time periods, counting the duration of mouth movement of multiple users, determining the core user, and combining voiceprint recognition and large language model to determine the identity attributes of core users. Avoid false alarms.
It effectively avoids false alarms for Mandarin, reduces the cost of manual review, and improves the accuracy and efficiency of the face-to-face review process.
Smart Images

Figure CN119028347A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of face-to-face interviews, and in particular to an audio and video recognition processing method, device, equipment and storage medium for face-to-face interviews with multiple people. Background Art
[0002] In the insurance industry, ensuring that the insured fully understands and agrees to the insurance terms is a very important part. In the traditional face-to-face interview process, by observing whether the insured's voice and lip shape are synchronized, it is possible to effectively identify whether the agent or other person is guiding or answering questions on behalf of the insured, thereby avoiding violations.
[0003] However, in actual scenarios, not all the words the agent speaks to the policyholder are instructions. Only certain words are instructions, such as "confirm" and "agree". In this case, it is hoped that no alarm will be issued for the agent's ordinary words. In addition, since the policyholder's answer is generally very brief, the lip synchronization model may not be accurate, which will lead to false alarms. Further transfer to manual approval will result in high review costs. Summary of the invention
[0004] The present application provides a method, device, equipment and storage medium for audio and video recognition processing for multi-person face-to-face interviews, which solves the technical problem that in actual scenarios, the agent's speech to the insured is not completely instructive, and no alarm is issued for ordinary speech. In addition, since the insured's answer is generally very brief, the lip synchronization model may not be accurate, which will lead to false alarms.
[0005] In view of this, the first aspect of the present application provides an audio and video recognition processing method for multi-person face-to-face review, the method comprising:
[0006] Step S1, obtaining corresponding text content based on the audio and video file to be identified, and marking each short sentence contained in the text content with a timestamp;
[0007] Step S2, locating at least one target speech and a time period corresponding to the target speech according to the text content;
[0008] Step S3, based on the lip synchronization model, respectively counting the mouth movement durations of two or more users included in the audio and video file to be identified in the time period corresponding to each target speech;
[0009] Step S4, determining a core user among two or more users according to the mouth movement duration;
[0010] Step S5, identifying the identity attribute of the core user, if the identity attribute is a proxy user, making an alarm judgment according to the preset list to which the text content corresponding to the target speech belongs.
[0011] Optionally, the step S1 specifically includes:
[0012] The preset ASR model is used to perform text recognition on the audio and video files to be recognized, to obtain the corresponding text content, and to timestamp each short sentence contained in the text content.
[0013] Optionally, the step S5 specifically includes:
[0014] Extracting a first voiceprint feature of the core user in the audio and video file to be identified based on a voiceprint recognition model;
[0015] Comparing the first voiceprint feature with the second voiceprint feature pre-stored in a database to determine the identity attributes of the core user, the identity attributes including the agent user and the insured user;
[0016] If the first voiceprint feature and the second voiceprint feature cannot match, the identity attribute of the core user is marked as other users.
[0017] Optionally, the step S5 further includes:
[0018] If the identity attribute is a proxy user or other user, and the text content corresponding to the target speech belongs to the preset reminder speech list, no alarm will be issued;
[0019] If the identity attribute is a proxy user or other user, and the text content corresponding to the target speech belongs to the preset warning speech list, an alarm is issued;
[0020] If the identity attribute is a proxy user or other user, but the text content corresponding to the target speech does not belong to the preset reminder speech list and the preset warning speech list, the length of the text content is calculated, and an alarm is issued when the length is less than the preset length threshold, otherwise no alarm is issued.
[0021] Optionally, the step S5 further includes:
[0022] If the identity attribute is a proxy user or other user, but the text content corresponding to the target speech does not belong to the preset reminder speech list and the preset warning speech list, the text content corresponding to the target speech is identified based on the preset large language model to determine whether there is any abnormality in the text content corresponding to the target speech. If there is an abnormality, an alarm is issued, otherwise no alarm is issued.
[0023] A second aspect of the present application provides an audio and video recognition processing device for multi-person face-to-face interviews, the device comprising:
[0024] An acquisition unit, configured to acquire corresponding text content based on the audio and video file to be identified, and timestamp each short sentence contained in the text content;
[0025] A positioning unit, configured to locate at least one target speech and a time period corresponding to the target speech according to the text content;
[0026] A statistical unit, configured to respectively count the mouth movement durations of two or more users included in the audio and video file to be identified in a time period corresponding to each target speech based on a lip synchronization model;
[0027] a determining unit, configured to determine a core user among two or more users according to the mouth movement duration;
[0028] The early warning unit is used to identify the identity attribute of the core user. If the identity attribute is a proxy user, an alarm judgment is made according to a preset list to which the text content corresponding to the target speech belongs.
[0029] Optionally, the acquiring unit is specifically used for:
[0030] The preset ASR model is used to perform text recognition on the audio and video files to be recognized, to obtain the corresponding text content, and to timestamp each short sentence contained in the text content.
[0031] Optionally, the early warning unit is specifically used for:
[0032] Extracting a first voiceprint feature of the core user in the audio and video file to be identified based on a voiceprint recognition model;
[0033] Comparing the first voiceprint feature with the second voiceprint feature pre-stored in a database to determine the identity attributes of the core user, the identity attributes including the agent user and the insured user;
[0034] If the first voiceprint feature and the second voiceprint feature cannot match, the identity attribute of the core user is marked as other users.
[0035] Optionally, the early warning unit is further used for:
[0036] If the identity attribute is a proxy user or other user, and the text content corresponding to the target speech belongs to the preset reminder speech list, no alarm will be issued;
[0037] If the identity attribute is a proxy user or other user, and the text content corresponding to the target speech belongs to the preset warning speech list, an alarm is issued;
[0038] If the identity attribute is a proxy user or other user, but the text content corresponding to the target speech does not belong to the preset reminder speech list and the preset warning speech list, the length of the text content is calculated, and an alarm is issued when the length is less than the preset length threshold, otherwise no alarm is issued.
[0039] Optionally, the early warning unit is further used for:
[0040] If the identity attribute is a proxy user or other user, but the text content corresponding to the target speech does not belong to the preset reminder speech list and the preset warning speech list, the text content corresponding to the target speech is identified based on the preset large language model to determine whether there is any abnormality in the text content corresponding to the target speech. If there is an abnormality, an alarm is issued, otherwise no alarm is issued.
[0041] The third aspect of the present application provides an audio and video recognition processing device for multi-person face-to-face interviews, the device comprising a processor and a memory:
[0042] The memory is used to store program code and transmit the program code to the processor;
[0043] The processor is used to execute the steps of the method for audio and video recognition processing for multi-person face-to-face review as described in the first aspect above according to the instructions in the program code.
[0044] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the method described in the first aspect above.
[0045] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:
[0046] In this application, a method, device, equipment and storage medium for multi-person face-to-face interviews are provided. The target speech is located in the audio and video files to be identified, and the mouth movement duration of each user in the time period corresponding to the target speech is counted based on the lip synchronization model to determine the core users in the time period of the target speech. Then, by identifying the identity attributes of the core users and combining the preset list, an alarm judgment is made on the text content corresponding to the target speech, avoiding the situation where a single lip synchronization model makes a wrong judgment. This solves the technical problem that in actual scenarios, the agent does not completely guide the speech when speaking to the insured, and does not alarm for ordinary speech. Moreover, since the insured's answer is generally very short, the lip synchronization model may not be accurate, which will lead to false alarms. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 A method flow chart of the audio and video recognition processing method for multi-person face-to-face review in an embodiment of the present application;
[0048] Figure 2 This is a structural diagram of an audio and video recognition processing device for multi-person face-to-face interview in an embodiment of the present application;
[0049] Figure 3 It is a structural diagram of the audio and video recognition processing equipment for multi-person face-to-face review in an embodiment of the present application. DETAILED DESCRIPTION
[0050] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0051] The present application designs a method, device, equipment and storage medium for audio and video recognition processing for multi-person face-to-face interviews, which solves the technical problem that in actual scenarios, the agent's speech to the insured is not completely instructive, and no alarm is issued for ordinary speech. In addition, since the insured's answers are generally very brief, the lip synchronization model may not be accurate, which will lead to false alarms.
[0052] For easier understanding, see Figure 1 , Figure 1 Flow chart of the method for audio and video recognition processing of multi-person face-to-face interview in the embodiment of the present application. Figure 1 As shown, specifically:
[0053] Step S1, obtaining corresponding text content based on the audio and video files to be identified, and marking each short sentence contained in the text content with a timestamp;
[0054] It should be noted that the audio and video files to be recognized are imported, processed by the ASR model, and the speech is converted into text. Timestamps are automatically added to each short sentence in the text to correspond to the corresponding part in the audio and video. The timestamps should be added accurately to the millisecond level to ensure the synchronization of text and audio and video.
[0055] Step S2: Locate at least one target speech and a time period corresponding to the target speech according to the text content;
[0056] It should be noted that, according to the preset target speech list, the text content is scanned, the target speech and its position in the text are determined, and the time period corresponding to the target speech is extracted.
[0057] Step S3: Based on the lip synchronization model, the mouth movement durations of two or more users in the audio and video files to be identified are counted in the time period corresponding to each target speech;
[0058] It should be noted that during the time period of the target speech, the audio and video files are analyzed and the mouth movement duration of each user is counted.
[0059] Step S4, determining a core user among two or more users according to the mouth movement duration;
[0060] It should be noted that core users are identified based on the duration of mouth movements.
[0061] Use computer vision techniques, such as convolutional neural networks (CNNs) in deep learning, to detect the mouth area of people in the video.
[0062] For each detected mouth movement, its start and end time points were recorded, and the duration of the mouth movement was calculated.
[0063] If there are multiple users in the video, it is necessary to distinguish the mouth movements of different users. This can be achieved through multimodal analysis, combining the sound signal and the video image.
[0064] The mouth movement durations of all users were aggregated and compared.
[0065] Core users may be the users who speak the most or play a leading role in the conversation.
[0066] Step S5: Identify the identity attribute of the core user. If the identity attribute is a proxy user, make an alarm judgment according to the preset list to which the text content corresponding to the target speech belongs.
[0067] It should be noted that voice samples of core users are extracted and analyzed using the voiceprint recognition model.
[0068] The voiceprint features are compared with those in the database to determine the user's identity.
[0069] Based on the user identity and the text content of the target speech, the following judgments are made:
[0070] If the user is a proxy user and the text belongs to the reminder script list, no alarm will be triggered.
[0071] If the user is a proxy and the text belongs to the warning script list, an alarm will be issued.
[0072] If the text content does not belong to any preset list, a decision on whether to alarm is made based on the text length and the analysis results of the large language model.
[0073] Furthermore, step S1 specifically includes:
[0074] The preset ASR model is used to perform text recognition on the audio and video files to be recognized, obtain the corresponding text content, and timestamp each short sentence contained in the text content.
[0075] It should be noted that the ASR model can ensure that the model can accurately recognize different accents and background noise. The ASR model can be an open source model or can be trained by the user.
[0076] Furthermore, step S5 specifically includes:
[0077] Extracting the first voiceprint feature of the core user in the audio and video file to be identified based on the voiceprint recognition model;
[0078] Compare the first voiceprint feature with the second voiceprint feature pre-stored in the database to determine the identity attributes of the core user, the identity attributes including the agent user and the insured user;
[0079] If the first voiceprint feature does not match the second voiceprint feature, the identity attribute of the core user is marked as another user.
[0080] It should be noted that a database containing multiple voiceprint features is first established to improve recognition accuracy. The database pre-stores voiceprints of several proxy users and several insured users.
[0081] Extract voice samples of core users and analyze them using the voiceprint recognition model.
[0082] The voiceprint features are compared with those in the database to determine the user's identity.
[0083] If the voiceprint feature does not belong to the database, the core user is marked as other users.
[0084] Furthermore, step S5 further includes:
[0085] If the identity attribute is a proxy user or other user, and the text content corresponding to the target speech belongs to the preset reminder speech list, no alarm will be issued;
[0086] If the identity attribute is a proxy user or other user, and the text content corresponding to the target speech belongs to the preset warning speech list, an alarm will be issued;
[0087] If the identity attribute is a proxy user or other user, but the text content corresponding to the target speech does not belong to the preset reminder speech list and the preset warning speech list, the length of the text content is calculated. When the length is less than the preset length threshold, an alarm is issued, otherwise no alarm is issued.
[0088] It should be noted that a warning script list and a reminder script list are given.
[0089] If the text recognized by ASR contains words in the warning speech list, and this speech is spoken by the proxy user, a direct warning will be issued; if the text recognized by ASR contains words in the reminder speech list, no warning will be issued.
[0090] Due to special circumstances such as noise or dialects, if the text recognized by ASR does not contain any words in the warning speech list or the reminder speech list, but the sentence is spoken by the proxy user, the length of the ASR-recognized text is used to determine whether it is a guidance speech.
[0091] If the agent user's words are long, they are usually reminder words. If the agent user's words are short, they are usually guidance words. Therefore, the length of the ASR recognition text is calculated. If the length is less than the length threshold, an alert is issued. If the length is greater than the length threshold, no alert is issued.
[0092] Furthermore, step S5 further includes:
[0093] If the identity attribute is a proxy user or other user, but the text content corresponding to the target speech does not belong to the preset reminder speech list and the preset warning speech list, the text content corresponding to the target speech is identified based on the preset large language model to determine whether there is any abnormality in the text content corresponding to the target speech. If there is an abnormality, an alarm is issued, otherwise no alarm is issued.
[0094] It should be noted that there is some doubt in identifying the words as reminder words or guidance words only from the length of the text. In this case, a preset large language model is further used to conduct an in-depth analysis of the text content to identify potential anomalies or risks.
[0095] As technology develops and user needs change, ASR models, voiceprint recognition models, and large language models need to be updated regularly to maintain the accuracy and effectiveness of the system.
[0096] See also Figure 2 , Figure 2 FIG. 1 is a structural diagram of an audio and video recognition processing device for multi-person face-to-face interview in an embodiment of the present application. Figure 2 As shown, specifically:
[0097] The acquisition unit 201 is used to acquire the corresponding text content based on the audio and video file to be identified, and to timestamp each short sentence contained in the text content;
[0098] It should be noted that the audio and video files to be recognized are imported, processed by the ASR model, and the speech is converted into text. Timestamps are automatically added to each short sentence in the text to correspond to the corresponding part in the audio and video. The timestamps should be added accurately to the millisecond level to ensure the synchronization of text and audio and video.
[0099] A positioning unit 202, configured to locate at least one target speech and a time period corresponding to the target speech according to the text content;
[0100] It should be noted that, according to the preset target speech list, the text content is scanned, the target speech and its position in the text are determined, and the time period corresponding to the target speech is extracted.
[0101] A counting unit 203 is used to count the mouth movement durations of two or more users in the audio and video file to be identified in a time period corresponding to each target speech based on the lip synchronization model;
[0102] It should be noted that during the time period of the target speech, the audio and video files are analyzed and the mouth movement duration of each user is counted.
[0103] A determination unit 204, configured to determine a core user among two or more users according to the mouth movement duration;
[0104] It should be noted that core users are identified based on the duration of mouth movements.
[0105] Use computer vision techniques, such as convolutional neural networks (CNNs) in deep learning, to detect the mouth area of people in the video.
[0106] For each detected mouth movement, its start and end time points were recorded, and the duration of the mouth movement was calculated.
[0107] If there are multiple users in the video, it is necessary to distinguish the mouth movements of different users. This can be achieved through multimodal analysis, combining the sound signal and the video image.
[0108] The mouth movement durations of all users were aggregated and compared.
[0109] The early warning unit 205 is used to identify the identity attribute of the core user. If the identity attribute is a proxy user, an alarm judgment is made according to the preset list to which the text content corresponding to the target speech belongs.
[0110] It should be noted that voice samples of core users are extracted and analyzed using the voiceprint recognition model.
[0111] The voiceprint features are compared with those in the database to determine the user's identity.
[0112] Based on the user identity and the text content of the target speech, the following judgments are made:
[0113] If the user is a proxy user and the text belongs to the reminder script list, no alarm will be triggered.
[0114] If the user is a proxy and the text belongs to the warning script list, an alarm will be issued.
[0115] If the text content does not belong to any preset list, a decision on whether to alarm is made based on the text length and the analysis results of the large language model.
[0116] Furthermore, the acquisition unit 201 is specifically used for:
[0117] The preset ASR model is used to perform text recognition on the audio and video files to be recognized, obtain the corresponding text content, and timestamp each short sentence contained in the text content.
[0118] It should be noted that the ASR model can ensure that the model can accurately recognize different accents and background noise. The ASR model can be an open source model or can be trained by the user.
[0119] Furthermore, the early warning unit 205 is specifically used for:
[0120] Extracting the first voiceprint feature of the core user in the audio and video file to be identified based on the voiceprint recognition model;
[0121] Compare the first voiceprint feature with the second voiceprint feature pre-stored in the database to determine the identity attributes of the core user, the identity attributes including the agent user and the insured user;
[0122] If the first voiceprint feature does not match the second voiceprint feature, the identity attribute of the core user is marked as another user.
[0123] It should be noted that a database containing multiple voiceprint features is first established to improve recognition accuracy. The database pre-stores voiceprints of several proxy users and several insured users.
[0124] Extract voice samples of core users and analyze them using the voiceprint recognition model.
[0125] The voiceprint features are compared with those in the database to determine the user's identity.
[0126] If the voiceprint feature does not belong to the database, the core user is marked as other users.
[0127] Furthermore, the early warning unit 205 is also used for:
[0128] If the identity attribute is a proxy user or other user, and the text content corresponding to the target speech belongs to the preset reminder speech list, no alarm will be issued;
[0129] If the identity attribute is a proxy user or other user, and the text content corresponding to the target speech belongs to the preset warning speech list, an alarm will be issued;
[0130] If the identity attribute is a proxy user or other user, but the text content corresponding to the target speech does not belong to the preset reminder speech list and the preset warning speech list, the length of the text content is calculated. When the length is less than the preset length threshold, an alarm is issued, otherwise no alarm is issued.
[0131] It should be noted that a warning script list and a reminder script list are given.
[0132] If the text recognized by ASR contains words in the warning speech list, and this speech is spoken by the proxy user, a direct warning will be issued; if the text recognized by ASR contains words in the reminder speech list, no warning will be issued.
[0133] Due to special circumstances such as noise or dialects, if the text recognized by ASR does not contain any words in the warning speech list or the reminder speech list, but the sentence is spoken by the proxy user, the length of the ASR-recognized text is used to determine whether it is a guidance speech.
[0134] If the agent user's words are long, they are usually reminder words. If the agent user's words are short, they are usually guidance words. Therefore, the length of the ASR recognition text is calculated. If the length is less than the length threshold, an alert is issued. If the length is greater than the length threshold, no alert is issued.
[0135] Furthermore, the early warning unit 205 is also used for:
[0136] If the identity attribute is a proxy user or other user, but the text content corresponding to the target speech does not belong to the preset reminder speech list and the preset warning speech list, the text content corresponding to the target speech is identified based on the preset large language model to determine whether there is any abnormality in the text content corresponding to the target speech. If there is an abnormality, an alarm is issued, otherwise no alarm is issued.
[0137] It should be noted that there is some doubt in identifying the words as reminder words or guidance words only from the length of the text. In this case, a preset large language model is further used to conduct an in-depth analysis of the text content to identify potential anomalies or risks.
[0138] As technology develops and user needs change, ASR models, voiceprint recognition models, and large language models need to be updated regularly to maintain the accuracy and effectiveness of the system.
[0139] The present application also provides another audio and video recognition processing device for multi-person face-to-face interviews, such as Figure 3For the sake of convenience, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. The terminal can be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (English full name: Personal Digital Assistant, English abbreviation: PDA), a sales terminal (English full name: Point of Sales, English abbreviation: POS), a car computer, etc., taking the terminal as a mobile phone as an example:
[0140] Figure 3 FIG. 1 is a block diagram showing a partial structure of a mobile phone related to a terminal provided in an embodiment of the present application. Figure 3 The mobile phone includes: a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090. Those skilled in the art can understand that Figure 3 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0141] Combine the following Figure 3 A detailed introduction to the various components of the mobile phone:
[0142] The RF circuit 1010 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 1080 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 1010 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (full name: Low Noise Amplifier, English abbreviation: LNA), a duplexer, etc. In addition, the RF circuit 1010 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communications can use any communication standard or protocol, including but not limited to Global System of Mobile communications (Global System of Mobile communication, English abbreviation: GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, English abbreviation: CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (Long Term Evolution, English abbreviation: LTE), e-mail, Short Messaging Service (SMS), etc.
[0143] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1020. The memory 1020 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1020 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0144] The input unit 1030 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 1031 or near the touch panel 1031 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 1080, and can receive and execute commands sent by the processor 1080. In addition, the touch panel 1031 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 1031, the input unit 1030 may further include other input devices 1032. Specifically, the other input devices 1032 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.
[0145] The display unit 1040 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1040 may include a display panel 1041. Optionally, the display panel 1041 may be configured in the form of a liquid crystal display (full name in English: Liquid Crystal Display, English abbreviation: LCD), an organic light-emitting diode (full name in English: Organic Light-Emitting Diode, English abbreviation: OLED), etc. Further, the touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it is transmitted to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides a corresponding visual output on the display panel 1041 according to the type of touch event. Although in Figure 3 In the embodiment, the touch panel 1031 and the display panel 1041 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone.
[0146] The mobile phone may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1041 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.
[0147] The audio circuit 1060, the speaker 1061, and the microphone 1062 can provide an audio interface between the user and the mobile phone. The audio circuit 1060 can transmit the received audio data to the speaker 1061 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 1062 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1060 and converted into audio data, and then the audio data is output to the processor 1080 for processing, and then sent to another mobile phone through the RF circuit 1010, or the audio data is output to the memory 1020 for further processing.
[0148] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the WiFi module 1070. It provides users with wireless broadband Internet access. Figure 3 A WiFi module 1070 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.
[0149] The processor 1080 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 1020 and calling data stored in the memory 1020, it executes various functions of the mobile phone and processes data, thereby monitoring the mobile phone as a whole. Optionally, the processor 1080 may include one or more processing units; preferably, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1080.
[0150] The mobile phone also includes a power supply 1090 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 1080 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.
[0151] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.
[0152] In the embodiment of the present application, the processor 1080 included in the terminal also has the following functions:
[0153] Step S1, obtaining corresponding text content based on the audio and video files to be identified, and marking each short sentence contained in the text content with a timestamp;
[0154] It should be noted that the audio and video files to be recognized are imported, processed by the ASR model, and the speech is converted into text. Timestamps are automatically added to each short sentence in the text to correspond to the corresponding part in the audio and video. The timestamps should be added accurately to the millisecond level to ensure the synchronization of text and audio and video.
[0155] Step S2: Locate at least one target speech and a time period corresponding to the target speech according to the text content;
[0156] It should be noted that, according to the preset target speech list, the text content is scanned, the target speech and its position in the text are determined, and the time period corresponding to the target speech is extracted.
[0157] Step S3: Based on the lip synchronization model, the mouth movement durations of two or more users in the audio and video files to be identified are counted in the time period corresponding to each target speech;
[0158] It should be noted that during the time period of the target speech, the audio and video files are analyzed and the mouth movement duration of each user is counted.
[0159] Step S4, determining a core user among two or more users according to the mouth movement duration;
[0160] It should be noted that core users are identified based on the duration of mouth movements.
[0161] Use computer vision techniques, such as convolutional neural networks (CNNs) in deep learning, to detect the mouth area of people in the video.
[0162] For each detected mouth movement, its start and end time points were recorded, and the duration of the mouth movement was calculated.
[0163] If there are multiple users in the video, it is necessary to distinguish the mouth movements of different users. This can be achieved through multimodal analysis, combining the sound signal and the video image.
[0164] The mouth movement durations of all users were aggregated and compared.
[0165] Core users may be the users who speak the most or play a leading role in the conversation.
[0166] Step S5: Identify the identity attribute of the core user. If the identity attribute is a proxy user, make an alarm judgment according to the preset list to which the text content corresponding to the target speech belongs.
[0167] It should be noted that voice samples of core users are extracted and analyzed using the voiceprint recognition model.
[0168] The voiceprint features are compared with those in the database to determine the user's identity.
[0169] Based on the user identity and the text content of the target speech, the following judgments are made:
[0170] If the user is a proxy user and the text belongs to the reminder script list, no alarm will be triggered.
[0171] If the user is a proxy and the text belongs to the warning script list, an alarm will be issued.
[0172] If the text content does not belong to any preset list, a decision on whether to alarm is made based on the text length and the analysis results of the large language model.
[0173] Furthermore, step S1 specifically includes:
[0174] The preset ASR model is used to perform text recognition on the audio and video files to be recognized, obtain the corresponding text content, and timestamp each short sentence contained in the text content.
[0175] It should be noted that the ASR model can ensure that the model can accurately recognize different accents and background noise. The ASR model can be an open source model or can be trained by the user.
[0176] Furthermore, step S5 specifically includes:
[0177] Extracting the first voiceprint feature of the core user in the audio and video file to be identified based on the voiceprint recognition model;
[0178] Compare the first voiceprint feature with the second voiceprint feature pre-stored in the database to determine the identity attributes of the core user, the identity attributes including the agent user and the insured user;
[0179] If the first voiceprint feature does not match the second voiceprint feature, the identity attribute of the core user is marked as another user.
[0180] It should be noted that a database containing multiple voiceprint features is first established to improve recognition accuracy. The database pre-stores voiceprints of several proxy users and several insured users.
[0181] Extract voice samples of core users and analyze them using the voiceprint recognition model.
[0182] The voiceprint features are compared with those in the database to determine the user's identity.
[0183] If the voiceprint feature does not belong to the database, the core user is marked as other users.
[0184] Furthermore, step S5 further includes:
[0185] If the identity attribute is a proxy user or other user, and the text content corresponding to the target speech belongs to the preset reminder speech list, no alarm will be issued;
[0186] If the identity attribute is a proxy user or other user, and the text content corresponding to the target speech belongs to the preset warning speech list, an alarm will be issued;
[0187] If the identity attribute is a proxy user or other user, but the text content corresponding to the target speech does not belong to the preset reminder speech list and the preset warning speech list, the length of the text content is calculated. When the length is less than the preset length threshold, an alarm is issued, otherwise no alarm is issued.
[0188] It should be noted that a warning script list and a reminder script list are given.
[0189] If the text recognized by ASR contains words in the warning speech list, and this speech is spoken by the proxy user, a direct warning will be issued; if the text recognized by ASR contains words in the reminder speech list, no warning will be issued.
[0190] Due to special circumstances such as noise or dialects, if the text recognized by ASR does not contain any words in the warning speech list or the reminder speech list, but the sentence is spoken by the proxy user, the length of the ASR-recognized text is used to determine whether it is a guidance speech.
[0191] If the agent user's words are long, they are usually reminder words. If the agent user's words are short, they are usually guidance words. Therefore, the length of the ASR recognition text is calculated. If the length is less than the length threshold, an alert is issued. If the length is greater than the length threshold, no alert is issued.
[0192] Furthermore, step S5 further includes:
[0193] If the identity attribute is a proxy user or other user, but the text content corresponding to the target speech does not belong to the preset reminder speech list and the preset warning speech list, the text content corresponding to the target speech is identified based on the preset large language model to determine whether there is any abnormality in the text content corresponding to the target speech. If there is an abnormality, an alarm is issued, otherwise no alarm is issued.
[0194] It should be noted that there is some doubt in identifying the words as reminder words or guidance words only from the length of the text. In this case, a preset large language model is further used to conduct an in-depth analysis of the text content to identify potential anomalies or risks.
[0195] As technology develops and user needs change, ASR models, voiceprint recognition models, and large language models need to be updated regularly to maintain the accuracy and effectiveness of the system.
[0196] An embodiment of the present application also provides a computer-readable storage medium for storing program code, which is used to execute any one of the implementation methods of the audio and video recognition processing method for multi-person face-to-face review described in the aforementioned embodiments.
[0197] In the embodiment of the present application, a method, device, equipment and storage medium for multi-person face-to-face audio and video recognition processing are provided, the target speech is located for the audio and video file to be recognized, and the mouth movement duration of each user in the time period corresponding to the target speech is counted based on the lip synchronization model, the core users in the time period of the target speech are determined, and then the identity attributes of the core users are identified, and the text content corresponding to the target speech is alarmed and judged in combination with the preset list, so as to avoid the situation where a single lip synchronization model makes a wrong judgment. It solves the technical problem that in actual scenarios, the agent’s speech to the insured is not completely instructive, and no alarm is given for ordinary speech. And because the insured’s answer is generally very short, the lip synchronization model may not be accurate, which will lead to false alarms.
[0198] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0199] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0200] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0201] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0202] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0203] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0204] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.
[0205] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A method for audio and video recognition processing for multi-person face-to-face review, characterized in that: include: Step S1, obtaining corresponding text content based on the audio and video file to be identified, and marking each short sentence contained in the text content with a timestamp; Step S2, locating at least one target speech and a time period corresponding to the target speech according to the text content; Step S3, based on the lip synchronization model, respectively counting the mouth movement durations of two or more users included in the audio and video file to be identified in the time period corresponding to each target speech; Step S4, determining a core user among two or more users according to the mouth movement duration; Step S5, identifying the identity attribute of the core user, if the identity attribute is a proxy user, making an alarm judgment according to the preset list to which the text content corresponding to the target speech belongs.
2. The method for audio and video recognition processing of multi-person face-to-face interview according to claim 1 is characterized in that: The step S1 specifically includes: The preset ASR model is used to perform text recognition on the audio and video files to be recognized, to obtain the corresponding text content, and to timestamp each short sentence contained in the text content.
3. The method for audio and video recognition processing of multi-person face-to-face interview according to claim 1, characterized in that: The step S5 specifically includes: Extracting a first voiceprint feature of the core user in the audio and video file to be identified based on a voiceprint recognition model; Comparing the first voiceprint feature with the second voiceprint feature pre-stored in a database to determine the identity attributes of the core user, the identity attributes including the agent user and the insured user; If the first voiceprint feature and the second voiceprint feature cannot match, the identity attribute of the core user is marked as other users.
4. The method for audio and video recognition processing for multi-person face-to-face interview according to claim 3 is characterized in that: The step S5 further comprises: If the identity attribute is a proxy user or other user, and the text content corresponding to the target speech belongs to the preset reminder speech list, no alarm will be issued; If the identity attribute is a proxy user or other user, and the text content corresponding to the target speech belongs to the preset warning speech list, an alarm is issued; If the identity attribute is a proxy user or other user, but the text content corresponding to the target speech does not belong to the preset reminder speech list and the preset warning speech list, the length of the text content is calculated, and an alarm is issued when the length is less than the preset length threshold, otherwise no alarm is issued.
5. The method for audio and video recognition processing for multi-person face-to-face interview according to claim 3 is characterized in that: The step S5 further comprises: If the identity attribute is a proxy user or other user, but the text content corresponding to the target speech does not belong to the preset reminder speech list and the preset warning speech list, the text content corresponding to the target speech is identified based on the preset large language model to determine whether there is any abnormality in the text content corresponding to the target speech. If there is an abnormality, an alarm is issued, otherwise no alarm is issued.
6. An audio and video recognition processing device for multi-person face-to-face interview, characterized in that: include: An acquisition unit, configured to acquire corresponding text content based on the audio and video file to be identified, and timestamp each short sentence contained in the text content; A positioning unit, configured to locate at least one target speech and a time period corresponding to the target speech according to the text content; A statistical unit, configured to respectively count the mouth movement durations of two or more users included in the audio and video file to be identified in a time period corresponding to each target speech based on a lip synchronization model; a determining unit, configured to determine a core user among two or more users according to the mouth movement duration; The early warning unit is used to identify the identity attribute of the core user. If the identity attribute is a proxy user, an alarm judgment is made according to a preset list to which the text content corresponding to the target speech belongs.
7. The audio and video recognition processing device for multi-person face-to-face interview according to claim 6, characterized in that: The acquisition unit is specifically used for: The preset ASR model is used to perform text recognition on the audio and video files to be recognized, to obtain the corresponding text content, and to timestamp each short sentence contained in the text content.
8. The audio and video recognition processing device for multi-person face-to-face interview according to claim 6, characterized in that: The early warning unit is specifically used for: Extracting a first voiceprint feature of the core user in the audio and video file to be identified based on a voiceprint recognition model; Comparing the first voiceprint feature with the second voiceprint feature pre-stored in a database to determine the identity attributes of the core user, the identity attributes including the agent user and the insured user; If the first voiceprint feature and the second voiceprint feature cannot match, the identity attribute of the core user is marked as other users.
9. An audio and video recognition processing device for multi-person face-to-face interview, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the audio and video recognition processing method for multi-person face-to-face review as described in any one of claims 1-5 according to the instructions in the program code.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program code, and the program code is used to execute the audio and video recognition processing method for multi-person face-to-face review as described in any one of claims 1-5.
