A financial service video post-event quality inspection method, device, equipment and medium

Abstract

This application provides a method, apparatus, equipment, and medium for post-event quality inspection of financial business videos, relating to the field of identity recognition technology. The method includes: acquiring video data from a financial business transaction scenario; separating audio and image data from the video data; determining the speaker's identity and a first time range in the audio data based on a neural network model; performing facial recognition on the image data to identify target objects and determine a second time range; cross-comparing the first and second time ranges to obtain an overlapping time interval; within the overlapping time interval, statistically analyzing the variance of the mouth aspect ratios of each target object using a sliding window, and summing the variances of each sliding window to obtain the matching confidence level between the target object and the speaker's identity; constructing a confidence matrix based on all matching confidence levels, and solving the confidence matrix using a global optimal matching algorithm to obtain the identity of each target object in the video data.

Description

Technical Field

[0001] This application relates to the field of identity recognition technology, and in particular to a method, apparatus, equipment and medium for post-event quality inspection of financial business videos. Background Technology

[0002] Currently, post-event quality inspection methods for financial business videos typically require pre-configuring the identity information of staff, customers, and other personnel, and preparing corresponding facial reference images or registered voiceprint data for each identity. Identity recognition is then completed based on facial or voiceprint comparison. In real-world financial business scenarios, it is often difficult to obtain timely and complete reference images or registered voiceprints of all personnel in the video, thus limiting the practical application of post-event quality inspection methods for financial business videos.

[0003] Furthermore, in financial business scenarios such as in-person dual recording, remote face-to-face signing, and wealth management sales recording, there are often issues in the videos, such as multiple people talking, people entering and leaving the frame, obstructions, changes in posture, and inconsistencies in the timing of audio and video. Currently, it is difficult to accurately determine the business identity of each face in the frame without relying on pre-registered identity samples, and to further statistically analyze the time points when different individuals enter and leave the frame, in order to support compliance quality inspection, penalty determination, and incident reporting.

[0004] Therefore, current methods for post-event quality inspection of financial business videos suffer from high deployment complexity in financial business scenarios and low accuracy in identity recognition. Summary of the Invention

[0005] This application provides a method, apparatus, device, and medium for post-event quality inspection of financial business videos to solve one or more problems existing in related technologies.

[0006] This application provides a post-event quality inspection method for financial business videos. The method includes: acquiring video data of a financial business transaction scenario; separating audio data and image data from the video data; determining the identity identifier of a speaker in the audio data and a first time range of the speaker's speech based on a pre-trained neural network model; performing face recognition on the image data to identify target objects in the image data and determining a second time range in which the target objects appear in the video data; cross-comparing the first time range and the second time range to obtain an overlapping time interval; statistically analyzing the variance of the aspect ratio of each mouth of the target object within the overlapping time interval using a sliding window, and summing the variances of each sliding window to obtain the matching confidence score between the target object and the speaker's identity identifier; constructing a confidence matrix based on all matching confidence scores, and solving the confidence matrix using a global optimal matching algorithm to obtain the identity identifier corresponding to each target object in the video data.

[0007] According to one embodiment of this application, determining the speaker's identity and the first time range of the speaker's speech in the audio data based on a pre-trained neural network model includes: performing speech recognition on the audio data to obtain text data segmented by sentence and start and end timestamps corresponding to each sentence; performing clustering based on the voiceprint features of the audio data to obtain speech segments of each speaker and start and end timestamps corresponding to each speech segment; determining the overlap duration between the start and end timestamps corresponding to the text data and the start and end timestamps corresponding to the speech segments; determining the speaker corresponding to each sentence of text data and generating corresponding dialogue text data based on the overlap duration; classifying the dialogue text data based on a pre-trained text classification model, and outputting the speaker's identity and the first time range of the speaker's speech in the audio data; the text classification model is trained based on a dialogue text dataset of a financial business scenario labeled with speaker identity.

[0008] According to one embodiment of this application, the step of performing face recognition on the image data to determine the target object in the image data and to determine the second time range in which the target object appears in the video data includes: performing face recognition on the image data to determine face location information and face feature information in each frame of image data; generating corresponding face trajectories based on face location information in multiple image data; performing clustering processing with time conflict constraints on the generated face trajectories based on the face feature information corresponding to the face trajectories to obtain clustering results; the clustering results represent at least one face trajectory corresponding to the same target object and the time range in which the at least one face trajectory corresponds to the video data; the time conflict constraints represent that the time ranges in which multiple face trajectories corresponding to the same target object appear in the video data do not overlap; and determining the target object in the image data and the second time range in which the target object appears in the video data based on the clustering results.

[0009] According to one embodiment of this application, the method further includes: determining the lip key points corresponding to the target object from the facial feature information based on the facial trajectory corresponding to each target object; and calculating the aspect ratio of the mouth of the target object in each frame of image data based on the lip key points.

[0010] According to one embodiment of this application, the step of calculating the variance of the mouth aspect ratios of the target object within the overlapping time interval using sliding windows, and summing the variances of each sliding window to obtain the matching confidence level between the target object and the speaker's identity identifier, includes: dividing the overlapping time interval based on a set window duration to obtain multiple sliding windows; determining the variance corresponding to each mouth aspect ratio of the target object within the sliding window; and summing the variances of all sliding windows to obtain the matching confidence level between the target object and the speaker's identity identifier.

[0011] According to one embodiment of this application, the step of constructing a confidence matrix based on all matching confidences and solving the confidence matrix using a global optimal matching algorithm to obtain the identity identifier corresponding to each target object in the video data includes: constructing a confidence matrix based on all matching confidences; the number of rows in the confidence matrix is ​​equal to the number of target objects, and the number of columns in the confidence matrix is ​​equal to the number of speaker identity identifiers; performing a global optimal solution on the confidence matrix using the LAPJV linear assignment problem solving algorithm to obtain the matching results between each identity identifier and each target object; and determining the identity identifier corresponding to each target object based on the matching results.

[0012] According to one embodiment of this application, the method further includes: in response to the identity identifier indicating that the target object is associated with the financial business transaction scenario, determining the face appearance time of the target object; performing quality inspection on the face appearance time based on preset financial business quality inspection rules, and obtaining a quality inspection result; the quality inspection result indicating whether the behavior of the target object in the financial business transaction scenario conforms to the financial business quality inspection rules; and outputting the face appearance time of the target object and the quality inspection result.

[0013] This application also provides a post-event quality inspection device for financial business videos. The device includes: an acquisition module for acquiring video data of a financial business transaction scenario and separating audio data and image data from the video data; an audio recognition module for determining the identity of a speaker in the audio data and a first time range of the speaker's speech based on a pre-trained neural network model; a video recognition module for performing face recognition on the image data to determine a target object in the image data and a second time range in which the target object appears in the video data; a comparison module for cross-comparing the first time range and the second time range to obtain an overlapping time interval; a matching module for statistically analyzing the variance of the mouth aspect ratio of each target object within the overlapping time interval using a sliding window and summing the variances of each sliding window to obtain a matching confidence level between the target object and the speaker's identity; and an identification module for constructing a confidence matrix based on all matching confidence levels, solving the confidence matrix using a global optimal matching algorithm, and obtaining the identity of each target object in the video data.

[0014] This application also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of the above-described embodiments.

[0015] This application also provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described above.

[0016] The method of this application embodiment acquires video data of a financial business transaction scenario, and separates audio data and image data from the video data; based on a pre-trained neural network model, it determines the identity identifier of the speaker in the audio data and the first time range of the speaker's speech; it performs face recognition on the image data to identify the target object in the image data and determine the second time range in which the target object appears in the video data; it cross-compares the first time range and the second time range to obtain an overlapping time interval; it uses a sliding window to calculate the variance of the aspect ratio of each mouth of the target object within the overlapping time interval, and accumulates the variances of each sliding window to obtain the matching confidence between the target object and the speaker's identity identifier; it constructs a confidence matrix based on all matching confidences, and solves the confidence matrix using a global optimal matching algorithm to obtain the identity identifier corresponding to each target object in the video data. This reduces the deployment complexity of identity recognition in financial business scenarios and improves the accuracy of identity recognition.

[0017] It should be understood that the teachings of this application are not required to achieve all the beneficial effects described above, but rather that a specific technical solution can achieve a specific technical effect, and other embodiments of this application can also achieve beneficial effects not mentioned above. Attached Figure Description

[0018] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of this application are illustrated in the drawings by way of example and not limitation, in which: In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0019] Figure 1 This illustration shows the processing flow of the post-event quality inspection method for financial business videos provided in an embodiment of this application. Figure 1 ; Figure 2 This illustration shows the processing flow of the post-event quality inspection method for financial business videos provided in an embodiment of this application. Figure 2 ; Figure 3 This illustration shows an application scenario diagram of the post-event quality inspection method for financial business videos provided in an embodiment of this application; Figure 4 This illustration shows an optional schematic diagram of a video post-inspection device for financial transactions provided in an embodiment of this application; Figure 5 A schematic diagram of the composition structure of the electronic device provided in the embodiments of this application is shown. Detailed Implementation

[0020] To make the objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0022] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0024] The processing flow of the post-event quality inspection method for financial business videos provided in the embodiments of this application is described below. See also Figure 1 , Figure 1 This is a schematic diagram of the processing flow of the post-event quality inspection method for financial business videos provided in the embodiments of this application. Figure 1 , will combine Figure 1 Steps S101-S106 shown will be explained.

[0025] Step S101: Obtain video data of the financial business transaction scenario, and separate audio data and image data from the video data.

[0026] In some embodiments, financial transaction scenarios may include: in-person dual recording, remote video signing, and playback of recorded wealth management sales videos, which require post-transaction quality inspection. Video data may be a file containing synchronized audio and video streams. The video stream may be a series of consecutive image frames. Audio data may include: an audio sequence separated from the video data and arranged in chronological order. Image data may include: an image frame sequence separated from the video data and arranged in chronological order.

[0027] Step S102: Based on a pre-trained neural network model, determine the speaker's identity and the first time range of the speaker's speech in the audio data.

[0028] In some embodiments, the neural network model may include a pre-trained neural network model specifically designed for financial scenarios. Speakers may include staff, customers, or unrelated personnel in the financial transaction scenario. Identification may include labels representing financial business roles such as staff, customers, and unrelated personnel. The first time range may be the start and end timestamps of each speaker's speech in the audio data.

[0029] Step S103: Perform face recognition on the image data to determine the target object in the image data and determine the second time range in which the target object appears in the video data.

[0030] In some embodiments, the target object may include a facial character generated by clustering facial trajectories in a continuous sequence of image frames. The second time range may be the start and end timestamps of each target object appearing in the video data.

[0031] Step S104: Cross-compare the first time range and the second time range to obtain the overlapping time interval.

[0032] Step S105: By using a sliding window to calculate the variance of the width-to-height ratio of each mouth of the target object within the overlapping time interval, and by summing the variances of each sliding window, the matching confidence between the target object and the speaker's identity identifier is obtained.

[0033] In some embodiments, the overlapping time interval can be the time interval where a first time range and a second time range intersect. The sliding window can be used to: statistically analyze the variance of the mouth aspect ratio within a local time range. The mouth aspect ratio can include: the ratio of mouth width to height calculated based on the coordinates of lip key points. The mouth aspect ratio can characterize the mouth opening and closing state of the target object in the image data. The matching confidence score can characterize the degree of probability that the target object and the speaker's identity are the same person.

[0034] Step S106: Construct a confidence matrix based on all matching confidences, solve the confidence matrix using the global optimal matching algorithm, and obtain the identity identifier corresponding to each target object in the video data.

[0035] In some embodiments, the confidence matrix can be the input data for the globally optimal match, where the number of rows in the confidence matrix equals the number of target objects, and the number of columns equals the number of speaker identifiers. The globally optimal matching algorithm can include the Linear Assignment Problem (LAPJV) algorithm. Solving this problem can be based on the LAPJV algorithm to obtain the best match, thus determining the one-to-one correspondence between target objects and identifiers.

[0036] As an example, after acquiring a video file from a bank's in-person dual-recording system, the video file is first separated into an audio stream and a video stream, where the video stream consists of multiple consecutive image frames. Speech recognition is performed on the audio stream to obtain transcribed text segmented by sentence and the start and end timestamps of the transcribed text. Simultaneously, voiceprint features are extracted for speaker clustering to obtain each speaker's speaking segment and its corresponding start and end timestamps. The speaker identifier is assigned to the transcribed text by calculating the overlap between the transcribed text and the speaking segments. The transcribed dialogue with speaker identifiers is then input into a pre-trained financial scenario text classification model, which outputs the identity identifiers of each speaker and their corresponding first time range. The speaker identifiers can be Speaker 1, Speaker 2, etc. The speaker's identity identifier can be staff member or customer.

[0037] Face detection is performed frame by frame in the video stream to extract face location information and key point information. Multiple face trajectories are generated through face tracking and hierarchical clustering with time conflict constraints is performed based on face feature vectors. Face trajectories with consistent features are merged into the same target object and the corresponding second time range is determined.

[0038] The first and second time ranges are cross-referenced to obtain overlapping time intervals. Within these overlapping time intervals, a sliding window is used to divide the space into segments with a preset window duration. For each window, the variance of the mouth aspect ratio of the target object is calculated, and the variances of all windows are summed to obtain the matching confidence score. A confidence matrix is ​​generated by iterating through all target objects and their corresponding identity identifiers. The best match is obtained using the LAPJV algorithm, thus solving for the one-to-one correspondence between target objects and identity identifiers.

[0039] The method in this application can complete post-event quality inspection of financial business videos using dialogue semantic features and video lip movement features in financial business scenarios without the need for pre-configuration of facial reference images or registration of voiceprint data. This reduces the deployment complexity of identity recognition in financial business scenarios and improves the accuracy of identity recognition in multi-person scenarios through global optimal matching. By comparing the timestamp overlap of speech recognition and voiceprint clustering, precise binding between transcribed text and speaker is achieved. The financial scenario text classification model is used to identify the speaker's role based on business semantic differences, thereby eliminating the dependence on a pre-registered voiceprint database. Through face trajectory clustering with time conflict constraints, multiple discrete trajectories of the same person at different time periods are stably merged into the same target object, while avoiding misjudging different faces existing at the same time as the same person. Therefore, even in cases of occlusion, head turning, or re-entry into the frame after leaving, the appearance and departure times of each person can be accurately counted, improving the reliability of subsequent quality inspection rule execution. By extracting lip key points from facial feature information and calculating the mouth width-to-height ratio, it is possible to effectively reflect whether the target face is in a speaking state within a specified time period. Compared to schemes that rely on complex deep neural networks to extract lip movement features, the statistical method adopted in this application has clearer logic, stronger interpretability, lower engineering implementation cost, and less dependence on the scale of training data and the generalization ability of the model. Therefore, the results are more stable and easier to implement in financial business scenarios. By using a sliding window to statistically analyze and accumulate the variance of the mouth width-to-height ratio within overlapping time intervals, the intensity of the target object's mouth movements during a specific speaking period is effectively characterized, replacing the complex deep neural network lip movement feature extraction with a low-computational-cost statistical method. By constructing a confidence matrix and using LAPJV for global optimal allocation, the most reasonable one-to-one correspondence between the target object and the identity identifier is found from the whole, effectively avoiding mismatches, cross-matches, and identity confusion caused by pairwise local judgments in multi-person scenarios. After assigning the identity identifier to the target object, the time of its face appearance is determined and quality inspection is carried out based on financial business quality inspection rules, directly outputting identity-based timeline results for compliance supervision, supporting score deduction judgment and event reporting.

[0040] In some embodiments, step S102, which uses a pre-trained neural network model to determine the speaker's identity and the first time range of the speaker's speech in the audio data, may include: performing speech recognition on the audio data to obtain text data segmented by sentence and the start and end timestamps corresponding to each sentence; performing clustering based on the voiceprint features of the audio data to obtain the speech segments of each speaker and the start and end timestamps corresponding to the speech segments; determining the overlap duration between the start and end timestamps corresponding to the text data and the start and end timestamps corresponding to the speech segments; determining the speaker corresponding to each sentence of text data based on the overlap duration and generating the corresponding dialogue text data; and classifying the dialogue text data based on a pre-trained text classification model to output the speaker's identity and the first time range of the speaker's speech in the audio data.

[0041] In this embodiment, the text classification model is trained on a dialogue text dataset of a financial business scenario labeled with the speaker's identity. The output of the text classification model is single-labeled and multi-class. The specific number of categories is determined by the types of roles that may appear in the actual financial business scenario, and in common financial business scenarios, it includes at least three categories: staff, customers, and irrelevant personnel. Speech recognition may include: converting audio data into time-stamped transcribed text data segmented by sentence. The text data may include transcribed text of dialogues in the financial business scenario. Start and end timestamps can represent the start and end times of speech segments or text data on the timeline. Voiceprint features can be used to distinguish different speakers. Clustering can be used to separate speech segments of each speaker from the audio. Speech segments may be audio data belonging to the same speaker. The overlap duration can be the duration of the intersection of the start and end timestamps corresponding to the text data and the start and end timestamps corresponding to the speech segments on the timeline. The dialogue text data may include: transcribed text data segmented by sentence with speaker identification.

[0042] As an example, after acquiring the audio data, the system first performs speech recognition to obtain sentence-by-sentence text data and the start and end timestamps for each sentence. It then extracts the voiceprint features of the audio data and performs clustering to obtain the speech segments of each speaker and their start and end timestamps. The system calculates the overlap between the start and end timestamps of the text data and the speech segments, identifying the speaker of the transcribed text as the speaker of the transcribed text. For example, if the transcribed text has the largest overlap with speaker 2's speech segment, then speaker 2 is identified as the speaker of the transcribed text. This process determines the speaker for each sentence of text data and generates dialogue text data with speaker tags. Finally, the dialogue text data is input into a text classification model pre-trained based on a dialogue text dataset from a financial business scenario labeled with speaker identities. The text classification model outputs the identity of each speaker and the first time range of their speech. For example, if speaker 2 is a staff member, the corresponding first time range is from 17:31:30 on May 11, 2026 to 17:32:14 on May 11, 2026.

[0043] In some embodiments, step S103, which involves performing face recognition on the image data to determine the target object in the image data and to determine the second time range in which the target object appears in the video data, may include: performing face recognition on the image data to determine the face location information and face feature information in each frame of image data; generating corresponding face trajectories based on the face location information in multiple image data; performing clustering processing with time conflict constraints on the generated face trajectories based on the face feature information corresponding to the face trajectories to obtain clustering results; and determining the target object in the image data and the second time range in which the target object appears in the video data based on the clustering results.

[0044] In this embodiment, the clustering result represents at least one face trajectory corresponding to the same target object and the time range of at least one face trajectory in the video data. The time conflict constraint represents that the time ranges of multiple face trajectories corresponding to the same target object in the video data do not overlap. Face location information may include: the coordinate position of the face region detected in each frame of image data within the frame. Face feature information may include: a face feature vector extracted from the face region to represent identity. Face trajectories can represent the time range and movement trajectory of a single face continuously appearing in the video data. The identities of faces in a single face trajectory are consistent. Clustering processing can merge multiple face trajectories belonging to the same person into the same target object. The target object may include: a face role entity representing a specific person generated after clustering all face trajectories in the video stream, which can be used as an object for which a financial business identity label needs to be assigned on the video side.

[0045] As an example, frame-by-frame face recognition is performed on image data to determine the face location and feature information in each frame. Face tracking is then performed based on the face location information across multiple consecutive frames to generate corresponding face trajectories. The identities of faces within a single face trajectory are consistent. The generated face trajectories are then subjected to clustering with temporal conflict constraints to obtain the clustering results.

[0046] The specific steps of clustering are: 1. Initially, each face trajectory is treated as an independent cluster.

[0047] 2. Calculate the similarity between all cluster pairs. That is, calculate the average similarity between all face feature vectors in one cluster and all face feature vectors in another cluster, and use this as the cluster similarity.

[0048] 3. Maintain a priority queue of cluster pairs sorted in descending order of similarity score.

[0049] 4. Repeat the following steps until merging is no longer possible: a. Analyze the candidate merge cluster pairs with the highest similarity. b. For all face trajectories within two candidate merge pairs, compare them pairwise to determine if the corresponding time ranges in the video data conflict. If all face trajectory pairs do not conflict, and the similarity between the candidate merge cluster pairs meets the face comparison similarity threshold, merge them into a new cluster, update the similarity with other clusters, and re-add it to the cluster pair priority queue. c. If any pair of face trajectories conflicts in the time range of the video data, or if the similarity between the candidate merge cluster pairs does not meet the face comparison similarity threshold, abandon this merge and remove the next candidate merge cluster pair from the cluster pair priority queue. d. Stop merging when it is impossible to continue, and obtain the final clustering result.

[0050] 5. At the end of the clustering process, each cluster in the clustering result is considered a target object. Each target object corresponds to one or more face trajectories. These face trajectories allow us to determine the second temporal range in which the target object appears in the video data.

[0051] In some embodiments, the post-event quality inspection method for financial business videos further includes: determining the lip key points corresponding to the target object from the facial feature information based on the facial trajectory corresponding to each target object; and calculating the aspect ratio of the mouth of the target object in each frame of image data based on the lip key points.

[0052] In this embodiment, lip key points may include feature information representing the shape of the lip contour, filtered from facial feature information. The mouth aspect ratio may include the ratio of mouth width to height calculated based on the coordinates of the lip key points. The mouth aspect ratio (MAR) can characterize the mouth opening and closing state of a target object in image data. In financial business scenarios, when the target object is not speaking, the mouth opening and closing state remains fixed or exhibits low-frequency changes; when the target object speaks, the mouth opening and closing state enters a high-frequency, rapid change process.

[0053] In some embodiments, step S105, which involves calculating the variance of the mouth aspect ratios of the target object within the overlapping time interval using sliding windows and summing the variances of each sliding window to obtain the matching confidence between the target object and the speaker's identity, may include: dividing the overlapping time interval based on a set window duration to obtain multiple sliding windows; determining the variance corresponding to each mouth aspect ratio of the target object within the sliding window; and summing the variances of all sliding windows to obtain the matching confidence between the target object and the speaker's identity.

[0054] In this embodiment, the variance summation of the sliding window is a statistical indicator that can characterize the drastic changes in MAR within a specified time period. Other statistical indicators may also include the summation of root mean square pointwise changes (RMS_DIFF), etc., which are not limited in this embodiment. Since the frequency of MAR changes when the target object speaks is within a certain range, the window duration in the sliding window variance accumulation method can be set according to the frequency of MAR changes. The setting of the window duration mainly considers the time length rather than the number of video frames.

[0055] As an example, the overlapping time interval between the second time range of target object A and the first time range corresponding to the identity of staff is first determined. This overlapping time interval is then divided into multiple sliding windows based on a set window duration. For each sliding window, the aspect ratio of the mouth of target object A in each image frame within the sliding window is obtained, and the variance of the mouth aspect ratio within the sliding window is calculated. The variances of all sliding windows are summed to obtain the matching confidence between target object A and the identity of staff.

[0056] In some embodiments, step S106, which involves constructing a confidence matrix based on all matching confidences and solving the confidence matrix using a global optimal matching algorithm to obtain the identity identifier corresponding to each target object in the video data, may include: constructing a confidence matrix based on all matching confidences; the number of rows in the confidence matrix equals the number of target objects, and the number of columns in the confidence matrix equals the number of speaker identity identifiers; performing a global optimal solution on the confidence matrix using LAPJV to obtain the matching results between each identity identifier and each target object; and determining the identity identifier corresponding to each target object based on the matching results.

[0057] In this embodiment, finding the global optimal solution may include: finding the method with the largest cumulative overall matching confidence value among all possible allocation methods. The matching result may include: the one-to-one correspondence between each identity identifier and each target object obtained after the global optimal solution.

[0058] As an example, the algorithm iterates through all target objects and all speaker identifiers, calculates the pairwise matching confidence, and constructs a confidence matrix based on all matching confidence scores. The number of rows in the confidence matrix equals the number of target objects, and the number of columns equals the number of speaker identifiers. The confidence matrix is ​​then globally optimized using LAPJV to obtain the matching results between each identifier and each target object. Finally, based on the matching results, the identifier corresponding to each target object in the video data is determined.

[0059] The processing flow of the post-event quality inspection method for financial business videos provided in the embodiments of this application is described below. See also Figure 2 , Figure 2 This is a schematic diagram of the processing flow of the post-event quality inspection method for financial business videos provided in the embodiments of this application. Figure 2 , will combine Figure 2 Steps S201-S203 are explained below.

[0060] Step S201: In response to the association between the identity identifier representing the target object and the financial business transaction scenario, determine the time when the target object's face appears.

[0061] Step S202: Based on the preset financial business quality inspection rules, the time of face appearance is inspected to obtain the inspection result.

[0062] Step S203: Output the appearance time of the target object's face and the quality inspection results.

[0063] In some embodiments, associating the identity identifier with the financial transaction scenario may mean that the identity identifier identifies the target object as a customer or staff member. The face appearance time may include the start and end timestamps of the target object corresponding to the identity identifier appearing in the video data. Financial transaction quality inspection rules may include pre-set rules such as the duration of simultaneous presence of staff and customers, timeout rules for personnel leaving the frame, and rules for unauthorized personnel entering the frame. This application embodiment does not limit specific financial transaction quality inspection rules. The quality inspection result can characterize whether the target object's behavior in the financial transaction scenario complies with the financial transaction quality inspection rules. Behavior may include the target object's appearance, departure, and continuous stay within the video frame during the financial transaction process.

[0064] As an example, in response to the target object's identity identifier indicating that the target object is a staff member in a financial business transaction scenario, the time when the target object's face appears in the entire video data is determined. Pre-defined financial business quality inspection rules are invoked to compare the target object's face appearance time with the required conditions such as the time of presence and absence from duty as specified in the rules, obtaining the comparison results. Quality inspection results are generated based on the comparison results. Finally, the target object's face appearance time and the quality inspection results are output synchronously.

[0065] Taking a bank counter dual recording scenario as an example, video data of the bank counter dual recording scenario is acquired, and audio data and image data are separated from the video data. Speech recognition is performed on the audio data to obtain text data segmented by sentence and start and end timestamps. Based on voiceprint features, voice segments of each speaker and start and end timestamps are obtained. By calculating the overlap duration between the text data and the voice segments, the speaker corresponding to each sentence of text is determined to generate dialogue text data. Then, the dialogue text data is input into a text classification model pre-trained based on a financial business scenario dialogue text dataset labeled with speaker identification, and the model outputs two types of identification: teller and customer, and their corresponding first time range. Face recognition is performed on image data to determine face location and feature information. After generating face trajectories based on face location information, clustering with time conflict constraints is performed based on face feature information. Trajectories with consistent features and no time conflicts are grouped into the same target object and their second time range is determined. The first time range of each identity identifier is cross-compared with the second time range of each target object to obtain overlapping time intervals. Within the overlapping time intervals, sliding windows are divided based on a set window duration. The variance of the mouth aspect ratio of the target object within each window is calculated, and the variances of all windows are summed to obtain the matching confidence. A confidence matrix is ​​constructed based on all matching confidences, and global optimal solution is obtained through LAPJV. Teller or customer identity identifiers are assigned to the corresponding target objects, and the face appearance time axis of the two is output based on the face appearance time of the target objects with identity labels.

[0066] Taking a remote video interview scenario as an example, the video data of the remote video interview scenario is acquired and separated into audio data and image data; speech recognition and voiceprint clustering are performed on the audio data, and the speaker corresponding to each sentence of text is determined by timestamp overlap comparison to generate dialogue text data. The text classification model is used to process and output the identity identifiers of customers and staff and their corresponding first time ranges; face recognition and tracking are performed on the image data to generate multiple facial trajectories. When the customer re-enters the screen due to network interruption, posture change, or brief absence of the face, clustering with time conflict constraints is performed on the multiple separated trajectories based on facial feature information. The system performs a class-based processing approach, associating trajectories with non-overlapping time ranges and consistent features with the same target object and summarizing the corresponding second time ranges. It then cross-compares the second time range of the target object with the first time range of the customer's identity identifier to obtain overlapping time intervals. Within the overlapping time intervals, it uses a sliding window to calculate the variance of the mouth's width-to-height ratio and accumulates it to obtain the matching confidence score. Based on all the matching confidence scores, it constructs a confidence matrix and then uses LAPJV to perform a global optimal solution. It assigns the customer or staff identifier to the corresponding target object and outputs the face appearance time axis of both based on the face appearance time of the target object with the identity label.

[0067] Taking a video scenario in a business hall as an example, video data from the business hall is acquired and separated into audio and image data. Speech recognition and voiceprint clustering are performed on the audio data. Dialogue text data with speaker identifiers is generated through timestamp overlap comparison. This text is input into a financial scenario text classification model, which outputs three categories of identity identifiers: staff, customers, and irrelevant personnel, along with their corresponding first time ranges. Face recognition and tracking are performed on the image data to generate face trajectories. Based on facial feature information, clustering with time conflict constraints is performed to generate each target object and its corresponding second time range. The first time ranges of the three identity identifiers are cross-compared with the second time ranges of each target object to obtain overlapping time intervals. Within these overlapping time intervals, the variance of the mouth width-to-height ratio is calculated using a sliding window and accumulated to obtain a matching confidence score. A confidence matrix is ​​constructed based on all matching confidence scores, and then global optimality is achieved using LAPJV. Customer or staff identifiers are assigned to the corresponding target objects, and the face appearance timelines of the identified target objects are output. Target objects that do not match are identified as irrelevant personnel.

[0068] refer to Figure 3 This application provides an application scenario diagram of a post-event quality inspection method for financial business videos, which is applied to determine the speaker corresponding to each sentence of text data and generate corresponding dialogue text data based on the overlap duration.

[0069] The audio data is segmented into transcribed text by sentence, and the start and end timestamps of each sentence are recorded. Based on voiceprint features, the audio is divided into different speaker segments, corresponding to speaker 1 segment and speaker 2 segment respectively.

[0070] The start and end timestamps of the transcribed text are compared with the start and end timestamps of each speaker's segment on the timeline for overlap calculation. As shown in the figure, the transcribed text has only a short overlap (1) with speaker 1's segment, while it has a longer overlap (2) with speaker 2's segment. The text is determined to have the longest overlap with speaker 2's speech segment, thus identifying speaker 2 as the speaker in the transcribed text. This process is used to determine the speaker corresponding to each sentence's text data and generate dialogue text data with speaker tags.

[0071] The following continues to describe the exemplary structure of the software modules included in the financial business video post-inspection device 90 provided in the embodiments of this application. In some embodiments, such as Figure 4 As shown, the financial business video post-event quality inspection device 90 may include: The acquisition module 901 is used to acquire video data from financial business processing scenarios and separate audio data and image data from the video data; The audio recognition module 902 is used to determine the speaker's identity and the first time range of the speaker's speech in the audio data based on a pre-trained neural network model. The video recognition module 903 is used to perform face recognition on image data, determine the target object in the image data, and determine the second time range in which the target object appears in the video data. The comparison module 904 is used to cross-compare the first time range and the second time range to obtain the overlapping time interval; The matching module 905 is used to calculate the variance of the width-to-height ratio of each mouth of the target object within the overlapping time interval through a sliding window, and to sum the variances of each sliding window to obtain the matching confidence between the target object and the speaker's identity identifier. The recognition module 906 is used to construct a confidence matrix based on all matching confidence scores, solve the confidence matrix through a global optimal matching algorithm, and obtain the identity identifier corresponding to each target object in the video data.

[0072] It should be noted that the description of the device in this application embodiment is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment, therefore it will not be repeated. For any technical details not covered in the financial business video post-inspection device provided in this application embodiment, please refer to... Figures 1 to 3 The meaning is understood in accordance with the description of any of the accompanying drawings.

[0073] According to embodiments of this application, this application also provides an electronic device and a non-transitory computer-readable storage medium.

[0074] Figure 5A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0075] like Figure 5 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0076] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0077] The computing unit 801 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the post-event quality inspection method for financial business videos. For example, in some embodiments, the post-event quality inspection method for financial business videos can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the post-event quality inspection method for financial business videos described above can be performed. Alternatively, in other embodiments, the computing unit 801 may be configured by any other suitable means (e.g., by means of firmware) to perform a post-event quality inspection method for financial business videos.

[0078] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0079] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0080] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0081] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0082] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0083] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0084] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0085] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for post-event quality inspection of financial transaction videos, characterized in that, The method includes: Acquire video data of financial business processing scenarios, and separate audio data and image data from the video data; Based on a pre-trained neural network model, the speaker's identity and the first time range of the speaker's speech are determined in the audio data; Perform face recognition on the image data to determine the target object in the image data and determine a second time range in which the target object appears in the video data; Cross-compare the first time range and the second time range to obtain the overlapping time interval; The variance of the mouth aspect ratio of the target object within the overlapping time interval is statistically analyzed by using a sliding window, and the variances of each sliding window are summed to obtain the matching confidence between the target object and the speaker's identity identifier. A confidence matrix is ​​constructed based on all matching confidence scores. The confidence matrix is ​​then solved using a global optimal matching algorithm to obtain the identity identifier corresponding to each target object in the video data.

2. The method according to claim 1, characterized in that, The pre-trained neural network model determines the speaker's identity and the first time range of the speaker's speech in the audio data, including: The audio data is subjected to speech recognition to obtain text data segmented by sentence and the start and end timestamps corresponding to the text data of each sentence; Clustering is performed based on the voiceprint features of the audio data to obtain the speech segments of each speaker and the start and end timestamps of the speech segments; Determine the overlap duration between the start and end timestamps corresponding to the text data and the start and end timestamps corresponding to the audio segments; Based on the overlap duration, the speaker corresponding to each sentence's text data is determined and the corresponding dialogue text data is generated; Based on a pre-trained text classification model, the dialogue text data is classified, and the speaker's identity identifier and the first time range of the speaker's speech are output in the audio data; the text classification model is trained on a dialogue text dataset of financial business scenarios labeled with speaker identity identifiers.

3. The method according to claim 1, characterized in that, The step of performing face recognition on the image data to determine the target object in the image data and to determine a second time range in which the target object appears in the video data includes: Face recognition is performed on the image data to determine the face location information and face feature information in each frame of image data; Based on the face location information in multiple image data, the corresponding face trajectory is generated; Based on the facial feature information corresponding to the facial trajectory, the generated facial trajectory is subjected to clustering processing with time conflict constraints to obtain clustering results; the clustering results represent at least one facial trajectory corresponding to the same target object and the time range corresponding to the at least one facial trajectory in the video data; the time conflict constraints represent that the time ranges of multiple facial trajectories corresponding to the same target object in the video data do not overlap. Based on the clustering results, the target object in the image data is determined, and a second time range in which the target object appears in the video data is determined.

4. The method according to claim 3, characterized in that, The method further includes: Based on the facial trajectory corresponding to each target object, the key lip points corresponding to the target object are determined from the facial feature information; Based on the lip key points, the aspect ratio of the mouth of the target object in each frame of image data is calculated.

5. The method according to claim 1, characterized in that, The method involves calculating the variance of the mouth aspect ratio of the target object within the overlapping time interval using a sliding window, and then summing the variances of each sliding window to obtain the matching confidence level between the target object and the speaker's identity identifier. This includes: The overlapping time interval is divided based on the set window duration to obtain multiple sliding windows; Determine the variance of each aspect ratio of the mouth of the target object within the sliding window; The variances of all sliding windows are summed to obtain the matching confidence between the target object and the speaker's identity.

6. The method according to claim 1, characterized in that, The process involves constructing a confidence matrix based on all matching confidence scores, solving the confidence matrix using a global optimal matching algorithm, and obtaining the identity identifier corresponding to each target object in the video data, including: A confidence matrix is ​​constructed based on all matching confidence scores; the number of rows in the confidence matrix is ​​equal to the number of target objects, and the number of columns in the confidence matrix is ​​equal to the number of speaker identifiers; The confidence matrix is ​​solved globally and optimally using the LAPJV linear assignment problem solving algorithm to obtain the matching results between each identity and each target object. Based on the matching results, the identity identifier corresponding to each target object is determined.

7. The method according to claim 1, characterized in that, The method further includes: In response to the identity identifier indicating that the target object is associated with the financial transaction scenario, the time when the target object's face appears is determined; Based on preset financial business quality inspection rules, the appearance time of the face is inspected to obtain the inspection result; the inspection result indicates whether the behavior of the target object in the financial business transaction scenario complies with the financial business quality inspection rules. Output the time of the appearance of the face of the target object and the quality inspection result.

8. A video post-processing quality inspection device for financial transactions, characterized in that, The device includes: The acquisition module is used to acquire video data of financial business processing scenarios and separate audio data and image data from the video data; An audio recognition module is used to determine the speaker's identity and the first time range of the speaker's speech in audio data based on a pre-trained neural network model. The video recognition module is used to perform face recognition on the image data, determine the target object in the image data, and determine a second time range in which the target object appears in the video data; The comparison module is used to cross-compare the first time range and the second time range to obtain the overlapping time interval; The matching module is used to calculate the variance of the width-to-height ratio of each mouth of the target object within the overlapping time interval by using a sliding window, and to sum the variances of each sliding window to obtain the matching confidence between the target object and the speaker's identity identifier. The identification module is used to construct a confidence matrix based on all matching confidence levels, solve the confidence matrix using a global optimal matching algorithm, and obtain the identity identifier corresponding to each target object in the video data.

9. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

Images