Method and device for quality inspection of double-recording interactive data and storage medium

Abstract

The application discloses a kind of dual recording interactive data's quality inspection method, device and storage medium.It relates to artificial intelligence field, the method includes: in the case where obtaining first object and second object authorization, obtain the video data set and audio data set of first object and second object in dual recording interaction process;Video data set and audio data set are fragmented according to time, and the video data and audio data of different stages are obtained;According to the video data and audio data of each stage, the fusion feature corresponding to stage is generated, and a plurality of fusion features are obtained, at least one fusion feature is obtained according to the noise interference degree of the feature of video data and audio data in the stage to which it belongs Fusion;Multiple fusion features are input into dual recording detection model, and dual recording quality inspection result is output according to dual recording detection model.By the present application, the problem that the quality inspection accuracy of dual recording interactive data is low in the related art is solved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to a method, apparatus, and storage medium for quality inspection of dual-recording interactive data. Background Technology

[0002] Dual recording interaction is the process by which financial institutions simultaneously collect audio and video recording data during business transactions. Quality inspection of dual recording interaction data refers to the inspection of the aforementioned audio and video recording data to ensure the compliance of business transactions and protect customer rights.

[0003] In related technologies, quality inspection of dual-recording interactive data is typically performed using methods based on speech recognition and video action recognition. For example, speech recognition converts the audio content during dual-recording into text, followed by analysis such as keyword capture, or object detection models are used to detect key actions in the video, such as whether relevant documents are correctly provided. However, during dual-recording interaction, video action recognition may be unclear due to environmental factors, and background noise may also interfere with the accuracy of speech recognition. Therefore, the accuracy of dual-recording quality inspection is insufficient.

[0004] There is currently no effective solution to the problem of low accuracy in quality inspection of dual-recording interactive data in related technologies. Summary of the Invention

[0005] The main objective of this application is to provide a method, apparatus, and storage medium for quality inspection of dual-recording interactive data, so as to solve the problem of low accuracy in quality inspection of dual-recording interactive data in related technologies.

[0006] To achieve the above objectives, according to one aspect of this application, a quality inspection method for dual-recording interactive data is provided. The method includes: obtaining, with authorization from a first object and a second object, a set of video data and a set of audio data of the first and second objects during the dual-recording interactive process; segmenting the video data and audio data sets according to time to obtain video data and audio data at different stages; generating fusion features corresponding to each stage of the video data and audio data to obtain multiple fusion features, wherein at least one fusion feature is obtained by fusing features of the video data and audio data according to the noise interference level of the stage, the noise interference level including video interference level and audio interference level; inputting the multiple fusion features into a dual-recording detection model, and outputting a dual-recording quality inspection result based on the dual-recording detection model.

[0007] Furthermore, the video and audio datasets are segmented according to time to obtain video and audio data for different stages, including: during the dual-recording interaction process, a first time range for the first stage, a second time range for the second stage, and a third time range for the third stage are determined. The first stage is used for identity verification, the second stage for product introduction and risk warning, and the third stage for contract confirmation. Based on the first time range, first video data is extracted from the video dataset and first audio data is extracted from the audio dataset; based on the second time range, second video data is extracted from the video dataset and second audio data is extracted from the audio dataset; and based on the third time range, third video data is extracted from the video dataset and third audio data is extracted from the audio dataset.

[0008] Furthermore, based on the fusion features corresponding to the generation stages of video and audio data at each stage, multiple fusion features are obtained, including: generating a first fusion feature based on the video interference level, first video data, and first audio data in the first stage; generating a second fusion feature based on the audio interference level, second video data, and second audio data in the second stage; generating a third fusion feature based on the third video data and third audio data in the third stage; and combining the first fusion feature, the second fusion feature, and the third fusion feature to obtain multiple fusion features.

[0009] Furthermore, generating the first fusion feature based on the video interference level of the first stage, the first video data, and the first audio data includes: determining multiple frames of video information based on the first video data, and extracting sub-video features corresponding to each frame of video information to obtain multiple sub-video features; determining the video interference level of each frame of video information through video interference detection; fusing the multiple sub-video features based on the video interference level to obtain the first video feature; extracting features from the first audio data to obtain the first audio feature, and fusing the first video feature and the first audio feature to generate the first fusion feature.

[0010] Furthermore, the first video feature is obtained by fusing multiple sub-video features based on the degree of video interference, including: assigning a first weight to sub-video features whose video interference level reaches the video interference threshold, and assigning a second weight to sub-video features whose video interference level does not reach the video interference threshold, wherein the first weight is less than the second weight; and weighting and fusing multiple sub-video features based on the first weight and the second weight to obtain the first video feature.

[0011] Furthermore, generating the second fusion feature based on the audio interference level in the second stage, the second video data, and the second audio data includes: splitting the second audio data and the second video data into preset sub-time periods to obtain multiple sub-time period audio sub-information and video sub-information; extracting audio sub-features from the audio sub-information and video sub-features from the video sub-information; performing audio interference detection on each audio sub-information to obtain multiple audio interference levels; fusing the audio sub-features and video sub-features of each sub-time period according to the audio interference levels to obtain multiple sub-time period features; and fusing the multiple sub-time period features to generate the second fusion feature.

[0012] Furthermore, the audio sub-features and video sub-features of each sub-time period are fused according to the degree of audio interference to obtain multiple sub-time period features, including: configuring a third weight for audio sub-features whose audio interference level reaches the audio interference threshold, and configuring a fourth weight for audio sub-features whose audio interference level does not reach the audio interference threshold, wherein the third weight is less than the fourth weight; and fusing the audio sub-features with each weight configuration with the video sub-features of the same sub-time period to obtain multiple sub-time period features.

[0013] Furthermore, the training steps of the dual-recording detection model include: receiving multiple dual-recording interaction samples and the label of each dual-recording interaction sample; extracting multiple sample fusion features of each dual-recording interaction sample, and training a preset neural network model based on the multiple sample fusion features and the corresponding labels to obtain the dual-recording detection model.

[0014] To achieve the above objectives, according to another aspect of this application, a quality inspection device for dual-recording interactive data is provided. The device includes: an acquisition unit, configured to acquire a set of video data and a set of audio data of the first and second objects during the dual-recording interactive process, provided that authorization has been obtained from a first object and a second object; a segmentation unit, configured to segment the video data and audio data according to time to obtain video data and audio data at different stages; a fusion unit, configured to generate fusion features corresponding to each stage based on the video data and audio data at each stage, obtaining multiple fusion features, wherein at least one fusion feature is obtained by fusing features of the video data and audio data according to the noise interference level of the stage, the noise interference level including video interference level and audio interference level; and an output unit, configured to input the multiple fusion features into a dual-recording detection model and output a dual-recording quality inspection result based on the dual-recording detection model.

[0015] Furthermore, the segmentation unit includes: a time range determination module, used to determine the first time range of the first stage, the second time range of the second stage, and the third time range of the third stage during the dual recording interaction process, wherein the first stage is used for identity verification, the second stage is used for product introduction and risk warning, and the third stage is used for contract confirmation; and a data extraction module, used to extract first video data from the video data set and first audio data from the audio data set according to the first time range, extract second video data from the video data set and second audio data from the audio data set according to the second time range, and extract third video data from the video data set and third audio data from the audio data set according to the third time range.

[0016] Furthermore, the fusion unit includes: a first fusion module, used to generate a first fusion feature based on the video interference level, the first video data, and the first audio data in the first stage; a second fusion module, used to generate a second fusion feature based on the audio interference level, the second video data, and the second audio data in the second stage; a third fusion module, used to generate a third fusion feature based on the third video data and the third audio data in the third stage; and a fusion result module, used to combine the first fusion feature, the second fusion feature, and the third fusion feature to obtain multiple fusion features.

[0017] Furthermore, the first fusion module includes: a first extraction submodule, used to determine multiple frames of video information based on the first video data, and extract sub-video features corresponding to each frame of video information to obtain multiple sub-video features; a first detection submodule, used to determine the degree of video interference for each frame of video information through video interference detection; a first fusion submodule, used to fuse the multiple sub-video features based on the degree of video interference to obtain a first video feature; and a first generation submodule, used to extract features from the first audio data to obtain a first audio feature, and fuse the first video feature and the first audio feature to generate a first fused feature.

[0018] Furthermore, the first fusion submodule includes: a first weight configuration component, used to configure a first weight for sub-video features whose video interference level reaches the video interference threshold, and configure a second weight for sub-video features whose video interference level does not reach the video interference threshold, wherein the first weight is less than the second weight; and a first weighted fusion component, used to perform weighted fusion on multiple sub-video features according to the first weight and the second weight to obtain a first video feature.

[0019] Furthermore, the second fusion module includes: a time-segmentation submodule, used to split the second audio data and the second video data according to preset sub-time periods to obtain audio sub-information and video sub-information of multiple sub-time periods; a second extraction submodule, used to extract audio sub-features of audio sub-information and video sub-features of video sub-information; a second detection submodule, used to perform audio interference detection on each audio sub-information to obtain multiple audio interference levels; a second fusion submodule, used to fuse the audio sub-features and video sub-features of each sub-time period according to the audio interference level to obtain multiple sub-time period features; and a second generation submodule, used to fuse the multiple sub-time period features to generate a second fused feature.

[0020] Furthermore, the second fusion submodule includes: a second weight configuration component, used to configure a third weight for audio sub-features whose audio interference level reaches the audio interference threshold, and to configure a fourth weight for audio sub-features whose audio interference level does not reach the audio interference threshold, wherein the third weight is less than the fourth weight; and a second weighted fusion component, used to fuse the audio sub-features configured with each weight with video sub-features of the same sub-time period to obtain multiple sub-time period features.

[0021] Furthermore, the device also includes: a receiving unit for receiving multiple dual-recording interactive samples and the label of each dual-recording interactive sample; and a training unit for extracting multiple sample fusion features of each dual-recording interactive sample and training a preset neural network model based on the multiple sample fusion features and the corresponding labels to obtain a dual-recording detection model.

[0022] According to another aspect of this application, a computer-readable storage medium is provided, which includes a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform any kind of quality inspection method for dual-recording interactive data.

[0023] According to another aspect of this application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a quality inspection method for performing any kind of dual-recording interactive data.

[0024] According to another aspect of this application, a computer program product is provided, including computer instructions, which, when executed by a processor, implement the steps of the dual-recording interactive data quality inspection method described above.

[0025] In this embodiment, by obtaining authorization from both the first and second objects, video and audio data sets of the first and second objects during the dual-recording interaction process are obtained; the video and audio data sets are segmented according to time to obtain video and audio data at different stages; multiple fusion features are obtained based on the fusion features corresponding to the generation stage of the video and audio data at each stage, wherein at least one fusion feature is obtained by fusing the features of the video and audio data according to the noise interference level of the stage, and the noise interference level includes the video interference level and the audio interference level; the multiple fusion features are input into the dual-recording detection model, and the dual-recording quality inspection result is output according to the dual-recording detection model, thus solving the technical problem of low quality inspection accuracy of dual-recording interaction data in the prior art.

[0026] By processing the video and audio data sets in the dual-recording interaction process into time-sliced ​​segments, video and audio data at different stages are obtained. Based on the noise interference level, the features of the video and audio data at at least one stage are fused to obtain the corresponding fused features. This achieves the focus of key features, improves the prediction accuracy of the dual-recording detection model, and enhances the quality inspection accuracy of dual-recording interaction data. Attached Figure Description

[0027] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0028] Figure 1 A hardware structure block diagram of a computer terminal for implementing a quality inspection method for dual-recording interactive data is shown.

[0029] Figure 2 This is a flowchart of a quality inspection method for dual-recording interactive data provided in the embodiments of this application;

[0030] Figure 3 This is a schematic diagram of a quality inspection device for dual-recording interactive data provided in the embodiments of this application;

[0031] Figure 4 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be used interchangeably to the extent appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0034] It should be noted that the information collected in this application (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of this data all comply with relevant laws, regulations, and standards, do not violate public order and good morals, and provide corresponding access points for users to choose to authorize or refuse. For example, this system has interfaces with relevant users or organizations, providing users with corresponding access points to choose to agree to or refuse automated decision results; if the user chooses to refuse, the process proceeds to the expert decision-making stage.

[0035] Example 1

[0036] According to an embodiment of this application, a method embodiment for quality inspection of dual-recording interactive data is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, the steps shown or described may be executed in a different order than that shown here to some extent.

[0037] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a quality inspection method for dual-recording interactive data is shown. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0038] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0039] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the quality inspection method for dual-recording interactive data in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned quality inspection method for dual-recording interactive data. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0040] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0041] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0042] Under the aforementioned operating environment, this application provides the following: Figure 2 The quality inspection method for dual-recording interactive data is shown. Figure 2 This is a flowchart of a quality inspection method for dual-recording interactive data according to Embodiment 1 of this application.

[0043] Step S201: With authorization from the first object and the second object, obtain the video data set and audio data set of the first object and the second object during the dual recording interaction process.

[0044] Optionally, the first subject can be a customer, and the second subject can be a client manager of the target financial institution. Both subjects can authorize each other through electronic signatures or facial recognition before the dual recording begins, ensuring that data collection complies with relevant laws, regulations, and standards. The video data set refers to the original, time-stamped video sequence continuously captured by the camera during the dual recording interaction. For example, the video data set may include visual information such as document display. The audio data set refers to the original, time-stamped voice data collected during the dual recording interaction. For example, the audio data set may include dialogue information between the first and second subjects.

[0045] Step S202: The video data set and audio data set are segmented according to time to obtain video data and audio data at different stages.

[0046] Optionally, based on the different interactive content at different time stages throughout the dual recording interaction process, the dual recording interaction can be divided into multiple stages according to time, and the video data set and audio data set can be segmented to divide the video data set into video data of multiple stages and the audio data into audio data of multiple stages, thereby obtaining video data and audio data within each stage. The video data is used to provide image information in the corresponding stage, while the audio data is used to provide sound information in the corresponding stage.

[0047] Step S203: Generate fusion features corresponding to each stage based on the video and audio data of each stage, and obtain multiple fusion features.

[0048] It should be noted that at least one fusion feature is obtained by fusing features of video and audio data based on the noise interference level of the respective stage. The noise interference level includes both video and audio interference levels. For example, multiple stages may include a first stage, a second stage, and a third stage. The fusion features of the first and second stages may be obtained by fusing features of video and audio data based on the noise interference level of the respective stage, while the fusion features of the third stage may be obtained by fusing features of video and audio data using only conventional methods.

[0049] Optionally, the fusion feature of any stage refers to the feature (i.e., feature vector, not the original data) of the video data and audio data corresponding to that stage after weighted fusion. The fusion feature is used to characterize the interaction characteristics of that stage. The noise interference level refers to the assessment value of the data quality degradation caused by environmental factors in a certain stage. The noise interference level can be divided into video interference level (e.g., overexposure, motion blur, camera shake) and audio interference level (e.g., low signal-to-noise ratio).

[0050] Optionally, for the fusion features corresponding to the generation stage of video and audio data at any stage, the fusion weights can be adjusted based on the degree of noise interference. The fusion weights can include cross-modal weights and cross-timeline weights. The focus of compliance quality inspection differs at different stages (e.g., the first stage emphasizes visual aspects, while the second stage emphasizes semantic aspects).

[0051] Optionally, cross-modal weights are used to represent the relative importance of the fusion of video features (i.e., features extracted from video data) and audio features (i.e., features extracted from audio data) at the same point in time. For example, if a certain stage is a product introduction and risk warning stage, and if the audio interference is high (e.g., high background noise), the weight of audio features can be reduced and the weight of video features (e.g., video footage such as nodding in agreement, displaying product text content, or risk warning text content) can be increased to compensate for the loss of audio information.

[0052] Optionally, cross-timeline weights refer to the relative importance of fusing features from different time points (e.g., different frames of a video, different segments of an audio file) within the same modality (e.g., video features). For example, during the identity verification stage, the first 3 seconds of the first person displaying their ID card may be blurry due to hand tremors, but the 5th second is clear and stable (the 5th second is the same as the first 3 seconds, only the clarity is different). The weight of the features from the first 3 seconds can be reduced, retaining only the contribution of the clear frame, so that the subsequent dual-recording detection model can focus more on the features of the clear frame.

[0053] Step S204: Input multiple fused features into the dual-recording detection model, and output the dual-recording quality inspection results based on the dual-recording detection model.

[0054] Optionally, the dual-recording detection model can be a supervised neural network classifier. Its input is a vector composed of three fused features (each fused feature corresponds to a stage), and its output is the dual-recording quality inspection result of the dual-recording interaction data. The dual-recording quality inspection result can be either pass or fail. For example, if there is no clear ID card image or product content introduction in the dual-recording interaction data, the dual-recording quality inspection result is fail. If all three stages have been completed and there are clear images of the data, the dual-recording quality inspection result is pass.

[0055] In summary, by processing the video and audio data sets in the dual-recording interaction process into time-sliced ​​segments, video and audio data at different stages are obtained. Furthermore, by fusing the features of the video and audio data at at least one stage according to the degree of noise interference, corresponding fused features are obtained. This achieves the focus of key features, improves the prediction accuracy of the dual-recording detection model, and enhances the quality inspection accuracy of dual-recording interaction data.

[0056] To improve the accuracy of quality inspection of dual-recording interactive data, optionally, the video data set and audio data set are segmented according to time to obtain video data and audio data at different stages, including: during the dual-recording interactive process, determining a first time range for the first stage, a second time range for the second stage, and a third time range for the third stage, wherein the first stage is used for identity verification, the second stage is used for product introduction and risk warning, and the third stage is used for contract confirmation; extracting first video data from the video data set and first audio data from the audio data set according to the first time range, extracting second video data from the video data set and second audio data from the audio data set according to the second time range, and extracting third video data from the video data set and third audio data from the audio data set according to the third time range.

[0057] Optionally, the first stage can be for identity verification, which is the initial step in the dual-recording process where the first and second parties jointly verify the authenticity of their identities. Core compliance actions may include the first party presenting their ID and facing the camera, and the second party presenting their ID and relevant qualification certificates. In this stage, video is susceptible to environmental interference, leading to poor interaction detection. The second stage can be for product introduction and risk warning. For example, in the second stage, the second party systematically introduces the product's relevant content and potential risks to the first party. Core compliance points may include the presentation, relaying, or verbal notification of product content and risk warnings, and the first party expressing informed consent. In this stage, audio is susceptible to environmental interference, leading to poor interaction detection. The third stage is for contract confirmation, where the first party, after fully understanding the relevant information, signs a contract using an electronic signature device. Compared to the first two stages, this stage is less susceptible to environmental noise interference. The above three stages are defined by a first time range, a second time range, and a third time range, which can be pre-set fixed time ranges.

[0058] In summary, by segmenting the video and audio datasets according to the first, second, and third time ranges, the corresponding video and audio data for each stage were obtained, thus improving the accuracy of quality inspection of dual-recording interactive data.

[0059] To improve the quality inspection accuracy of dual-recording interactive data, optionally, multiple fusion features are obtained based on the fusion features corresponding to the generation stages of video and audio data at each stage. These include: generating a first fusion feature based on the video interference level, first video data, and first audio data in the first stage; generating a second fusion feature based on the audio interference level, second video data, and second audio data in the second stage; generating a third fusion feature based on the third video data and third audio data in the third stage; and combining the first, second, and third fusion features to obtain multiple fusion features.

[0060] Optionally, the video interference level can be used to represent the decrease in recognition confidence caused by factors such as abnormal ambient lighting, occlusion, camera shake, or motion blur, while the audio interference level can be used to represent the decrease in audio data quality caused by background noise, multiple people talking, etc.

[0061] Optionally, for the first stage, the cross-timeline weights of the single modality for video features can be adjusted using the degree of video interference (see the foregoing explanation for details) to obtain the first video feature for the first stage. This first video feature is then fused with the corresponding features of the first audio data to obtain the first fused feature. For the second stage, the cross-modal weights of audio and video features can be adjusted using the degree of audio interference (see the foregoing explanation for details) to obtain the second fused feature for the second stage. For the third stage, features can be extracted from the third video data to obtain the third video feature, and features can be extracted from the third audio data to obtain the third audio feature. The third video feature and the third audio feature are then directly fused by concatenation to obtain the third fused feature.

[0062] In summary, by generating a first fusion feature based on the video interference level, the first video data, and the first audio data in the first stage, generating a second fusion feature based on the audio interference level, the second video data, and the second audio data in the second stage, and generating a third fusion feature based on the third video data and the third audio data in the third stage, multiple fusion features were obtained, which improved the quality inspection accuracy of dual-recording interactive data.

[0063] To improve the accuracy of quality inspection of dual-recording interactive data, optionally, generating a first fusion feature based on the video interference level, the first video data, and the first audio data in the first stage includes: determining multiple frames of video information based on the first video data, and extracting sub-video features corresponding to each frame of video information to obtain multiple sub-video features; determining the video interference level of each frame of video information through video interference detection; fusing the multiple sub-video features based on the video interference level to obtain a first video feature; extracting features from the first audio data to obtain a first audio feature, and fusing the first video feature and the first audio feature to generate the first fusion feature.

[0064] Optionally, the video interference level may include one or more of the following: an abnormal lighting index, an occlusion detection index, and a motion blur index. If there are multiple indices, the video interference level can be obtained by weighted averaging of the different indices. If there is only one index, that index can be directly determined as the video interference level. The exposure of the video information can be statistically analyzed using an image histogram to obtain the lighting conditions. If the lighting conditions are overexposed or underexposed, the abnormal lighting index is set to a preset first value (e.g., 0.8). If the lighting conditions are normal, the abnormal lighting index is set to a preset second value (e.g., 0). Occlusion results can be obtained using an occlusion detection model (e.g., a face region occlusion detection model or an ID card occlusion detection model). If neither the face nor the ID card is occluded, the occlusion detection index is set to a preset third value (e.g., 0). If at least one of the face or the ID card is occluded, the occlusion detection index is set to a preset fourth value (e.g., 0.9). Open-source blur image recognition models can be used to detect motion blur in video information and obtain blur probability values ​​as motion blur indicators. Alternatively, image sharpness can be calculated based on the variance of Laplacian (an open-source sharpness calculation method) to obtain the variance, and the motion blur indicator can be determined based on the reciprocal of the variance.

[0065] Optionally, multi-frame video information refers to video information from multiple frames acquired frame by frame from the first video data, where each frame can be treated as a still image. Sub-video features refer to the features obtained by feature extraction from a single frame of video information (e.g., using 3D-CNN or 2D-CDD, a spatial feature extraction network), used to encode the visual semantics related to actions and document display in that frame. Extracting the sub-video features corresponding to each frame of video information yields multiple sub-video features. Weights can be assigned to each sub-video feature in the following manner:

[0066] w i =1-d i

[0067] Where i represents the video information of the i-th frame, w i d represents the weight of the sub-video features of the i-th frame video information. iThis represents the degree of video interference in the i-th frame. In other words, the higher the degree of video interference, the more severe the interference, and the lower the weight; conversely, the less interference, the higher the weight. Multiple sub-video features are fused by weighted averaging according to the above weights to obtain the first video feature. Mel-frequency cepstral coefficients can be used to extract features from the first audio data to obtain the first audio feature. The first video feature (e.g., a-dimensional) and the first audio feature (e.g., b-dimensional) can be concatenated along the dimensional direction to obtain the first fused feature (e.g., a+b-dimensional); alternatively, the first video feature or the first audio feature can be reshaped to the same dimension, and then the first video feature and the first audio feature of the same dimension can be linearly fused to obtain the first fused feature.

[0068] In summary, by performing video interference detection on each frame of video information and fusing the sub-video features of different frames based on the degree of video interference, a first video feature is obtained. The first video feature and the first audio feature are then fused to generate a first fused feature, which improves the quality inspection accuracy of dual-recording interactive data.

[0069] To improve the quality inspection accuracy of dual-recording interactive data, optionally, multiple sub-video features are fused according to the degree of video interference to obtain the first video feature, including: assigning a first weight to sub-video features whose video interference level reaches the video interference threshold, and assigning a second weight to sub-video features whose video interference level does not reach the video interference threshold, wherein the first weight is less than the second weight; and weighting and fusing multiple sub-video features according to the first weight and the second weight to obtain the first video feature.

[0070] For example, a video interference threshold can be used as a criterion to distinguish between interfering and clear frames in sub-video features. The video interference threshold can be set to 0.65, the first weight can be set to 0.1, and the second weight can be set to 0.9. This allows for weighted fusion of multiple sub-video features based on the first and second weights, significantly suppressing the contribution of interfering frames to the final features and making the dual-recording detection model more focused on the sub-video features of clear frames. Each sub-video feature can be multiplied by its corresponding configured weight to obtain multiple product results. The sum of these products is then divided by the sum of all configured weights to obtain the first video feature.

[0071] In summary, by assigning lower weights to sub-video features with video interference levels exceeding the video interference threshold and higher weights to sub-video features with video interference levels below the video interference threshold, the quality inspection accuracy of dual-recording interactive data is improved.

[0072] To improve the accuracy of quality inspection of dual-recording interactive data, optionally, generating a second fusion feature based on the audio interference level, second video data, and second audio data in the second stage includes: splitting the second audio data and second video data into preset sub-time periods to obtain multiple sub-time period audio sub-information and video sub-information; extracting audio sub-features from the audio sub-information and video sub-features from the video sub-information; performing audio interference detection on each audio sub-information to obtain multiple audio interference levels; fusing the audio sub-features and video sub-features of each sub-time period based on the audio interference levels to obtain multiple sub-time period features; and fusing the multiple sub-time period features to generate the second fusion feature.

[0073] Optionally, the audio interference level can include at least one of the SNR (Signal-to-Noise Ratio) and STOI (Speech Proficiency Indication) metrics. If both metrics are included, a weighted average can be taken to obtain the audio interference level. The preset sub-time period can be 2 seconds, meaning the second audio and video data are split into multiple sub-time periods (e.g., 1-2 seconds, 3-4 seconds, 5-6 seconds, etc.) with audio sub-information (original audio waveform segments within 2 seconds) and video sub-information (continuous frame sequences within 2 seconds). Mel-frequency cepstral coefficients can be used to extract audio sub-features from the audio sub-information, and 3D-CNN or 2D-CDD can be used to extract video sub-features from the video sub-information. For any given sub-time period, the higher the audio interference level, the higher the weight of the video sub-features when fusing them to obtain sub-time period features (correspondingly, the lower the weight of the audio sub-features). Audio and video sub-features can be fused by splicing along the dimensional direction or by first reshaping to the same dimension and then linearly fusing to obtain the sub-time period features corresponding to the sub-time period. A temporal aggregation network can be used to fuse features from multiple sub-time periods to generate a second fused feature. For example, the temporal aggregation network can be a two-layer long short-term memory neural network, and the hidden state output at the last time step in the second layer can be used as the second fused feature.

[0074] In summary, by fusing audio and video sub-features within each sub-time period based on the degree of audio interference, and by fusing features from multiple sub-time periods to generate a second fused feature, the quality inspection accuracy of dual-recording interactive data is improved.

[0075] To improve the quality inspection accuracy of dual-recording interactive data, optionally, the audio sub-features and video sub-features of each sub-time period are fused according to the degree of audio interference to obtain multiple sub-time period features, including: configuring a third weight for audio sub-features whose audio interference level reaches the audio interference threshold, and configuring a fourth weight for audio sub-features whose audio interference level does not reach the audio interference threshold, wherein the third weight is less than the fourth weight; and fusing the audio sub-features with each weight configuration with the video sub-features of the same sub-time period to obtain multiple sub-time period features.

[0076] Alternatively, for any given sub-time period, the audio sub-features and video sub-features can be fused in the following ways:

[0077] F k =a k ×α k +b k ×β k

[0078] Among them, F k Let a represent the sub-time characteristics of the k-th sub-time period. k Represents audio sub-features, b k Represents video sub-features, α k +β k =1. The audio interference threshold can be 0.6. If the audio interference level reaches the audio interference threshold, the weight of the audio sub-feature is the third weight (e.g., 0.2, corresponding to the weight of the video sub-feature being 0.8). Otherwise, the weight of the audio sub-feature is the fourth weight (e.g., 0.6, corresponding to the weight of the video sub-feature being 0.4).

[0079] In summary, by assigning lower weights to audio sub-features whose audio interference reaches the audio interference threshold, the importance of video sub-features is amplified, thereby improving the quality inspection accuracy of dual-recording interactive data.

[0080] To improve the accuracy of quality inspection of dual-recording interactive data, the training steps of the dual-recording detection model may optionally include: receiving multiple dual-recording interactive samples and the label of each dual-recording interactive sample; extracting multiple sample fusion features of each dual-recording interactive sample; and training a preset neural network model based on the multiple sample fusion features and the corresponding labels to obtain the dual-recording detection model.

[0081] Optionally, the dual-recording interactive samples may include a sample video dataset and a sample audio dataset. Referring to the feature extraction method described above in the prediction stage, those skilled in the art can complete the feature extraction of the dual-recording interactive samples in the same way to obtain multiple sample fusion features for each dual-recording interactive sample (e.g., sample fusion features in the first stage, sample fusion features in the second stage, and sample fusion features in the third stage). The label for each dual-recording interactive sample can be "pass" or "fail". The preset neural network model can be a multilayer perceptron, specifically including an input layer, four hidden layers, and an output layer. Each hidden layer uses ReLU (an activation function) and dropout (i.e., the operation of discarding neurons) to prevent overfitting. The output layer includes a sigmoid (an activation function for binary classification) classifier, and the training process can use the cross-entropy loss function.

[0082] In summary, by training a pre-defined neural network model based on the fusion features of multiple samples and their corresponding labels, a dual-recording detection model was obtained, which improved the accuracy of quality inspection of dual-recording interactive data.

[0083] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, the steps shown or described may be performed in a different order than that shown here to some extent.

[0084] Example 2

[0085] This application also provides a quality inspection device for dual-recording interactive data. It should be noted that this quality inspection device can be used to execute the quality inspection method for dual-recording interactive data provided in this application. The following describes the quality inspection device for dual-recording interactive data provided in this application.

[0086] According to an embodiment of this application, an apparatus for implementing the above-described quality inspection method for dual-recording interactive data is also provided, such as... Figure 3 As shown, the device includes:

[0087] The acquisition unit 301 is used to acquire the video data set and audio data set of the first object and the second object during the dual recording interaction process, provided that the first object and the second object have been authorized.

[0088] Slicing unit 302 is used to slice the video data set and audio data set according to time to obtain video data and audio data at different stages.

[0089] The fusion unit 303 is used to generate fusion features corresponding to each stage based on the video data and audio data of each stage, and obtain multiple fusion features. At least one fusion feature is obtained by fusing the features of the video data and audio data according to the noise interference level of the stage. The noise interference level includes the video interference level and the audio interference level.

[0090] The output unit 304 is used to input multiple fused features into the dual-recording detection model and output the dual-recording quality inspection results based on the dual-recording detection model.

[0091] The dual-recording interactive data quality inspection device provided in this application embodiment obtains video and audio data sets of the first and second objects during the dual-recording interaction process through the acquisition unit 301, which, upon obtaining authorization from the first and second objects, performs time-based segmentation processing on the video and audio data sets to obtain video and audio data at different stages. The fusion unit 303 generates fusion features corresponding to each stage of the video and audio data, resulting in multiple fusion features. At least one fusion feature is obtained by fusing features of the video and audio data based on the noise interference level of the respective stage. The noise interference level includes both video and audio interference levels. The output unit 304 inputs the multiple fusion features into a dual-recording detection model and outputs the dual-recording quality inspection result based on the model. This solves the problem of low accuracy in the quality inspection of dual-recording interactive data in related technologies, thereby improving the accuracy of dual-recording interactive data quality inspection.

[0092] Optionally, in the quality inspection device for dual-recording interactive data provided in this application embodiment, the segmentation unit 302 includes: a time range determination module, used to determine a first time range for a first stage, a second time range for a second stage, and a third time range for a third stage during the dual-recording interaction process, wherein the first stage is used for identity verification, the second stage is used for product introduction and risk warning, and the third stage is used for contract confirmation; and a data extraction module, used to extract first video data from a video data set and first audio data from an audio data set according to the first time range, extract second video data from a video data set and second audio data from an audio data set according to the second time range, and extract third video data from a video data set and third audio data from an audio data set according to the third time range.

[0093] Optionally, in the quality inspection device for dual-recording interactive data provided in this application embodiment, the fusion unit 303 includes: a first fusion module, used to generate a first fusion feature based on the video interference level, first video data, and first audio data in the first stage; a second fusion module, used to generate a second fusion feature based on the audio interference level, second video data, and second audio data in the second stage; a third fusion module, used to generate a third fusion feature based on the third video data and third audio data in the third stage; and a fusion result module, used to combine the first fusion feature, the second fusion feature, and the third fusion feature to obtain multiple fusion features.

[0094] Optionally, in the quality inspection device for dual-recording interactive data provided in this application embodiment, the first fusion module includes: a first extraction submodule, used to determine multiple frames of video information based on the first video data, and extract sub-video features corresponding to each frame of video information to obtain multiple sub-video features; a first detection submodule, used to determine the video interference level of each frame of video information through video interference detection; a first fusion submodule, used to fuse the multiple sub-video features based on the video interference level to obtain a first video feature; and a first generation submodule, used to extract features from the first audio data to obtain a first audio feature, and fuse the first video feature and the first audio feature to generate a first fused feature.

[0095] Optionally, in the quality inspection device for dual-recording interactive data provided in this application embodiment, the first fusion submodule includes: a first weight configuration component, used to configure a first weight for sub-video features whose video interference level reaches the video interference threshold, and configure a second weight for sub-video features whose video interference level does not reach the video interference threshold, wherein the first weight is less than the second weight; and a first weighted fusion component, used to perform weighted fusion of multiple sub-video features according to the first weight and the second weight to obtain a first video feature.

[0096] Optionally, in the quality inspection device for dual-recording interactive data provided in this application embodiment, the second fusion module includes: a time-segmentation submodule, used to split the second audio data and the second video data according to preset sub-time periods to obtain audio sub-information and video sub-information of multiple sub-time periods; a second extraction submodule, used to extract audio sub-features of audio sub-information and video sub-features of video sub-information; a second detection submodule, used to perform audio interference detection on each audio sub-information to obtain multiple audio interference levels; a second fusion submodule, used to fuse the audio sub-features and video sub-features of each sub-time period according to the audio interference level to obtain multiple sub-time period features; and a second generation submodule, used to fuse the multiple sub-time period features to generate a second fused feature.

[0097] Optionally, in the quality inspection device for dual-recording interactive data provided in this application embodiment, the second fusion sub-module includes: a second weight configuration component, used to configure a third weight for audio sub-features whose audio interference level reaches the audio interference threshold, and configure a fourth weight for audio sub-features whose audio interference level does not reach the audio interference threshold, wherein the third weight is less than the fourth weight; and a second weighted fusion component, used to fuse the audio sub-features configured with each weight with video sub-features of the same sub-time period to obtain multiple sub-time period features.

[0098] Optionally, in the quality inspection device for dual-recording interactive data provided in the embodiments of this application, the device further includes: a receiving unit, used to receive multiple dual-recording interactive samples and the label of each dual-recording interactive sample; and a training unit, used to extract multiple sample fusion features of each dual-recording interactive sample, and train a preset neural network model based on the multiple sample fusion features and the corresponding labels to obtain a dual-recording detection model.

[0099] It should be noted that the acquisition unit 301, the slicing unit 302, the fusion unit 303, and the output unit 304 mentioned above correspond to steps S201 to S204 in Embodiment 1. The instances and application scenarios implemented by the units and the corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of a device and can run in the computer terminal 10 provided in Embodiment 1.

[0100] Example 3

[0101] Embodiments of this application may provide an electronic device. Figure 4 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 4 As shown, the electronic device may include: one or more ( Figure 4 (Only one is shown) Processor 402, memory 404, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0102] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the above-described methods. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0103] The processor can invoke information and applications stored in the memory via the transmission device to perform the following steps: With authorization from both the first and second objects, obtain video and audio data sets of the first and second objects during the dual-recording interaction process; segment the video and audio data sets according to time to obtain video and audio data at different stages; generate fusion features corresponding to each stage based on the video and audio data at each stage, obtaining multiple fusion features, wherein at least one fusion feature is obtained by fusing features of the video and audio data according to the noise interference level of the stage, the noise interference level including video interference level and audio interference level; input the multiple fusion features into the dual-recording detection model, and output the dual-recording quality inspection result based on the dual-recording detection model.

[0104] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: During the dual recording interaction process, determine the first time range of the first stage, the second time range of the second stage, and the third time range of the third stage, wherein the first stage is used for identity verification, the second stage is used for product introduction and risk warning, and the third stage is used for contract confirmation; extract first video data from the video data set and first audio data from the audio data set according to the first time range, extract second video data from the video data set and second audio data from the audio data set according to the second time range, and extract third video data from the video data set and third audio data from the audio data set according to the third time range.

[0105] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: generating a first fusion feature based on the video interference level, the first video data and the first audio data of the first stage; generating a second fusion feature based on the audio interference level, the second video data and the second audio data of the second stage; generating a third fusion feature based on the third video data and the third audio data of the third stage; and combining the first fusion feature, the second fusion feature and the third fusion feature to obtain multiple fusion features.

[0106] The processor can also call the information and application program stored in the memory through the transmission device to perform the following steps: determine multiple frames of video information based on the first video data, and extract the sub-video features corresponding to each frame of video information to obtain multiple sub-video features; determine the video interference level of each frame of video information through video interference detection; fuse the multiple sub-video features according to the video interference level to obtain the first video feature; extract features from the first audio data to obtain the first audio feature, and fuse the first video feature and the first audio feature to generate the first fused feature.

[0107] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: assign a first weight to the sub-video features whose video interference level reaches the video interference threshold, and assign a second weight to the sub-video features whose video interference level does not reach the video interference threshold, wherein the first weight is less than the second weight; and perform weighted fusion of multiple sub-video features according to the first weight and the second weight to obtain the first video feature.

[0108] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: splitting the second audio data and the second video data into preset sub-time periods to obtain multiple sub-time period audio sub-information and video sub-information; extracting audio sub-features of the audio sub-information and video sub-features of the video sub-information; performing audio interference detection on each audio sub-information to obtain multiple audio interference levels; fusing the audio sub-features and video sub-features of each sub-time period according to the audio interference levels to obtain multiple sub-time period features; fusing the multiple sub-time period features to generate a second fused feature.

[0109] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: configure a third weight for audio sub-features whose audio interference level reaches the audio interference threshold, and configure a fourth weight for audio sub-features whose audio interference level does not reach the audio interference threshold, wherein the third weight is less than the fourth weight; fuse the audio sub-features configured with each weight with the video sub-features of the same sub-time period to obtain multiple sub-time period features.

[0110] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: receiving multiple dual-recording interaction samples and the label of each dual-recording interaction sample; extracting multiple sample fusion features of each dual-recording interaction sample, and training a preset neural network model based on the multiple sample fusion features and the corresponding labels to obtain a dual-recording detection model.

[0111] This application provides a scheme for quality inspection of dual-recording interactive data. By obtaining authorization from both a first and a second object, video and audio data sets of the first and second objects during the dual-recording interaction process are obtained. The video and audio data sets are then segmented according to time to obtain video and audio data at different stages. Multiple fusion features are obtained based on the fusion features corresponding to each stage of the video and audio data generation process. At least one fusion feature is obtained by fusing features of the video and audio data according to the noise interference level of the respective stage, where the noise interference level includes both video and audio interference levels. These multiple fusion features are input into a dual-recording detection model, and the dual-recording quality inspection result is output based on the model. This solves the technical problem of low accuracy in quality inspection of dual-recording interactive data in the prior art.

[0112] Those skilled in the art will understand that Figure 4 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 4 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 4 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 4 The different configurations shown.

[0113] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0114] Example 4

[0115] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the quality inspection method for dual-recording interactive data provided in Embodiment 1.

[0116] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0117] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: obtaining a set of video data and a set of audio data of the first and second objects during the dual-recording interaction process, after obtaining authorization from the first and second objects; segmenting the set of video data and the set of audio data by time to obtain video data and audio data at different stages; generating fusion features corresponding to each stage of video data and audio data to obtain multiple fusion features, wherein at least one fusion feature is obtained by fusing features of video data and audio data according to the noise interference level of the stage, the noise interference level including video interference level and audio interference level; inputting multiple fusion features into the dual-recording detection model, and outputting dual-recording quality inspection results according to the dual-recording detection model.

[0118] Optionally, in this embodiment, the computer-readable storage medium is further configured to store program code for performing the following steps: during the dual-recording interaction process, determining a first time range for a first stage, a second time range for a second stage, and a third time range for a third stage, wherein the first stage is used for identity verification, the second stage is used for product introduction and risk warning, and the third stage is used for contract confirmation; extracting first video data from a video data set and first audio data from an audio data set according to the first time range, extracting second video data from a video data set and second audio data from an audio data set according to the second time range, and extracting third video data from a video data set and third audio data from an audio data set according to the third time range.

[0119] Optionally, in this embodiment, the computer-readable storage medium is further configured to store program code for performing the following steps: generating a first fusion feature based on the video interference level of the first stage, the first video data, and the first audio data; generating a second fusion feature based on the audio interference level of the second stage, the second video data, and the second audio data; generating a third fusion feature based on the third video data and the third audio data of the third stage; and combining the first fusion feature, the second fusion feature, and the third fusion feature to obtain multiple fusion features.

[0120] Optionally, in this embodiment, the computer-readable storage medium is further configured to store program code for performing the following steps: determining multiple frames of video information based on the first video data, and extracting sub-video features corresponding to each frame of video information to obtain multiple sub-video features; determining the video interference level of each frame of video information through video interference detection; fusing the multiple sub-video features according to the video interference level to obtain a first video feature; extracting features from the first audio data to obtain a first audio feature, and fusing the first video feature and the first audio feature to generate a first fused feature.

[0121] Optionally, in this embodiment, the computer-readable storage medium is further configured to store program code for performing the following steps: configuring a first weight for sub-video features whose video interference level reaches a video interference threshold, configuring a second weight for sub-video features whose video interference level does not reach a video interference threshold, wherein the first weight is less than the second weight; and performing weighted fusion of multiple sub-video features according to the first weight and the second weight to obtain a first video feature.

[0122] Optionally, in this embodiment, the computer-readable storage medium is further configured to store program code for performing the following steps: splitting the second audio data and the second video data into preset sub-time periods to obtain multiple sub-time period audio sub-information and video sub-information; extracting audio sub-features from the audio sub-information and extracting video sub-features from the video sub-information; performing audio interference detection on each audio sub-information to obtain multiple audio interference levels; fusing the audio sub-features and video sub-features of each sub-time period according to the audio interference levels to obtain multiple sub-time period features; and fusing the multiple sub-time period features to generate a second fused feature.

[0123] Optionally, in this embodiment, the computer-readable storage medium is further configured to store program code for performing the following steps: configuring a third weight for audio sub-features whose audio interference level reaches the audio interference threshold, configuring a fourth weight for audio sub-features whose audio interference level does not reach the audio interference threshold, wherein the third weight is less than the fourth weight; fusing the audio sub-features with each weight configuration with video sub-features of the same sub-time period to obtain multiple sub-time period features.

[0124] Optionally, in this embodiment, the computer-readable storage medium is further configured to store program code for performing the following steps: receiving multiple dual-recording interaction samples and the label of each dual-recording interaction sample; extracting multiple sample fusion features of each dual-recording interaction sample, and training a preset neural network model based on the multiple sample fusion features and the corresponding labels to obtain a dual-recording detection model.

[0125] This application also provides a computer program product that, when executed on a data processing device, is suitable for performing quality inspection method steps for dual-recording interactive data.

[0126] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0127] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0128] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0129] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0130] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0131] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0132] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A quality inspection method for dual-recording interactive data, characterized in that, include: With authorization from both the first and second objects, obtain the video data set and audio data set of the first and second objects during the dual-recording interaction process; The video and audio datasets are segmented according to time to obtain video and audio data at different stages. Based on the video data and audio data of each stage, a fusion feature corresponding to that stage is generated, resulting in multiple fusion features. At least one of the fusion features is obtained by fusing the features of the video data and the audio data according to the noise interference level of the stage. The noise interference level includes the video interference level and the audio interference level. The multiple fusion features are input into the dual recording detection model, and the dual recording quality inspection results are output based on the dual recording detection model.

2. The method according to claim 1, characterized in that, The video and audio datasets are segmented according to time, resulting in video and audio data at different stages, including: During the dual-recording interaction process, a first time range for the first stage, a second time range for the second stage, and a third time range for the third stage are determined. The first stage is used for identity verification, the second stage is used for product introduction and risk warning, and the third stage is used for contract confirmation. Based on the first time range, first video data is extracted from the video data set and first audio data is extracted from the audio data set; based on the second time range, second video data is extracted from the video data set and second audio data is extracted from the audio data set; based on the third time range, third video data is extracted from the video data set and third audio data is extracted from the audio data set.

3. The method according to claim 2, characterized in that, Based on the video and audio data of each stage, a fusion feature corresponding to that stage is generated, resulting in multiple fusion features including: A first fusion feature is generated based on the video interference level in the first stage, the first video data, and the first audio data. A second fusion feature is generated based on the degree of audio interference in the second stage, the second video data, and the second audio data; A third fusion feature is generated based on the third video data and the third audio data of the third stage; The first fusion feature, the second fusion feature, and the third fusion feature are combined to obtain the plurality of fusion features.

4. The method according to claim 3, characterized in that, The first fusion feature generated based on the video interference level in the first stage, the first video data, and the first audio data includes: Based on the first video data, determine multiple frames of video information, and extract sub-video features corresponding to each frame of video information to obtain multiple sub-video features; The degree of video interference in each frame of video information is determined by video interference detection. The multiple sub-video features are fused based on the degree of video interference to obtain the first video feature; Feature extraction is performed on the first audio data to obtain the first audio feature, and the first video feature and the first audio feature are fused to generate the first fused feature.

5. The method according to claim 4, characterized in that, The multiple sub-video features are fused based on the degree of video interference to obtain the first video feature, which includes: A first weight is assigned to sub-video features whose video interference level reaches the video interference threshold, and a second weight is assigned to sub-video features whose video interference level does not reach the video interference threshold, wherein the first weight is less than the second weight; The first video feature is obtained by weighting and fusing the multiple sub-video features according to the first weight and the second weight.

6. The method according to claim 3, characterized in that, The second fusion feature is generated based on the audio interference level in the second stage, the second video data, and the second audio data, including: The second audio data and the second video data are split into preset sub-time periods to obtain multiple sub-time period audio sub-information and video sub-information. Extract the audio sub-features from the audio sub-information, and extract the video sub-features from the video sub-information; Audio interference detection is performed on each of the audio sub-information to obtain multiple audio interference levels; Based on the degree of audio interference, the audio sub-features and video sub-features of each sub-time period are fused to obtain multiple sub-time period features; The features of the multiple sub-time periods are fused to generate the second fused feature.

7. The method according to claim 6, characterized in that, Based on the degree of audio interference, the audio sub-features and video sub-features of each sub-time period are fused to obtain multiple sub-time period features, including: A third weight is assigned to audio sub-features whose audio interference level reaches the audio interference threshold, and a fourth weight is assigned to audio sub-features whose audio interference level does not reach the audio interference threshold, wherein the third weight is less than the fourth weight; The audio sub-features with each weight configuration are fused with the video sub-features of the same sub-time period to obtain multiple sub-time period features.

8. The method according to claim 1, characterized in that, The training steps of the dual-recording detection model include: Receive multiple dual-recording interaction samples and the tag of each dual-recording interaction sample; Multiple sample fusion features are extracted from each of the dual-recording interaction samples, and a preset neural network model is trained based on the multiple sample fusion features and the corresponding labels to obtain the dual-recording detection model.

9. A quality inspection device for dual-recording interactive data, characterized in that, include: The acquisition unit is used to acquire, upon obtaining authorization from the first object and the second object, a set of video data and a set of audio data of the first object and the second object during the dual recording interaction process; The segmentation unit is used to segment the video data set and the audio data set according to time to obtain video data and audio data at different stages; A fusion unit is used to generate a fusion feature corresponding to each stage based on the video data and the audio data of each stage, thereby obtaining multiple fusion features. At least one of the fusion features is obtained by fusing the features of the video data and the audio data according to the noise interference level of the stage to which it belongs. The noise interference level includes the video interference level and the audio interference level. The output unit is used to input the multiple fused features into the dual-recording detection model and output the dual-recording quality inspection result based on the dual-recording detection model.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the computer-readable storage medium is located to perform the quality inspection method for dual-recording interactive data as described in any one of claims 1 to 8.

11. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program executes the quality inspection method for dual-recording interactive data as described in any one of claims 1 to 8.

12. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the quality inspection method for dual-recording interactive data as described in any one of claims 1 to 8.

Images