Video customer service control method and device and storage medium

Abstract

The invention discloses a video customer service control method and device and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining the multi-modal information of a customer, extracting the multi-dimensional features of the multi-modal information, carrying out the screening and matching from a biological feature library through combining with a feature similarity threshold value, and obtaining a customer identity authentication result. And fusing and analyzing the multi-modal information and an identity authentication result through a multi-scale collaborative model, and judging the semantic, emotion and business scene information of the customer. And calling corresponding customer portrait data, processing customer semantic information, emotion information and business scene information through a multi-dimensional fusion decision, generating an optimal service processing path and a personalized service strategy, and analyzing and generating a personalized service video according to the optimal service processing path and the personalized service strategy. According to the method, through multi-modal authentication, multi-scale analysis, portrait decision, personalized video generation and closed-loop optimization, the problem of poor service efficiency is solved, and optimization and upgrading of customer experience are realized.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a control method, device and storage medium for video customer service. Background Technology

[0002] In the securities industry, customer service is a crucial link in ensuring compliant business operations, maintaining customer trust, and controlling business risks. Currently, related technologies still rely on manual verification of customer identification information and service delivery. This method suffers from highly volatile accuracy rates due to human experience, making it difficult to quickly and accurately identify customers and adapt to their individual needs, ultimately leading to inefficient service.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a control method, device and storage medium for video customer service, which aims to solve the technical problem of poor service efficiency.

[0005] To achieve the above objectives, this application proposes a video customer service control method, the method comprising: After obtaining the customer's multimodal information, extract the multidimensional features of the multimodal information; Based on the multidimensional features and feature similarity thresholds, matches are filtered from the biometric database to obtain customer identity authentication results; By fusing and parsing the multimodal information and the customer identity authentication results through a multi-scale collaborative model, customer semantic information, emotional information, and business scenario information are determined. Retrieve customer profile data corresponding to the customer identity authentication result, and use multi-dimensional fusion decision processing to generate the optimal service processing path and personalized service strategy based on the customer semantic information, the emotional information and the business scenario information. Based on the optimal service processing path and the personalized service strategy, a personalized service video is generated.

[0006] In one embodiment, the multimodal information is preprocessed to remove invalid information, thereby obtaining target multimodal information; According to the modality type corresponding to the target multimodal information, extract the single-modal feature data from the target multimodal information respectively; The multimodal feature data is correlated, calibrated, and dimension-unified according to preset feature weights, and the multidimensional features are output.

[0007] In one embodiment, based on the multidimensional features, a set of customer candidate biometric features corresponding to the multidimensional features is retrieved from the preset biometric feature library to obtain a set of customer candidate biometric features; Calculate the feature similarity between the customer candidate biometric feature set and the multidimensional feature in each single modality dimension, and sum them according to the preset modality weights to obtain the comprehensive similarity value corresponding to each candidate feature, and output the comprehensive similarity value corresponding to each candidate feature; The comprehensive similarity value corresponding to each candidate feature is compared with the feature similarity threshold, and the customer information with the highest comprehensive similarity value higher than the feature similarity threshold is selected to generate the customer identity authentication result.

[0008] In one embodiment, key information fields are extracted from the multiple modal information to obtain multimodal basic parsing data containing text transcription content, facial micro-expression features, and normalized text information; Based on the multimodal basic parsing data, the small model in the multi-scale collaborative model calls the preset sensitive word library to intercept obviously illegal sensitive content, and obtain the sensitive word interception result; Based on text keywords and facial micro-expression features, the business scenario is initially labeled to its category, and preliminary business scenario labeling data is output. Based on the sensitive word interception results, the preliminary business scenario annotation data, and the customer identity authentication results, the customer semantic information, the emotion information, and the business scenario information are generated by deep analysis of the multimodal sentiment semantic feature information through the large model in the multi-scale collaborative model.

[0009] In one embodiment, customer profile data corresponding to the customer identity authentication result is retrieved, and the customer profile data is associated and aligned with three types of information: customer semantic information, emotional information, and business scenario information, to obtain comprehensive customer information; The comprehensive customer information is quantitatively evaluated from three dimensions: emotional urgency, business scenario complexity, and customer service preference matching. The weight of each dimension is determined and an evaluation score is formed. The multi-dimensional quantitative evaluation results are then output. Based on the multi-dimensional quantitative evaluation results, combined with the preset business processing rules and real-time service resource configuration, the optimal processing path is dynamically matched, and the personalized service strategy is customized.

[0010] In one embodiment, the core elements required for video generation in the optimal service processing path and the personalized service strategy are extracted and decomposed to generate a list of core elements for video generation; Based on the list of core elements for video generation, a digital human model consistent with the list of core elements for video generation, a segment of investor education content corresponding to the theme, and background materials suitable for the scene are matched and retrieved from the preset material library to obtain a video generation resource package. The video generation resource package integrates materials according to the process requirements of the optimal service processing path, synchronizes digital human voice and lip movements, embeds personalized investor education content and customer-related business data, and generates the personalized service video.

[0011] In one embodiment, the core attributes of the video content in the personalized service video are checked item by item, the location and type of the illegal content are marked, and a preliminary compliance verification result is generated; Based on the preliminary compliance verification results, the marked violations are updated according to the preset compliance standards. After verification confirms that there are no violations, a formal verification report detailing the compliance points is generated. The personalized service video that has passed the compliance verification will be pushed to the designated customer according to the optimal service processing path. The video push time, terminal delivery status and trigger identifier of customer click to view will be recorded simultaneously, and the video push execution result will be output.

[0012] In one embodiment, the video push execution result corresponding to the client terminal is monitored, and the real-time feedback content submitted by the client through the terminal feedback entry is recorded simultaneously to obtain the client's video viewing behavior data and real-time feedback information; Based on the customer video viewing behavior data and the real-time feedback information, the customer profile data and the current business scenario information are retrieved to generate integrated full customer service interaction data. Extract key indicators reflecting service adaptability, customer pain points, and video content effectiveness from the full volume of customer service interaction data, and generate service interaction feedback data and strategy optimization reference information.

[0013] In addition, to achieve the above objectives, this application also proposes a video customer service device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the video customer service control method as described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the video customer service control method described above.

[0015] This application provides a video customer service control method, which includes: firstly, acquiring multimodal customer information and extracting multidimensional features; secondly, filtering and matching customer identity authentication results from a biometric database based on these multidimensional features and feature similarity thresholds; thirdly, fusing and analyzing the multimodal information and customer identity authentication results through a multi-scale collaborative model to determine customer semantic information, emotional information, and business scenario information; fourthly, retrieving customer profile data corresponding to the customer identity authentication results; and fifthly, generating an optimal service processing path and personalized service strategy through multidimensional fusion decision processing of the aforementioned semantic, emotional, and business scenario information. Finally, generating personalized service videos based on these paths and strategies, and simultaneously pushing the videos to customer terminals after compliance quality inspection to verify the core attributes of the video content's compliance quality, collecting viewing interaction data and real-time feedback, and linking customer profiles and business scenario information to an intelligent decision engine. This technical implementation solves the technical problem of low overall process efficiency, improves the accuracy and security of customer identity authentication, enhances service response efficiency and customer experience, and achieves dynamic iterative optimization of service strategies.

[0016] In summary, this application solves the technical problem of low overall process efficiency by generating personalized videos and optimizing them in a closed loop through multimodal authentication, multi-scale parsing, and profiling decisions. It improves the overall efficiency of identity authentication, service adaptation and response, enhances authentication security and service compliance, and optimizes and upgrades the customer experience. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the first embodiment of the video customer service control method of this application; Figure 2 This is the overall flowchart of this application; Figure 3 This is a flowchart illustrating the seventh embodiment of the video customer service control method of this application; Figure 4 This is a flowchart illustrating the eighth embodiment of the video customer service control method of this application; Figure 5 This is a structural diagram of the video customer service equipment used in this application.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] In related technologies, it is still necessary to manually verify customer identification information item by item and provide services. In this way, the accuracy of identification is greatly affected by human experience, making it difficult to quickly and accurately identify customer identities and to quickly match differentiated needs for customers, thus resulting in poor service efficiency.

[0023] This application provides a solution: First, after acquiring the customer's multimodal information, multidimensional features of the multimodal information are extracted. Then, based on the multidimensional features and feature similarity thresholds, matching is performed from a biometric database to obtain the customer identity authentication result. Next, the multimodal information and the customer identity authentication result are fused and processed through a multi-scale collaborative model to determine the customer's semantic information, emotional information, and business scenario information. Then, the customer profile data corresponding to the customer identity authentication result is retrieved, and the customer's semantic information, emotional information, and business scenario information are processed through multidimensional fusion decision processing to generate the optimal service processing path and personalized service strategy. Finally, based on the optimal service processing path and the personalized service strategy, a personalized service video is generated.

[0024] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or video customer service device capable of performing the above functions. The following description uses a video customer service device as an example to illustrate this embodiment and the subsequent embodiments.

[0025] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0026] This application provides a video customer service control method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the video customer service control method of this application.

[0027] In this embodiment, the video customer service control method includes steps S10 to S50: Step S10: After obtaining the customer's multimodal information, extract the multidimensional features of the multimodal information.

[0028] In this embodiment, multimodal information refers to various types of raw information provided by the client, covering information from different carriers such as voice, images, and text. Multidimensional features refer to key representations of different dimensions extracted from various types of multimodal information, including core information at the acoustic, visual, and text structure levels.

[0029] As an optional implementation method, the acquired multimodal information is classified and sorted, splitting into speech, image, and text information according to the information carrier type. Speech information undergoes signal denoising and segmentation, image information undergoes contour extraction and feature point localization, and text information undergoes word segmentation and key information annotation. Then, the basic features corresponding to each type of information are extracted. Subsequently, the different types of basic features are simply concatenated and integrated according to preset rules to form multidimensional features. This method has a simple processing flow, low resource consumption, and fast execution speed.

[0030] As an alternative implementation, a cross-modal association mapping rule for multimodal information is first established. Simultaneous preprocessing of speech, image, and text information is performed. While extracting the basic features of each single modality, the correlation between different modal information is captured and associated features are generated. Then, through operations such as feature dimension calibration, redundant information removal, and core feature enhancement, the basic features of single modalities and cross-modal association features are deeply fused to form multidimensional features. This method provides sufficient feature fusion, preserves cross-modal association information, and has strong feature discrimination ability.

[0031] Step S20: Based on the multidimensional features and feature similarity threshold, a match is selected from the biometric database to obtain the customer identity authentication result.

[0032] In this embodiment, the feature similarity threshold is a preset standard for determining whether features match. The biometric database is a database that stores a customer's historical multidimensional biometric features and associated identity information.

[0033] As an optional implementation, this method first retrieves all customers' historical multidimensional features from the biometric database. These historical features are then split into single-modal types and the multidimensional features to be matched. The similarity between the feature to be matched and the historical features is calculated for each single-modal. All historical features with single-modal similarity scores higher than the feature similarity threshold are marked as candidate features. If a candidate feature is unique, it is directly associated with the corresponding identity information. If multiple candidate features exist, the one with the highest number of single-modal similarity scores is selected as the matching result. If no candidate feature meets the criteria, authentication fails. This method is logically simple, has a fast matching speed, and is suitable for high-concurrency scenarios.

[0034] As an alternative implementation, a preset weight is first assigned based on the importance of each modality. A subset of historical features consistent with the modality type of the multidimensional feature to be matched is selected from the biometric database. The similarity between the feature to be matched and each historical feature in the subset is calculated under the corresponding modality. A weighted summation is then performed to obtain a comprehensive similarity score. Historical features with a comprehensive similarity score higher than the feature similarity threshold are sorted in descending order of score. The historical feature with the highest score is associated with identity information as a preliminary result. Cross-modal feature association verification confirms the logical consistency between the preliminary result and the feature to be matched. Finally, the identity authentication result is output; if no matching feature is found, the authentication fails. The advantages of this implementation are that it considers modality weight differences and cross-modal verification, resulting in strong anti-interference capabilities. The disadvantages are that the calculation process is complex and time-consuming, requiring high processing resources. The effect is that it can accurately distinguish subtle differences in features, significantly reduce the false positive rate, improve the reliability of identity authentication, and adapt to complex feature matching scenarios. However, sufficient computing power is required to maintain efficient processing under high concurrency.

[0035] Step S30: The multimodal information and the customer identity authentication result are fused and processed by a multi-scale collaborative model to determine the customer semantic information, emotion information and business scenario information.

[0036] In this embodiment, the multi-scale collaborative model is a fusion processing model that can adapt to different information granularity parsing needs. It includes a large model and a small model. The small model uses precise keyword search and matching, is suitable for relatively certain and fixed scenarios, and has a faster response speed, but cannot understand semantics. The large model has strong generalization ability and is applicable to a wide range of scenarios, but consumes more resources. Semantic information is the core intent and content expressed by the customer. Emotional information is the emotional state conveyed by the customer. Business scenario information is the specific business type and demand scenario to which the customer belongs.

[0037] As an optional implementation, multimodal information is split according to carrier type. The basic scale module of a multi-scale collaborative model is used to analyze the text content in speech, facial expression features in images, and key information in text, respectively. Then, the model's fusion scale module integrates the single-modal analysis results, combines them with customer authentication results to verify information consistency, and eliminates contradictory data. Subsequently, core intent is extracted to form semantic information, sentiment tendencies are summarized to form emotional information, and business type is matched to form business scenario information. This method has a clear division of labor, a clear analysis process, is easy to maintain, and is adaptable to simple information scenarios.

[0038] As an alternative implementation, multimodal information and customer authentication results are simultaneously input into a multi-scale collaborative model. The bottom-level scale module extracts cross-modal correlation features, the mid-level scale module simultaneously parses semantic cores and sentiment tendencies based on these correlation features, and the top-level scale module combines customer identity information with the mid-level parsing results to accurately match business scenario types. Simultaneously, a built-in cross-scale verification mechanism corrects deviations in parsing results across different scales, ultimately integrating and outputting semantic information, sentiment information, and business scenario information. This method features thorough cross-modal feature fusion, coherent parsing logic, and a robust verification mechanism.

[0039] Step S40: Retrieve customer profile data corresponding to the customer identity authentication result, and generate the optimal service processing path and personalized service strategy by multi-dimensional fusion decision processing of the customer semantic information, the emotional information and the business scenario information.

[0040] In this embodiment, customer profile data is associated with customer identity authentication results and includes customer-specific information such as investment experience, risk preference, and historical service records. Multi-dimensional fusion decision-making is a decision-making method that integrates customer semantic information, emotional information, business scenario information, and customer profile data for comprehensive analysis and judgment. Optimal service processing path is an efficient service execution process that adapts to customer needs and resource allocation. Personalized service strategy is a customized service plan tailored to individual customer characteristics and the current scenario.

[0041] As an optional implementation, customer profile data is retrieved based on customer authentication results. Customer semantic information, emotional information, and business scenario information are categorized and labeled according to preset categories. Preset service rules are then matched hierarchically according to business scenario priority, emotional urgency, and core semantic needs. Service execution processes are selected based on service preferences in the customer profile, and service plans including reception methods and content delivery priorities are simultaneously customized. Finally, the optimal service processing path and personalized service strategy are integrated to generate the final result. This method features clear decision-making logic, fast execution speed, and low resource consumption.

[0042] As an alternative implementation, customer profile data corresponding to the customer's authentication results is first retrieved. Customer semantic information, emotional information, and business scenario information are then integrated with the customer profile data through feature association. Core features reflecting customer demand priority and service suitability are extracted, and the decision weights of each feature are dynamically allocated. A comprehensive assessment is then conducted based on preset business specifications and real-time service resource status to select a service execution process that balances efficiency and suitability. Simultaneously, based on individual characteristics in the customer profile and current scenario requirements, a service plan including exclusive content and interaction methods is customized, generating the optimal service processing path and personalized service strategy. This method features thorough feature fusion, high decision-making flexibility, and strong adaptability.

[0043] Step S50: Based on the optimal service processing path and the personalized service strategy, a personalized service video is generated by parsing.

[0044] In this embodiment, personalized service video is a video format that carries exclusive service content and is adapted to customer needs.

[0045] As an optional implementation, this method first breaks down the core process nodes of the optimal service processing path and the key content and style requirements of the personalized service strategy. It then matches scene segments corresponding to the process nodes, digital human avatars matching customer preferences, and voice materials with appropriate tones from a pre-set video clip library, digital human avatar library, and voice material library. The materials are then spliced ​​and integrated in the process sequence, with the visual rhythm and voice synchronization adjusted synchronously. Key information required for personalized services is embedded to generate personalized service videos. This method features a simple, fast, and low-resource-consumption generation process, making it suitable for high-concurrency service scenarios.

[0046] As an alternative implementation method, this approach deeply analyzes the execution logic of the optimal service processing path and the detailed requirements of personalized service strategies. It adapts the voice tone and rhythm to the customer's emotional state, adjusts the content depth according to the complexity of the business scenario, and modifies basic materials retrieved from the resource library to generate customized transition effects, subtitle styles, and interactive trigger points. It integrates customer-specific information and service priorities, and ensures accurate matching between content and strategy through multi-dimensional adaptation and verification, ultimately generating personalized service videos. This method offers a high degree of customization, strong content adaptability, and outstanding uniqueness, adapting to complex personalized service needs.

[0047] For example, refer to Figure 2 , Figure 2This is the overall flowchart of this application. In the video customer service scenario of the securities industry, after a customer initiates a video service request, the system obtains multimodal information such as their voice, facial image, and business text, and extracts multi-dimensional features such as voice acoustic features (fundamental frequency average 115Hz), coordinates of 128 facial feature points, and text keywords "credit card bill installment". Based on these features and a feature similarity threshold of 0.88, a match is selected from the biometric database to obtain the customer identity authentication result (match degree 0.95). Through multi-scale collaborative model fusion analysis, the semantic information is determined to be "consulting on bill installment rates", the emotional information is "neutral to urgent", and the business scenario information is "credit card service scenario". Customer profile data (credit card usage duration of 5 years, excellent credit rating) is retrieved, and the optimal service processing path is generated through multi-dimensional fusion decision-making: "assisting human agents, rate explanation" and personalized service strategy (pushing "exclusive installment discount for excellent credit rating" video). Based on this path and strategy, a personalized service video containing the digital human "Financial Advisor C" and the content "Exclusive Installment Rate Calculation" is generated. The video is 1 minute and 30 seconds long, and the key frames at 20 seconds and 50 seconds are marked with "Rate Comparison".

[0048] By using multimodal authentication, multi-scale parsing, and profiling decisions to generate personalized videos and optimize them in a closed loop, the technical problem of low overall process efficiency has been solved, the overall efficiency of identity authentication, service adaptation and response has been improved, authentication security and service compliance have been enhanced, and the customer experience has been optimized and upgraded.

[0049] Based on any of the above embodiments, in Embodiment 2 of this application, step S10 includes steps A11 to A13: Step A11: Preprocess the multimodal information, remove invalid information from the multimodal information, and obtain the target multimodal information.

[0050] In this embodiment, preprocessing is an operation that performs preliminary processing on the raw multimodal information. Invalid information refers to interfering content in the multimodal information that has no value for subsequent processing. Target multimodal information is the valid information that can be used for subsequent processing such as feature extraction after removing invalid information.

[0051] As an optional implementation method, the multimodal information is first classified according to carrier type. For speech information, silence segments are detected and removed; for image information, blurred regions are identified and cropped; and for text information, meaningless characters are filtered and content that is too short is selected. Interference components in each type of information are then identified and marked. Subsequently, the marked invalid information is completely removed, and the remaining valid components are integrated to form the target multimodal information. This method has an intuitive processing flow, is simple to operate, and consumes few resources.

[0052] Step A12: Extract the single-modal feature data from the target multimodal information according to the modality type corresponding to the target multimodal information.

[0053] In this embodiment, modality type refers to the carrier category of multimodal information, such as speech, image, and text. Single-modal feature data is key representation information extracted from a single modality type.

[0054] As an optional implementation, the modal types of the target multimodal information are first distinguished and categorized into speech, image, and text. For speech information, key acoustic representations are extracted. For image information, key visual representations are extracted. For text information, key text structure representations are extracted. Each category is extracted independently to form corresponding single-modal feature data. This method has a simple processing flow, the extraction of each modality does not interfere with each other, and the execution speed is fast.

[0055] Step A13: The multimodal feature data is correlated, calibrated, and dimension-unified according to preset feature weights for different modal features, and the multidimensional features are output.

[0056] In this embodiment, preset feature weights are preset values ​​assigned to the importance of different modal features. Association calibration is the operation of adjusting the association relationships between different modal features. Dimensional unification is the process of adjusting the dimensions of different modal features to be consistent.

[0057] As an optional implementation method, this approach first analyzes the intrinsic correlations between multimodal feature data. Based on preset feature weights, the correlation strength between different modal features is dynamically adjusted. When the dimensionality is uniform, a nonlinear mapping method is employed to preserve the nonlinear correlations while maintaining dimensionality consistency among different modal features. Then, a feature fusion algorithm is used to deeply fuse the modal features, generating multidimensional features. This method achieves thorough modal fusion, preserves nonlinear correlations, and exhibits strong feature discrimination capabilities.

[0058] For example, in a video customer service scenario within the securities industry, when a customer initiates a video customer service request, the system acquires multimodal information including their voice (inquiring about "opening a ChiNext stock account"), facial image, and business text (including "risk assessment, permission activation"). During preprocessing, an 8-second silent segment in the voice, a 20% blurred background area in the image, and meaningless characters such as "Hello" in the text are removed to obtain the target multimodal information. Single-modal feature data is extracted according to modality type: 16-dimensional acoustic features are extracted from the voice using MFCC, 256-dimensional visual features are extracted from the image using the VGG model, and 80-dimensional keyword features are extracted from the text using the bag-of-words model. Subsequently, the multimodal feature data is correlated and calibrated according to preset feature weights (voice 0.35, image 0.4, text 0.25), and LDA is used to unify the dimensions to 128, outputting multi-dimensional features.

[0059] By preprocessing to remove invalid information from multimodal information, extracting single-modal features by modality, and generating multidimensional features through weight calibration and dimension unification, the problems of inaccurate identity authentication and incorrect business scenario parsing are solved, thereby improving the accuracy of identity authentication, business efficiency, and customer service experience of securities video customer service.

[0060] Based on any of the above embodiments, in Embodiment 3 of this application, step S20 includes steps B11 to B13: Step B11: Based on the multidimensional features, retrieve the set of customer candidate biometric features corresponding to the multidimensional features from the preset biometric feature library to obtain the customer candidate biometric feature set.

[0061] In this embodiment, the customer candidate biometric set is a summary of the customer's historical biometric features retrieved from the biometric database that are matched and associated with multidimensional features.

[0062] As an optional implementation, the core feature dimensions and weights of each dimension of the multidimensional feature are first deeply analyzed, and historical biometric features of customers containing the core feature dimensions are retrieved from the biometric database. By analyzing the dimensional correlation strength between the multidimensional features and historical features, historical features with correlation strength below a basic threshold are eliminated. Simultaneously, feature data with potential complementary correlations are retained, prioritized according to feature matching fit, and integrated to form a priority-labeled set of customer candidate biometric features. This method yields a candidate set with high accuracy, low redundancy, and high efficiency for subsequent matching.

[0063] Step B12: Calculate the feature similarity between the customer candidate biometric feature set and the multidimensional feature in each single modality dimension, and sum them according to the preset modality weights to obtain the comprehensive similarity value corresponding to each candidate feature, and output the comprehensive similarity value corresponding to each candidate feature.

[0064] In this embodiment, a single modality dimension is the feature dimension corresponding to a single modality. Feature similarity is an indicator that measures the degree of fit between candidate biometric features and multidimensional features in a single modality dimension. Preset modality weights are preset values ​​assigned to the importance of different single modality dimensions. The comprehensive similarity value is the final fit index after weighting and summing the similarities of each single modality feature according to the preset weights.

[0065] As an optional implementation, this method splits the customer candidate biometric feature set into individual modal dimensions of the multidimensional features. For each individual modal dimension, the feature fit is calculated to obtain the feature similarity corresponding to each modal dimension. Then, preset modal weights are retrieved, and the similarity of each individual modal feature is multiplied one-to-one with its corresponding weight. All products are summed to obtain the comprehensive similarity value for each candidate feature, and the comprehensive similarity values ​​are output in order of candidate feature order. This method has intuitive calculation logic, simple steps, fast execution speed, low resource consumption, and is suitable for simple feature matching needs.

[0066] Step B13: Compare the comprehensive similarity value corresponding to each candidate feature with the feature similarity threshold, filter out the customer information with the highest comprehensive similarity value that is higher than the feature similarity threshold, and generate the customer identity authentication result.

[0067] In this embodiment, customer information refers to customer identity-related data associated with candidate biometric features.

[0068] As an optional implementation, the comprehensive similarity values ​​corresponding to all candidate features are collected, and each value is compared with a feature similarity threshold. Candidate features with comprehensive similarity values ​​higher than the threshold, along with their associated customer information, are then identified. The identified candidate features are then sorted from highest to lowest comprehensive similarity value, and the customer information at the top of the sorted list is selected to directly generate the customer authentication result. This method is simple, fast, and suitable for high-concurrency, fast-response scenarios.

[0069] For example, in a video customer service scenario within the securities industry, when a securities client initiates a video customer service inquiry about margin trading, the system, based on previously generated 128-dimensional multi-dimensional features (including voice, face, and text fusion features), retrieves a set of matching candidate biometric features from a pre-set biometric database (storing historical biometric features of 100,000 securities clients). This set includes 10 highly relevant candidate features. A cosine similarity algorithm is used to calculate the similarity between each candidate feature and the multi-dimensional features in a single modality dimension: 0.88 for voice, 0.93 for face, and 0.85 for text. These are then weighted and summed according to pre-set modal weights (voice 0.3, face 0.4, text 0.3) to obtain the comprehensive similarity value for each candidate feature (highest value 0.92, others between 0.75 and 0.83). The comprehensive similarity value is compared to a feature similarity threshold of 0.85. Customer information with a comprehensive similarity value of 0.92 or higher is selected, generating a customer identity authentication result: authentication successful, matching the customer's name and securities account number.

[0070] By retrieving a set of candidate biometric features for customers through multi-dimensional feature association, calculating single-modal similarity and weighted summation, and selecting the highest matching customer information through threshold comparison, the problem of low accuracy and easy impersonation of traditional identity authentication is solved, thus improving the authentication efficiency and security of video customer service.

[0071] Based on any of the above embodiments, in Embodiment 4 of this application, step S30 includes steps C11 to C14: Step C11: Extract key information fields from the multiple modal information to obtain multimodal basic parsing data containing text transcription content, facial micro-expression features, and normalized text information.

[0072] In this embodiment, the key information field refers to the core information units that are valuable for subsequent parsing. The text-to-text content is the result of converting speech information into text. Facial micro-expression features are subtle facial movement representations extracted from image information. The normalized text information is the standardized text content. The multimodal basic parsing data is the comprehensive parsing result formed by integrating the above three types of information.

[0073] As an optional implementation method, this approach first analyzes the contextual relationships of multimodal information. After transcribing the speech information, it corrects typos and grammatical inconsistencies by incorporating the context, resulting in accurate text transcription. Frame sequence analysis is then performed on the image information to extract facial micro-expression features at different time points and label their duration. Semantic correction, keyword extraction, and sentence normalization are then performed on the original text information to obtain structured and normalized text information. Simultaneously, temporal and logical association markers for the three types of information are established and integrated to form multimodal basic parsing data. This method offers accurate information extraction, high completeness, and strong correlation.

[0074] Step C12: Based on the multimodal basic parsing data, the preset sensitive word library is called through the small model in the multi-scale collaborative model to intercept obviously illegal sensitive content, and the sensitive word interception result is obtained.

[0075] In this embodiment, the small model is a submodule responsible for specific sensitive word interception tasks. The preset sensitive word library is a database that pre-stores prohibited sensitive words. Obviously prohibited sensitive content is the violation information that directly hits the sensitive word library. The sensitive word interception result is a determination result that identifies whether a violation exists and the corresponding prohibited content.

[0076] As an optional implementation, this method extracts text transcription, normalized text information, and facial micro-expression features from multimodal basic parsing data. It then calls a small model within a multi-scale collaborative model to first determine the contextual tendency of the text expression based on facial micro-expression features. Next, a pre-defined sensitive word library is loaded to match the text, and contextual verification is performed on the matched sensitive words. This distinguishes between normal usage scenarios and violation scenarios, removes falsely judged content, and marks the truly violating sensitive words and their contextual features, generating sensitive word interception results with contextual verification labels. This method combines multimodal information verification, achieving high interception accuracy and a low false positive rate.

[0077] Step C13: Based on text keywords and facial micro-expression features, preliminarily label the major category to which the business scenario belongs, and output the preliminary business scenario labeling data.

[0078] In this embodiment, text keywords are core semantic words extracted from text information. Facial micro-expression features are subtle facial movements captured from images. The business scenario category is a macro-level classification of the business scenario. The preliminary business scenario labeling data is the scenario category identification result based on the preliminary determination of the two types of features.

[0079] As an optional implementation, text keywords and facial micro-expression features are extracted from multimodal basic parsing data to establish a keyword mapping library for predefined business scenario categories. The extracted text keywords are then directly matched with words in the library. Simultaneously, the basic tendencies of facial micro-expression features are correlated. If the keyword matching degree meets the basic standard and there is no significant conflict in the micro-expression tendencies, the corresponding business scenario category is labeled. All labeling results are integrated to form preliminary business scenario labeling data. This method is simple, fast, and resource-efficient, enabling rapid preliminary labeling of large-scale scenarios.

[0080] Step C14: Based on the sensitive word interception results, the preliminary business scenario annotation data, and the customer identity authentication results, the customer semantic information, the emotion information, and the business scenario information are generated by deep analysis of the multimodal sentiment semantic feature information through the large model in the multi-scale collaborative model.

[0081] In this embodiment, multimodal sentiment semantic feature information refers to the sentiment tendencies and core semantic features contained in multimodal information.

[0082] As an optional implementation method, a mapping relationship is first established between sensitive word interception results, preliminary business scenario annotation data, and customer identity authentication results. Through a large-scale multi-scale collaborative model, implicit associations, semantic levels, and emotional details in multimodal sentiment semantic features are deeply mined. This is then cross-validated using historical service records and preference data corresponding to the customer's identity, dynamically adjusting the parsing weights of semantics, emotion, and scenario to accurately extract core semantic information, refine the emotional state dimension, determine the specific needs of the business scenario, and generate three types of information with parsing evidence and confidence levels. This method offers sufficient parsing depth, high accuracy, and strong anti-interference capabilities, making it suitable for complex and diverse service scenarios.

[0083] For example, in a video customer service scenario within the securities industry, when a client initiates a video customer service request, the system extracts key information fields from the multimodal information. Through ASR speech-to-text transcription, the system obtains the text "Inquiry about the eligibility for IPO subscriptions on the Science and Technology Innovation Board and the requirements for portfolio value." A CNN model extracts 20 facial micro-expression features, such as those around the eyes and mouth. Redundant characters are removed and keywords are extracted from the original text, resulting in a regularized text containing "Science and Technology Innovation Board, IPO eligibility, and portfolio value," forming the multimodal basic parsing data. Based on this multimodal basic parsing data, a lightweight BERT mini-model within the multi-scale collaborative model calls a pre-defined sensitive word library containing securities violation terms such as "insider trading" and "market manipulation." No violations are detected, resulting in a sensitive word interception result of "no violation." Based on the text keywords and "calm" facial micro-expression features, the business scenario is initially categorized as "investment consulting," and preliminary business scenario annotation data is output. Combining the labeled data, sensitive word interception results, and the previously generated customer identity authentication result of "Customer A, account assets of 500,000, risk level R4", the multimodal sentiment semantic feature information is deeply analyzed through the GPT-4 lightweight large model in the multi-scale collaborative model. This generates customer semantic information "consulting on the opening conditions and minimum holding market value requirements for IPOs on the Science and Technology Innovation Board", sentiment information "peaceful", and business scenario information "Science and Technology Innovation Board investment consultation - IPO qualification inquiry scenario".

[0084] By extracting key multimodal information to form basic analytical data, the small model intercepts illegal and sensitive content, combines text and micro-expressions to initially label the major categories of scenarios, and the large model links multiple results to deeply analyze semantics, emotions and business scenarios. This solves the problems of fuzzy semantic analysis, misjudgment of emotions and broad scenario positioning, and improves the accuracy of analysis and service efficiency of securities video customer service.

[0085] Based on any of the above embodiments, in Embodiment 5 of this application, step S40 includes steps D11 to D13: Step D11: Retrieve the customer profile data corresponding to the customer identity authentication result, and align the customer profile data with the three types of information: customer semantic information, emotional information, and business scenario information, to obtain comprehensive customer information.

[0086] In this embodiment, association alignment is the operation of associating and integrating multiple types of information according to identity identifiers and logical relationships. Comprehensive customer information is a complete set of customer data formed after integrating multiple types of associated information.

[0087] As an optional implementation method, this approach retrieves customer profile data corresponding to the customer's identity authentication results, breaks down the core dimensions of the profile, and then analyzes the core elements of customer semantic information, emotional information, and business scenario information. A semantic association rule base is established, and dynamic association between the profile data and these three types of information is achieved through element matching. If information conflicts are found, a conflict coordination mechanism is activated. After alignment, the data is structured according to the logic of identity, needs, emotions, and scenarios, and the association relationships and conflict resolution results are labeled to obtain comprehensive customer information. This method features flexible association logic, high data integration, strong integrity, and is suitable for personalized service scenarios.

[0088] Step D12: Quantitatively evaluate the comprehensive customer information from three dimensions: emotional urgency, business scenario complexity, and customer service preference matching degree. Determine the weight ratio of each dimension and form an evaluation score, and output the multi-dimensional quantitative evaluation result.

[0089] In this embodiment, emotional urgency measures the customer's emotional urgency. Business scenario complexity determines the difficulty and cumbersomeness of the business scenario and process. Customer service preference matching degree is the degree of fit between customer service preferences and potential service types. Dimension weight percentage is the importance percentage of each evaluation dimension in the overall evaluation. Evaluation score is the specific numerical value of each dimension after quantification. The multi-dimensional quantitative evaluation result is a comprehensive evaluation data that integrates the weights and scores of emotional urgency, business scenario complexity, and customer service preference matching degree.

[0090] As an optional implementation method, this approach first deeply analyzes the correlation logic among the various dimensions of comprehensive customer information. Based on business scenario types and customer characteristics, the weighting of the three dimensions is dynamically adjusted, and multi-level, detailed evaluation indicators are set for each dimension. By cross-validating historical customer service data with current information, each indicator is scored individually, and the total dimension score is obtained. Then, a comprehensive evaluation score is calculated according to the dynamic weights, supplementing the evaluation basis and confidence level for each dimension, and outputting a multi-dimensional quantitative evaluation result with confidence level indicators. This method features dynamic weight adaptation, high quantitative accuracy, strong anti-interference capability, and adaptability to complex and personalized service scenarios.

[0091] As an alternative implementation method, this approach deeply analyzes customer characteristics within comprehensive customer information, including risk level, business familiarity, and service preferences. It dynamically adjusts the weights of dimensions such as emotional urgency, business scenario complexity, and customer service preference matching, designing adaptive evaluation indicators for each dimension. Through a process of indicator scoring, feature adaptation verification, and score correction, each dimension's indicators are first scored individually. Then, scores are corrected based on customer characteristics, and a comprehensive evaluation score is calculated using dynamic weights. Adaptation explanations for features, weights, and indicators are supplemented, outputting multi-dimensional quantitative evaluation results with customer characteristic adaptation indicators. This method offers strong personalization, high accuracy, and broad adaptability, providing a personalized basis for subsequent path matching.

[0092] Step D13: Based on the multi-dimensional quantitative evaluation results, combined with the preset business processing rules and real-time service resource configuration, dynamically match the optimal processing path and customize the personalized service strategy.

[0093] In this embodiment, the preset business processing rules are pre-defined service execution specifications adapted to different scenarios. The real-time service resource configuration status refers to the current real-time status of available service personnel, system resources, etc.

[0094] As an optional implementation method, this approach breaks down the core indicators of multi-dimensional quantitative evaluation results and combines them with details such as real-time service resource load, professional matching degree, and response timeliness to dynamically adjust the adaptation priority of preset business processing rules, selecting the optimal processing path that balances efficiency, adaptability, and resource capacity. Furthermore, it links historical service feedback and preference details from comprehensive customer information to customize personalized service strategies that include service content focus, interaction methods, and response pace. Simultaneously, it marks the adaptation basis of the path and strategy, ultimately outputting the results. This method boasts strong dynamic adaptability, high matching accuracy, and significant strategy differentiation.

[0095] As an alternative implementation, the service resource configuration is first monitored in real time. Combining the comprehensive score and indicators from multi-dimensional quantitative evaluation results, a resource load and evaluation result adaptation model is established. Path matching priorities are dynamically adjusted to select the optimal processing path that balances service efficiency, quality, and resource capacity. Then, based on path requirements and comprehensive customer information, a personalized service strategy is customized, including service channels, response times, and content formats. The correlation between resource load status and path matching is simultaneously marked, and the optimal processing path and personalized service strategy are output. This method offers high resource utilization, strong path adaptability, and good service flexibility, making it suitable for high-concurrency service scenarios.

[0096] For example, in a video customer service scenario within the securities industry, when customer B initiates a video customer service request, the system retrieves customer profile data corresponding to their identity authentication result (customer B, securities account number XXX, account assets 800,000, risk level R4) (3 historical service records, preference for video explanations, previous consultation on fund fixed investment, service feedback score of 85). This profile data is then aligned with three categories of information—customer semantic information "urgent consultation on margin trading account liquidation line and margin call method," emotional information "urgency," and business scenario information "margin trading business - liquidation consultation scenario"—based on identity identifiers and business logic to obtain comprehensive customer information. The Analytic Hierarchy Process (AHP) is used to quantify and evaluate this comprehensive customer information across three dimensions: emotional urgency score of 85 (weight 0.35), business scenario complexity score of 78 (weight 0.4), and customer service preference matching score of 90 (weight 0.25). The calculated comprehensive evaluation score is 83.45, and the multi-dimensional quantitative evaluation result is output. Based on the pre-set processing rules of "prioritizing dedicated agents for high-urgency and medium-to-high-complexity businesses" and the real-time service resource allocation (3 dedicated human agents for margin trading, response time ≤ 2 minutes, and video material library including a demonstration video of the liquidation process), the optimal processing path is dynamically matched: "prioritize access to dedicated human agents for margin trading and synchronized video demonstrations," and personalized service strategies are customized: "push example videos of liquidation line calculations, quick access to online channels for margin calls, and real-time Q&A from agents."

[0097] By linking and aligning customer profiles with semantic, emotional, and business scenario information to form comprehensive customer information, core indicators are quantitatively evaluated from multiple dimensions. Combined with business rules and real-time resource dynamic matching paths and customized strategies, the problems of rigid matching of traditional customer service paths, lack of service strategy differentiation, and unreasonable resource allocation are solved, thereby improving the resource utilization efficiency, service adaptation accuracy, and customer experience of securities video customer service.

[0098] Based on any of the above embodiments, in Embodiment Six of this application, step S50 includes steps E11 to E13: Step E11: Deconstruct and extract the core elements required for video generation from the optimal service processing path and the personalized service strategy, and generate a list of core elements for video generation.

[0099] In this embodiment, the core elements required for video generation are the key content and format elements that support the production of personalized service videos. The list of core elements for video generation is a list-style data that integrates all extracted key elements for video generation.

[0100] As an optional implementation method, the core process nodes of the optimal service processing path and the core requirements of the personalized service strategy are first decomposed. Based on fixed categories such as content focus, digital human image, voice style, and presentation duration, basic elements are extracted for each category. These elements include the business content corresponding to the process node, the image type matching customer preferences, the voice tone matching emotions, and the duration range that conforms to path efficiency. These are then directly listed and integrated into a list of core elements for video generation. This method has simple decomposition logic, fast execution speed, low resource consumption, and is suitable for standardized video generation scenarios.

[0101] Step E12: Based on the list of core elements for video generation, match and retrieve digital human models, corresponding topic-based educational content clips, and background materials suitable for the scene from the preset material library to obtain a video generation resource package.

[0102] In this embodiment, the preset resource library is a database that pre-stores resources such as digital human models, investor education content clips, and background materials. Digital human models are virtual avatar resources used for video presentation. Investor education content clips corresponding to the theme are investment education content materials related to the service theme. Background materials adapted to the scene are video background resources that fit the style of the business scenario. The video generation resource package is a collection of video production resources formed by integrating and retrieving various materials.

[0103] As an optional implementation method, this approach first involves in-depth analysis of the detailed requirements and interrelationships of each element in the core elements list for video generation. This determines the style compatibility standards between the digital human image and the educational content, as well as the required level of fit between background materials and the business scenario's atmosphere. Preliminary matching resources are retrieved from a pre-set resource library, and then these resources undergo detailed verification. Resources with insufficient compatibility are eliminated, and alternative materials that meet the detailed requirements are added. These are then integrated to form a video generation resource package with compatibility verification tags. This method offers high material matching accuracy, strong overall coordination, and excellent resource quality, making it suitable for complex, high-quality video generation scenarios.

[0104] As an alternative implementation, the details and adaptation standards of the core elements in the video generation list are first analyzed. Preliminary matching digital human models, educational content clips, and background materials are retrieved from a pre-set material library, triggering an intelligent redundancy filtering mechanism. Based on the priorities of element adaptability, resource lightweighting, and presentation effect, materials with insufficient adaptability, redundancy, or excessive resource consumption are eliminated. The educational content clips are deduplicated and logically sorted, and the style and motion adaptability of the digital human models are verified. Finally, the selected high-quality materials are integrated to generate a lightweight, highly adaptable video generation resource package, and the selection criteria are recorded. This method features a lightweight resource package, high adaptability, and guaranteed generation efficiency, making it suitable for scenarios requiring rapid generation and lightweight deployment.

[0105] Step E13: The video generation resource package is integrated with the materials according to the process requirements of the optimal service processing path, and the digital human voice and lip movements are synchronized, personalized investor education content and customer-related business data are embedded to generate the personalized service video.

[0106] In this embodiment, synchronizing digital human speech and lip movements is a process that ensures that the digital human's speech output and lip movements are consistent.

[0107] As an optional implementation method, the process nodes and timing requirements of the optimal service processing path are first identified. The materials in the video generation resource package are then layered and arranged, and the consistency between the digital human's voice and lip movements is calibrated using precise synchronization technology. Personalized investor education content and customer-related business data are embedded layer by layer according to content logic. Simultaneously, the presentation rhythm and visual effects of the materials are adjusted to ensure harmony and consistency between the digital human's expression, content presentation, and background style. Finally, overall verification and optimization are performed to generate personalized service videos. This method is meticulously integrated, precisely synchronized, and naturally embedded, perfectly meeting the service path and strategy requirements.

[0108] As an alternative implementation, the materials in the video generation resource package are integrated according to the process requirements of the optimal service processing path. A real-time compliance verification module is embedded at each stage, including digital human voice and lip-syncing, personalized investor education content embedding, and customer-related business data integration. If a violation is detected, generation is paused in real time, and material replacement or content correction is initiated. After correction, the material integration, synchronization, and embedding operations continue, generating a personalized service video that combines compliance and personalization, and recording the compliance verification nodes. This method offers strong video compliance, high fault tolerance, and eliminates the need for subsequent rework, making it suitable for scenarios with high compliance standards.

[0109] For example, in a video customer service scenario within the securities industry, when a client C initiates a video customer service request for "ETF option account opening consultation," the system first breaks down the core elements of video generation from the optimal service processing path ("process explanation, qualification verification, operation guidance") and the personalized service strategy ("professional digital human, easy-to-understand explanation, real-time business data embedding"), generating a list of core elements for video generation: digital human type (financial professional), voice style (clear and calm, speech rate 120 words / minute), investor education content theme (ETF option account opening conditions, process steps), background material (simple financial chart background), embedded data (client C's account assets 600,000, risk level R5), and video duration (3 minutes). Based on this list, the system matches and retrieves the "Financial Advisor Virtual Human B" digital human model, 15 investor education content segments (each 8-12 seconds) of "ETF Option Account Opening Full Process," and "blue-themed financial data visualization" background material from a pre-set resource library (containing more than 20 digital human models, more than 500 securities investor education segments, and more than 30 background materials), resulting in a video generation resource package. Following the optimal service processing path, the Unity engine was used to integrate materials, and Wwise audio synchronization tool was used to achieve precise synchronization of digital human voice and lip movements (synchronization error ≤ 0.1 seconds). Personalized business data such as customer C account assets and risk level, as well as popular investment education content on "5 conditions for opening an options account", were embedded to generate a personalized service video with a resolution of 1080P and a duration of 2 minutes and 58 seconds.

[0110] By breaking down the core elements of video generation, accurately retrieving resources from the material library, integrating and synchronizing them according to the service path, and embedding personalized data, the problems of traditional investor education videos being difficult for customers to understand and having weak service targeting have been solved, thereby improving the accuracy of video services and the efficiency of customer reception.

[0111] Based on any of the above embodiments, in Embodiment Seven of this application, referring to Figure 3 , Figure 3 This is a flowchart illustrating the seventh embodiment of the video customer service control method of this application. Following step S50, steps F11-F13 are also included: Step F11: Verify the core attributes of the video content in the personalized service video item by item, mark the location and type of the illegal content, and generate preliminary compliance verification results.

[0112] In this embodiment, the core attributes of the video content are the key features that affect compliance within the video. The location of the violation content refers to the time point or segment range within the video where the violation information is present. The type of violation content refers to the specific category of the violation information. The preliminary compliance verification result is preliminary compliance assessment data formed after labeling relevant violation information.

[0113] As an optional implementation method, this approach breaks down the core attribute dimensions of video content and refines the verification indicators. It then checks video frames, audio content, and embedded data frame by frame along the timeline. Combined with preset compliance standards, it deeply verifies core attributes and performs multi-dimensional cross-validation on suspected violations. This determines the precise time segment and specific type of the violation, while recording the verification basis and confidence level, generating preliminary compliance verification results with confidence level indicators. This method offers detailed and in-depth verification dimensions and high accuracy in violation identification.

[0114] As an alternative implementation, a multi-model compliance verification system is first constructed. The core attributes of the video content are broken down item by item and assigned to corresponding models for specific verification. The verification results of each model are then summarized. If a single model determines a violation but other models do not, cross-validation is initiated to determine the location, type, and verification basis of the violation, generating preliminary compliance verification results with multi-model verification indicators. This method boasts high accuracy, low false negative rate, and strong reliability, significantly reducing compliance risks.

[0115] Step F12: Based on the preliminary compliance verification results, update the marked violations according to the preset compliance standards. After verifying that there are no violations, generate a formal verification report with detailed compliance points.

[0116] In this embodiment, the preset compliance standard refers to the pre-defined compliance judgment criteria and correction specifications. Updating the marked violations involves correcting the initially marked violations according to the compliance standard. The formal verification report is the final compliance certificate that details the compliance points after confirming that there are no violations.

[0117] As an optional implementation method, the violation labeling information in the preliminary compliance verification results is retrieved, and the type and location labeling of each violation are checked against the preset compliance standards one by one. Violation information with judgment errors or labeling deviations is corrected according to the standards. After a comprehensive review of the video to confirm that no violations have been omitted, the core points of video compliance are summarized and integrated into a formal verification report in a fixed format. This method is simple in process, fast in execution, and low in resource consumption.

[0118] As an alternative implementation method, based on the preliminary compliance verification results, and in accordance with the detailed clauses of the preset compliance standards, the marked violations are corrected and confirmed. Simultaneously, the verification process for each compliance point is traced, and the core dimensions of video compliance are identified. The report is presented in a logically structured manner, following compliance points, verification process, applicable standards, and conclusions. This supplements the compliance control nodes throughout the entire video generation process, generating a formal verification report with traceability capabilities. This allows compliance auditors to trace the formation process of each compliance conclusion. This method offers high report detail, strong traceability, and comprehensive compliance assurance, making it suitable for industry scenarios with high compliance requirements.

[0119] Step F13: Push the personalized service video that has passed the compliance verification to the designated customer according to the optimal service processing path, and simultaneously record the video push time, terminal delivery status and the trigger identifier of the customer clicking to view, and output the video push execution result.

[0120] In this embodiment, the personalized service video that passes compliance verification is a dedicated service video that has been confirmed by compliance review to be free of violations and suitable for customer needs. The terminal delivery status is the feedback result of whether the video has been successfully delivered to the customer's terminal. The trigger identifier for customer click to view is a marker recording whether the customer clicked to watch the video. The video push execution result is execution feedback data formed by integrating push-related records.

[0121] As an optional implementation, this method analyzes the optimal service processing path's push rules and client terminal types, selects suitable multi-channel synchronous push notifications, and pushes compliant, verified personalized service videos to designated customers. It records the initial push time and detailed delivery status of each channel in real time, continuously tracks the customer's complete click-to-view behavior and associates it with trigger identifiers, integrates multi-channel push data, full-process status feedback, and complete trigger identifiers, supplements preliminary explanations of push anomalies, and outputs video push execution results with detailed tracking records. This method boasts strong channel adaptability, comprehensive information recording, and strong traceability.

[0122] For example, in a video customer service scenario within the securities industry, a personalized service video (2 minutes 30 seconds long, 1080P resolution) generated by client D for "convertible bond conversion rule consultation" is checked item by item using a video content compliance detection model. The video content compliance model identifies a violation at 1 minute 12 seconds: "deviation in conversion price description" (type: data error), generating a preliminary compliance verification result. Based on this result, the annotation is updated against the preset standards of the "xxx Content Compliance Specification." After three rounds of cross-verification confirming no violations, a formal verification report is generated, including five compliance points such as "data source is official exchange rules, description conforms to regulatory language, and investor education content is not misleading." Following the optimal service processing path of "real-time push from video customer service terminal and assisted delivery via APP message", the compliant video is pushed to the designated customer D. The video push time (14:35:20), terminal delivery status (successful customer service terminal, successful APP), and customer click to view trigger indicator (1. Clicked, click time 14:36:10) are recorded simultaneously through the message push system. The video push execution result containing the full process record is output.

[0123] By accurately verifying each video's core attributes to identify violations, correcting and verifying according to preset compliance standards to generate detailed reports, and pushing content along the optimal path while recording the entire process status, the system solves the problems of high compliance omission risk, lack of compliance traceability, and difficulty in tracking push effects in traditional video services, thereby improving the accuracy of service push and the efficiency of compliance management.

[0124] Based on any of the above embodiments, in Embodiment Eight of this application, referring to Figure 4 , Figure 4 This is a flowchart illustrating the eighth embodiment of the video customer service control method of this application. Following step F13, steps G11-G13 are also included: Step G11: Monitor the video push execution results corresponding to the customer terminal, and synchronously record the real-time feedback content submitted by the customer through the terminal feedback entry to obtain customer video viewing behavior data and real-time feedback information.

[0125] In this embodiment, the client terminal is the device carrier through which the customer receives personalized service videos. The video push execution result is execution feedback data that integrates the push time, terminal delivery status, and customer click markers. The real-time feedback content consists of opinions, evaluations, or questions submitted by the customer through a preset feedback entry on the terminal. The customer video viewing behavior data records information related to the customer's behavior during the video viewing process. The real-time feedback information is a structured information set formed by integrating the real-time feedback content submitted by the customer.

[0126] As an optional implementation method, this approach monitors the entire process of video push execution results on client terminals, meticulously recording detailed customer video viewing behaviors and capturing real-time feedback content in various forms, such as text and ratings, submitted through the terminal feedback portal. The feedback content is categorized and organized, and core requests are extracted. A correlation mapping is established between behavioral data and feedback information, labeling feedback types and importance, and integrating this data to form structured, high-value customer video viewing behavior data and real-time feedback information. This method offers comprehensive behavior recording, in-depth feedback mining, and high data value.

[0127] Step G12: Based on the customer video viewing behavior data and the real-time feedback information, retrieve the customer profile data and the current business scenario information to generate integrated full-volume customer service interaction data.

[0128] In this embodiment, the current business scenario information refers to the specific business type and demand scenario currently belonging to the customer. The full customer service interaction data is a complete service interaction data set formed by integrating the above-mentioned multiple types of information.

[0129] As an optional implementation method, this approach analyzes the core elements of customer video viewing behavior data and real-time feedback information to extract customer needs and pain points. It retrieves customer profile data and current business scenario information based on identity identifiers, breaks down the core dimensions of various information types, and establishes implicit association rules. Through logical verification, it achieves deep multi-dimensional fusion, performs structured classification of the fused data, supplements association criteria and priority identifiers, and ultimately generates a structured full dataset of customer service interactions. This method features deep information fusion, strong data integrity, and high value density, making it suitable for complex and refined analysis scenarios.

[0130] As an alternative implementation, customer video viewing behavior data and real-time feedback information are used to retrieve customer profile data and information related to the current business scenario. First, all types of data are hashed and encrypted, and the data generation and flow nodes are recorded in timestamp order. The encrypted data is then uploaded to a blockchain-based evidence storage system. Next, the data is structured and integrated according to identity identification, the entire service process, and feedback data, synchronously linked to the blockchain evidence storage number, and the association rules and processing steps during the data integration process are recorded, generating a complete set of traceable customer service interaction data with blockchain evidence storage identification. This method offers high data security, strong credibility, and good traceability, protecting the data rights of customers and institutions, and is suitable for industry scenarios with high compliance and security requirements.

[0131] Step G13: Extract key indicators reflecting service adaptability, customer pain points, and video content effectiveness from the full amount of customer service interaction data, and generate service interaction feedback data and strategy optimization reference information.

[0132] In this embodiment, service adaptability refers to the degree to which the service solution matches the customer's needs. Customer pain points are the core demands of the customer that are not met during the service process. Service interaction feedback data is a set of service effect feedback formed by integrating key indicators. Strategy optimization reference information is targeted reference data that supports the iteration of service strategies and video content.

[0133] As an optional implementation method, this approach breaks down service adaptability, customer pain points, and video content effectiveness into detailed dimensions. A multi-level key indicator system is established, extracting indicator data from the full volume of customer service interaction data and analyzing implicit relationships between data points. This uncovers potential customer needs and shortcomings in video content, structurally classifies and prioritizes feedback data, and combines customer profiles and business scenario characteristics to extract targeted strategy optimization directions. This generates structured service interaction feedback data and precise strategy optimization reference information. This method offers comprehensive indicator dimensions, in-depth data mining, and highly accurate reference information.

[0134] As an alternative implementation method, first define the service optimization closed-loop goal, and then break it down into a detailed indicator system based on service adaptability, customer pain points, and video content effectiveness. Intelligently extract indicator data from the full volume of customer service interaction data, analyze the correlation between indicators through data mining, extract targeted service interaction feedback data, and combine this with the optimization goals to generate actionable strategy optimization reference information. Simultaneously, label the correlation between indicators and optimization directions to obtain service interaction feedback data and strategy optimization reference information. This method features highly targeted indicators, practical optimization references, and high data value, improving optimization efficiency and effectiveness, and is suitable for continuously iterating service scenarios.

[0135] For example, in a video customer service scenario within the securities industry, a customer anxiously inquires about the sharp drop in the price of a stock they hold. The system first completes rapid identity verification through voiceprint recognition. Subsequently, the multimodal perception engine works simultaneously: speech recognition converts the dialogue into text, and the sentiment analysis module identifies the customer's anxiety through speech rate analysis. Meanwhile, the semantic understanding module determines that the question involves the complex scenario of "individual stock analysis." Based on this, the intelligent decision engine quickly makes the following judgments: First, this is a professional question and needs to be transferred to a human agent. Second, the customer is anxious and needs to be prioritized and reassured. Therefore, the system immediately transfers the call to a senior investment advisor and displays an "anxious expression" on the agent's screen in real time, along with recommended scripts: "Mr. Wang, I understand your feelings. Let's first look at the fundamentals of this stock together..." Simultaneously, the system automatically pushes the latest research report and fund flow data for the stock for the agent's reference. Throughout the conversation, the compliance quality inspection module monitors in real time to ensure that the agent's interpretation does not contain illegal content such as promises of returns.

[0136] By monitoring customer viewing behavior and immediate feedback in real time, integrating customer profiles and business scenario information, and extracting core indicators to generate feedback and optimization references, the system solves the problems of fragmented service feedback collection, insufficient exploration of pain points, and lack of accurate data support for strategy optimization in traditional services. This improves the completeness of service feedback and the accuracy of strategy optimization, and helps the securities video customer service system to continuously iterate and upgrade.

[0137] This application provides a video customer service device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the video customer service control method in Embodiment 1 above.

[0138] The following is for reference. Figure 5The diagram illustrates a structural schematic of a video customer service device suitable for implementing embodiments of this application. The video customer service device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, multi-dimensional authentication terminals, personal digital assistants (PDAs), tablet computers (PADs), portable media players (PMPs), and smart interactive terminals, as well as fixed terminals such as virtual human interactive all-in-one machines and desktop computers. Figure 5 The video customer service device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0139] like Figure 5 As shown, the video customer service device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the video customer service device. The processing unit 1001, the ROM 1002, and the RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the video client equipment to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show video client equipment with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0140] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0141] The video customer service device provided in this application, employing the video customer service control method described in the above embodiments, can solve the technical problem of poor service efficiency. Compared with the prior art, the beneficial effects of the video customer service device provided in this application are the same as those of the video customer service control method described in the above embodiments, and other technical features of the video customer service device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0142] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0143] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0144] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the video customer service control method described in the above embodiments.

[0145] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0146] The aforementioned computer-readable storage medium may be included in the video customer service device; or it may exist independently and not be assembled into the video customer service device.

[0147] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the video customer service device, the video customer service device performs the following actions: After acquiring the customer's multimodal information, it extracts multidimensional features from the multimodal information; based on the multidimensional features and feature similarity thresholds, it filters and matches from a biometric database to obtain a customer identity authentication result; it fuses and processes the multimodal information and the customer identity authentication result using a multi-scale collaborative model to determine customer semantic information, emotional information, and business scenario information; it retrieves customer profile data corresponding to the customer identity authentication result, and processes the customer semantic information, emotional information, and business scenario information through multidimensional fusion decision processing to generate an optimal service processing path and a personalized service strategy; and it parses and generates a personalized service video based on the optimal service processing path and the personalized service strategy.

[0148] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0149] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0150] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0151] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described video customer service control method, thereby solving the technical problem of poor service efficiency. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the video customer service control method provided in the above embodiments, and will not be repeated here.

[0152] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A control method of video customer service, characterized by, The method comprises: After obtaining the multi-modal information of the customer, the multi-dimensional features of the multi-modal information are extracted; Based on the multi-dimensional features and the feature similarity threshold, the matching is screened from the biological feature library to obtain the customer identity authentication result; The multi-modal information and the customer identity authentication result are processed by a multi-scale collaborative model fusion analysis to determine the customer semantic information, emotional information and business scenario information; The customer portrait data corresponding to the customer identity authentication result is called, and the customer semantic information, the emotional information and the business scenario information are processed by multi-dimensional fusion decision to generate the optimal service processing path and the individualized service strategy; According to the optimal service processing path and the individualized service strategy, the individualized service video is generated.

2. The control method of video customer service according to claim 1, wherein After obtaining the multi-modal information of the customer, the multi-dimensional features of the multi-modal information are extracted, which comprises: The multi-modal information is preprocessed to eliminate invalid information in the multi-modal information to obtain target multi-modal information; According to the modal type corresponding to the target multi-modal information, the single modal feature data in the target multi-modal information is extracted; The multi-modal feature data is associated, calibrated and dimensioned according to the preset feature weight of different modal features to output the multi-dimensional features.

3. The control method of video customer service according to claim 1, wherein Based on the multi-dimensional features, the customer candidate biological feature set corresponding to the multi-dimensional features is called from the preset biological feature library to obtain the customer candidate biological feature set; The feature similarity of the customer candidate biological feature set and the multi-dimensional features in each single modal dimension is calculated respectively, and the comprehensive similarity value corresponding to each candidate feature is obtained by weighted summation according to the preset modal weight to output the comprehensive similarity value corresponding to each candidate feature; The comprehensive similarity value corresponding to each candidate feature is compared with the feature similarity threshold to screen out the customer information with the highest comprehensive similarity value higher than the feature similarity threshold to generate the customer identity authentication result. The multi-modal information and the customer identity authentication result are processed by a multi-scale collaborative model fusion analysis to determine the customer semantic information, emotional information and business scenario information, which comprises:

4. The control method of video customer service according to claim 1, wherein Extract the key information field in the multi-modal information to obtain multi-modal basic analysis data containing text transcription content, facial micro-expression features and regularized text information; Based on the multi-modal basic analysis data, the preset sensitive word library is called by the small model in the multi-scale collaborative model to intercept obvious rule-sensitive content to obtain a sensitive word interception result; According to the text keyword and the facial micro-expression feature, the business scenario is preliminarily labeled to output preliminary business scenario labeling data; Based on the sensitive word interception result, the preliminary business scenario labeling data and the customer identity authentication result, the multi-modal emotional semantic feature information is deeply analyzed by the large model in the multi-scale collaborative model to generate the customer semantic information, the emotional information and the business scenario information. ​ 5. The control method of video customer service according to claim 1, wherein The step of calling the customer portrait data corresponding to the customer identity authentication result, processing the customer semantic information, the emotional information and the business scene information through multi-dimensional fusion decision, generating the optimal service processing path and the personalized service strategy comprises: Call the customer portrait data corresponding to the customer identity authentication result, and align the customer portrait data with the customer semantic information, the emotional information and the business scene information to obtain customer comprehensive information; Quantitative evaluation of the customer comprehensive information from the emotional emergency degree, the business scene complexity and the customer service preference matching degree, determine the weight proportion of each dimension and form the evaluation score, output the multi-dimensional quantitative evaluation result; According to the multi-dimensional quantitative evaluation result, combining the preset business processing rule and the real-time service resource configuration, dynamically matching the optimal processing path, and customizing the personalized service strategy.

6. The control method of video customer service according to claim 1, wherein The step of generating a personalized service video according to the optimal service processing path and the personalized service strategy comprises: Disassemble and extract the core elements required for video generation in the optimal service processing path and the personalized service strategy, generate a video generation core element list; Based on the video generation core element list, match and call the digital human model, the corresponding theme teaching content segment and the adaptive scene background material consistent with the video generation core element list from the preset material material library to obtain a video generation resource package; Integrate the video generation resource package according to the process requirements of the optimal service processing path, synchronize the digital human voice and lip movement, embed personalized teaching content and customer associated business data, and generate the personalized service video.

7. The control method of video customer service according to claim 1, wherein After the step of generating a personalized service video according to the optimal service processing path and the personalized service strategy, the control method of the video customer service further comprises: Check the core attributes of the video content in the personalized service video item by item, mark the location and type of the illegal content, and generate a preliminary compliance verification result; Based on the preliminary compliance verification result, update the marked illegal content according to the preset compliance standard, and generate a formal verification report detailing the compliance points after verification and confirmation of no illegal content. The personalized service video that passes the compliance verification is pushed to the specified customer according to the optimal service processing path, and the video push time, terminal delivery state and customer click view trigger identifier are recorded synchronously, and the video push execution result is output.

8. The control method of video customer service according to claim 7, wherein After the step of pushing the personalized service video that passes the compliance verification to the specified customer according to the optimal service processing path, synchronously recording the video push time, terminal delivery state and customer click view trigger identifier, and outputting the video push execution result, the control method of the video customer service further comprises: Monitor the video push execution result of the customer terminal, synchronously record the instant feedback content submitted by the customer through the terminal feedback entrance, obtain the customer video viewing behavior data and instant feedback information; Based on the customer video watching behavior data and the instant feedback information, the customer portrait data and the current service scene information are called to generate integrated customer service interaction full data; Key indicators reflecting service adaptation, customer demand pain points and video content effectiveness in the customer service interaction full data are extracted to generate service interaction feedback data and strategy optimization reference information.

9. A video customer service device, characterized by The video customer service device comprises a memory, a processor and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the control method of the video customer service according to any one of claims 1 to 8.

10. A storage medium, characterized by The storage medium is a computer readable storage medium, and the storage medium stores a computer program. When the computer program is executed by the processor, the steps of the control method of the video customer service according to any one of claims 1 to 8 are implemented.

Images