Collection digital human interaction system based on multi-modal large model
Through a collection digital human interaction system based on multimodal large model, integrating voice, text, images and video data, the customer dissatisfaction caused by low collection efficiency and system machine characteristics in the existing technology is solved, and a more natural and efficient Interactive experience improves collection efficiency and personalized services.
Patent Information
- Application Number
- CN202411187454.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-12-03
- Estimated Expiration
- Not applicable · inactive patent
Smart Images

Figure CN119067768A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of information processing technology, and in particular to a debt collection digital human interaction system based on a multimodal large model. Background Art
[0002] With the development and popularization of artificial intelligence, intelligent robots such as virtual digital humans are widely used to replace manual customer service in areas including customer service, outbound calls, and voice interaction. Although this can reduce costs and improve work efficiency, the field of intelligent collection interaction has defects such as complex original information, strong customer personalization, and if the system exhibits strong machine characteristics, it may cause customers to be irritated or even directly interrupt the interaction. The existing collection digital human interaction system has insufficient information processing capabilities and can only process information of one mode. It makes outbound calls to overdue customers through pre-set processes and fixed interaction methods. It performs poorly when processing multi-modal information or requires comprehensive information to make decisions, or requires human participation, and cannot achieve automated collection.
[0003] The multimodal big model is a machine learning technology based on deep learning. Its core idea is to fuse different media data (such as text, images, audio and video, etc.) and achieve more intelligent information processing by learning the associations between different modalities. The multimodal big model can make full use of the information of different media data to extract richer and more comprehensive features, thereby improving the performance and generalization ability of the model. In the existing technology, multimodal functions are mostly achieved by combining and training a single model in a collaborative manner. A single model is trained on multiple classification tasks at the same time. Each model needs to be trained and inferred, while the combined model needs to fuse or integrate the outputs of multiple models. This leads to an increase in computing resources and a decrease in inference speed.
[0004] The above-mentioned record of background technical knowledge is intended to help ordinary technicians in this field understand the existing technologies that are relatively close to the present invention, and at the same time facilitate the understanding of the inventive concept and technical solution of the present invention. It should be clear that in the absence of clear evidence that the above-mentioned content has been disclosed before the filing date of this patent application, the above-mentioned background technology should not be used to evaluate the novelty of the technical solution of this application. Summary of the invention
[0005] Technical issues In order to solve the above problems, the purpose of the present invention is to provide a collection digital human interaction system based on a multimodal large model, which provides users with a more natural and rich interactive experience by integrating multiple interaction methods such as voice, text, images and videos. The system can enhance the communication and understanding of information, process and display information in multiple modes, and solve the problem of low collection repayment rate of existing collection virtual digital humans.
[0006] Technical Solution In order to achieve the above objectives, the present application provides a debt collection digital human interaction system based on a multimodal large model, including: The data collection and input module is used to collect voice data, image data and text data related to overdue customers, and input the above data and preset overdue collection reminder words into the multimodal big model; The multimodal large model module is used to aggregate the input information of the data collection and input module and the interactive information of the interactive module, and output the processing results to the collection digital human module and the interactive module; The debt collection digital human module is used to interact with the outside world through the interactive module based on the processing results provided by the multimodal large model module; The interactive module is used to send letters, messages, and make voice calls, and can also conduct interactive Q&A with users; Among them, the multimodal large model module includes a speech recognition submodule, an image recognition submodule, a natural language processing submodule and a machine learning submodule. The speech recognition submodule, the image recognition submodule and the natural language processing submodule respectively recognize, analyze and extract speech, image and text information to obtain corresponding information, which is input into the machine learning submodule for machine learning and outputs the processing results. The multimodal large model aggregates the speech data, image data and text data related to overdue customers, and analyzes, refines and integrates the data information related to debt collection through the speech recognition submodule, the image recognition submodule and the natural language processing submodule respectively, and combines the preset overdue debt collection prompt words, and learns the correlation between the contents derived from different modalities through the machine learning submodule to achieve more intelligent information processing. The multimodal large model can make full use of the information of different media data to extract richer and more comprehensive features, thereby improving the performance and generalization ability of the model.
[0007] The debt collection digital human interaction system provided by the present invention is mainly based on a multimodal large model. The system model results are simplified, the redundancy and complexity between models are reduced, the difficulty of model development and maintenance is reduced, and the ease of use and scalability of the model are improved; the multimodal virtual digital human supports multiple interaction methods, including voice, text, image, and video, which improves the naturalness and convenience of the interaction. At the same time, multimodal interaction also helps to better understand and meet user needs and provide personalized services.
[0008] Furthermore, the voice data, text data and image data related to overdue customers include: overdue customer information, historical repayment record data, and historical communication behavior data of overdue customers.
[0009] Furthermore, the data collection and input module also includes inputting legal documents into the multimodal large model module to fill the underlying legal framework.
[0010] Furthermore, the legal documents include current legal provisions, cases, judgments, rulings, etc.
[0011] Furthermore, the processing results in the multimodal large model module include text information and / or voice information.
[0012] Furthermore, the speech recognition submodule analyzes and recognizes the overdue customer call recordings, converts them into text to obtain initial speech text, and inputs it into the machine learning submodule as input data.
[0013] Furthermore, the image recognition submodule identifies and extracts relevant information about overdue customers in the video and / or image, converts it into text to obtain initial image text, and inputs it into the machine learning submodule as input data.
[0014] Furthermore, the natural language processing submodule obtains initial text by identifying, analyzing and extracting text information from the communication records between the system and the overdue customer and / or persons involved in the case related to the overdue customer, and inputs it into the machine learning submodule as input data.
[0015] Furthermore, the machine learning submodule integrates, identifies, and analyzes the initial voice text, initial image text, and initial text, matches the reply nodes in the multimodal large model corpus to generate processing results, and enables the multimodal large model to automatically learn, train, and optimize, and transmits the processing results to the collection digital human module and the interaction module.
[0016] Furthermore, the specific steps of the machine learning submodule fusing, identifying and analyzing the initial voice text, the initial image text and the initial text include: S1, respectively extracting features from the initial voice text, the initial image text and the initial text text to obtain initial voice features, initial image features and initial text features; S2, applying a root mean square normalization layer to normalize the initial text features to obtain normalized text features; S3, applying a self-attention mechanism layer to calculate the weight of each position in the normalized text feature and combining the normalized text feature to perform weighted addition to obtain a weighted text feature; S4, adding the weighted text feature to the initial text feature to obtain a text addition result; S5, using a cross attention layer to learn the correlation matrix between the initial speech features and the normalized feature information output after the text addition result is normalized by the root mean square normalization layer, and performing a weighted summation on each position in the initial speech features and all positions of the normalized feature information to obtain a first associated information feature; S6, using a cross attention layer to learn a correlation matrix between the initial image features and the normalized feature information output after the text addition result is normalized by the root mean square normalization layer, and performing a weighted summation on each position in the initial image features and all positions of the normalized feature information to obtain a second association information feature; S7: Add the first associated information feature and the second associated information feature to obtain a multimodal associated information feature, thereby completing feature fusion.
[0017] The present invention utilizes a machine learning submodule to fuse the multimodal information of the initial voice text, the initial image text, and the initial text text. Aiming at complex text environments, the self-attention mechanism layer is used to better handle the problem of long-sequence texts and long-distance dependencies in complex contexts, thereby improving the model's ability to process long-sequence texts. At the same time, the cross-attention mechanism is applied to realize the information fusion interaction between modal data of text, voice, and images, thereby realizing deep integration of multiple fields such as computer vision, natural language processing, speech recognition and synthesis, and further innovating continuously, effectively combining multiple interaction methods such as voice, text, image, and video to provide highly realistic virtual characters and a natural and rich interactive experience.
[0018] Furthermore, the machine learning submodule performs machine learning and outputs processing results, and further includes: T1. The interactive module receives the reply information, voice information and / or interactive information of overdue customers, generates interactive text information, performs word segmentation operation to obtain keyword extraction results, performs vectorization to obtain word vectors, and fills the corresponding keywords, parts of speech and word vectors into different categories of the multimodal large model corpus; T2. Merge the word vector with the positional encoding to obtain the word vector matrix, and multiply it with the weight matrix to obtain the query matrix, key matrix, and value matrix respectively: T3. Optimize the Transformer model to focus on the emotional tendencies of overdue customers:
[0019] in, denote the query matrix, key matrix, and value matrix respectively; represents the dimension of the key vector; Represents relative position encoding; the Transformer model improves its ability to understand context through a multi-head attention mechanism, and retrieves matching reply nodes and / or keywords from the multimodal large model corpus for interactive replies. By introducing the relative position encoding of words , can effectively enhance the system's ability to perceive the order of words, and verify the unexpected relative position encoding The introduction of can make the system pay attention to the probability of occurrence of high-frequency words, so that when the overdue customer shows good intentions, bad intentions or other tendencies, it can retrieve historical reply content or keywords from the corpus to generate reply information that matches their emotions. It does not exceed the legal and regulatory level, and is more deterrent and normative at the language expression level. It can even adjust the details of the debt and repayment requirements according to the specific circumstances of the overdue customer, so as to achieve efficient collection to the greatest extent.
[0020] Furthermore, the interaction module is equipped with a collection submodule and a transmission submodule. The collection submodule takes the audio and video images of the overdue customers during the interaction with the overdue customers, and transmits the audio and video images to the multimodal large model module through the transmission submodule for identification and analysis, thereby identifying the current emotion value of the overdue customers, comparing the current emotion value with the historical emotion value, predicting their willingness to repay based on the change in the emotion value, and then adaptively changing the wording tendency of the text information and / or voice information output by the multimodal large model module.
[0021] Furthermore, the specific steps of transmitting the audio and video images to the multimodal large model module through the transmission submodule for identification and analysis, and then identifying the current emotion value of the overdue customer include: Performing part-of-speech analysis on keywords based on the multimodal large model corpus, and determining text sentiment values of overdue customers based on the analysis results; Identify and analyze the tone information, volume information, and speech speed information of the overdue customer in the audio and video images, and determine the voice emotion value of the overdue customer based on the analysis results; Identify and analyze facial and / or body behavior feature information of overdue customers in the audio and video images, and determine the behavior emotion value of the overdue customers based on the analysis results; The current emotional value of the overdue customer is determined according to the text emotional value, the sound emotional value and the behavior emotional value. In the process of interacting with overdue customers through emails, voice, video and other means, the multimodal large model can analyze and identify the current emotional state of the overdue customer, including anger, anxiety, resistance, appointment, sincerity, sincerity, etc., and compare and analyze the current emotion with the historical emotion, comprehensively predict their willingness to repay, and then adaptively change the wording tendency of the text information and / or voice information output by the multimodal large model module, and adaptively perform more humanized interactive operations such as topic switching, semantic transition, interruption, recovery, and severity during the interaction process, so as not to exceed the legal and regulatory level, and be more kind, humane or deterrent and standardized at the language expression level, and even adjust the details of the arrears and repayment requirements according to the specific circumstances of the customer, so as to achieve efficient collection to the greatest extent.
[0022] The present invention provides a post-loan collection digital human interaction system based on a multimodal large model, which needs to access overdue customer information, historical repayment record data, and historical communication behavior data of overdue customers (including text, letter images, audio and video files), plus industry overdue collection reminder words and legal documents, train and optimize the multimodal large model, generate personalized collection virtual digital human reply content through the multimodal large model, and perform automatic outbound calls and personalized information sending to the corresponding overdue customers. The multimodal large model can analyze and identify the current emotional state of overdue customers, including anger, anxiety, resistance, appointment, sincerity, sincerity, etc., and compare and analyze the current emotions with historical emotions, comprehensively predict their repayment willingness, and can adaptively perform more humanized interactive operations such as topic conversion, semantic transition, interruption, recovery, and severity in the process of interaction with users. It is more kind, humane or deterrent and standardized at the language expression level, and can even adjust the details of the arrears and repayment requirements according to the specific circumstances of the customer, so as to achieve efficient collection to the greatest extent. The realization of multimodal virtual digital humans involves technologies in multiple fields such as computer vision, natural language processing, speech recognition and synthesis, and emotional computing. These technologies are deeply integrated into a multimodal large model, and are constantly innovated to achieve highly realistic virtual characters and natural interactive experiences. Based on the above system, the automatic upgrade and automated outbound call contact of virtual digital humans for debt collection can be realized, reducing the consumption of manpower and material resources, and improving the accuracy of contact with overdue customers, so as to solve the problem of low efficiency of existing debt collection methods.
[0023] A post-loan collection digital human interaction method based on a multimodal large model, the method is based on the above-mentioned system and performs the following steps: Users interact with the debt collection digital person through the interactive module; The multimodal large model module generates a processing result based on the interactive content and delivers it to the collection digital human module; The next step of interaction is achieved with the user through the interaction module.
[0024] A computer device, comprising a memory, a processor, a communication interface and a communication bus; wherein the memory, the processor and the communication interface communicate with each other via the communication bus; the memory is used to store computer programs; the processor is used to execute the computer programs stored in the memory, and when the processor executes the computer program, it implements at least one step of the aforementioned post-loan collection digital human interaction method based on a multimodal large model.
[0025] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements at least one step of the above-mentioned post-loan collection digital human interaction method based on a multimodal large model.
[0026] Based on the common knowledge in this field, the above-mentioned preferred conditions can be combined with each other to obtain a specific implementation method.
[0027] Beneficial Effects According to the present invention, by accessing overdue customer information, historical repayment record data, and historical communication behavior data of overdue customers, the multimodal big model is trained and optimized, and personalized collection virtual digital human reply content is generated through the multimodal big model, and automatic outbound calls and personalized information sending are performed to the corresponding overdue customers.
[0028] The debt collection digital human interaction system based on the multimodal large model of the present invention involves technologies in multiple fields such as computer vision, natural language processing, speech recognition and synthesis, and emotional computing. These technologies are deeply integrated into a multimodal large model and continuously innovated to achieve highly realistic virtual characters and natural and rich interactive experiences. The multimodal large model can make full use of information from different media data and extract richer and more comprehensive features, thereby improving the performance and generalization ability of the model.
[0029] The multimodal large model introduces the relative position encoding of words in the machine learning process , so that the system pays attention to the probability of occurrence of high-frequency words, so that when an overdue customer shows good intentions, bad intentions or other tendencies, it can retrieve historical reply content or keywords from the corpus to generate reply information that matches their emotions. It does not exceed the legal and regulatory level, and is more deterrent and normative at the language expression level. It can even adjust the details of the debt and repayment requirements according to the specific circumstances of the overdue customer, so as to achieve efficient collection to the greatest extent.
[0030] In the process of interacting with overdue customers, the multimodal big model can analyze and identify the current emotional state of overdue customers, including anger, anxiety, resistance, reservation, sincerity, sincerity, etc., and compare and analyze the current emotions with historical emotions, comprehensively predict their repayment willingness, and then adaptively change the wording tendency of the text information and / or voice information output by the multimodal big model module, and adaptively perform more humane interactive operations such as topic switching, semantic transitions, interruptions, recovery, and severity during the interaction process, so as not to exceed the legal and regulatory level, and be more kind, humane or deterrent and normative at the language expression level, and even adjust the details of the outstanding debts and repayment requirements according to the specific circumstances of the customer, so as to achieve efficient collection to the greatest extent.
[0031] The present invention adopts the above technical solution to achieve the above purpose, which makes up for the shortcomings of the prior art and has reasonable design and convenient operation. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to make the above and / or other purposes, features, advantages and examples of the present invention more obvious and easy to understand, the drawings required for use in the specific implementation of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying creative work.
[0033] Figure 1 A logical block diagram showing the collection digital human interaction system based on a multimodal large model; Figure 2 A schematic diagram showing the logic block diagram of the machine learning submodule; Figure 3 A schematic diagram of a flow chart representing the machine learning submodule; Figure 4 A flowchart showing how the machine learning submodule processes the interaction information and outputs the processing results. DETAILED DESCRIPTION
[0034] Those skilled in the art can refer to the content of this article and appropriately replace and / or modify the process parameters to achieve the same. However, it should be particularly noted that all similar replacements and / or modifications are obvious to those skilled in the art and are considered to be included in the present invention. The products and preparation methods described in the present invention have been described through preferred examples. It is obvious that relevant personnel can modify or appropriately change and combine the products and preparation methods described herein without departing from the content, spirit and scope of the present invention to implement and apply the technology of the present invention.
[0035] Unless otherwise defined, technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the invention belongs. The present invention uses the methods and materials described herein; however, other suitable methods and materials known in the art may also be used. The materials, methods, and examples described herein are illustrative only and are not intended to be limiting. All publications, patent applications, patents, provisional applications, database entries, and other references mentioned herein are incorporated herein by reference in their entirety. In the event of a conflict, the present specification, including definitions, shall prevail.
[0036] Unless specifically stated, the materials, methods, and examples described herein are illustrative only and not limiting. Although methods and materials similar or equivalent to those described herein can be used in the practice or testing of the present invention, suitable methods and materials are still described herein.
[0037] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention. At the same time, the embodiments in this application and the features in the embodiments can be combined with each other without conflict.
[0038] The present invention is described in detail below Embodiment 1: A debt collection digital human interaction system based on a multimodal large model is provided, including: a data acquisition and input module, a multimodal large model module, a debt collection digital human module, and an interaction module. The logic block diagram is as follows Figure 1 shown.
[0039] The data collection and input module is used to collect voice data, image data and text data related to overdue customers, and input the above data and preset overdue collection reminder words into the multimodal big model. It also inputs legal documents including current legal provisions, cases, judgments, rulings, etc. into the multimodal big model.
[0040] The multimodal large model module is used to summarize the input information of the data collection and input module and the interactive information of the interactive module, and output the processed text information and / or voice information to the collection digital human module and the interactive module.
[0041] The multimodal large model module includes a speech recognition submodule, an image recognition submodule, a natural language processing submodule and a machine learning submodule. The speech recognition submodule, the image recognition submodule and the natural language processing submodule respectively recognize, analyze and extract speech, image and text information to obtain corresponding information. Specifically, the speech recognition submodule analyzes and recognizes the call recordings of overdue customers, converts them into text to obtain the initial speech text, and inputs it into the machine learning submodule as input data; the image recognition submodule recognizes and extracts the relevant information about overdue customers in the video and / or image, and converts it into text. The initial image text is obtained and input into the machine learning submodule as input data; the natural language processing submodule obtains the initial text text by identifying, analyzing and extracting text information from the communication records between the system and the overdue customer and / or persons involved in the case related to the overdue customer, and inputs it into the machine learning submodule as input data; the machine learning submodule integrates, identifies and analyzes the initial voice text, initial image text and initial text text, matches the reply nodes in the multimodal large model corpus to generate processing results, and enables the multimodal large model to automatically learn, train and optimize, and transmits the processing results to the collection digital human module and the interaction module.
[0042] like Figure 2The logic block diagram and Figure 3 As shown in the flowchart, the specific steps of the machine learning submodule using the root mean square normalization layer, the self-attention mechanism layer and the cross attention layer to fuse and recognize and analyze the initial voice text, the initial image text and the initial text text include: S1, respectively extracting features from the initial voice text, the initial image text and the initial text text to obtain initial voice features, initial image features and initial text features; S2, applying a first root mean square normalization layer to normalize the initial text features to obtain normalized text features; S3, applying a self-attention mechanism layer to calculate the weight of each position in the normalized text feature and combining the normalized text feature to perform weighted addition to obtain a weighted text feature; S4, adding the weighted text feature to the initial text feature to obtain a text addition result; S5, applying the first cross attention layer to learn the correlation matrix between the initial speech features and the normalized feature information output after the text addition result is normalized by the second root mean square normalization layer, and performing weighted summation on each position in the initial speech features and all positions of the normalized feature information to obtain the first associated information feature; S6, applying the second cross attention layer to learn the correlation matrix between the initial image features and the normalized feature information output after the text addition result is normalized by the second root mean square normalization layer, and performing weighted summation on each position in the initial image features and all positions of the normalized feature information to obtain the second association information feature; S7: Add the first associated information feature and the second associated information feature to obtain a multimodal associated information feature, thereby completing feature fusion.
[0043] like Figure 4 As shown, the machine learning submodule performs machine learning and outputs processing results, and further includes: T1. The interactive module receives the reply information, voice information and / or interactive information of overdue customers, generates interactive text information, performs word segmentation operation to obtain keyword extraction results, performs vectorization to obtain word vectors, and fills the corresponding keywords, parts of speech and word vectors into different categories of the multimodal large model corpus. In the corpus, the keywords and parts of speech are marked with sentiment values, and the keywords with repayment willingness ranging from extremely low to extremely high are marked with repayment willingness scores of 0-10 to distinguish the repayment willingness represented by the corresponding keywords; T2. Merge the word vector with the positional encoding to obtain the word vector matrix, and multiply it with the weight matrix to obtain the query matrix, key matrix, and value matrix respectively: T3. Optimize the Transformer model to focus on the emotional tendencies of overdue customers:
[0044] in, denote the query matrix, key matrix, and value matrix respectively; represents the dimension of the key vector; Represents relative position encoding; the Transformer model improves its ability to understand context through a multi-head attention mechanism, and retrieves matching reply nodes and / or keywords from the multimodal large model corpus for interactive replies. Actual case verification shows that the relative position encoding in the optimized Transformer model The introduction of can make the processing results of the model output adapt to the good or bad intentions of the debtor. When the overdue customer has malicious input information such as "I will never repay", "You are breaking the law", "What can you do if you don't repay", "It doesn't matter", etc., the output processing information will appear "credit investigation", "illegal", "further accountability", "prosecution" and other more deterrent or deterrent words; and when the debtor has good input information such as "sorry", "as soon as possible", "timely", "difficult", etc., the model output processing information will appear "can be delayed", "please contact by phone", "please go to the law firm for detailed discussion" and other more humane soothing terms. Therefore, it can be seen that relative position encoding The introduction of can enable the model to automatically learn the corresponding tendencies in the multimodal large model corpus according to the good and evil tendencies of overdue customers and materialize them into similar expressions of interactive output information, thereby realizing personalized and targeted generation of collection digital people, thereby achieving efficient collection to a greater extent.
[0045] The debt collection digital human module is used to interact externally through the interactive module based on the processing results provided by the multimodal large model module.
[0046] The interactive module is used to send letters, send messages, make voice calls, and conduct interactive Q&A with users. The interactive module is equipped with a collection submodule and a transmission submodule. The collection submodule takes the audio and video images of the overdue customers during the interaction with the overdue customers, and transmits the audio and video images to the multimodal large model module through the transmission submodule for identification and analysis, and then identifies the current emotional value of the overdue customers, compares the current emotional value with the historical emotional value, predicts their repayment willingness based on the change of emotional value, marks the current repayment willingness of the overdue customers according to 0 points for extremely low repayment willingness and 10 points for extremely high repayment willingness, retrieves the keyword insertion processing information suitable for the overdue customers from the multimodal large model corpus, and then adaptively changes the wording tendency of the text information and / or voice information output by the multimodal large model module based on the current repayment willingness. Specifically, the current emotion value of the overdue customer is identified through the following steps: performing part-of-speech analysis on the keywords based on the multimodal large model corpus, and determining the text emotion value of the overdue customer according to the analysis results; identifying and analyzing the intonation information, volume information and speaking speed information of the overdue customer in the audio and video images, and determining the voice emotion value of the overdue customer according to the analysis results; identifying and analyzing the facial and / or body behavior feature information of the overdue customer in the audio and video images, and determining the behavior emotion value of the overdue customer according to the analysis results; determining the current emotion value of the overdue customer according to the text emotion value, the voice emotion value and the behavior emotion value. System A is a system that predicts repayment willingness based on changes in emotion values and adaptively changes the wording of text and / or voice information output by the multimodal large model module. System B is a system that does not predict repayment willingness based on changes in emotion values. Internal testing was conducted by a professional team of 10 people. The test results showed that System A is extremely sensitive in adjusting the digital human interaction strategy quickly according to the tester's tone, volume, speaking speed, facial behavior, and body behavior. That is, if the tester's comprehensive emotions switch between goodwill and malice, the digital human interaction strategy can quickly switch accordingly; while the switching mobility of the digital human interaction strategy of System B is slow. When the tester's comprehensive emotions switch between goodwill and malice, the interaction strategy of the virtual collection digital human still uses the previous strategy and does not switch accordingly. Therefore, the system adaptively changes the wording of the text information and / or voice information output by the multimodal large model module, and adaptively performs more humane interactive operations such as topic switching, semantic transitions, interruptions, recovery, and severity during the interaction process. It does not exceed the legal and regulatory levels, and is more benevolent, humane or deterrent, and normative at the language expression level. It can even adjust the details of the outstanding debts and repayment requirements according to the specific circumstances of the customer, so as to achieve efficient debt collection to the greatest extent possible.
[0047] Embodiment 2: A post-loan collection digital human interaction method based on a multimodal large model is provided. The method is based on the system described in the aforementioned embodiment 1 and performs the following steps: Users interact with the debt collection digital person through the interactive module; The multimodal large model module generates a processing result based on the interactive content and delivers it to the collection digital human module; The next step of interaction is achieved with the user through the interaction module.
[0048] Embodiment 3: A computer device, comprising a memory, a processor, a communication interface and a communication bus; wherein the memory, the processor and the communication interface communicate with each other via the communication bus; the memory is used to store computer programs; the processor is used to execute the computer programs stored in the memory, and when the processor executes the computer program, it implements at least one step of the aforementioned post-loan collection digital human interaction method based on a multimodal large model, and can achieve the same technical effect. To avoid repetition, this embodiment will not be described in detail.
[0049] Embodiment 4: A computer-readable storage medium is provided, in which a computer program executable by a processor is stored. When the computer program is executed by the processor, at least one step of the aforementioned post-loan collection digital human interaction method based on a multimodal large model is executed, and the same technical effect can be achieved. To avoid repetition, this embodiment will not be described in detail.
[0050] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PR AM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0051] The conventional techniques in the above embodiments are prior arts known to those skilled in the art, and thus will not be described in detail here.
[0052] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.
[0053] Although the present invention has been described in detail and some specific embodiments have been cited, it is obvious to those skilled in the art that various changes and modifications can be made without departing from the spirit and scope of the present invention.
[0054] Although the above-mentioned specific embodiments have been shown, described and pointed out the novel features applied to various embodiments, it should be understood that various omissions, substitutions and changes can be made to the form and details of the described device or method without departing from the spirit of the present disclosure. In addition, the above-mentioned various features and methods can be used independently of each other, or can be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. Many of the above-mentioned embodiments include similar components, and therefore, these similar components are interchangeable in different embodiments. Although the present invention has been disclosed in the context of certain embodiments and examples, it should be understood by those skilled in the art that the present invention can extend beyond the specifically disclosed embodiments to other alternative embodiments and / or applications and obvious modifications and equivalents thereof. Therefore, the present invention is not intended to be limited by the specific disclosure of the preferred embodiments herein.
[0055] Matters not covered in the present invention are all known technologies.
Claims
1. Collection digital human interaction system based on multimodal large model, including: The data collection and input module is used to collect voice data, image data and text data related to overdue customers, and input the above data and preset overdue collection reminder words into the multimodal big model; The multimodal large model module is used to aggregate the input information of the data collection and input module and the interactive information of the interactive module, and output the processing results to the collection digital human module and the interactive module; The debt collection digital human module is used to interact with the outside world through the interactive module based on the processing results provided by the multimodal large model module; The interactive module is used to send letters, messages, and make voice calls, and can also conduct interactive Q&A with users; Among them, the multimodal large model module includes a speech recognition submodule, an image recognition submodule, a natural language processing submodule and a machine learning submodule. The speech recognition submodule, the image recognition submodule and the natural language processing submodule respectively recognize, analyze and extract the speech, image and text information to obtain the corresponding information, which is input into the machine learning submodule for machine learning and outputs the processing results.
2. The system according to claim 1, characterized in that: The speech recognition submodule analyzes and recognizes the overdue customer call recordings, converts them into text to obtain initial speech text, and inputs it into the machine learning submodule as input data; The image recognition submodule identifies and extracts relevant information about overdue customers in the video and / or image, converts it into text to obtain initial image text, and inputs it into the machine learning submodule as input data; The natural language processing submodule obtains initial text by identifying, analyzing and extracting text information from the communication records between the system and the overdue customer and / or persons involved in the case related to the overdue customer, and inputs it into the machine learning submodule as input data.
3. The system according to claim 2, characterized in that: The machine learning submodule integrates, identifies, and analyzes the initial voice text, initial image text, and initial text, matches the reply nodes in the multimodal large model corpus to generate processing results, and enables the multimodal large model to automatically learn, train, and optimize, and transmits the processing results to the collection digital human module and the interaction module.
4. The system according to claim 3, characterized in that: The specific steps of the machine learning submodule fusing, identifying and analyzing the initial voice text, the initial image text and the initial text text include: S1, respectively extracting features from the initial voice text, the initial image text and the initial text text to obtain initial voice features, initial image features and initial text features; S2, applying a root mean square normalization layer to normalize the initial text features to obtain normalized text features; S3, applying a self-attention mechanism layer to calculate the weight of each position in the normalized text feature and combining the normalized text feature to perform weighted addition to obtain a weighted text feature; S4, adding the weighted text feature to the initial text feature to obtain a text addition result; S5, using a cross attention layer to learn the correlation matrix between the initial speech features and the normalized feature information output after the text addition result is normalized by the root mean square normalization layer, and performing a weighted summation on each position in the initial speech features and all positions of the normalized feature information to obtain a first associated information feature; S6, using a cross attention layer to learn a correlation matrix between the initial image features and the normalized feature information output after the text addition result is normalized by the root mean square normalization layer, and performing a weighted summation on each position in the initial image features and all positions of the normalized feature information to obtain a second association information feature; S7: Add the first associated information feature and the second associated information feature to obtain a multimodal associated information feature, thereby completing feature fusion.
5. The system according to claim 1, characterized in that: The machine learning submodule performs machine learning and outputs processing results, and further includes: T1. The interactive module receives the reply information, voice information and / or interactive information of overdue customers, generates interactive text information, performs word segmentation operation to obtain keyword extraction results, performs vectorization to obtain word vectors, and fills the corresponding keywords, parts of speech and word vectors into different categories of the multimodal large model corpus; T2. Merge the word vector with the positional encoding to obtain the word vector matrix, and multiply it with the weight matrix to obtain the query matrix, key matrix, and value matrix respectively: T3. Optimize the Transformer model to focus on the emotional tendencies of overdue customers: in, denote the query matrix, key matrix, and value matrix respectively; represents the dimension of the key vector; Represents relative position encoding; the Transformer model improves its ability to understand the context through a multi-head attention mechanism, and retrieves matching reply nodes and / or keywords from the multimodal large model corpus for interactive replies.
6. The system according to claim 1, characterized in that: The interaction module is equipped with a collection submodule and a transmission submodule. The collection submodule takes the audio and video images of the overdue customers during the interaction with the overdue customers, and transmits the audio and video images to the multimodal large model module through the transmission submodule for recognition and analysis, thereby identifying the current emotional value of the overdue customers, comparing the current emotional value with the historical emotional value, predicting their willingness to repay based on the change in the emotional value, and then adaptively changing the wording tendency of the text information and / or voice information output by the multimodal large model module.
7. The system according to claim 6, characterized in that: The specific steps of transmitting the audio and video images to the multimodal large model module through the transmission submodule for identification and analysis, and then identifying the current emotion value of the overdue customer include: Performing part-of-speech analysis on keywords based on the multimodal large model corpus, and determining text sentiment values of overdue customers based on the analysis results; Identify and analyze the tone information, volume information, and speech speed information of the overdue customer in the audio and video images, and determine the voice emotion value of the overdue customer based on the analysis results; Identify and analyze facial and / or body behavior feature information of overdue customers in the audio and video images, and determine the behavior emotion value of the overdue customers based on the analysis results; The current emotion value of the overdue customer is determined according to the text emotion value, the sound emotion value and the behavior emotion value.
8. A post-loan collection digital human interaction method based on a multimodal large model, characterized by: The method is based on the system according to any one of claims 1 to 7, and performs the following steps: Users interact with the debt collection digital person through the interactive module; The multimodal large model module generates a processing result based on the interactive content and delivers it to the collection digital human module; The next step of interaction is achieved with the user through the interaction module.
9. A computer device, comprising a memory, a processor, a communication interface and a communication bus; wherein: The memory, processor, and communication interface communicate with each other via the communication bus; the memory is used to store computer programs; the processor is used to execute the computer programs stored in the memory, and is characterized in that: when the processor executes the computer program, it implements at least one step of the post-loan collection digital human interaction method based on a multimodal large model as described in claim 8.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, at least one step of the post-loan collection digital human interaction method based on a multimodal large model as described in claim 7 is implemented.
Citation Information
Cited By
Digital human voice generation system based on large language model
CN119314465A
AI digital human-based behavior prediction method, system and device, and storage medium
CN119762930A
A service fulfillment processing method and device, a storage medium and an electronic device
CN122679065A
