Loan review system based on digital human technology

Abstract

The present application discloses a loan review system based on digital human technology, which relates to the field of digital human technology. The system includes: when the digital human control center receives the digital human face-to-face review instruction, it receives the text response file fed back by the intelligent robot module, and conducts the user face-to-face review based on the text response file; when the user face-to-face review is completed, the front-end digital human central control module sends the user identity review instruction to the digital human control center; when the digital human control center receives the user identity review instruction, it pulls the user's real-time audio and video stream from the streaming media service module, and sends the user's real-time audio and video stream to the identity review module; the identity review module conducts identity review on the user to be reviewed based on the user's real-time audio and video stream. The application of the above technical solution solves the technical problems that the approval process of the loan face-to-face review process in the prior art is cumbersome, and the approval accuracy and efficiency may be low due to human factors.

Description

Technical Field

[0001] This application relates to the field of digital human technology, and in particular to a loan review system based on digital human technology. Background Art

[0002] The loan review is a crucial step in the loan approval process for financial institutions. It verifies the borrower's identity and assesses their repayment ability and loan risk. This process is crucial for ensuring loan security and maintaining financial market stability.

[0003] In traditional loan scenarios, the face-to-face review process typically involves a remote interview between the loan officer and the company's face-to-face review specialist. During this process, the review specialist will ask the loan officer a set of pre-set questions and conduct a risk analysis based on the loan officer's responses to determine whether to proceed with the subsequent approval process. However, this loan review process typically requires a dedicated face-to-face reviewer to conduct a face-to-face interview via video call. It involves multiple steps, including identity verification, questioning and answering, risk analysis and assessment, and approval decision-making. Each step requires multiple interactions between the loan applicant and the face-to-face review specialist, making the approval process cumbersome. Human factors (such as the face-to-face review specialist's experience, emotions, bias, or negligence) can lead to low approval accuracy and efficiency.

[0004] The above content is only used to assist in understanding the technical solution of the present invention and does not constitute an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide a loan review system based on digital human technology, which aims to solve the technical problems in the existing technology that the loan face-to-face review process is cumbersome and may lead to low approval accuracy and efficiency due to human factors.

[0006] To achieve the above objectives, this application proposes a loan review system based on digital human technology, which includes: a digital human control center, a front-end digital human central control module, an intelligent robot module, a streaming media service module, and an identity review module;

[0007] The digital human control center is configured to receive the text response file fed back by the intelligent robot module upon receiving the digital human face-to-face review instruction, and conduct a user face-to-face review based on the text response file;

[0008] The front-end digital human central control module is used to send a user identity verification instruction to the digital human control center when the user face-to-face review is completed;

[0009] The digital human control center is further configured to, upon receiving the user identity verification instruction, pull the user's real-time audio and video stream from the streaming media service module and send the user's real-time audio and video stream to the identity verification module;

[0010] The identity verification module is used to perform identity verification on the user to be verified based on the user's real-time audio and video stream.

[0011] In one embodiment, the front-end digital human central control module is used to send a first communication connection request to the digital human control center;

[0012] The Digital Human Control Center is configured to establish a connection with the front-end Digital Human Central Control Module upon receiving the first communication connection request, and to feedback the real-time communication address and session description information of the streaming media service module upon successful connection;

[0013] The front-end digital human central control module is further configured to send a second communication connection request to the streaming media service module upon receiving the real-time communication address and the session description information;

[0014] The streaming media service module is configured to establish a connection with the front-end digital human central control module upon receiving the second communication connection request, start the transcoding and recording process upon successful connection, and return a connection success instruction;

[0015] The front-end digital human central control module is further configured to send a digital human initialization instruction to the digital human control center via the first communication protocol upon receiving the connection success instruction;

[0016] The Digihuman control center is further configured to perform Digihuman initialization processing upon receiving the Digihuman initialization instruction, and to feedback an initialization success instruction upon completion of the initialization;

[0017] The front-end digital human central control module is further configured to display the digital human image and send a digital human face-to-face review instruction to the digital human control center upon receiving the initialization success instruction.

[0018] In one embodiment, the system further comprises: a digital human engine module and a speech recognition module;

[0019] The digital human control center is further configured to establish an audio and video stream connection with the streaming media service module upon receiving the digital human initialization instruction to obtain the audio and video stream;

[0020] The Digital Human Control Center is further configured to establish a communication connection with the Digital Human Engine Module and, when the connection is successful, send the real-time communication address to the Digital Human Engine Module;

[0021] The digital human engine module is configured to establish a connection with the streaming media service module via a second communication protocol upon receiving the real-time communication address;

[0022] The digital human engine module is further configured to convert the audio and video stream into a first digital human image audio and video stream when the connection is successful, and push the first digital human image audio and video stream to the streaming media service module;

[0023] The digital human control center is also used to establish a connection with the voice recognition module and the identity verification module, and when the connection is successful, it feeds back an initialization success instruction to the front-end digital human central control module.

[0024] In one embodiment, the digital human control center is configured to obtain the user identity information of the user to be reviewed based on the digital human face-to-face review instruction upon receiving the digital human face-to-face review instruction;

[0025] The digital human control center is further configured to generate a face-to-face interview start instruction based on the user identity information, and send the face-to-face interview start instruction to the intelligent robot module;

[0026] The intelligent robot module is configured to generate a text response file based on the user identity information in the face-to-face interview start instruction, and send the text response file to the digital human control center;

[0027] The digital human control center is also used to conduct user interviews based on the text response file.

[0028] In one embodiment, the system further comprises: a speech synthesis module;

[0029] The digital human control center is further configured to call the speech synthesis module to convert the text response file into a speech response file;

[0030] The digital human control center is further configured to send the voice response file to the digital human engine module via a first communication protocol;

[0031] The digital human engine module is used to convert the voice response file into an audio and video stream of a second digital human image;

[0032] The digital human engine module is further configured to push the second digital human image audio and video stream to the streaming media service module and respond to the user that the face-to-face review is completed.

[0033] In one embodiment, the identity verification module includes: a liveness detection module;

[0034] The digital human control center is further configured to send the user's real-time video stream in the user's real-time audio and video stream to the living body detection module;

[0035] The liveness detection module is used to generate a liveness test result based on the real-time video stream of the user and send the liveness test result to the front-end digital human central control module;

[0036] The front-end digital human central control module is further used to determine the number of face matching failures based on the liveness test results;

[0037] The front-end digital human central control module is also used to display a face review failure prompt message to the user to be reviewed when the number of face matching failures reaches a first risk number threshold.

[0038] In one embodiment, the living body detection module is used to determine whether there is a human face in each frame of the user's real-time video stream;

[0039] The liveness detection module is further configured to display gesture action prompt information if the gesture action exists, and the user to be reviewed responds based on the gesture action prompt information;

[0040] The liveness detection module is further configured to compare the face with the face on the ID card of the user to be reviewed when the response is passed, and generate a liveness test result when the comparison fails.

[0041] In one embodiment, the identity verification module further includes: a voiceprint detection module;

[0042] The digital human control center is further configured to send the user's real-time audio stream in the user's real-time audio and video stream to the voiceprint detection module;

[0043] The voiceprint detection module is used to generate voiceprint information corresponding to the user's real-time audio stream and search for the identity information of the current auditing user based on the voiceprint information;

[0044] The voiceprint detection module is further configured to compare the identity information with the user identity information of the user to be reviewed, and if the comparison fails, send the voiceprint comparison result to the front-end digital human central control module;

[0045] The front-end digital human central control module is further configured to record the number of identity matching failures if the voiceprint comparison result shows that the current audited user does not match the user to be audited;

[0046] The front-end digital human central control module is also used to display a face-to-face review failure prompt message to the user to be reviewed when the number of identity matching failures reaches a second risk number threshold.

[0047] In one embodiment, the front-end digital human central control module is further configured to collect user audio information of the user to be reviewed when the identity of the user to be reviewed is passed, and send the user audio information to the streaming media service module. The user audio information is information generated when the user to be reviewed answers questions asked by the digital human.

[0048] The digital human control center is further configured to pull the user audio information from the streaming media service module in real time and send the user audio information to the voice recognition module;

[0049] The speech recognition module is further configured to convert the user audio information into user text information and return the user text information to the digital human control center;

[0050] The digital human control center is further configured to send the user text information to the intelligent robot module;

[0051] The intelligent robot module is used to generate an intelligent robot text speech file based on the user text information, and send the intelligent robot text speech file to the digital human control center;

[0052] The digital human control center is further configured to call the speech synthesis module to convert the intelligent robot text speech file into an intelligent robot speech file, and send the intelligent robot speech file to the digital human engine module;

[0053] The digital human engine module is also used to convert the intelligent robot voice file into a third digital human image audio and video stream, and push the third digital human image audio and video stream to the streaming media service module to conduct face-to-face question and answer with the user to be reviewed.

[0054] In one embodiment, the system further comprises: a risk assessment module;

[0055] The intelligent robot module is further used to label the intelligent robot text speech in the intelligent robot text speech file to obtain a number of risk labels;

[0056] The risk assessment module is used to determine whether the risk label meets the risk requirements;

[0057] The risk assessment module is further configured to generate an alarm message to instruct manual review if the risk is not met.

[0058] The present application provides a loan review system based on digital human technology. The present application discloses that upon receiving a digital human face-to-face review instruction, the digital human control center receives a text response file fed back by an intelligent robot module and conducts a user face-to-face review based on the text response file. Upon completion of the user face-to-face review, the front-end digital human central control module sends a user identity review instruction to the digital human control center. Upon receiving the user identity review instruction, the digital human control center pulls the user's real-time audio and video stream from the streaming media service module and sends the user's real-time audio and video stream to the identity review module. The identity review module conducts an identity review on the user to be reviewed based on the user's real-time audio and video stream. Compared to the prior art, in which a face-to-face review specialist asks loan officers preset fixed questions and conducts a risk analysis based on the loan officers' answers to determine whether to continue with the subsequent approval process, which is a cumbersome approval process, the present invention conducts a user face-to-face review based on the text response file fed back by the intelligent robot and conducts an identity review on the user based on the user's real-time audio and video stream upon completion of the face-to-face review. This solves the technical problems of the prior art's loan face-to-face review process being cumbersome and of low approval accuracy and efficiency due to human factors. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0060] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0061] Figure 1 This is a structural diagram of the first embodiment of the loan review system based on digital human technology provided in this application;

[0062] Figure 2 This is a structural diagram of the second embodiment of the loan review system based on digital human technology provided in this application;

[0063] Figure 3 This is a structural diagram of the third embodiment of the loan review system based on digital human technology provided in this application;

[0064] Figure 4 This is the overall flow chart of the loan review system based on digital human technology for this application.

[0065] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0066] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0067] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0068] The main solution of the embodiment of the present application is: the digital human control center is used to receive the text response file fed back by the intelligent robot module when receiving the digital human face-to-face review instruction, and conduct the user face-to-face review based on the text response file; the front-end digital human central control module is used to send the user identity review instruction to the digital human control center when the user face-to-face review is completed; the digital human control center is also used to pull the user's real-time audio and video stream from the streaming media service module when receiving the user identity review instruction, and send the user's real-time audio and video stream to the identity review module; the identity review module is used to conduct identity review on the user to be reviewed based on the user's real-time audio and video stream.

[0069] The loan approval process in existing technologies requires that the interviewer ask the loan officer questions according to preset fixed questions, and conducts risk analysis based on the loan officer's answers to determine whether to continue the subsequent approval process. The interview involves multiple links, including identity verification, questioning and answering, risk analysis and assessment, approval decision-making, etc. Each step requires multiple interactions between the loan applicant and the interviewer, making the approval process cumbersome.

[0070] This application provides a solution that can conduct user face-to-face review based on the text response file fed back by the intelligent robot, and conduct identity verification of the user based on the user's real-time audio and video stream when the face-to-face review is completed, thereby solving the technical problems in the existing technology of the loan face-to-face review process being cumbersome and resulting in low approval accuracy and efficiency due to human factors.

[0071] Reference Figure 1 , Figure 1 This is a structural diagram of the first embodiment of the loan review system based on digital human technology in this application.

[0072] like Figure 1 As shown, the loan review system based on digital human technology in this embodiment includes: a digital human control center 10, a front-end digital human central control module 20, an intelligent robot module 30, a streaming media service module 40 and an identity review module 50.

[0073] In this solution, the digital human control center 10 mainly includes front-end instruction reception and processing, back-end instruction reception and processing, rtmp protocol stream parsing, docking digital human engine module 60, ASR module (i.e., speech recognition module 70), TTS module (i.e., speech synthesis module 80), intelligent robot module 30, real-time face detection module (i.e., liveness detection module 501), etc. These instructions include establishing a connection with this module, initializing module functions, streaming the entire process, etc.; the front-end digital human central control module 20 is used to initiate websoket to interact with the digital human control center 10 for instruction information, to receive and execute instruction information responded by the digital human control center 10, and to process webrtc real-time video push and pull streams; the main function of the intelligent robot module 30 is to enable the face-to-face review specialist to use the big model for training, learn user information, and the big model Based on user information, including but not limited to occupation, identity, gender, region, work information, credit information, etc., a large model is used for each user, with digital humans replacing real people to ask questions. Risk assessment is performed based on the user's answers, and different questions are generated for further questioning. For example, if the user fills in that their occupation is electronic engineer, the intelligent robot module will generate the corresponding professional knowledge of electronic engineers for questioning. Based on the user's answers, it is identified whether the user has committed fraud. The streaming service module 40 is used to receive the video stream of the WebRTC protocol for analysis, and then convert it into an RTMP protocol stream and push it to the subscriber for the digital human control center 10 to connect to the RTMP stream for consumption. In addition, it also records the real-time stream for later analysis. The identity verification module 50 is used to perform face detection and voice detection on the user to verify the user's identity.

[0074] The digital human control center 10 is used to receive the text response file fed back by the intelligent robot module 30 when receiving the digital human face-to-face review instruction, and conduct a user face-to-face review based on the text response file.

[0075] It should be understood that the above-mentioned digital human face-to-face review instruction may be an instruction for instructing the digital human control center 10 to control the digital human to conduct a face-to-face review of the user who needs a loan.

[0076] It should be noted that the above-mentioned text response file can be a file generated based on the text speech in response to the text speech by the intelligent robot module 30. In actual application, when the digital human control center 10 receives the digital human face-to-face review instruction from the front-end digital human central control module 20, it can send a face-to-face review start event to the intelligent robot module 30 via the HTTP protocol. After receiving the face-to-face review start event, the intelligent robot in the intelligent robot module 30 can generate the first text speech in response to the user information carried in the digital human face-to-face review instruction through the large model: for example, "Welcome to so-and-so, are you so-and-so?", record it, and generate a text response file based on the text speech obtained in the response. At the same time, the text response file is fed back to the digital human control center 10. After receiving the text response file, the digital human control center 10 can conduct a user face-to-face review based on the text response file.

[0077] The front-end digital human central control module 20 is used to send a user identity verification instruction to the digital human control center 10 when the user face-to-face review is completed.

[0078] It should be understood that the above-mentioned user identity review instruction may be an instruction for instructing the Digital Human Control Center 10 to review the identity information of the user who needs a loan.

[0079] The digital human control center 10 is further configured to pull the user's real-time audio and video stream from the streaming media service module 40 upon receiving the user identity verification instruction, and send the user's real-time audio and video stream to the identity verification module 50 .

[0080] It should be noted that the above-mentioned user real-time audio and video stream can be the user's audio information and video information collected in real time by the front-end digital human central control module 20, and these audio information and video information can be transmitted to the streaming media service module 40 through the WebRTC protocol.

[0081] The identity verification module 50 is used to perform identity verification on the user to be verified based on the user's real-time audio and video stream.

[0082] In actual applications, upon completing the face-to-face review, the Digihuman control center 10 can send a face-to-face review completion instruction to the front-end Digihuman central control module 20. Upon receiving the face-to-face review completion instruction, the front-end Digihuman central control module 20 sends a user identity verification instruction to the Digihuman control center 10, instructing the Digihuman control center 10 to verify the identity information of the user requiring a loan. In this embodiment, the user identity information verification can include liveness detection and voiceprint detection. Specifically, upon receiving the user identity verification instruction sent by the front-end Digihuman central control module 20, the Digihuman control center 10 can pull the user's real-time audio and video stream from the streaming media service module 40 and send the user's real-time audio and video stream to the identity verification module 50, so that the identity verification module 50 can conduct an identity verification on the user requiring a loan (i.e., the aforementioned user to be reviewed) based on the user's real-time audio and video stream.

[0083] In this solution, first, digital face-to-face interviews significantly reduce the cumbersome processes and steps associated with traditional face-to-face interviews through automation and intelligent technologies. For example, through the online application system, customers can submit applications anytime, anywhere, without having to be present in person. Furthermore, automated information verification and intelligent risk assessments can rapidly verify customer information and assess risk, significantly shortening approval times. Second, digital face-to-face interviews eliminate human interference, such as the experience, emotions, biases, or negligence of interviewers, thereby improving the accuracy and fairness of approval results. Third, digital face-to-face interviews reduce significant manpower input and lower operating costs for financial institutions. Traditional face-to-face interviews require significant human resources for document compilation, information verification, and risk assessment, while digital face-to-face interviews can perform these functions through automation and intelligent technologies. Furthermore, digital face-to-face interviews are not restricted by time or location, providing round-the-clock service. Customers can apply for loans and check approval status at any time and from any location through the online application system, significantly improving service convenience and flexibility. Digital face-to-face interviews streamline the loan application process, reducing customer wait time and trips, and improving customer satisfaction and loyalty. At the same time, digital face-to-face interviews also provide more convenient and efficient service methods, such as self-service terminals and AI customer service, further enhancing the customer experience.

[0084] Furthermore, before conducting a face-to-face interview with the user, each module in the system needs to establish a communication connection for communication. Therefore, the front-end digital human central control module 20 is used to send a first communication connection request to the digital human control center 10.

[0085] It should be noted that the above-mentioned first communication connection request can be a communication request initiated by the front-end digital human central control module 20 to the digital human control center 10. In this embodiment, the first communication connection request can be a websocket connection request, wherein the websocket connection is a technology for establishing a persistent connection between the client and the server, allowing two-way communication and real-time data transmission.

[0086] The digital human control center 10 is used to establish a connection with the front-end digital human central control module 20 upon receiving the first communication connection request, and to feedback the real-time communication address and session description information of the streaming media service module 40 upon successful connection.

[0087] It is understood that after receiving the communication connection request initiated by the front-end digital human central control module 20, the digital human control center 10 can establish a communication connection with the front-end digital human central control module 20. After the front-end digital human central control module 20 and the digital human control center 10 successfully establish communication, the digital human control center 10 can return the real-time communication address and session description information of the streaming service module 40. In this embodiment, the streaming service module 40 can be an SRS streaming service. SRS (Simple Realtime Server) is a simple and efficient real-time streaming server capable of processing low-latency streaming media. Accordingly, the above-mentioned real-time communication address and session description information can be the SRS's WebRTC address and SDP information, respectively. The WebRTC address can be the HTTP interface address used for WebRTC publishing and playing. These addresses allow WebRTC clients to communicate with the SRS server to achieve the transmission of real-time audio and video streams. In addition, in the communication process of WebRTC, SDP information plays a vital role. SDP is a protocol used to describe multimedia sessions. It contains detailed information about the session, such as media type, codec, IP address, port number, etc., that is, the above-mentioned SDP information.

[0088] The front-end digital human central control module 20 is further configured to send a second communication connection request to the streaming media service module 40 upon receiving the real-time communication address and the session description information.

[0089] It should be noted that the above-mentioned second communication connection request can be a communication request initiated by the front-end digital human central control module 20 to the streaming media service module 40. In this embodiment, the second communication connection request can be a WebRTC connection request, wherein the WebRTC connection is a real-time communication (RTC) technology based on web pages, which allows audio, video and data to be directly exchanged between browsers.

[0090] The streaming media service module 40 is used to establish a connection with the front-end digital human central control module 20 when receiving the second communication connection request, start the transcoding and recording process when the connection is successful, and return a connection success instruction.

[0091] It can be understood that after receiving the communication connection request initiated by the front-end digital human central control module 20, the streaming service module 40 can establish a communication connection with the front-end digital human central control module 20. After the streaming service module 40 and the front-end digital human central control module 20 successfully establish communication, the webrtc to rtmp transcoding can be started and recording can be performed (that is, the above-mentioned transcoding and recording process), and a successful establishment instruction can be returned.

[0092] The front-end digital human central control module 20 is further configured to send a digital human initialization instruction to the digital human control center 10 via the first communication protocol upon receiving the successful connection instruction.

[0093] It should be understood that the first communication protocol mentioned above is the websocket protocol. In this embodiment, after receiving the SRS establishment success instruction, the front-end digital human central control module 20 can send a digital human initialization instruction to the digital human control center 10 via the websocket protocol, transmitting the webrtc address of the push-pull stream.

[0094] The digital human control center 10 is further configured to perform digital human initialization processing upon receiving the digital human initialization instruction, and to feedback an initialization success instruction upon completion of the initialization.

[0095] The front-end digital human central control module 20 is further configured to display the digital human image and send a digital human face-to-face review instruction to the digital human control center 10 upon receiving the initialization success instruction.

[0096] In practice, after receiving the Digihuman initialization command from the front-end Digihuman central control module 20, the Digihuman control center 10 can initialize the Digihuman and, upon successful initialization, feedback a successful initialization command to the front-end Digihuman central control module 20. Upon receiving the successful initialization command, the front-end Digihuman central control module 20 begins playing the WebRTC stream, displaying the Digihuman image, and collecting camera and audio information, transmitting it to the SRS server via the WebRTC protocol. Simultaneously, the front-end Digihuman central control module 20 sends a Digihuman face-to-face review command to the Digihuman control center 10, instructing the Digihuman control center 10 to control the Digihuman to conduct a face-to-face review of the loan applicant.

[0097] In this embodiment, the digital human control center receives a text response file fed back by the intelligent robot module upon receiving a digital human face-to-face review instruction, and conducts a user face-to-face review based on the text response file. The front-end digital human central control module sends a user identity verification instruction to the digital human control center upon completion of the user face-to-face review instruction. Upon receiving the user identity verification instruction, the digital human control center pulls the user's real-time audio and video stream from the streaming media service module and sends the user's real-time audio and video stream to the identity verification module. The identity verification module conducts an identity verification on the user to be reviewed based on the user's real-time audio and video stream. Compared to the prior art, in which a face-to-face review specialist asks loan officers preset fixed questions and conducts a risk analysis based on the loan officers' answers to determine whether to continue the subsequent approval process, which is a cumbersome approval process, this embodiment conducts a user face-to-face review based on the text response file fed back by the intelligent robot, and conducts an identity verification on the user based on the user's real-time audio and video stream upon completion of the face-to-face review. This solves the technical problems of the prior art loan face-to-face review process being cumbersome and of low approval accuracy and efficiency due to human factors.

[0098] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above embodiment 1 can be referred to the above introduction and will not be described in detail later. Figure 2 , Figure 2 This is a structural diagram of the second embodiment of the loan review system based on digital human technology in this application.

[0099] Based on the above-mentioned first embodiment of the loan review system based on digital human technology, a second embodiment of the loan review system based on digital human technology of the present invention is proposed.

[0100] like Figure 2 As shown, the system further includes: a digital human engine module 60 and a speech recognition module 70.

[0101] In this solution, the digital human engine module 60 is used to receive the voice stream transmitted from the digital human control center 10, convert the voice stream into the digital human image output audio and video stream, and push it to the SRS streaming service module 40 through the WebRTC protocol; the voice recognition module 70 is used to receive the voice stream transmitted from the digital human control center 10 and convert the voice stream into corresponding text.

[0102] The digital human control center 10 is further configured to establish an audio and video stream connection with the streaming media service module 40 upon receiving the digital human initialization instruction, so as to obtain the audio and video stream.

[0103] It can be understood that the above-mentioned audio and video streams are the audio information and video information of the user collected by the front-end digital human central control module 20.

[0104] The digital human control center 10 is further configured to establish a communication connection with the digital human engine module 60 and send the real-time communication address to the digital human engine module 60 when the connection is successful.

[0105] The digital human engine module 60 is configured to establish a connection with the streaming media service module 40 via a second communication protocol upon receiving the real-time communication address.

[0106] In actual applications, upon receiving the Digihuman initialization command, the Digihuman control center 10 can establish an RTMP real-time audio and video stream connection from the SRS and enable recording. It can then establish a WebSocket communication connection with the Digihuman engine module 60 and send the WebRTC address to the Digihuman engine module 60. After receiving the WebRTC address, the Digihuman engine module 60 can connect to the SRS server using the WebRTC protocol (i.e., the second communication protocol described above).

[0107] The digital human engine module 60 is further configured to convert the audio and video stream into a first digital human image audio and video stream when the connection is successful, and push the first digital human image audio and video stream to the streaming media service module 40 .

[0108] It should be noted that after the digital human engine module 60 and the streaming service module 40 are successfully connected, the digital human static video stream, that is, the above-mentioned first digital human image audio and video stream, can be pushed to the streaming service module 40 through the WebRTC protocol.

[0109] The digital human control center 10 is also used to establish a connection with the voice recognition module 70 and the identity verification module 50, and when the connection is successful, it feeds back an initialization success instruction to the front-end digital human central control module 20.

[0110] In actual applications, after the digital human control center 10 successfully connects with the digital human engine module 60, it can continue to establish a communication connection with the voice recognition module 70 and the identity verification module 50. When the connection is successful, it indicates that the initialization is completed. At this time, the digital human control center 10 can feedback the initialization success instruction to the front-end digital human central control module 20.

[0111] Furthermore, the digital human control center 10 is configured to obtain the user identity information of the user to be reviewed based on the digital human face-to-face review instruction when receiving the digital human face-to-face review instruction.

[0112] It should be noted that the digital human face-to-face review instruction in this embodiment carries the user identity information of the user to be reviewed. Upon receiving the digital human face-to-face review instruction sent by the front-end digital human central control module 20, the digital human control center 10 can parse the digital human face-to-face review instruction to obtain the user identity information of the user to be reviewed. The user identity information may include the name, ID number, and ID card face image of the user to be reviewed, and this embodiment does not impose any restrictions on this.

[0113] The digital human control center 10 is further configured to generate an interview start instruction based on the user identity information and send the interview start instruction to the intelligent robot module 30 .

[0114] The intelligent robot module 30 is used to generate a text response file based on the user identity information in the face-to-face interview start instruction, and send the text response file to the digital human control center 10.

[0115] The digital human control center 10 is also used to conduct user interviews based on the text response file.

[0116] In this embodiment, the Digital Human Control Center 10 can generate a start-interview instruction based on the user identity information of the user to be reviewed, instructing the Digital Human to begin the face-to-face interview. This instruction can then be sent to the Intelligent Robot Module 30 via the HTTP protocol. Upon receiving the start-interview instruction, the Intelligent Robot Module 30 can parse it to obtain the user identity information of the user to be reviewed. Then, the large model generates a response with the first text sentence: for example, "Welcome to XXX, are you XXX?" This is recorded and a text response file is generated. This text response file is then sent to the Digital Human Control Center 10. Upon receiving the text response file, the Digital Human Control Center 10 can control the Digital Human to conduct the face-to-face interview with the user based on the text response file.

[0117] Furthermore, the system further includes: a speech synthesis module 80.

[0118] In this solution, the speech synthesis module 80 can be used to receive text transmitted from the digital human control center 10 and convert the text into a speech stream.

[0119] The digital human control center 10 is further configured to call the speech synthesis module 80 to convert the text response file into a speech response file; the digital human control center 10 is further configured to send the speech response file to the digital human engine module 60 via the first communication protocol; the digital human engine module 60 is configured to convert the speech response file into a second digital human image audio and video stream; the digital human engine module 60 is further configured to push the second digital human image audio and video stream to the streaming media service module 40 and respond to the user that the face-to-face review is completed.

[0120] In actual applications, after receiving the text response file sent by the intelligent robot module 30, the digital human control center 10 can call the speech synthesis module 80 via the HTTP protocol to convert the text response file into a corresponding voice response file through the speech synthesis module 80, and transmit the voice response file to the digital human engine module 60 via the websocket protocol (i.e., the above-mentioned first communication protocol). After receiving the voice response file, the digital human engine module 60 can convert it into an audio and video stream with the digital human image, that is, obtain the above-mentioned second digital human image audio and video stream, and push the second digital human image audio and video stream to the SRS via the webrtc protocol, and at the same time respond to the user that the face-to-face review is completed.

[0121] In this embodiment, it is disclosed that upon receiving a digital human initialization instruction, the digital human control center establishes an audio and video stream connection with the streaming media service module to obtain the audio and video stream; establishes a communication connection with the digital human engine module, and upon successful connection, sends a real-time communication address to the digital human engine module; upon receiving the real-time communication address, the digital human engine module establishes a connection with the streaming media service module via a second communication protocol; upon successful connection, the audio and video stream is converted into a first digital human image audio and video stream, and the first digital human image audio and video stream is pushed to the streaming media service module; the digital human control center establishes a connection with the voice recognition module and the identity verification module, and upon successful connection, feeds back an initialization success instruction to the front-end digital human central control module, so that after the initialization is completed, the user can be face-to-face reviewed by the digital human, thereby improving the approval efficiency and the accuracy of the approval results.

[0122] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the above embodiments can be referred to the above introduction and will not be described in detail later. Figure 3 , Figure 3 This is a block diagram of the third embodiment of the loan review system based on digital human technology.

[0123] Based on the first embodiment and / or second embodiment of the loan review system based on digital human technology described above, a third embodiment of the loan review system based on digital human technology of the present invention is proposed.

[0124] like Figure 3 As shown, the identity verification module 50 includes: a liveness detection module 501.

[0125] In this solution, the liveness detection module 501 receives video frames from the Digihuman control center 10 and, based on the continuous images, determines whether a face is present in the video, whether it is alive, whether there is fraud, or whether it is the real person. This module cooperates with the Digihuman engine module 60 to prompt the user to perform corresponding gestures, such as drawing a circle, a line, a triangle, or pointing up, down, left, or right.

[0126] The digital human control center 10 is further configured to send the user real-time video stream in the user real-time audio and video stream to the living body detection module 501 .

[0127] In this embodiment, the digital human control center 10 can pull the user's real-time video stream from the SRS, analyze each frame in the user's real-time video stream in real time, and transmit it to the living body detection module 501 through the websocket protocol.

[0128] The liveness detection module 501 is used to generate a liveness test result based on the user's real-time video stream and send the liveness test result to the front-end digital human central control module 20.

[0129] Specifically, the liveness detection module 501 is used to determine whether there is a face in each frame of the user's real-time video stream; the liveness detection module 501 is also used to display gesture action prompt information if it exists, and the user to be reviewed responds based on the gesture action prompt information; the liveness detection module 501 is also used to compare the face with the face on the ID card of the user to be reviewed when the response is passed, and generate a liveness test result when the comparison fails.

[0130] In actual applications, after receiving each frame of the user's real-time video stream, the liveness detection module 501 can determine whether there is a face. If the face exists, the user is prompted to make random gestures, such as drawing circles or triangles with the hands, pointing up, down, left, and right with the hands, etc. If it fails, it directly responds to the front-end digital human central control module 20. If it passes, the face is compared with the ID card face in the user identity information of the user to be reviewed. If the comparison result is not the same person or the face does not exist, it means that the comparison failed. At this time, a liveness test result of failed comparison is generated and sent to the front-end digital human central control module 20.

[0131] The front-end digital human central control module 20 is further used to determine the number of face matching failures based on the liveness test results.

[0132] It should be understood that the aforementioned number of face matching failures refers to the number of times the user's gestures failed, and / or the number of times the face did not match the face on the ID card. In this embodiment, after receiving the liveness test results, the front-end digital human central control module 20 can analyze them. If the liveness test result indicates that the face does not exist, the user is prompted to "Please face the camera." If the liveness test result indicates that the gesture failed, the number of gesture failures is recorded. If the liveness test result indicates that the face does not match the face on the ID card, the number of times the face does not match the face on the ID card is also recorded.

[0133] The front-end digital human central control module 20 is further configured to display a face review failure prompt message to the user to be reviewed when the number of face matching failures reaches a first risk number threshold.

[0134] It should be noted that the first risk threshold may be the maximum number of attempts that indicate a failed face matching attempt. In this embodiment, if the number of failed face matching attempts reaches the first risk threshold, a notification indicating that the person is not the person in question will be displayed, and the face-to-face interview will be terminated. In this case, a new face-to-face interview will be initiated.

[0135] Furthermore, if Figure 3 As shown, the identity verification module 50 further includes: a voiceprint detection module 502.

[0136] In this solution, the voiceprint detection module 502 can be used to receive the voice stream transmitted from the digital human control center 10 and analyze whether the voiceprint is that of the person according to the voice stream.

[0137] The digital human control center 10 is further configured to send the user's real-time audio stream in the user's real-time audio and video stream to the voiceprint detection module 502 .

[0138] It should be understood that the Digital Human Control Center 10 can pull the user's real-time audio stream from the SRS and transmit the user's real-time audio stream to the voiceprint detection module 502 via the websocket protocol in real time.

[0139] The voiceprint detection module 502 is used to generate voiceprint information corresponding to the user's real-time audio stream, and search for the identity information of the currently audited user based on the voiceprint information.

[0140] The voiceprint detection module 502 is further configured to compare the identity information with the user identity information of the user to be reviewed, and when the comparison fails, send the voiceprint comparison result to the front-end digital human central control module 20 .

[0141] The front-end digital human central control module 20 is further used to record the number of identity matching failures if the voiceprint comparison result shows that the current audited user does not match the user to be audited.

[0142] The front-end digital human central control module 20 is further configured to display a face-to-face review failure prompt message to the user to be reviewed when the number of identity matching failures reaches a second risk number threshold.

[0143] In actual applications, after receiving the user's real-time audio stream, the voiceprint detection module 502 can convert the user's real-time audio stream into a voiceprint and search for the identity information of the current audit user based on the voiceprint. At this time, the identity information of the current audit user can be compared with the user identity information of the user to be audited. If the comparison result indicates that the current audit user and the user to be audited are not the same person, the comparison fails and the comparison result can be sent to the front-end digital human central control module 20. The front-end digital human central control module 20 receives the voiceprint comparison result. If the result is not the person, it records the number of times to obtain the above-mentioned identity matching failure number. When the number of identity matching failures reaches the second risk number threshold, it prompts the user to conduct an interview, and the interview is terminated. The interview needs to be re-initiated.

[0144] Furthermore, the front-end digital human central control module 20 is also used to collect user audio information of the user to be reviewed when the identity review of the user to be reviewed is passed, and send the user audio information to the streaming media service module 40. The user audio information is information generated when the user to be reviewed answers the questions asked by the digital human.

[0145] It should be noted that when the identity of the user to be reviewed is passed, the digital human can ask questions to the user to be reviewed, and the user can answer based on the digital human's questions. At this time, the front-end digital human central control module 20 can collect the user audio information of the user to be reviewed and transmit the user audio information to the streaming media service module 40 through the WebRTC protocol.

[0146] The digital human control center 10 is further configured to pull the user audio information from the streaming service module 40 in real time and send the user audio information to the voice recognition module 70 .

[0147] The speech recognition module 70 is further configured to convert the user audio information into user text information and return the user text information to the digital human control center 10;

[0148] The digital human control center 10 is also used to send the user text information to the intelligent robot module 30;

[0149] The intelligent robot module 30 is used to generate an intelligent robot text speech file based on the user text information, and send the intelligent robot text speech file to the digital human control center 10;

[0150] The digital human control center 10 is further configured to call the speech synthesis module 80 to convert the intelligent robot text speech file into an intelligent robot speech file, and send the intelligent robot speech file to the digital human engine module 60;

[0151] The digital human engine module 60 is also used to convert the intelligent robot voice file into a third digital human image audio and video stream, and push the third digital human image audio and video stream to the streaming media service module 40 to conduct face-to-face interviews with the user to be reviewed.

[0152] In actual applications, the digital human control center 10 can pull real-time audio information from the SRS and forward the audio information to the voice recognition module 70 via the websokcet protocol. At this time, the voice recognition module 70 translates the audio information into text information in real time and sends it to the digital human control center 10. After receiving the text information translated by the voice recognition module 70, the digital human control center 10 sends it to the intelligent robot in the intelligent robot module 30 via the HTTP protocol. The intelligent robot analyzes the user's intention through a large model, identifies risks, and automatically generates a process text response for the next sentence (i.e., the above-mentioned intelligent robot text file) and gives it to the digital human control center 10. After receiving the robot's text speech file, the Digihuman Control Center 10 invokes the speech synthesis module 80 via the HTTP protocol to convert the text file into a speech file. This converts the text file into the speech file, obtaining the aforementioned intelligent robot speech file. The Digihuman Control Center 10 then transmits the intelligent robot speech file to the Digihuman Engine Module 60 via the WebSocket protocol. Upon receiving the intelligent robot speech file, the Digihuman Engine Module 60 converts it into an audio and video stream with the digital human image (i.e., the aforementioned third digital human image audio and video stream) and pushes it to the SRS via the WebRTC protocol. Thereafter, the process returns to the step of collecting the user's audio information until the user has answered all the interview questions.

[0153] Furthermore, the system further includes: a risk assessment module 90 .

[0154] The intelligent robot module 30 is also used to label the intelligent robot text speech in the intelligent robot text speech file to obtain several risk labels; the risk assessment module 90 is used to determine whether the risk label meets the risk requirements; the risk assessment module 90 is also used to generate an alarm message to indicate manual review if it does not meet the requirements.

[0155] In this embodiment, after receiving the user text message, the intelligent robot module 30 can find out the user's intention to identify risks and label the generated intelligent robot text to obtain multiple risk labels. After the user to be reviewed has answered all the face-to-face review questions, the process enters the next approval stage. At this time, the risk assessment module 90 ( Figure 3 (not shown) can determine whether the risk label meets the risk requirements. If not, manual review of the video is required, which can greatly improve efficiency.

[0156] In the specific implementation, refer to Figure 4 , Figure 4 This is the overall flow chart of the loan review system based on digital human technology for this application. Figure 4 As shown, the proposed loan review system based on digital human technology includes a WebRTC push-pull streaming module, a front-end digital human central control SDK module (i.e., the front-end digital human central control module 20), a digital human control center 10, a digital human engine module 60, an intelligent robot module 30, an ASR (Automatic Speech Recognition) module (i.e., a speech recognition module 70), a TTS (Text To Speech) module (i.e., a speech synthesis module 80), an SRS module (i.e., a streaming media service module 40), a liveness detection module 501, and a voiceprint detection module 502. The front-end digital human central control SDK module and the digital human control center 10 are connected via WebSockets, while the SRS module is connected to the front-end digital human central control SDK module and the digital human engine module via WebRTC. The digital human control center 10 is connected to the digital human engine module, the ASR module, the liveness detection module, and the voiceprint detection module via the WebSocket protocol. Once all modules in the system have established communication connections, face-to-face user reviews based on digital human technology can be conducted, improving the efficiency and accuracy of loan reviews.

[0157] In this embodiment, it is disclosed that the digital human control center sends the user's real-time video stream in the user's real-time audio and video stream to the liveness detection module; the liveness detection module generates a liveness test result based on the user's real-time video stream, and sends the liveness test result to the front-end digital human central control module; the front-end digital human central control module determines the number of face matching failures based on the liveness test result; when the number of face matching failures reaches a first risk threshold, a face-to-face review failure prompt message is displayed to the user to be reviewed, thereby preventing others from conducting the face-to-face review on behalf of the loan user himself, thereby improving the accuracy of the face-to-face review.

Claims

1. A loan review system based on digital human technology, characterized in that: The system includes: a digital human control center, a front-end digital human central control module, an intelligent robot module, a streaming media service module and an identity verification module; The digital human control center is used to receive the text response file fed back by the intelligent robot module when receiving the digital human face-to-face review instruction, and conduct a user face-to-face review based on the text response file; The front-end digital human central control module is used to send a user identity verification instruction to the digital human control center when the user face-to-face review is completed; The digital human control center is further used to pull the user's real-time audio and video stream from the streaming media service module upon receiving the user identity verification instruction, and send the user's real-time audio and video stream to the identity verification module; The identity verification module is used to verify the identity of the user to be verified based on the real-time audio and video stream of the user; The front-end digital human central control module is used to send a websocket connection request to the digital human control center; The digital human control center is used to establish a connection with the front-end digital human central control module upon receiving the websocket connection request, and to feedback the real-time communication address and session description information of the streaming media service module upon successful connection, wherein the real-time communication address and the session description information are the webrtc address and SDP information respectively; The front-end digital human central control module is also used to send a webrtc connection request to the streaming media service module when receiving the real-time communication address and the session description information; The streaming media service module is used to establish a connection with the front-end digital human central control module upon receiving the webrtc connection request, start the transcoding and recording process when the connection is successful, and return a connection success instruction, wherein the transcoding and recording process is a process of transcoding from webrtc to rtmp and recording; The front-end digital human central control module is further used to send a digital human initialization instruction to the digital human control center through the first communication protocol when receiving the connection success instruction; The digital human control center is further used to perform digital human initialization processing upon receiving the digital human initialization instruction, and to feedback an initialization success instruction upon completion of the initialization; The front-end digital human central control module is also used to display the digital human image and send a digital human face-to-face review instruction to the digital human control center upon receiving the initialization success instruction; The system also includes: a digital human engine module and a speech recognition module; The digital human control center is further used to establish an audio and video stream connection with the streaming media service module to obtain the audio and video stream when receiving the digital human initialization instruction; The digital human control center is also used to establish a communication connection with the digital human engine module, and when the connection is successful, send the real-time communication address to the digital human engine module; The digital human engine module is used to establish a connection with the streaming media service module through a second communication protocol when receiving the real-time communication address; The digital human engine module is further used to convert the audio and video stream into a first digital human image audio and video stream when the connection is successful, and push the first digital human image audio and video stream to the streaming media service module; The digital human control center is also used to establish a connection with the voice recognition module and the identity verification module, and when the connection is successful, feedback an initialization success instruction to the front-end digital human central control module; The identity verification module includes: a liveness detection module; The digital human control center is further used to send the user real-time video stream in the user real-time audio and video stream to the living body detection module; The liveness detection module is used to generate a liveness test result based on the real-time video stream of the user, and send the liveness test result to the front-end digital human central control module; The front-end digital human central control module is also used to determine the number of face matching failures based on the liveness test result; The front-end digital human central control module is also used to display a face-to-face review failure prompt message to the user to be reviewed when the number of face matching failures reaches a first risk number threshold; The identity verification module also includes: a voiceprint detection module; The digital human control center is further used to send the user's real-time audio stream in the user's real-time audio and video stream to the voiceprint detection module; The voiceprint detection module is used to generate voiceprint information corresponding to the real-time audio stream of the user, and search for the identity information of the current auditing user based on the voiceprint information; The voiceprint detection module is further used to compare the identity information with the user identity information of the user to be reviewed, and when the comparison fails, send the voiceprint comparison result to the front-end digital human central control module; The front-end digital human central control module is further used to record the number of identity matching failures if the voiceprint comparison result shows that the current audited user does not match the user to be audited; The front-end digital human central control module is also used to display a face-to-face review failure prompt message to the user to be reviewed when the number of identity matching failures reaches a second risk number threshold.

2. The system according to claim 1, characterized in that The digital human control center is used to obtain the user identity information of the user to be reviewed based on the digital human face-to-face review instruction when receiving the digital human face-to-face review instruction; The digital human control center is further used to generate a face-to-face interview start instruction based on the user identity information, and send the face-to-face interview start instruction to the intelligent robot module; The intelligent robot module is used to generate a text response file based on the user identity information in the face-to-face review start instruction, and send the text response file to the digital human control center; The digital human control center is also used to conduct user interviews based on the text response file.

3. The system according to claim 2, characterized in that The system further comprises: a speech synthesis module; The digital human control center is further used to call the speech synthesis module to convert the text response file into a speech response file; The digital human control center is further used to send the voice response file to the digital human engine module through the first communication protocol; The digital human engine module is used to convert the voice response file into a second digital human image audio and video stream; The digital human engine module is also used to push the second digital human image audio and video stream to the streaming media service module and respond to the user that the face-to-face review is completed.

4. The system according to claim 1, characterized in that The living body detection module is used to determine whether there is a human face in each frame of the user's real-time video stream; The liveness detection module is further configured to display gesture action prompt information if it exists, and the user to be reviewed responds based on the gesture action prompt information; The liveness detection module is also used to compare the face with the face on the ID card of the user to be reviewed when the response is passed, and generate a liveness test result when the comparison fails.

5. The system according to claim 3, characterized in that The front-end digital human central control module is also used to collect user audio information of the user to be reviewed when the identity review of the user to be reviewed is passed, and send the user audio information to the streaming media service module, wherein the user audio information is information generated when the user to be reviewed answers questions asked by the digital human; The digital human control center is also used to pull the user audio information from the streaming media service module in real time and send the user audio information to the voice recognition module; The speech recognition module is further used to convert the user audio information into user text information, and return the user text information to the digital human control center; The digital human control center is also used to send the user text information to the intelligent robot module; The intelligent robot module is used to generate an intelligent robot text speech file based on the user text information, and send the intelligent robot text speech file to the digital human control center; The digital human control center is further used to call the speech synthesis module to convert the intelligent robot text speech file into an intelligent robot voice file, and send the intelligent robot voice file to the digital human engine module; The digital human engine module is also used to convert the intelligent robot voice file into a third digital human image audio and video stream, and push the third digital human image audio and video stream to the streaming media service module to conduct face-to-face questioning with the user to be reviewed.

6. The system according to claim 5, characterized in that The system further comprises: a risk assessment module; The intelligent robot module is further used to label the intelligent robot text speech in the intelligent robot text speech file to obtain a number of risk labels; The risk assessment module is used to determine whether the risk label meets the risk requirements; The risk assessment module is also used to generate an alarm message to instruct manual review if the risk is not met.

Images