Product recommendation virtual digital human interaction method, apparatus and device, and program product

Abstract

The invention provides a product recommendation virtual digital human interaction method, device and equipment and a program product, relates to the field of artificial intelligence and is applied to terminal equipment, a product recommendation virtual digital human runs on the terminal equipment, and login authorization information of a user is acquired in response to login operation of the user; obtaining user interaction information in response to an interaction operation of a user; determining user account information according to the login authorization information; inputting the user account information into the customer classification model, and outputting customer category information; inputting the user interaction information and the customer category information into the multi-modal large model, and outputting digital human feedback information; generating voice reply information and / or image reply information according to the digital human feedback information; controlling an audio output unit to play voice content based on the voice reply information; and / or, based on the image reply information, controlling the display unit to display the image content through the virtual digital person, thereby solving the problem that the information replied by the digital person to the user is poor in accuracy.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and particularly to a product recommendation virtual digital human interaction method, device, equipment, and program product. Background Art

[0002] With the development of artificial intelligence technology, digital human technology is widely used in human-computer interaction systems. By interacting with users, corresponding reply information is provided to users. For example, when the human-computer interaction system is a customer service Q&A system, the digital human can reply to the user with corresponding answer information based on the user's question content. Another example is that when the human-computer interaction system is a product recommendation system, the digital human will recommend product information with a high degree of matching to the user according to the user needs and user information provided by the user.

[0003] In the prior art, when a human-computer interaction system uses a digital human to reply to relevant information to a user, it usually adopts the method of calling the information pre-stored in the database to achieve interaction with the user.

[0004] However, the solution of the prior art results in the problem of poor accuracy when the digital human replies to the user. Summary of the Invention

[0005] This application provides a product recommendation virtual digital human interaction method, device, equipment, and program product to solve the technical problem of poor accuracy when the digital human replies to the user.

[0006] In a first aspect, this application provides a product recommendation virtual digital human interaction method, which is applied to a terminal device on which a product recommendation virtual digital human runs. The method includes: in response to a user's login operation, obtaining the user's login authorization information; in response to a user's interaction operation, obtaining user interaction information; determining the user's user account information according to the login authorization information; inputting the user account information into a customer classification model to output customer category information; inputting the user interaction information and the customer category information into a multi-modal large model to output digital human feedback information; generating a voice reply information and / or an image reply information according to the digital human feedback information; controlling an audio output unit to play voice content based on the voice reply information; and / or controlling a display unit to display image content through the virtual digital human based on the image reply information.

[0007] In a second aspect, this application provides a product recommendation virtual digital human interaction device, which is applied to a terminal device on which a product recommendation virtual digital human runs. The device includes:

[0008] An acquisition unit, configured to acquire the user's login authorization information in response to the user's login operation; and acquire user interaction information in response to the user's interaction operation.

[0009] A processing unit, configured to determine the user's user account information according to the login authorization information; input the user account information into a customer classification model to output customer category information; input the user interaction information and the customer category information into a multi-modal large model to output digital human feedback information; and generate a voice reply information and / or an image reply information according to the digital human feedback information.

[0010] A control unit, configured to control an audio output unit to play voice content based on the voice reply information; and / or control a display unit to display image content through a virtual digital human based on the image reply information.

[0011] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0012] The memory stores computer-executable instructions;

[0013] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the first aspect and / or various possible implementation manners of the first aspect as above.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, they are used to implement the first aspect and / or various possible implementation manners of the first aspect as above.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the first aspect and / or various possible implementation manners of the first aspect as above.

[0016] The product recommendation virtual digital human interaction method, device, equipment and program product provided by the present application are applied to a terminal device on which a product recommendation virtual digital human runs. By responding to the user's login operation, the user's login authorization information is obtained; by responding to the user's interaction operation, user interaction information is obtained; according to the login authorization information, the user's user account information is determined; the user account information is input into a customer classification model, and customer category information is output; the user interaction information and the customer category information are input into a multi-modal large model, and digital human feedback information is output; according to the digital human feedback information, a voice reply information and / or an image reply information are generated; based on the voice reply information, an audio output unit is controlled to play voice content; and / or, based on the image reply information, a display unit is controlled to display image content through the virtual digital human. On the basis of obtaining the user's login authorization information by responding to the user's login operation and obtaining the user interaction information by responding to the user's interaction operation, the user's user account information is obtained by processing the login authorization information, and then the customer category information of the user is determined according to the user account information. Further, the user interaction information and the customer category information are input into the multi-modal large model, so that the multi-modal large model performs semantic analysis processing on the input user interaction information and customer category information, and then outputs corresponding digital human feedback information. Further, the terminal device processes the digital human feedback information, that is, generates a voice reply information and / or an image reply information; still further, the terminal device controls the audio output unit to play voice content based on the voice reply information; and / or, based on the image reply information, controls the display unit to display image content through the virtual digital human, that is, realizes the interaction with the user, that is, replies relevant information to the user, and solves the problem of poor accuracy of the information replied by the digital human to the user caused by the existing technical solutions. Description of the Drawings

[0017] The drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0018] Figure 1 It is an application scenario diagram of the product recommendation virtual digital human interaction method provided by an embodiment of the present application;

[0019] Figure 2 It is a flowchart of the product recommendation virtual digital human interaction method provided by an embodiment of the present application;

[0020] Figure 3 It is a flowchart of the product recommendation virtual digital human interaction method provided by another embodiment of the present application;

[0021] Figure 4Schematic structural diagram of a product recommendation virtual digital human interaction device provided by an embodiment of the present application;

[0022] Figure 5 Schematic structural diagram of an electronic device provided by the present application.

[0023] Through the above-mentioned drawings, specific embodiments of the present application have been shown, and there will be more detailed descriptions hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. Specific Embodiments

[0024] Here, exemplary embodiments will be described in detail, and their examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0025] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data and other processing all comply with the relevant laws, regulations, and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good customs, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0026] And the present application involves big data analysis of user information (including but not limited to personal biometric characteristics, identity data, consumption data, asset data, electronic terminal operation data, etc.), and uses artificial intelligence technology for automated decision-making. For a technical solution that makes a decision having a significant impact on personal rights and interests based on the results of automated decision-making, an operation entrance is provided for the user to choose to agree or refuse the results of automated decision-making; if the user chooses to refuse, the expert decision-making process will be entered.

[0027] It should be noted that the product recommendation virtual digital human interaction method, device, equipment, and program product provided by the present application can be used in the field of artificial intelligence or any field other than artificial intelligence. The application fields of the product recommendation virtual digital human interaction method, device, equipment, and program product in the present application are not limited.

[0028] The specific application scenario of the present application is a scenario where a user interacts with a virtual digital human through a terminal device. Figure 1This is an application scenario diagram of the product recommendation virtual digital human interaction method provided by the embodiments of the present application. As Figure 1 shown, the execution subject of the method provided by the embodiments of the present application can be an electronic control unit, a terminal device, or a server. Taking the terminal device as the execution subject for illustration, the terminal device responds to the user's operation, obtains the user's login authorization information and user interaction information, and then processes the login authorization information and user interaction information based on the customer classification model and the multimodal large model provided by the embodiments of the present application, that is, outputs digital human feedback information; further, according to the digital human feedback information, generates voice reply information and / or image reply information, and then controls the audio output unit to play voice content based on the voice reply information; and / or, based on the image reply information, controls the display unit to display image content through the virtual digital human; that is, the function of the product recommendation virtual digital human interacting with the user and the function of recommending products to the user are realized.

[0029] In the prior art, when a human-computer interaction system (product recommendation system) uses a digital human to reply relevant information to a user and / or recommend products to the user, it usually adopts the method of calling the information pre-stored in the database to realize interaction with the user; since the amount of information pre-stored in the database is limited, when the match degree between the user's question and the preset questions stored in the database is low, the information replied by the digital human to the user will have the problem of poor accuracy.

[0030] The product recommendation virtual digital human interaction method, device, equipment and program product provided by the present application are intended to solve the above technical problems in the prior art.

[0031] The following uses specific embodiments to detail the technical solutions of the present application and how the technical solutions of the present application solve the above technical problems. These specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0032] Figure 2 This is a flowchart of the product recommendation virtual digital human interaction method provided by an embodiment of the present application. As Figure 2 shown, the execution subject of the product recommendation virtual digital human interaction method provided by this embodiment can be an electronic control unit, a terminal device, or a server. Exemplarily, this embodiment takes the terminal device as the execution subject of the method of this embodiment for illustration. The product recommendation virtual digital human interaction method provided by this embodiment is applied to the terminal device, and a product recommendation virtual digital human runs on the terminal device, including the following steps:

[0033] Step S101, in response to the user's login operation, obtain the user's login authorization information.

[0034] Exemplarily, in response to a user's login operation, for example, the terminal device responds to the user's personal account login operation via a keyboard or a display unit (touchable), and obtains the user's login authorization information; specifically, for example, in response to the user's login operation, the terminal device displays the login authorization information to be authorized to the user via the display unit, and then the user performs a login authorization operation via the keyboard or the display unit (touchable), and further, based on the user's login authorization operation, the terminal device obtains the user's login authorization information from the server.

[0035] Step S102: In response to the user's interaction operation, obtain the user interaction information.

[0036] Exemplarily, after the terminal device obtains the user's login authorization information, it generates a product recommendation virtual digital human that matches the user. Then, in response to the user's interaction operation, the terminal device can obtain the user interaction information. Specifically, for example, after the terminal device generates a product recommendation virtual digital human that matches the user, it collects the user's interaction operations (body movements, audio, key inputs) via an image acquisition unit, and / or an audio acquisition unit, and / or a text input unit (keyboard), and then obtains the user interaction information.

[0037] Step S103: According to the login authorization information, determine the user's user account information.

[0038] Exemplarily, the terminal device analyzes the user's investment preferences, and / or consultation preferences, and / or personal assets based on the obtained login authorization information, and then the user account information can be obtained; specifically, for example, if it is determined according to the login authorization information that the product invested by the user is a stable product, the consultation content is about stable products, and the personal assets are within the stable asset range, then the above information is summarized to obtain the user's user account information.

[0039] Step S104: Input the user account information into the customer classification model and output the customer category information.

[0040] Exemplarily, the customer classification model is a pre-trained model. By processing the input user account information and the pre-trained reference account information, the customer category information can be output. Specifically, for example, the product invested by the user corresponding to the user account information info_1 is a stable product, the consultation content is about the stable product, and the personal assets are within the stable asset range. The product invested by the user corresponding to the user account information info_2 is a stable product, the consultation content is about the risky product, and the personal assets are within the stable asset range. Quantitative analysis is performed on the user account information info_1 and the user account information info_2 respectively according to the reference account information, and then the quantization value data_1 corresponding to the user account information info_1 is 5, and the quantization value data_2 corresponding to the user account information info_2 is 8. For example, the reference quantization value data_0 is 6. Furthermore, according to the quantization value data_1 being less than the reference quantization value data_0, the corresponding customer category is a conservative financial management customer, that is, the customer category information corresponding to the user account information info_1 is obtained. According to the quantization value data_2 being greater than the reference quantization value data_0, the corresponding customer category is a risky financial management customer, that is, the customer category information corresponding to the user account information info_2 is obtained.

[0041] Step S105: Input the user interaction information and the customer category information into the multi-modal large model to output the digital human feedback information.

[0042] Exemplarily, after obtaining the user interaction information and the customer category information, the terminal device inputs the user interaction information and the customer category information into the multi-modal large model. The multi-modal large model can process the user interaction information and the customer category information according to the preset processing logic, and then output the digital human feedback information. Specifically, for example, the user interaction information is "Am I suitable to buy product AAA", and the customer category corresponding to the customer category information is a conservative financial management customer. The preset processing logic is to analyze the user interaction information according to the customer category information. Furthermore, based on the customer category of this customer being a conservative financial management customer, the multi-modal large model determines whether this customer is suitable to buy product AAA. If product AAA is a stable product, then this customer is suitable to buy product AAA. If product AAA is a risky product, then this customer is not suitable to buy product AAA. Furthermore, the digital human feedback information "You are suitable to buy product AAA" or "You are not suitable to buy product AAA" is output.

[0043] Furthermore, in a possible implementation manner, the user interaction information includes the user consultation content and the user interaction duration; the multi-modal large model includes a first intention determination module, a first information processing module, and a second information processing module; the specific implementation manner of step S105 includes:

[0044] Step S1051: Input the user interaction duration into the first intention determination module to generate the first user intention information.

[0045] Step S1052: Input the first user intention information and the user consultation content into the first information processing module to output the user intention consultation content.

[0046] Exemplarily, the user interaction information includes the user consultation content and the user interaction duration. There is a positive correlation between the user interaction duration and the degree of the user's attention to the product, that is, the longer the user interaction duration, the higher the degree of the user's attention to the product corresponding to the user consultation content. Further, the first intention determination module generates the corresponding degree of attention to the product according to the input user interaction duration, that is, generates the first user intention information. Then, input the first user intention information and the user consultation content into the first information processing module. The first information processing module performs user intention analysis on the user consultation content according to the first user intention information, and then outputs the user intention consultation content. The longer the user interaction duration, the higher the correlation between the generated user intention consultation content and the user consultation content. The shorter the user interaction duration, the lower the correlation between the generated user intention consultation content and the user consultation content. For example, the user interaction information is "Am I suitable to buy product AAA?". If the user interaction duration is longer, the generated user intention consultation content is another product "AAB product" close to "product AAA". If the user interaction duration is shorter, the generated user intention consultation content is another product "CCC product" not close to "product AAA". Among them, "product AAA" is a stable product, "AAB product" is a stable product, and "CCC product" is a risky product.

[0047] Step S1053: Input the user intention consultation content and the customer category information into the second information processing module to output the digital human feedback information.

[0048] Exemplarily, after obtaining the user intention consultation content, the second information processing module further processes the user intention consultation content according to the customer category information, so that the digital human feedback information determined based on the user intention consultation content matches the customer category. Specifically, for example, based on the customer category of this customer being a conservative wealth management customer, if the product corresponding to the user intention consultation content is the CCC product (risky product), the output digital human feedback information will be bound with the corresponding prompt identifier. For example, the digital human feedback information is "The other product recommended by the system is the CCC product. Please choose carefully."; if the product corresponding to the user intention consultation content is the AAB product (stable product), the output digital human feedback information will be bound with the corresponding prompt identifier. For example, the digital human feedback information is "The other product recommended by the system is the AAB product. Please choose according to your needs."

[0049] In the steps of this embodiment, by parsing and processing the user interaction duration, the consultation intention of the user is determined, that is, the first user intention information is generated. Furthermore, in combination with the user consultation content, the user intention consultation content is output, and further, the user intention consultation content is processed by the customer category information, and then the digital human feedback information matching the customer consultation content is obtained, improving the accuracy of the voice reply information and / or image reply information generated in the subsequent steps.

[0050] Furthermore, in another possible implementation manner, the user interaction information includes the user consultation content and the user interaction frequency; the multimodal large model includes a second intention determination module and a third information processing module; the specific implementation manner of step S105 includes:

[0051] Step S105A, the second intention determination module screens the user consultation content according to the user interaction frequency and outputs the second user intention information.

[0052] Exemplarily, the user interaction information includes the user consultation content and the user interaction frequency; there is a positive correlation between the user interaction frequency and the user's attention degree to the product, that is, if the user interaction frequency is higher, it indicates that the user's attention degree to the product corresponding to the user consultation content is higher; furthermore, the second intention determination module counts two or more products corresponding to the user consultation content, that is, the user interaction frequency corresponding to each product is obtained; further, according to the user interaction frequency corresponding to each product, the average interaction frequency is calculated by summing and averaging; furthermore, the size relationship between the average interaction frequency and the user interaction frequency is judged, the product corresponding to the user interaction frequency greater than the average interaction frequency is determined, and then, according to the product information corresponding to the determined product, the second user intention information is output. For example, if the determined product is the "AAA product", the products corresponding to the second user intention information are the "AAB product" and the "AAC product", where the "AAA product", the "AAB product" and the "AAC product" are all stable products.

[0053] Step S105B, the second user intention information and the customer category information are input into the third information processing module, and the digital human feedback information is output.

[0054] Exemplarily, after obtaining the second user intention information, the third information processing module further processes the second user intention information according to the customer category information, so that the digital human feedback information determined based on the second user intention information matches the customer category. Specifically, for example, based on the customer category of this customer being a conservative financial management customer, if the product corresponding to the second user intention information is a CCC product (a risky product), the output digital human feedback information will be bound with a corresponding prompt identifier. For example, the digital human feedback information is "The other product recommended by the system is a CCC product. Please choose carefully."; if the product corresponding to the second user intention information is an AAB product (a stable product), the output digital human feedback information will be bound with a corresponding prompt identifier. For example, the digital human feedback information is "The other product recommended by the system is an AAB product. Please choose according to your needs."

[0055] In the steps of this embodiment, by calculating the average interaction frequency according to the user interaction frequency corresponding to each product, and then by comparing the size relationship between the user interaction frequency of each product and the average interaction frequency, the product corresponding to the user interaction frequency greater than the average interaction frequency is determined, that is, the determined product is the product with a high degree of user attention. Then, based on the determined product, the corresponding second user intention information is output. Further, the second user intention information is processed by the customer category information, and then the digital human feedback information matching the customer consultation content is obtained, improving the accuracy of the voice reply information and / or image reply information generated in the subsequent steps; among them, the average interaction frequency changes in real time with the customer consultation content, improving the matching degree between the second user intention information and the customer consultation content.

[0056] Step S106, generate voice reply information and / or image reply information according to the digital human feedback information.

[0057] Step S107, based on the voice reply information, control the audio output unit to play the voice content; and / or, based on the image reply information, control the display unit to display the image content through the virtual digital human.

[0058] Exemplarily, after obtaining the digital human feedback information, the terminal device decodes the digital human feedback information, and then generates voice reply information and / or image reply information; further, based on the voice reply information, the audio output unit can be controlled to play the voice content; and / or, based on the image reply information, the display unit is controlled to display the image content through the virtual digital human.

[0059] Further, in a possible implementation manner, the specific implementation manner of step S106 includes:

[0060] Step S1061, confirm the user information receiving ability according to the digital human feedback information.

[0061] Step S1062, if the user information receiving ability is the audio information receiving ability, then according to the digital human feedback information, generate a voice reply message.

[0062] Step S1063, if the user information receiving ability is the image information receiving ability, then according to the digital human feedback information, generate a digital human limb movement image reply message and a storable image reply message.

[0063] Step S1064, if the user information receiving ability is the audio information receiving ability and the image information receiving ability, then according to the digital human feedback information, generate a voice reply message, a digital human limb movement image reply message and a storable image reply message.

[0064] Exemplarily, after obtaining the digital human feedback information, the terminal device confirms the user information receiving ability according to the digital human feedback information; specifically, for example, the terminal device determines whether the user is a disabled person according to the digital human feedback information. If the user is a disabled person, then further determine whether the user is a deaf-mute disabled person or a blind person, and then determine the user information receiving ability. That is, if the user is a deaf-mute disabled person, confirm that the user information receiving ability is the image information receiving ability. If the user is a blind person, confirm that the user information receiving ability is the audio information receiving ability. If the user is not a disabled person, confirm that the user information receiving ability is the audio information receiving ability and the image information receiving ability. Further, if the user information receiving ability is the audio information receiving ability, the terminal device generates a voice reply message according to the digital human feedback information; if the user information receiving ability is the image information receiving ability, the terminal device generates a digital human limb movement image reply message and a storable image reply message according to the digital human feedback information; if the user information receiving ability is the audio information receiving ability and the image information receiving ability, the terminal device generates a voice reply message, a digital human limb movement image reply message and a storable image reply message according to the digital human feedback information.

[0065] Correspondingly, if the user is a deaf-mute disabled person, after the terminal device generates a digital human limb movement image reply message and a storable image reply message, the specific implementation manner of step S107 includes:

[0066] Step S1071, based on the digital human limb movement image reply message, control the virtual digital human to perform limb movements and display them on the display unit.

[0067] Step S1072, based on the storable image reply message, display the storable image on the display unit and display the corresponding storable interaction component.

[0068] Exemplarily, the terminal device generates corresponding digital human limb movement control instructions based on the digital human limb movement images in the reply information, and then controls the virtual digital human to perform limb movements and display them on the display unit, where the limb movement is sign language that can be understood by deaf and mute disabled people; the terminal device displays the storable images on the display unit based on the storable image reply information and displays the corresponding storable interaction components; after the user clicks on the "storable interaction component", the storable images are stored in the user's personal cloud storage or personal terminal device.

[0069] In the steps of this embodiment, by judging the user information receiving ability of the user, the reply form of the information generated by the terminal device is determined, and then the display unit of the terminal device is controlled to display the virtual digital human performing limb movements and storable images, and / or the audio output unit is controlled to play voice content, so that the virtual digital human can interact not only with non-disabled people, but also with deaf and mute disabled people and blind people, improving the user experience.

[0070] The product recommendation virtual digital human interaction method provided in this embodiment is applied to a terminal device on which a product recommendation virtual digital human runs. By responding to the user's login operation, the user's login authorization information is obtained; by responding to the user's interaction operation, the user interaction information is obtained; according to the login authorization information, the user's user account information is determined; the user account information is input into the customer classification model to output customer category information; the user interaction information and customer category information are input into the multi-modal large model to output digital human feedback information; according to the digital human feedback information, voice reply information and / or image reply information are generated; based on the voice reply information, the audio output unit is controlled to play voice content; and / or, based on the image reply information, the display unit is controlled to display image content through the virtual digital human. On the basis of obtaining the user's login authorization information by responding to the user's login operation and obtaining the user interaction information by responding to the user's interaction operation, the user's user account information is obtained by processing the login authorization information, and then the user's customer category information is determined according to the user account information. Further, the user interaction information and customer category information are input into the multi-modal large model to realize the semantic analysis and processing of the input user interaction information and customer category information by the multi-modal large model, and then the corresponding digital human feedback information is output. Further, the terminal device generates voice reply information and / or image reply information by processing the digital human feedback information; still further, the terminal device controls the audio output unit to play voice content based on the voice reply information; and / or, based on the image reply information, controls the display unit to display image content through the virtual digital human, that is, the interaction with the user is realized, that is, relevant information is replied to the user, solving the problem of poor accuracy of the information replied by the digital human to the user caused by the existing technical solutions.

[0071] Furthermore, during the interaction between the user and the virtual digital human, the system responds in real time to the user's feedback click operation to generate user feedback information. Then, the user feedback information, user interaction information, and customer category information are input into the multi-modal large model to output digital human feedback information. For example, the terminal device displays "BBC products" to the user through the display unit, along with the corresponding feedback interaction components "Click to view" and "Not interested". If the user clicks on the "Click to view" interaction component, user feedback information (the user is interested) is generated. If the user clicks on the "Not interested" interaction component, user feedback information (the user is not interested) is generated. Then, after inputting the user feedback information, user interaction information, and customer category information into the multi-modal large model, the multi-modal large model outputs the corresponding digital human feedback information. For example, if the user feedback information corresponds to the user being interested, the corresponding digital human feedback information includes product information of products similar to "BBC products". If the user feedback information corresponds to the user being not interested, the corresponding digital human feedback information does not include product information of products similar to "BBC products". Specifically, for example, in another possible implementation, the user feedback information is quantified. The first quantization coefficient corresponding to "the user is interested" is 1, and the second quantization coefficient corresponding to "the user is not interested" is 0.5. Based on the user's customer category information, a reference quantization coefficient of 0.5 is determined. Then, based on the user being interested, the proportion of product information of products similar to "BBC products" in the digital human feedback information output by the multi-modal large model is 50% (the product of the first quantization coefficient and the reference quantization coefficient). Based on the user being not interested, the proportion of product information of products similar to "BBC products" in the digital human feedback information output by the multi-modal large model is 25% (the product of the second quantization coefficient and the reference quantization coefficient).

[0072] Figure 3 The flowchart of the product recommendation virtual digital human interaction method provided by another embodiment of the present application is as follows Figure 3 As shown, the product recommendation virtual digital human interaction method provided in this embodiment is based on the product recommendation virtual digital human interaction method provided in the embodiment shown Figure 2 and further refines step S104. Then, the product recommendation virtual digital human interaction method provided in this embodiment includes the following steps:

[0073] Step S201: In response to the user's login operation, obtain the user's login authorization information.

[0074] Step S202: In response to the user's interaction operation, obtain user interaction information.

[0075] Step S203: According to the login authorization information, determine the user's user account information; wherein, the user account information includes historical investment information, historical consultation information, and user asset information.

[0076] Step S204: Obtain reference investment information, reference consultation information, and reference asset information through a customer classification model; among them, the reference investment information includes first-class investment information and second-class investment information, the reference consultation information includes first-class consultation information and second-class consultation information, and the reference asset information includes first-class asset information and second-class asset information.

[0077] Step S205: Obtain a first-category distance based on historical investment information, historical consultation information, user asset information, first-class investment information, first-class consultation information, and first-class asset information.

[0078] Step S206: Obtain a second-category distance based on historical investment information, historical consultation information, user asset information, second-class investment information, second-class consultation information, and second-class asset information.

[0079] Step S207: Output customer category information based on the first-category distance and the second-category distance.

[0080] Exemplarily, the terminal device obtains reference investment information, reference consultation information, and reference asset information based on the reference information preset by the customer classification model; among them, the reference investment information includes first-class investment information and second-class investment information, the reference consultation information includes first-class consultation information and second-class consultation information, and the reference asset information includes first-class asset information and second-class asset information; furthermore, vector calculations are performed on historical investment information, historical consultation information, user asset information, first-class investment information, first-class consultation information, and first-class asset information to obtain a first-category distance; vector calculations are performed on historical investment information, historical consultation information, user asset information, second-class investment information, second-class consultation information, and second-class asset information to obtain a second-category distance; furthermore, by judging the magnitude relationship between the first-category distance and the second-category distance, the customer category information is determined and output. For example, if the first-category distance is less than or equal to the second-category distance, the reference category information corresponding to the first-category distance is determined as the customer category information; if the first-category distance is greater than the second-category distance, the reference category information corresponding to the second-category distance is determined as the customer category information.

[0081] Demonstratively, in a possible implementation manner, the reference investment information includes first-class investment information, second-class investment information, and third-class investment information, and the first-class investment information includes a first-class investment mean vector u a1 , the second-class investment information includes a second-class investment mean vector u a2 , and the third-class investment information includes a third-class investment mean vector u a3 ; the reference consultation information includes first-class consultation information, second-class consultation information, and third-class consultation information, and the first-class consultation information includes a first-class consultation mean vector u b1, the second type of consultation information includes the second type of consultation mean vector u b2 , the third type of consultation information includes the third type of consultation mean vector u b3 ; The reference asset information includes the first type of asset information, the second type of asset information, and the third type of asset information. The first type of asset information includes the first type of asset mean vector u c1 , the second type of asset information includes the second type of asset mean vector u c2 , the third type of asset information includes the third type of asset mean vector u c3 ; Further, the user account information includes historical investment information, historical consultation information, and user asset information. The historical investment information includes the first type of historical investment information, the second type of historical investment information, and the third type of historical investment information. The first type of historical investment information includes the first type of historical investment mean vector v a1 , the second type of historical investment information includes the second type of historical investment mean vector v a2 , the third type of historical investment information includes the third type of historical investment mean vector v a3 ; The historical consultation information includes the first type of historical consultation information, the second type of historical consultation information, and the third type of historical consultation information. The first type of historical consultation information includes the first type of historical consultation mean vector v b1 , the second type of historical consultation information includes the second type of historical consultation mean vector v b2 , the third type of historical consultation information includes the third type of historical consultation mean vector v b3 ; The user asset information includes the user asset mean vector v c1 .

[0082] Further, substituting the first type of investment mean vector u a1 , the first type of consultation mean vector u b1 , the first type of asset mean vector u c1 , the first type of historical investment mean vector v a1 , the first type of historical consultation mean vector v b1 , and the user asset mean vector v c1 into formula (1), the first category distance dist_1 can be calculated;

[0083] (1)

[0084] Substituting the second type of investment mean vector u a2 , the second type of consultation mean vector u b2 , the second type of asset mean vector u c2 , the second type of historical investment mean vector v a2 , the second type of historical consultation mean vector v b2 , and the user asset mean vector v c1Substitute into formula (2), and the distance of the second category dist_2 can be calculated accordingly;

[0085] (2)

[0086] Substitute the mean vector u of the third - type investment a3 , the mean vector u of the third - type consultation b3 , the mean vector u of the third - type assets c3 , the mean vector v of the third - type historical investment a3 , the mean vector v of the third - type historical consultation b3 , and the mean vector v of the user's assets c1 into formula (3), and the distance of the third category dist_3 can be calculated accordingly;

[0087] (3)

[0088] Furthermore, compare the magnitudes of the distance of the first category dist_1, the distance of the second category dist_2, and the distance of the third category dist_3. For example, if the distance of the second category dist_2 is the smallest, then determine the reference category information corresponding to the distance of the second category as the customer category information. That is, the smaller the distance value, the higher the similarity between the corresponding reference investment information, reference consultation information, reference asset information and the user account information. Furthermore, determine the reference category information corresponding to the reference investment information, reference consultation information, and reference asset information as the customer category information.

[0089] In the steps of this embodiment, by calculating the vector distance between the reference investment information, reference consultation information, reference asset information and the user account information, accurate customer category information is determined, thereby improving the accuracy of the voice reply information and / or image reply information generated in the subsequent steps.

[0090] Further, in another possible implementation manner, the specific implementation steps of step S207 include: obtaining the inverse distance of the first category according to the distance of the first category; obtaining the inverse distance of the second category according to the distance of the second category; calculating the weights of the first category and the second category for the inverse distance of the first category and the inverse distance of the second category; determining the product information of the first category according to the weight of the first category; determining the product information of the second category according to the weight of the second category; and generating customer category information according to the product information of the first category and the product information of the second category.

[0091] Exemplarily, take the reciprocal of the distance of the first category dist_1 to obtain the inverse distance of the first category , take the reciprocal of the distance of the second category dist_2 to obtain the inverse distance of the second category ; furthermore, calculate the weight W1 of the first category and the weight W2 of the second category according to formula (4);

[0092] (4)

[0093] Further, according to the first category weight W1, the first category product information is obtained, and according to the second category weight W2, the second category product information is obtained; furthermore, according to the first category product information and the second category product information, customer category information is generated; specifically, for example, if the first category product is a stable product and the second category product is a risky product, then according to the first category weight W1, the first category product information obtained is that the user has a tendency of W1 to choose the stable product, and according to the second category weight W2, the second category product information obtained is that the user has a tendency of W2 to choose the risky product; furthermore, according to the first category product information and the second category product information, the generated customer category information is that the user has a tendency of W1 to choose the stable product and a tendency of W2 to choose the risky product.

[0094] In the steps of this embodiment, by further processing the vector distance, the corresponding category weights are obtained respectively, and then according to the category weights, the tendency of the user to choose the corresponding category of products is determined, that is, the customer category information with quantitative data is obtained, which improves the accuracy of the voice reply information and / or image reply information generated in the subsequent steps.

[0095] Step S208, input the user interaction information and the customer category information into the multi-modal large model, and output the digital human feedback information.

[0096] Step S209, generate voice reply information and / or image reply information according to the digital human feedback information.

[0097] Step S210, based on the voice reply information, control the audio output unit to play the voice content; and / or, based on the image reply information, control the display unit to display the image content through the virtual digital human.

[0098] In this embodiment, the implementation manners of steps S201 - S203 and steps S208 - S210 Figure 2 are the same as those of steps S101 - S103 and steps S105 - S107 in the embodiment shown in this application, and will not be elaborated here one by one.

[0099] Figure 4 is a structural schematic diagram of a product recommendation virtual digital human interaction device provided by an embodiment of this application. As Figure 4 shown, the product recommendation virtual digital human interaction device 3 provided in this embodiment is applied to a terminal device, and a product recommendation virtual digital human runs on the terminal device, including:

[0100] An acquisition unit 31, configured to acquire the user's login authorization information in response to the user's login operation; and acquire the user interaction information in response to the user's interaction operation;

[0101] A processing unit 32, configured to determine the user's user account information according to the login authorization information; input the user account information into a customer classification model, and output customer category information; input the user interaction information and the customer category information into a multi-modal large model, and output digital human feedback information; generate a voice reply information and / or an image reply information according to the digital human feedback information;

[0102] A control unit 33, configured to control the audio output unit to play voice content based on the voice reply information; and / or control the display unit to display image content through a virtual digital human based on the image reply information.

[0103] In a possible implementation manner, the user account information includes historical investment information, historical consultation information, and user asset information; when the processing unit 32 inputs the user account information into the customer classification model and outputs the customer category information, it is specifically configured to: obtain reference investment information, reference consultation information, and reference asset information through the customer classification model; wherein, the reference investment information includes first-class investment information and second-class investment information, the reference consultation information includes first-class consultation information and second-class consultation information, and the reference asset information includes first-class asset information and second-class asset information; obtain a first-category distance according to the historical investment information, historical consultation information, user asset information, first-class investment information, first-class consultation information, and first-class asset information; obtain a second-category distance according to the historical investment information, historical consultation information, user asset information, second-class investment information, second-class consultation information, and second-class asset information; output the customer category information according to the first-category distance and the second-category distance.

[0104] In a possible implementation manner, when the processing unit 32 outputs the customer category information according to the first-category distance and the second-category distance, it is specifically configured to: obtain a first-category inverse distance according to the first-category distance; obtain a second-category inverse distance according to the second-category distance; perform a weight calculation on the first-category inverse distance and the second-category inverse distance to obtain a first-category weight and a second-category weight; determine first-category product information according to the first-category weight; determine second-category product information according to the second-category weight; generate the customer category information according to the first-category product information and the second-category product information.

[0105] In a possible implementation, the user interaction information includes the user's consultation content and the user interaction duration; the multi-modal large model includes a first intention determination module, a first information processing module, and a second information processing module; when the processing unit 32 inputs the user interaction information and the customer category information into the multi-modal large model and outputs the digital human feedback information, it is specifically used to: input the user interaction duration into the first intention determination module to generate the first user intention information; input the first user intention information and the user's consultation content into the first information processing module to output the user intention consultation content; input the user intention consultation content and the customer category information into the second information processing module to output the digital human feedback information.

[0106] In a possible implementation, the user interaction information includes the user's consultation content and the user interaction frequency; the multi-modal large model includes a second intention determination module and a third information processing module; when the processing unit 32 inputs the user interaction information and the customer category information into the multi-modal large model and outputs the digital human feedback information, it is specifically used to: the second intention determination module screens the user's consultation content according to the user interaction frequency to output the second user intention information; input the second user intention information and the customer category information into the third information processing module to output the digital human feedback information.

[0107] In a possible implementation, when the processing unit 32 generates the voice reply information and / or the image reply information according to the digital human feedback information, it is specifically used to: confirm the user information reception ability according to the digital human feedback information; if the user information reception ability is the audio information reception ability, generate the voice reply information according to the digital human feedback information; if the user information reception ability is the image information reception ability, generate the digital human body movement image reply information and the storable image reply information according to the digital human feedback information; if the user information reception ability is the audio information reception ability and the image information reception ability, generate the voice reply information, the digital human body movement image reply information, and the storable image reply information according to the digital human feedback information; correspondingly, when the control unit 33 controls the display unit of the terminal device to display the image content through the virtual digital human based on the image reply information, it is specifically used to: control the virtual digital human to perform body movements and display them on the display unit based on the digital human body movement image reply information; display the storable image and the corresponding storable interaction component on the display unit based on the storable image reply information.

[0108] Among them, the acquisition unit 31, the processing unit 32, and the control unit 33 are connected in sequence. The product recommendation virtual digital human interaction device 3 provided in this embodiment can execute the technical solutions of the method embodiments as shown in Figures 2 - 3 any one, and its implementation principle and technical effects are similar, which will not be elaborated here.

[0109] Figure 5 The structural schematic diagram of the electronic device provided for this application. As Figure 5 shown, the electronic device 50 provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the device 50 further includes a communication component 503. Among them, the processor 501, the memory 502, and the communication component 503 are connected through a bus 504.

[0110] In the specific implementation process, at least one processor 501 executes the computer-executable instructions stored in the memory 502, so that at least one processor 501 executes the above-mentioned method.

[0111] For the specific implementation process of the processor 501, reference can be made to the above method embodiment, and its implementation principle and technical effect are similar, which will not be elaborated here in this embodiment.

[0112] In the above embodiment, it should be understood that the processor may be a central processing unit (English: Central Processing Unit, abbreviated: CPU), or other general-purpose processors, digital signal processors (English: Digital Signal Processor, abbreviated: DSP), application-specific integrated circuits (English: Application Specific Integrated Circuit, abbreviated: ASIC), etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.

[0113] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0114] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0115] This application also provides a computer program product, including a computer program, which implements the above-mentioned method when executed by a processor.

[0116] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above-mentioned method is implemented.

[0117] It should be noted that, for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0118] Furthermore, it should be noted that although the steps in the flowchart are shown in sequence according to the indication of the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps has no strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

[0119] It should be understood that the above-mentioned device embodiments are only illustrative, and the devices of the present application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0120] In addition, without special explanation, in each embodiment of the present application, each functional unit / module can be integrated in one unit / module, or each unit / module can exist physically alone, or two or more units / modules can be integrated together. The above-mentioned integrated unit / module can be implemented in the form of hardware or in the form of a software program module.

[0121] When the integrated unit / module is implemented in the form of hardware, the hardware can be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic storage medium or magneto-optical storage medium, such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), etc.

[0122] When the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in various embodiments of this application. And the aforementioned memory includes: USB flash drives, read-only memory (ROM), random access memory (RAM), mobile hard disks, magnetic disks, or optical discs and other various media that can store program codes.

[0123] In the above embodiments, the descriptions of each embodiment have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combinations of these technical features do not conflict, they should all be considered as the scope recorded in this specification.

[0124] Other embodiments of the present application will be readily apparent to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include known common knowledge or conventional technical means in the technical field not disclosed in the present application. The specification and examples are only illustrative, and the true scope and spirit of the present application are pointed out by the following claims.

[0125] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. A product recommendation virtual digital human interaction method, characterized in that: Applied to a terminal device, on which a product recommendation virtual digital human is running, the method comprises: In response to a login operation of a user, obtaining login authorization information of the user; Responding to the user's interactive operation, obtaining user interaction information; Determining user account information of the user according to the login authorization information; Inputting the user account information into a customer classification model and outputting customer category information; Inputting the user interaction information and the customer category information into a multimodal large model, and outputting digital human feedback information; Generating voice reply information and / or image reply information according to the digital human feedback information; Based on the voice reply information, controlling the audio output unit to play the voice content; And / or, based on the image reply information, controlling the display unit to display the image content through a virtual digital human.

2. The method according to claim 1, characterized in that The user account information includes historical investment information, historical consulting information and user asset information; The step of inputting the user account information into a customer classification model and outputting customer category information includes: By using the customer classification model, reference investment information, reference consulting information and reference asset information are obtained; wherein the reference investment information includes first-category investment information and second-category investment information, the reference consulting information includes first-category consulting information and second-category consulting information, and the reference asset information includes first-category asset information and second-category asset information; Obtaining a first category distance according to the historical investment information, the historical consulting information, the user asset information, the first category investment information, the first category consulting information, and the first category asset information; Obtaining a second category distance according to the historical investment information, the historical consulting information, the user asset information, the second category investment information, the second category consulting information, and the second category asset information; The customer category information is output according to the first category distance and the second category distance.

3. The method according to claim 2, characterized in that The step of outputting the customer category information according to the first category distance and the second category distance includes: According to the first category distance, a first category inverse distance is obtained; According to the second category distance, a second category inverse distance is obtained; Performing weight calculation on the first category inverse distance and the second category inverse distance to obtain a first category weight and a second category weight; Determining first category product information according to the first category weight; Determining second category product information according to the second category weight; The customer category information is generated according to the first category product information and the second category product information.

4. The method according to claim 1, characterized in that: The user interaction information includes user consultation content and user interaction duration; the multimodal large model includes a first intention determination module, a first information processing module and a second information processing module; The step of inputting the user interaction information and the customer category information into the multimodal large model and outputting digital human feedback information includes: Inputting the user interaction duration into a first intention determination module to generate first user intention information; Inputting the first user intention information and the user consultation content into the first information processing module, and outputting the user intention consultation content; The user's intended consultation content and the customer category information are input into the second information processing module, and the digital human feedback information is output.

5. The method according to claim 1, characterized in that The user interaction information includes user consultation content and user interaction frequency; the multimodal large model includes a second intention determination module and a third information processing module; The step of inputting the user interaction information and the customer category information into the multimodal large model and outputting digital human feedback information includes: The second intention determination module filters the user consultation content according to the user interaction frequency and outputs second user intention information; The second user intention information and the customer category information are input into the third information processing module, and the digital human feedback information is output.

6. The method according to claim 1, characterized in that Generating voice reply information and / or image reply information according to the digital human feedback information includes: Confirming the user's information receiving capability based on the digital human's feedback information; If the user information receiving capability is audio information receiving capability, generating voice reply information according to the digital human feedback information; If the user information receiving capability is image information receiving capability, generating digital human body movement image reply information and storable image reply information according to the digital human feedback information; If the user information receiving capability is audio information receiving capability and image information receiving capability, then generating voice reply information, digital human body movement image reply information and storable image reply information according to the digital human feedback information; Accordingly, based on the image reply information, controlling the display unit of the terminal device to display the image content through the virtual digital human includes: Based on the digital human body movement image reply information, controlling the virtual digital human to perform body movement and displaying it on the display unit; Based on the storable image reply information, the storable image is displayed on the display unit, and the corresponding storable interactive component is displayed.

7. A product recommendation virtual digital human interaction device, characterized in that: Applied to a terminal device, on which a product recommendation virtual digital person is running, the device comprises: an acquisition unit, configured to acquire the login authorization information of the user in response to the user's login operation; and acquire the user interaction information in response to the user's interaction operation; A processing unit, configured to determine the user account information of the user according to the login authorization information; input the user account information into a customer classification model and output customer category information; input the user interaction information and the customer category information into a multimodal large model and output digital human feedback information; generate voice reply information and / or image reply information according to the digital human feedback information; A control unit is used to control the audio output unit to play the voice content based on the voice reply information; and / or, based on the image reply information, control the display unit to display the image content through a virtual digital human.

8. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed by a processor.

Images