Business data processing method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a business data processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring multi-modal data input by a user, and converting the multi-modal data into high-dimensional features; based on the high-dimensional features, maintaining a session state through a session manager, and generating a recommendation task; based on the recommendation task, performing recall processing on the candidate content of the product to obtain a recommendation result; fusing the environment-related characteristics of the recommendation results and then scoring to obtain a score of each recommendation result; taking a recommendation result corresponding to the highest score as a final recommendation result of the digital person; and outputting a final recommendation result to the user in at least one mode of voice, expression and action based on the digital human, and outputting an interpretable text of the final recommendation result. By adopting the method, the accuracy of digital person recommendation can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a business data processing method and device, computer equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] With the rapid development of artificial intelligence and multimedia technology, recommendation systems have been widely used in e-commerce, finance, entertainment, smart home and many other scenarios, and have become the core engine for improving user experience and business conversion.

[0003] In related technologies, the mainstream implementation is to display the recommendation results in the form of text and pictures or short video cards in APP, web pages and small programs, and the triggering mechanism is usually the page or session behavior (page opening, search, sliding) of the user or the timing push (message push, notification). Some scenarios combine customer service robots or voice assistants, but most recommendation systems are still mainly in the form of passive recommendation, not active recommendation, and the interaction is limited, resulting in inaccurate final push results. SUMMARY

[0004] Therefore, it is necessary to provide a business data processing method, device, computer equipment, computer readable storage medium and computer program product capable of achieving accurate push through digital people in view of the above technical problems.

[0005] In a first aspect, the present application provides a business data processing method, comprising:

[0006] obtaining multi-modal data input by a user, and converting the multi-modal data into high-dimensional features;

[0007] based on the high-dimensional features, maintaining a session state through a dialogue manager, and generating a recommendation task;

[0008] based on the recommendation task, performing recall processing on candidate content of a product to obtain a recommendation result;

[0009] fusing and scoring environment-related features of the recommendation result to obtain a score of each recommendation result;

[0010] taking the recommendation result corresponding to the highest score as the final recommendation result of the digital person;

[0011] based on the digital person, outputting the final recommendation result to the user in at least one of voice, expression and action, and outputting an interpretable text of the final recommendation result.

[0012] In one embodiment, the multi-modal data includes audio data, image data and touch data; and the conversion of the multi-modal data into high-dimensional features comprises:

[0013] transforming the audio data into text data, and determining a text feature of the multi-modal data based on the text data;

[0014] identifying an emotional change in the audio data and a facial emotion of the user in the image data, and determining an emotional feature of the user based on the emotional change and the facial emotion;

[0015] identifying a gesture action of the user in the image data, and determining a behavioral feature of the user based on the gesture action and the touch data.

[0016] In one of the embodiments, the maintaining the conversation state and generating the recommended task by the dialogue manager based on the high-dimensional feature comprises:

[0017] In a case where at least one of the user staying, the expression change and the gaze focus is identified, a user intention in the high-dimensional feature is identified, a recommended dialogue is triggered based on the user intention, and a recommended task is generated.

[0018] In one of the embodiments, the performing the recall processing on the candidate content of the product based on the recommended task to obtain a recommended result comprises:

[0019] transforming the recommended task into a user state vector, and transforming each candidate content of the product into a product feature vector;

[0020] performing a similarity matching between the user state vector and the product feature vector, and determining the recommended result in the candidate content of the product based on a similarity matching result.

[0021] In one of the embodiments, the environment-related feature comprises a user immediate intention feature, an emotional stability degree feature, a user historical behavior feature and an environment context feature.

[0022] In one of the embodiments, the method further comprises:

[0023] collecting real-time feedback of the user, and adjusting the final recommended result based on the real-time feedback; the real-time feedback comprises at least one of voice feedback, expression feedback and gesture feedback.

[0024] In a second aspect, the application further provides a business data processing apparatus, comprising:

[0025] an acquisition module configured to acquire multi-modal data input by a user, and transform the multi-modal data into high-dimensional features;

[0026] a generation module configured to maintain a conversation state by a dialogue manager based on the high-dimensional features, and generate a recommended task;

[0027] The recall module is used to recall candidate content of the product based on the recommendation task and obtain recommendation results;

[0028] The fusion module is used to fuse the environmental-related features of the recommendation results and score them to obtain a score for each recommendation result;

[0029] The determination module is used to select the recommendation result corresponding to the highest score as the final recommendation result for the digital human;

[0030] The output module is used to output the final recommendation result to the user based on the digital human using at least one of voice, facial expression, and action, and to output the interpretable text of the final recommendation result.

[0031] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0032] Acquire multimodal data input by the user and transform the multimodal data into high-dimensional features;

[0033] Based on the aforementioned high-dimensional features, the conversation manager maintains the session state and generates recommended tasks.

[0034] Based on the recommendation task, the candidate content of the product is recalled to obtain the recommendation result;

[0035] The environmental-related features of the recommendation results are fused and scored to obtain a score for each recommendation result;

[0036] The recommendation result corresponding to the highest score will be used as the final recommendation result for the digital human.

[0037] Based on the digital human, the final recommendation result is output to the user in at least one of voice, facial expression, and gesture, and an interpretable text of the final recommendation result is also output.

[0038] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0039] Acquire multimodal data input by the user and transform the multimodal data into high-dimensional features;

[0040] Based on the aforementioned high-dimensional features, the conversation manager maintains the session state and generates recommended tasks.

[0041] Based on the recommendation task, the candidate content of the product is recalled to obtain the recommendation result;

[0042] The environmental-related features of the recommendation results are fused and scored to obtain a score for each recommendation result;

[0043] The recommendation result corresponding to the highest score will be used as the final recommendation result for the digital human.

[0044] Based on the digital human, the final recommendation result is output to the user in at least one of voice, facial expression, and gesture, and an interpretable text of the final recommendation result is also output.

[0045] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0046] Acquire multimodal data input by the user and transform the multimodal data into high-dimensional features;

[0047] Based on the aforementioned high-dimensional features, the conversation manager maintains the session state and generates recommended tasks.

[0048] Based on the recommendation task, the candidate content of the product is recalled to obtain the recommendation result;

[0049] The environmental-related features of the recommendation results are fused and scored to obtain a score for each recommendation result;

[0050] The recommendation result corresponding to the highest score will be used as the final recommendation result for the digital human.

[0051] Based on the digital human, the final recommendation result is output to the user in at least one of voice, facial expression, and gesture, and an interpretable text of the final recommendation result is also output.

[0052] The aforementioned business data processing methods, devices, computer equipment, computer-readable storage media, and computer program products first acquire multimodal data input by the user and transform it into high-dimensional features. Based on these high-dimensional features, a dialogue manager maintains the session state and generates recommendation tasks. Based on the recommendation tasks, candidate content is recalled to obtain recommendation results. The environment-related features of the recommendation results are fused and scored to obtain a score for each recommendation result. The recommendation result with the highest score is taken as the final recommendation result for the digital human. Based on the digital human, the final recommendation result is output to the user using at least one of voice, facial expression, or gesture, along with interpretable text of the final recommendation result. Thus, by integrating multimodal input technologies such as speech recognition, facial expression recognition, and posture perception, real-time interactive perception between the digital human and the user is achieved. During the interaction, unstructured feedback information from the user is collected, including voice pauses, facial expression changes, and posture responses. The acceptance level of the recommended content is implicitly modeled to form a dynamic profile of user preferences, and the recommendation strategy is adjusted in real time to improve the accuracy and personalization of the recommendations. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is an application environment diagram of a business data processing method in one embodiment;

[0055] Figure 2 This is a flowchart illustrating a business data processing method in one embodiment;

[0056] Figure 3 This is a structural block diagram of a business data processing device in one embodiment;

[0057] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0059] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0060] The business data processing method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located on the cloud or other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0061] In one exemplary embodiment, such as Figure 2 As shown, a business data processing method is provided, which is applied to... Figure 1 Taking terminal 102 as an example, the explanation includes the following steps 202 to 212. Wherein:

[0062] Step 202: Obtain the multimodal data input by the user and convert the multimodal data into high-dimensional features.

[0063] For example, multimodal data input by the user is received, and the multimodal data is converted into high-dimensional features according to a preset transformation method.

[0064] Multimodal data includes audio data, image data, and touch data, such as browsing page information, microphone audio, front-facing camera video, touch events, browsing pause duration, swiping and clicking behaviors, etc.

[0065] In one embodiment, the specific transformation process can be streaming processing and batch processing, namely, a low-latency streaming path and a high-precision batch path, wherein the low-latency streaming path is used for immediate response, and the high-precision batch path is used for offline training and model iteration.

[0066] Step 204: Based on high-dimensional features, maintain the session state through the dialogue manager and generate recommended tasks.

[0067] Optionally, the conversation manager (DM) receives high-dimensional features from the NLU (Intent Recognition and Slot Extraction) output, maintains the conversation state (DST), determines the next action (such as asking for clarification, direct recommendation, etc.) according to a preset strategy, maps the conversation intent to the recommendation task, and ensures the compliance of the recommendation task.

[0068] Step 206: Based on the recommendation task, recall the candidate content of the product to obtain the recommendation results.

[0069] For example, the recommendation task is transformed into a user state vector, and a two-tower recall structure is adopted. The left tower is input to the user state vector, and the right tower is input to the product features. The candidate content of the product is recalled to obtain the recommendation result.

[0070] Step 208: After fusing the environmental-related features of the recommendation results, a score is assigned to each recommendation result.

[0071] Optionally, the environmental-related features of each recommendation result are determined, and a weighted sum is performed after assigning a preset weight to each environmental-related feature to obtain the score of each recommendation result.

[0072] Step 210: The recommendation result corresponding to the highest score is taken as the final recommendation result for the digital human.

[0073] For example, the recommendation result corresponding to the highest score is taken as the final recommendation result for the digital human.

[0074] Step 212: Based on the digital human, output the final recommendation result to the user in at least one of voice, facial expression, and action, and output the interpretable text of the final recommendation result.

[0075] Optionally, the final recommendation result is output to the user through a digital human using at least one of voice, facial expression, or gesture, and an interpretable text of the final recommendation result is also output.

[0076] In one embodiment, the digital human directly broadcasts response content and automatically displays relevant pop-ups or pages. The specific appearance of the digital human can be adaptively adjusted according to the real-time scene, and this application embodiment does not limit this.

[0077] In one embodiment, the terminal also includes a static feature library and a dynamic feature library. The static feature library stores low-frequency updated data such as user profiles, consumption preferences, and long-term interest tags; the dynamic feature cache stores high-frequency features such as short-term behavior, emotional trends, and current interaction context.

[0078] A hybrid architecture combining Redis distributed key-value database and Milvus vector database is employed to achieve low-latency, high-concurrency reads. All features are tagged with timestamps and context identifiers to distinguish feature snapshots across different session states.

[0079] In the aforementioned business data processing method, multimodal data input by the user is acquired and transformed into high-dimensional features. Based on these high-dimensional features, a dialogue manager is used to maintain the session state and generate recommendation tasks. Based on these recommendation tasks, candidate content for the product is recalled to obtain recommendation results. The environment-related features of the recommendation results are fused and scored to obtain a score for each recommendation result. The recommendation result with the highest score is taken as the final recommendation result for the digital human. Based on the digital human, the final recommendation result is output to the user using at least one of voice, facial expression, or gesture, along with interpretable text of the final recommendation result. Thus, by integrating multimodal input technologies such as speech recognition, facial expression recognition, and posture perception, real-time interactive perception between the digital human and the user is achieved. During the interaction, unstructured feedback information from the user is collected, including voice pauses, facial expression changes, and posture responses. This implicitly models the acceptance level of the recommended content, forming a dynamic profile of user preferences, and adjusting the recommendation strategy in real time to improve the accuracy and personalization of the recommendations.

[0080] In an exemplary embodiment, the multimodal data includes audio data, image data, and touch data; converting the multimodal data into high-dimensional features includes: converting the audio data into text data, and determining the text features of the multimodal data based on the text data; identifying emotional changes in the audio data and the user's facial emotions in the image data, and determining the user's emotional features based on the emotional changes and facial emotions; identifying the user's hand gestures in the image data, and determining the user's behavioral features based on the hand gestures and touch data.

[0081] In practical implementation, real-time streaming ASR (Automatic Speech Recognition) is used to convert audio data into text data. NLU (Non-Limited Intent Recognition and Slot Extraction) is based on a lightweight BERT or Transformer model to perform intent classification and slot extraction, outputting semantic labels as well as confidence scores and uncertainty estimates, i.e., text features. Emotional changes in audio data (such as the fundamental frequency, rhythm, and energy of emotions) and facial emotions of users in image data are identified and fused to determine the user's emotion category (such as joy, confusion, annoyance, etc.), intensity, and emotion confidence, i.e., emotion features. Based on keypoint detection (such as mediapipe / openpose), the user's gestures in image data are identified, and combined with touch data, the user's gestures, i.e., behavioral features, are determined and classified.

[0082] The facial emotions of the user can be determined based on AUS or expression categories, or other methods with facial emotion recognition functions can be used. This application does not limit this method.

[0083] In one embodiment, user attention / attention is identified by determining whether the user is looking at the screen or a specific element by determining the user's facial alignment and gaze estimation, in order to trigger automatic product recommendations.

[0084] In one embodiment, the confidence level and uncertainty estimate of the semantic label can be used to trigger a clarification strategy. In cases where the confidence level and uncertainty estimate are below a threshold, or where there is a conflict in the high-dimensional features of the multimodal data (e.g., the voice is affirmative, but the facial expression is doubtful), DM triggers clarification to avoid misjudgment.

[0085] In the above embodiments, by mapping heterogeneous multimodal data to a unified high-dimensional feature space, cross-modal semantic alignment and information fusion are achieved, which significantly improves the accuracy of user intent recognition and the robustness of recommendation.

[0086] In an exemplary embodiment, based on high-dimensional features, the conversation state is maintained through a dialogue manager, and a recommendation task is generated, including: when at least one of the user's pause, facial expression change, and gaze focus is detected, the user's intent in the high-dimensional features is identified, and based on the user's intent, a recommendation dialogue is triggered, and a recommendation task is generated.

[0087] In practice, when at least one of the following behaviors is detected: user pause, facial expression change, and eye focus, the user intent in the high-dimensional features is identified. Based on the user intent and the preset strategy, a recommendation dialogue is triggered and a recommendation task is generated.

[0088] The default strategy is the Contextual Bandit strategy, which selects candidate strategies based on long-term benefits and immediate feedback (such as exploration and utilization weight adjustment, clarification question template selection, and display template selection).

[0089] In the above embodiments, the session manager continuously records user intent, slots, and emotions, which solves the semantic gap caused by the "independent recognition in each round" of the existing technology, making the final recommendation results more accurate.

[0090] In an exemplary embodiment, based on the recommendation task, the candidate content of the product is recalled to obtain the recommendation result, including: converting the recommendation task into a user state vector and converting the candidate content of each product into a product feature vector; performing similarity matching between the user state vector and the product feature vector, and determining the recommendation result from the candidate content of the product based on the similarity matching result.

[0091] In practical implementation, the text features, emotion features and behavioral features in the recommendation task are determined. The Transformer fusion network is used to map the text features, emotion features and behavioral features to a unified representation space and transform them into user state vectors. The specific formula is shown in formula (1).

[0092]

[0093] in, For text features, As an emotional characteristic, These are behavioral characteristics.

[0094] In one embodiment, the recall strategy includes semantic similarity recall, behavioral preference recall, and emotional state recall. Based on the three recall strategies, the product feature vectors of the candidate content of the product are matched for similarity, and a preset number of products are determined from the candidate content of the product according to the matching results.

[0095] Among them, semantic similarity recall is based on semantic similarity matching, behavioral preference is based on long-term interest embedding for similarity matching, and emotional state recall is based on user emotions. For example, when a user's emotion is detected as anxiety, stable financial products are recalled, and when a user's emotion is detected as joy, short-term return products are recalled.

[0096] In the above embodiments, multiple methods are used to recall candidate content for the product, which solves the problem of the failure of a single strategy in cold start, new product, and emotional change scenarios, while also taking into account compliance and explainability.

[0097] In one exemplary embodiment, environment-related features include user immediate intent features, emotional stability features, user historical behavior features, and environmental context features.

[0098] In practice, the recommendation results are scored and ranked, and environmental features are integrated. Reinforcement learning is used to calculate the score for each feature. The specific calculation formula is shown in formula (2).

[0099]

[0100] in, For the user's immediate intent characteristics, As a characteristic of emotional stability, For user historical behavior characteristics, For environmental context features, , , and Preset weights are assigned to each feature.

[0101] Among them, the user's immediate intent features come from the context of the dialogue interaction, the user's historical behavior features come from the user's historical clicks, dwell time, purchase records, and other historical behaviors, and the environmental context features come from the current time, weather, and geographical location.

[0102] In the above embodiments, by comprehensively considering multiple factors, the optimal recommendation result is further determined, making the final result more accurate.

[0103] In one exemplary embodiment, the business data processing method further includes: collecting real-time feedback from users and adjusting the final recommendation result based on the real-time feedback.

[0104] Real-time feedback includes at least one of voice feedback, facial expression feedback, and gesture feedback, such as nodding, negation, facial expression changes, and tone of voice.

[0105] In practice, user feedback is collected in real time, and the preset weights of the calculated scores are updated through a reinforcement learning mechanism to further adjust the final recommendation results.

[0106] In the above embodiments, by monitoring users' reactions in real time and making adaptive adjustments based on users' specific behaviors, a closed-loop feedback of user behavior is formed, which makes the recommendation form evolve from a static fixed form to a dynamic self-learning form.

[0107] To illustrate the business data processing method in this application in detail, an embodiment is described below. For example, this application describes a business data processing method in a specific scenario.

[0108] First, it receives multimodal data input by the user and transforms the multimodal data into high-dimensional features according to a preset transformation method.

[0109] The Dialogue Manager (DM) receives high-dimensional features from the NLU (Intent Recognition and Slot Extraction) output, maintains the Session State (DST), determines the next action (such as asking for clarification, direct recommendation, etc.) based on preset policies, maps the session intent to the recommendation task, and ensures the compliance of the recommendation task.

[0110] The recommendation task is transformed into a user state vector. A two-tower recall model is adopted, with the user state vector as the input on the left tower and the product features as the input on the right tower. The candidate content of the product is recalled to obtain the recommendation result.

[0111] The environmental features of each recommendation result are determined, and a weighted sum is calculated after assigning a preset weight to each environmental feature to obtain the score of each recommendation result.

[0112] The recommendation result corresponding to the highest score is taken as the final recommendation result for the digital human. The digital human outputs the final recommendation result to the user using at least one of the following methods: voice, facial expression, or gesture, along with interpretable text.

[0113] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0114] Based on the same inventive concept, this application also provides a business data processing apparatus for implementing the business data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more business data processing apparatus embodiments provided below can be found in the limitations of the business data processing method described above, and will not be repeated here.

[0115] In one exemplary embodiment, such as Figure 3As shown, a business data processing device is provided, including: an acquisition module 301, a generation module 302, a recall module 303, a fusion module 304, a determination module 305, and an output module 306, wherein:

[0116] The acquisition module is used to acquire multimodal data input by the user and convert the multimodal data into high-dimensional features;

[0117] The generation module is used to maintain the session state through the dialogue manager and generate recommended tasks based on the high-dimensional features;

[0118] The recall module is used to recall candidate content of the product based on the recommendation task and obtain recommendation results;

[0119] The fusion module is used to fuse the environmental-related features of the recommendation results and score them to obtain a score for each recommendation result;

[0120] The determination module is used to select the recommendation result corresponding to the highest score as the final recommendation result for the digital human;

[0121] The output module is used to output the final recommendation result to the user based on the digital human using at least one of voice, facial expression, and action, and to output the interpretable text of the final recommendation result.

[0122] In one exemplary embodiment, the above-described apparatus further includes a conversion module for:

[0123] The audio data is converted into text data, and the text features of the multimodal data are determined based on the text data.

[0124] Identify emotional changes in the audio data and facial emotions of the user in the image data, and determine the user's emotional characteristics based on the emotional changes and facial emotions;

[0125] The user's hand gestures in the image data are identified, and the user's behavioral characteristics are determined based on the hand gestures and the touch data.

[0126] In one exemplary embodiment, the above-described generation module is further configured to:

[0127] If at least one of the following behaviors is detected: user lingering, facial expression changes, and eye focus, the user intent in the high-dimensional features is identified, and based on the user intent, a recommendation dialogue is triggered and a recommendation task is generated.

[0128] In one exemplary embodiment, the recall module described above is further configured to:

[0129] The recommendation task is transformed into a user state vector, and the candidate content of each product is transformed into a product feature vector.

[0130] The user state vector is matched with the product feature vector for similarity, and the recommendation result is determined from the candidate content of the product based on the similarity matching result.

[0131] In one exemplary embodiment, the environment-related features include user immediate intent features, emotional stability features, user historical behavior features, and environmental context features.

[0132] In one exemplary embodiment, the above-described apparatus further includes an adjustment module for:

[0133] The system collects real-time feedback from the user and adjusts the final recommendation result based on the real-time feedback; the real-time feedback includes at least one of voice feedback, facial expression feedback, and gesture feedback.

[0134] Each module in the aforementioned business data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0135] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), a communication interface, a display unit, and input devices. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface, display unit, and input devices are also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores the output data of the digital human. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a business data processing method.

[0136] The display unit of this computer device is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of this computer device can be a touch layer covering the display screen, or buttons, a trackball, or a touchpad set on the casing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0137] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0138] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0139] Acquire multimodal data input by the user and transform the multimodal data into high-dimensional features;

[0140] Based on the aforementioned high-dimensional features, the conversation manager maintains the session state and generates recommended tasks.

[0141] Based on the recommendation task, the candidate content of the product is recalled to obtain the recommendation result;

[0142] The environmental-related features of the recommendation results are fused and scored to obtain a score for each recommendation result;

[0143] The recommendation result corresponding to the highest score will be used as the final recommendation result for the digital human.

[0144] Based on the digital human, the final recommendation result is output to the user in at least one of voice, facial expression, and gesture, and an interpretable text of the final recommendation result is also output.

[0145] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0146] Acquire multimodal data input by the user and transform the multimodal data into high-dimensional features;

[0147] Based on the aforementioned high-dimensional features, the conversation manager maintains the session state and generates recommended tasks.

[0148] Based on the recommendation task, the candidate content of the product is recalled to obtain the recommendation result;

[0149] The environmental-related features of the recommendation results are fused and scored to obtain a score for each recommendation result;

[0150] The recommendation result corresponding to the highest score will be used as the final recommendation result for the digital human.

[0151] Based on the digital human, the final recommendation result is output to the user in at least one of voice, facial expression, and gesture, and an interpretable text of the final recommendation result is also output.

[0152] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0153] Acquire multimodal data input by the user and transform the multimodal data into high-dimensional features;

[0154] Based on the aforementioned high-dimensional features, the conversation manager maintains the session state and generates recommended tasks.

[0155] Based on the recommendation task, the candidate content of the product is recalled to obtain the recommendation result;

[0156] The environmental-related features of the recommendation results are fused and scored to obtain a score for each recommendation result;

[0157] The recommendation result corresponding to the highest score will be used as the final recommendation result for the digital human.

[0158] Based on the digital human, the final recommendation result is output to the user in at least one of voice, facial expression, and gesture, and an interpretable text of the final recommendation result is also output.

[0159] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0160] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0161] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0162] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A business data processing method, characterized in that, The method includes: Acquire multimodal data input by the user and transform the multimodal data into high-dimensional features; Based on the aforementioned high-dimensional features, the conversation manager maintains the session state and generates recommended tasks. Based on the recommendation task, the candidate content of the product is recalled to obtain the recommendation result; The environmental-related features of the recommendation results are fused and scored to obtain a score for each recommendation result; The recommendation result corresponding to the highest score will be used as the final recommendation result for the digital human. Based on the digital human, the final recommendation result is output to the user in at least one of voice, facial expression, and gesture, and an interpretable text of the final recommendation result is also output.

2. The method according to claim 1, characterized in that, The multimodal data includes audio data, image data, and touch data; the process of converting the multimodal data into high-dimensional features includes: The audio data is converted into text data, and the text features of the multimodal data are determined based on the text data. Identify emotional changes in the audio data and facial emotions of the user in the image data, and determine the user's emotional characteristics based on the emotional changes and facial emotions; The user's hand gestures in the image data are identified, and the user's behavioral characteristics are determined based on the hand gestures and the touch data.

3. The method according to claim 1, characterized in that, The step of maintaining the session state and generating recommendation tasks based on the high-dimensional features through a dialogue manager includes: If at least one of the following behaviors is detected: user lingering, facial expression changes, and eye focus, the user intent in the high-dimensional features is identified, and based on the user intent, a recommendation dialogue is triggered and a recommendation task is generated.

4. The method according to claim 1, characterized in that, The process of recalling candidate content for the product based on the recommendation task to obtain recommendation results includes: The recommendation task is transformed into a user state vector, and the candidate content of each product is transformed into a product feature vector. The user state vector is matched with the product feature vector for similarity, and the recommendation result is determined from the candidate content of the product based on the similarity matching result.

5. The method according to claim 1, characterized in that, The environment-related features include user immediate intent features, emotional stability features, user historical behavior features, and environmental context features.

6. The method according to claim 1, characterized in that, The method further includes: The system collects real-time feedback from the user and adjusts the final recommendation result based on the real-time feedback; the real-time feedback includes at least one of voice feedback, facial expression feedback, and gesture feedback.

7. A business data processing device, characterized in that, The device includes: The acquisition module is used to acquire multimodal data input by the user and convert the multimodal data into high-dimensional features; The generation module is used to maintain the session state through the dialogue manager and generate recommended tasks based on the high-dimensional features; The recall module is used to recall candidate content of the product based on the recommendation task and obtain recommendation results; The fusion module is used to fuse the environmental-related features of the recommendation results and score them to obtain a score for each recommendation result; The determination module is used to select the recommendation result corresponding to the highest score as the final recommendation result for the digital human; The output module is used to output the final recommendation result to the user based on the digital human using at least one of voice, facial expression, and action, and to output the interpretable text of the final recommendation result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images