Business processing method and device based on emotion recognition driving, equipment and medium

Abstract

This application provides a business processing method, apparatus, device, and medium based on emotion recognition, which can be applied to the field of big data technology. The method includes: in response to being in a business assistance mode, collecting the user's visual and voice data; performing emotion recognition based on the visual and voice data collected within a first preset time period to obtain an emotion recognition result; performing assistance strategy matching based on the emotion recognition result to determine a target assistance strategy; and executing the target assistance strategy to assist the user in business processing.

Description

Technical Field

[0001] This application relates to the field of big data technology, and specifically to a business processing method, apparatus, device, and medium based on emotion recognition. Background Technology

[0002] With the ongoing digital transformation of financial services, bank self-service terminals have played a crucial role in improving service efficiency and optimizing customer experience, gradually becoming one of the main channels for customers to handle various transactions. However, in this process, existing self-service systems have exposed some operational issues when serving diverse user groups.

[0003] Moreover, in actual use, due to reasons such as the operation process not being simple enough and the response speed being slow, users find it difficult to quickly obtain effective support when they encounter problems, which greatly limits the promotion and application of self-service among various user groups. Summary of the Invention

[0004] In view of the above problems, embodiments of this application provide a business processing method, apparatus, device and medium based on emotion recognition.

[0005] According to a first aspect of this application, a business processing method based on emotion recognition is provided, comprising: in response to being in a business assistance mode, collecting visual data and voice data of a user; performing emotion recognition based on the visual data and voice data collected within a first preset time period to obtain an emotion recognition result; performing assistance strategy matching based on the emotion recognition result to determine a target assistance strategy; and executing the target assistance strategy to assist the user in business processing.

[0006] According to an embodiment of this application, matching assistance strategies based on emotion recognition results to determine a target assistance strategy includes: obtaining a user's first emotional state based on multiple emotion recognition results determined within a second preset time period; and determining the target assistance strategy as a machine assistance strategy when the first emotional state is expressed as a negative state.

[0007] According to an embodiment of this application, executing a target assistance strategy to assist a user in business processing includes: when the target assistance strategy is a machine assistance strategy, collecting the user's current operation data; processing the current operation data using an intelligent agent to obtain operation instruction information; and displaying the operation instruction information on a display interface based on the operation location related to the current operation data, so as to guide the user to perform business processing based on the operation instruction information.

[0008] According to embodiments of this application, an intelligent agent processes current operation data to obtain operation instructions, including: generating prompt text based on current operation data, interface context, and user's historical behavior; and inputting the prompt text into a large language model included in the intelligent agent to utilize the large language model to perform semantic understanding and dynamic interpretation of the user's operation steps to generate operation instructions.

[0009] According to an embodiment of this application, the business processing method based on emotion recognition further includes: obtaining a user's second emotion state based on multiple emotion recognition results determined within a third preset time period after the execution of a machine assistance strategy; and determining the target assistance strategy as a human assistance strategy when the second emotion state is a negative state.

[0010] According to an embodiment of this application, executing a target assistance strategy to assist users in business processing includes: when the target assistance strategy is a manual assistance strategy, sending a task notification to the branch where the self-service terminal belongs so that the branch staff can assist users in business processing on-site.

[0011] According to an embodiment of this application, executing a target assistance strategy to assist users in business processing includes: when the target assistance strategy is a manual assistance strategy, sending a task notification to the customer service center via a message queue so that the staff of the customer service center can remotely assist users in business processing.

[0012] According to an embodiment of this application, emotion recognition is performed based on visual data and speech data collected within a first preset time period to obtain an emotion recognition result. This includes: modeling the contextual relationship of a user's facial expressions based on the visual data collected within the first preset time period using a hierarchical structure and a self-attention mechanism to extract the user's facial muscle movements, facial expressions, and micro-expression changes to obtain visual modal features; performing feature encoding on the speech data collected within the first preset time period to generate a vector representation containing contextual speech relationships to obtain speech modal features; and performing emotion recognition based on the visual modal features and speech modal features to obtain an emotion recognition result.

[0013] According to embodiments of this application, emotion recognition is performed based on visual modal features and speech modal features to obtain emotion recognition results, including: extracting the temporal dependency relationship between visual modal features and speech modal features through a multi-head attention mechanism to obtain associated features; normalizing and mapping the associated features to obtain an emotion state probability distribution; and obtaining the emotion recognition result based on the emotion state probability distribution.

[0014] According to an embodiment of this application, user operation status indicators are collected, including at least one of the following: duration of stay on a single interface, number of clicks per unit time, number of consecutive invalid clicks, number of consecutive return operations, and menu level depth of the business operation path; operation validity is judged based on the operation status indicators to obtain a judgment result; if the judgment result indicates that the user operation is difficult, an interface layout scheme associated with the operation status indicators is determined from the rule base; and the layout of interface elements of the displayed interface is updated based on the interface layout scheme.

[0015] According to an embodiment of this application, the emotion recognition-driven business processing method further includes: when the determination result indicates that the user's operation is abnormal, determining and executing an assistance strategy to assist the user in business processing.

[0016] According to an embodiment of this application, the emotion recognition-driven business processing method further includes: in response to a user's usage request, determining the user group to which the user belongs; and if it is determined that the user group to which the user belongs is an older user group, switching the working mode to a business assistance mode.

[0017] According to a second aspect of this application, an emotion recognition-driven business processing apparatus is provided, comprising: a data acquisition module for acquiring visual and voice data of a user in response to being in a business assistance mode; a recognition module for performing emotion recognition based on the visual and voice data acquired within a first preset time period to obtain an emotion recognition result; a matching module for performing assistance strategy matching based on the emotion recognition result to determine a target assistance strategy; and an execution module for executing the target assistance strategy to assist the user in business processing.

[0018] According to a third aspect of this application, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0019] According to a fourth aspect of this application, a computer-readable storage medium is also provided, on which a computer program or instructions are stored, wherein the computer program or instructions, when executed by a processor, implement the steps of the above-described method.

[0020] According to a fifth aspect of this application, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0021] In the embodiments of this application, the above-described business processing method can improve the user's experience during the operation of bank self-service equipment by collecting the user's visual and voice data in real time for emotion recognition and dynamically matching and executing assistance strategies based on the emotional state. For example, by proactively sensing and automatically recognizing the user's emotional changes, and simultaneously matching assistance strategies in a timely manner, targeted assistance and guidance can be provided. The aforementioned assistance strategies can reduce negative emotions caused by operational difficulties, lower the error rate, improve the success rate and efficiency of business processing, and enhance user acceptance and satisfaction with bank self-service. Attached Figure Description

[0022] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0023] Figure 1 The illustration shows an application scenario diagram of the emotion recognition-driven business processing method, apparatus, device, and medium according to embodiments of this application.

[0024] Figure 2 A flowchart illustrating an emotion recognition-driven business processing method according to an embodiment of this application is shown.

[0025] Figure 3 This illustration shows a data flow diagram that assists users in performing business processes according to an embodiment of this application.

[0026] Figure 4 The diagram illustrates a data flow diagram for determining the emotion recognition result according to an embodiment of this application.

[0027] Figure 5 A flowchart illustrating a method for determining emotion recognition results according to an embodiment of this application is shown schematically.

[0028] Figure 6 The illustration shows a data flow diagram of the layout of interface elements in an updated display interface according to an embodiment of this application.

[0029] Figure 7 The diagram illustrates a structural block diagram of an emotion recognition-driven business processing apparatus according to an embodiment of this application.

[0030] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing an emotion recognition-driven business processing method according to an embodiment of this application. Detailed Implementation

[0031] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0032] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0033] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0034] In the embodiments disclosed herein, the collection, updating, analysis, processing, use, transmission, provision, invention, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to safeguard user personal information security and network security.

[0035] In the embodiments disclosed herein, user authorization or consent is obtained before acquiring or collecting user personal information.

[0036] Currently, so-called "age-friendly" designs are mostly limited to static interface adjustments, such as enlarging fonts or providing voice prompts, lacking the ability to dynamically adapt to the user's real-time operating status and cognitive load. The system's interaction mode is fixed, failing to provide personalized guidance based on the user's operational proficiency, behavioral habits, or emotional fluctuations. More importantly, existing systems completely lack a proactive perception and recognition mechanism for user emotions and operational difficulties, failing to intervene promptly when users experience confusion or anxiety, resulting in a persistently high error rate. When users genuinely need assistance, they can only rely on on-site human assistance, a method limited by branch human resources, making it difficult to guarantee real-time and reliable responses, causing frequent interruptions in business processing and severely impacting the service experience and efficiency for elderly users.

[0037] Therefore, existing self-service terminals still have significant technological gaps in terms of age-friendly design, emotion perception, and remote collaboration.

[0038] In view of this, this disclosure proposes a business processing method based on emotion recognition, applied to a self-service terminal. The method includes: in response to being in a business assistance mode, collecting the user's visual data and voice data; performing emotion recognition based on the visual data and voice data collected within a first preset time period to obtain an emotion recognition result; performing assistance strategy matching based on the emotion recognition result to determine a target assistance strategy; and executing the target assistance strategy to assist the user in business processing.

[0039] Figure 1 The illustration schematically depicts an application scenario of a business processing method, apparatus, device, and medium based on emotion recognition-driven according to embodiments of this application. It should be noted that... Figure 1 The examples shown are merely examples of scenarios in which the embodiments of this application can be applied, to help those skilled in the art understand the technical content of this application, but do not mean that the embodiments of this application cannot be used in other devices, systems, environments or scenarios.

[0040] like Figure 1 As shown, application scenario 100 according to this embodiment may include an application scenario of a financial trading machine. Network 104 is used as a medium to provide a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0041] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be devices with displays that support financial transactions such as deposits and withdrawals, such as self-service terminals.

[0042] Server 105 can be a server providing various services, such as a backend management server (for example only) supporting transactions supported by the first terminal device 101, the second terminal device 102, and the third terminal device 103. The backend management server can analyze and process received user requests and other data, and then feed the processing results back to the terminal devices. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services such as cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and basic cloud computing services such as big data.

[0043] It should be noted that the emotion recognition-driven business processing method provided in this application embodiment can generally be executed by server 105 and / or terminal devices 101-103. Accordingly, the emotion recognition-driven business processing device provided in this application embodiment can generally be set in server 105 and / or terminal devices 101-103.

[0044] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0045] Figure 2 A flowchart illustrating an emotion recognition-driven business processing method according to an embodiment of this application is shown.

[0046] like Figure 2 As shown, the emotion recognition-driven business processing method 200 according to the embodiments of this application may include steps S210 to S240.

[0047] In step S210, in response to being in business assistance mode, the user's visual data and voice data are collected.

[0048] In embodiments of this application, user consent or authorization can be obtained before acquiring user information. For example, a request to acquire user information can be sent to the user before step S210. If the user consents or authorizes the acquisition of user information, step S210 is executed.

[0049] In the embodiments of this application, the aforementioned business assistance mode can be a preset business mode for users who need assistance. In the business assistance mode, the user's visual and voice data can be acquired in real time through a data acquisition device. The data acquisition device may include a camera or a microphone, the visual data may be the user's facial expression data, and the voice data may be the user's tone of voice or speech rate data, etc.

[0050] In step S220, emotion recognition is performed based on the visual and speech data collected within the first preset time period to obtain the emotion recognition result.

[0051] In the embodiments of this application, after collecting visual and speech data from the user within a preset time period, emotion recognition can be performed on the data. Specifically, through a period of data collection, facial expression features can be extracted from the visual data using computer vision technology, and speech tone features can be extracted using speech recognition technology. By comparison, data on changes in facial expression features and speech tone features can be obtained; then, emotion recognition can be performed based on these changes. For example, if a user exhibits characteristics such as a change from expressionless to frowning, increased tone, and faster speech during operation, the emotion recognition result could indicate that the user is experiencing anxiety.

[0052] In step S230, assistance strategy matching is performed based on the emotion recognition results to determine the target assistance strategy.

[0053] In step S240, the target assistance strategy is executed to assist the user in business processing.

[0054] In the embodiments of this application, based on the above-mentioned emotion recognition results, a suitable assistance strategy can be selected from the assistance strategy library according to the different emotions. The assistance strategy library may include a variety of preset assistance strategies, such as: explaining the operation method of the corresponding business to the user by triggering voice guidance; playing operation animation guidance to guide the user; notifying staff to provide on-site assistance, etc.

[0055] In the embodiments of this application, the current emotion recognition result of the user is matched to determine the target assistance strategy, and then the target assistance strategy is executed. For example, if the user is in a relatively anxious mood, the intelligent semantic interpretation strategy may be triggered first to provide immediate operation instructions and guidance. If the user's emotion remains abnormal and the operation fails multiple times, staff will be notified to provide on-site assistance.

[0056] In the embodiments of this application, the above-described business processing method can improve the user's experience during the operation of bank self-service equipment by collecting the user's visual and voice data in real time for emotion recognition and dynamically matching and executing assistance strategies based on the emotional state. For example, by proactively sensing and automatically recognizing the user's emotional changes, and simultaneously matching assistance strategies in a timely manner, targeted assistance and guidance can be provided. The aforementioned assistance strategies can reduce negative emotions caused by operational difficulties, lower the error rate, improve the success rate and efficiency of business processing, and enhance user acceptance and satisfaction with bank self-service.

[0057] In embodiments of this application, matching assistance strategies based on emotion recognition results to determine a target assistance strategy includes: obtaining a user's first emotional state based on multiple emotion recognition results determined within a second preset time period; and determining the target assistance strategy as a machine assistance strategy when the first emotional state is expressed as a negative state.

[0058] In the embodiments of this application, the aforementioned second preset time period differs from the first preset time period, allowing for continuous detection of the user's emotional changes after the first preset time period. For example, visual and speech data of the user can be collected multiple times during the second preset time period, and emotion recognition can be performed on this data to obtain multiple emotion recognition results. Then, these multiple emotion recognition results can be combined to calculate a comprehensive emotional state. The average or weighted average of the emotion recognition results during this period can be taken, thereby more accurately reflecting the user's emotional trend.

[0059] In an embodiment of this application, further, if the user's initial emotional state is still negative, the target assistance strategy can be designated as a machine assistance strategy. The aforementioned machine assistance strategy may include voice explanation, interface adjustment, or enhanced prompts. Voice explanation may involve playing pre-recorded operation instructions; interface adjustment may involve optimizing the layout of the user interface, simplifying operation steps, and highlighting key information requiring operation; enhanced prompts may involve soothing the user's emotions through voice prompts and playing animated operation prompts.

[0060] In the embodiments of this application, the above steps, by collecting and analyzing the user's emotional data multiple times within a specific time period, can more accurately identify changes in the user's emotional state. The aforementioned multi-time period analysis method avoids misjudgments caused by the randomness of a single emotion recognition, thus ensuring a more accurate and reliable assessment of the user's emotional state. Furthermore, through machine-assisted strategies, targeted assistance can be quickly provided to users without the need for human customer service intervention. For example, through intelligent semantic interpretation, interface optimization, or voice prompts, the system can dynamically adjust the interaction method based on the user's real-time emotional state, making self-service more humanized and intelligent.

[0061] In embodiments of this application, executing a target assistance strategy to assist users in business processing includes: when the target assistance strategy is a machine assistance strategy, collecting the user's current operation data; using an intelligent agent to process the current operation data to obtain operation instructions; and displaying the operation instructions on a display interface based on the operation location related to the current operation data, so as to guide the user to perform business processing based on the operation instructions.

[0062] Figure 3This illustration shows a data flow diagram that assists users in performing business processes according to an embodiment of this application.

[0063] In the embodiments of this application, reference is made to Figure 3 After determining the target assistance strategy as a machine assistance strategy according to the above steps, user operation data can be collected. The operation data can include the user's specific operation behavior on the self-service device, such as the user's current interface position, the user's specific operation actions, such as clicking, swiping, inputting, etc.; it can also include the user's current operation behavior pattern, which can include repeated clicking, invalid swiping, etc.

[0064] In the embodiments of this application, the intelligent agent processes the current operation data to obtain operation instructions, including: generating prompt text based on the current operation data, interface context and user's historical behavior; and inputting the prompt text into a large language model included in the intelligent agent, so as to use the large language model to perform semantic understanding and dynamic interpretation of the user's operation steps to generate operation instructions.

[0065] In the embodiments of this application, reference is made to Figure 3 Then, this operational data can be input into the intelligent agent, which can analyze it to derive operational instructions. Specifically, the intelligent agent can be a large language model or other artificial intelligence module. The intelligent agent can combine the current operational data, business processes, and other interface context information to determine the problems that the user may encounter or the causes of those problems. Furthermore, the intelligent agent can generate corresponding operational instructions based on the causes of the problems.

[0066] In the embodiments of this application, the aforementioned interface context may be the name, functional module, or displayed content of the current interface, and the user's historical behavior may be the user's habits in similar operations, such as frequently made mistakes or operational preferences. Then, a prompt text can be generated based on the above various information to enable the intelligent agent to input corresponding operation instructions.

[0067] In the embodiments of this application, taking a large language model as an example, the large language model can understand the user's current operation intent or other contextual information by inputting prompt text, and then output operation information descriptions for the user's current operation to guide the user's operation. The large model can combine the real-time context of the business process, dynamically adjust the focus and level of detail of the explanation content, and automatically convert professional terminology into everyday language, ensuring that users receive clear and accurate guidance for key operations, thereby effectively enhancing their understanding and confidence during the operation process.

[0068] In the embodiments of this application, reference is made to Figure 3Next, the operation instructions can be displayed at the user's location based on the instructions provided by the intelligent agent, thus clearly guiding the user in processing the business. Specifically: if the user is filling out a form, the operation instructions can be displayed next to the form to guide the user; or if the user needs to click on a target location, the operation instructions can be displayed as a pop-up next to the target location to guide the user to click on the correct location.

[0069] In the embodiments of this application, an intelligent agent collects and analyzes user operation data and provides corresponding operation instructions, which are then displayed at the operation location. By providing real-time assistance during the user's operation, operation interruptions caused by user confusion or operational errors can be effectively avoided. This real-time, continuous operation guidance ensures that users can successfully complete business transactions, thereby improving the overall usability and stability of the self-service terminal. Furthermore, the intelligent agent dynamically generates operation instructions by analyzing user operation data and historical behavior in real-time through prompt text. It can provide immediate assistance based on the user's needs at different operation steps, ensuring that the user receives appropriate operation guidance at each stage, thereby improving operational efficiency and success rate.

[0070] In the embodiments of this application, a user's second emotional state is obtained based on multiple emotion recognition results determined within a third preset time period after the execution of the machine-assisted strategy; and if the second emotional state is expressed as a negative state, the target assistance strategy is determined to be a human assistance strategy.

[0071] In embodiments of this application, multiple emotion recognition results can be determined within a third preset time period in addition to the first and second preset time periods, thereby obtaining a second emotional state. For example, the second emotional state can be obtained by averaging or weighted averaging the multiple emotion recognition results, which can more accurately reflect the user's emotional changes.

[0072] In the embodiments of this application, if the user's mood remains negative after the first preset time period and the second preset time period, it indicates that the machine assistance strategy has failed to effectively alleviate the user's emotional problem. Further, the target assistance strategy can be determined as a human assistance strategy, whereby the user's problem can be transferred to back-end staff or notified on-site staff. Staff can communicate directly with the user through real-time video or voice calls to provide more detailed guidance and assistance.

[0073] In the embodiments of this application, different target assistance strategies can be adjusted in a timely manner based on the second emotional state identified during a third preset time period. This not only provides machine assistance strategies in the initial stage (either the second or first preset time period) but also dynamically adjusts the assistance method according to the user's emotional response. This allows for more precise responses to changes in the user's emotions, especially when machine assistance fails to effectively resolve the problem, enabling a timely switch to human assistance to prevent further deterioration of the user's emotions. Furthermore, the dual assistance mechanism consisting of machine and human intervention ensures that the user receives support under any circumstances. This multi-layered protection mechanism enhances the overall reliability of the system and improves the user's business processing experience.

[0074] In the embodiments of this application, the execution of a target assistance strategy to assist users in business processing includes: when the target assistance strategy is a manual assistance strategy, sending a task notification to the branch where the self-service terminal belongs so that the staff of the branch can assist users in business processing on-site.

[0075] In embodiments of this application, the aforementioned task notification may include the location of the self-service terminal currently being used by the user, the problem the user needs to solve, and a description of the user's emotional state, etc. Furthermore, staff can guide the user through the current operation steps; soothe the user's emotions and alleviate anxiety or confusion through face-to-face communication; and help the user solve specific problems encountered during the operation to ensure smooth business operations.

[0076] In the embodiments of this application, when the target assistance strategy is a manual assistance strategy, a task notification is sent to the customer service center through a message queue so that the staff of the customer service center can remotely assist users in handling business.

[0077] In the embodiments of this application, the aforementioned message queue serves as middleware for transmitting messages between different systems or modules, ensuring reliable message transmission and processing. It is typically used for asynchronous communication, improving system response speed and reliability. Furthermore, staff can guide users through the current operation steps via voice or video; staff can soothe users' emotions and alleviate anxiety or confusion through communication; and help users resolve specific problems encountered during operation, ensuring smooth business operations. Staff can also view the user interface in real time using DOM (Document Object Model) mirroring technology and perform non-intrusive visual annotations (such as a circle cursor) to guide user operations.

[0078] For example, during remote assistance, a structured remote assistance session request can be initiated via a WebSocket long-lived connection, transmitting user interface information, operational data, and a second emotional state to the customer service center. The customer service center's client can present an enhanced interactive view, displaying the user's emotional curve and operational issues, enabling staff to understand the user's current situation. Furthermore, staff can initiate two-way interactive guidance, either through direct communication via a high-definition voice channel or through non-intrusive visual annotations on the interface using DOM mirroring. Users can then execute instructions from staff at self-service terminals, generating collaborative logs of the problem and resolution process upon completion, providing data support for subsequent optimization.

[0079] In the embodiments of this application, the aforementioned human assistance strategies can improve user experience by helping users encountering difficulties. Specifically: On-site assistance provides immediate help, allowing branch staff to quickly respond to user needs and provide guidance directly next to the user's self-service terminal. This face-to-face interaction quickly resolves issues, reduces user waiting time, and improves business processing efficiency. Remote assistance sends task notifications to the customer service center via message queues, allowing staff to respond to user needs from any location. This not only improves service flexibility but also allows for dynamic adjustment of customer service resources based on actual needs, enhancing system scalability. These two human assistance strategies ensure that users receive timely and effective assistance when needed, improving service quality and user satisfaction.

[0080] In the embodiments of this application, emotion recognition is performed based on visual data and speech data collected within a first preset time period to obtain an emotion recognition result. This includes: modeling the contextual relationship of a user's facial expressions based on the visual data collected within the first preset time period using a hierarchical structure and a self-attention mechanism to extract the user's facial muscle movements, expressions, and micro-expression changes to obtain visual modal features; performing feature encoding on the speech data collected within the first preset time period to generate a vector representation containing contextual speech relationships to obtain speech modal features; and performing emotion recognition based on the visual modal features and speech modal features to obtain an emotion recognition result.

[0081] Figure 4 A flowchart illustrating a method for determining emotion recognition results according to an embodiment of this application is shown schematically.

[0082] Figure 5 The diagram illustrates a data flow diagram for determining the emotion recognition result according to an embodiment of this application.

[0083] like Figure 4 As shown, the method for determining the emotion recognition result based on emotion recognition according to the embodiments of this application may include steps S310 to S330.

[0084] In step S310, the contextual relationship of the user's facial expressions is modeled based on the visual data collected within the first preset time period through a hierarchical structure and self-attention mechanism, so as to extract the user's facial muscle movements, expressions and micro-expression changes to obtain visual modal features.

[0085] In the embodiments of this application, reference is made to Figure 5 Based on visual data collected within a first preset time period, a hierarchical structure and self-attention mechanism can be used for modeling. This allows for the acquisition of the contextual relationships of user facial expressions, revealing information such as changes in expression, thereby determining visual modal features. The hierarchical structure can be achieved by progressively extracting high-level features from the data through multi-layer neural networks. For example, the bottom layer might extract pixel-level edge information, the middle layer extracts facial contours, and the top layer extracts expression features. The self-attention mechanism allows the model to dynamically focus on more important parts of the data during processing. For instance, in facial expression recognition, the model might pay more attention to changes in the state of key areas such as the eyes, eyebrows, and mouth.

[0086] In step S320, the speech data collected within the first preset time period is feature encoded to generate a vector representation containing the contextual speech relationship, thereby obtaining speech modal features.

[0087] In the embodiments of this application, reference is made to Figure 5 Simultaneously, speech data collected within a first preset time period can be encoded using a speech processing model to determine vector representations that include contextual speech relationships, thus obtaining speech modal features. This speech processing model focuses not only on individual speech segments but also on contextual information such as speech rate, intonation variations, and pause patterns. Furthermore, the speech features are encoded into vector form, which captures emotional information within the speech. The resulting speech data features ultimately reflect the user's emotional state.

[0088] In step S330, emotion recognition is performed based on visual modal features and speech modal features to obtain the emotion recognition result.

[0089] In the embodiments of this application, reference is made to Figure 5 After obtaining visual and speech modal features, feature fusion can be performed, followed by emotion recognition based on the fused features. Specifically, a deep learning model can be used to process the fused features to identify the user's emotional state. Ultimately, the user's emotional state, such as anxiety, calmness, or confusion, can be output.

[0090] In the embodiments of this application, the method described above integrates and fuses visual and vocal modal features for multimodal recognition, which can more comprehensively capture the user's emotional state and reduce misjudgments that may arise from a single modality. Furthermore, before obtaining the aforementioned two modal features, information such as contextual relationships and changes in vocal expressions over time are also considered, thereby further improving the accuracy of emotion recognition. Accurate emotion recognition allows for more personalized interaction methods in subsequent steps, enhancing user trust in self-service terminals.

[0091] In the embodiments of this application, emotion recognition is performed based on visual modal features and speech modal features to obtain emotion recognition results, including: extracting the temporal dependency relationship between visual modal features and speech modal features through a multi-head attention mechanism to obtain associated features; normalizing and mapping the associated features to obtain an emotion state probability distribution; and obtaining the emotion recognition result based on the emotion state probability distribution.

[0092] In the embodiments of this application, the multi-head attention mechanism described above can simultaneously process visual modal features and speech modal features, learning the correlation between them to obtain associated features. Specifically, there is a temporal dependency between a user's facial expressions and speech tone. For example, a user may simultaneously display specific facial expressions while speaking. After obtaining the associated features, normalization mapping can be performed to obtain an emotional state probability distribution. Normalization mapping can scale the associated features to a fixed range, such as between 0 and 1, and the sum of all probability values ​​is 1. Finally, based on the emotional state probability distribution, the emotional state with the highest probability can be selected as the final emotion recognition result. For example, if the probability distribution is anxiety (0.7), calm (0.2), and confusion (0.1), then the emotion recognition result is "anxiety".

[0093] In the embodiments of this application, speech coding can employ a pre-trained speech model for deep feature encoding, directly generating a vector representation rich in contextual speech relationships from the original audio waveform. Emotion recognition uses a cross-modal attention mechanism to jointly model visual and speech features, dynamically capturing the temporal dependencies between the two modalities through a multi-head attention mechanism, and finally outputting the final emotion state probability distribution through Softmax.

[0094] For example, for a time step The video frame system extracts visual feature vectors. As a query (Q), the speech coding feature vector The attention weights are calculated as keys (K) and values ​​(V), as shown in Equation (1):

[0095] (1)

[0096] in, Where is the feature dimension scaling factor, and T is the transpose factor. Represents the set of real numbers. This represents the dimension of the feature vector.

[0097] By using this attention mechanism to weight and fuse speech features based on visual context, a multimodal feature vector can be obtained. For example, visual features capture a user's frowning motion while voice features capture a rise in pitch; merging these features can enhance the recognition of anxiety signals.

[0098] Multimodal feature vectors The classifier outputs the probability distribution of discrete emotion categories. These categories can include five subcategories: "calm," "anxious," "confused," "angry," and "happy," each corresponding to a probability value. For example, the output... This indicates that the system determines the current user to be anxious with a probability of 0.7.

[0099] Furthermore, it's possible to define a user's current anxiety state when the probability of the anxiety category is greater than 0.6, and consider it normal when it's below 0.4. This method defines a probability range for each category to achieve quantitative judgment and provide real-time reference for subsequent intelligent decision-making. Simultaneously, continuous emotion dimension values, including Arousal and Valence, can be calculated, with values ​​ranging from [value range missing]. This continuous value is obtained through a weighted mapping of discrete category probabilities; for example, the anxiety category corresponds to... It is used to quantify the user's psychological load or emotional intensity, thereby obtaining emotion recognition results.

[0100] In the embodiments of this application, a multi-head attention mechanism is used in the above method to extract the temporal correlation between visual modality features and speech modality features, which can more comprehensively capture the user's emotional state and reduce misjudgments or individual differences. Furthermore, through normalization processing, the feature vector can be mapped to a probability distribution of emotional states, thereby reducing misjudgments caused by differences in feature values. The normalized probability distribution more intuitively reflects the likelihood of the user's emotional state, improving the reliability of emotion recognition.

[0101] In the embodiments of this application, the business processing method based on emotion recognition further includes: collecting user operation status indicators, which include at least one of the following: single interface dwell time, number of clicks per unit time, number of consecutive invalid clicks, number of consecutive return operations, and menu level depth of the business operation path; judging the validity of the operation based on the operation status indicators to obtain a judgment result; when the judgment result indicates that the user has difficulty operating, determining the interface layout scheme associated with the operation status indicators from the rule base; and updating the layout of interface elements of the display interface based on the interface layout scheme.

[0102] Figure 6 The illustration shows a data flow diagram of the layout of interface elements in an updated display interface according to an embodiment of this application.

[0103] In the embodiments of this application, reference is made to Figure 6 The aforementioned operational status indicators can be used to measure user behavior when operating self-service devices, thereby determining whether users are confused or encountering difficulties. Specifically, if a user stays on a single screen for an excessively long time, it may indicate confusion about that screen's operation. Frequent clicking within a certain timeframe may indicate the user is trying to find the correct operation path. Continuous invalid clicks (such as clicking on invalid areas or repeated clicks) may indicate the user cannot find the correct operation method. Repeatedly clicking the back button may indicate the user is lost in the operation process. Entering excessively deep menu levels during operation may indicate the user is performing complex operations, easily leading to disorientation.

[0104] In the embodiments of this application, the validity of a user's operation can be determined by setting thresholds or rules, and further, it can be determined whether the user has encountered operational difficulties. When a user encounters operational difficulties, a corresponding interface layout scheme can be selected from the rule base. For example, if the user stays on the screen for too long, the interface layout scheme could be to enlarge the font or add prompts. If the user makes too many consecutive invalid clicks, the interface layout scheme could be to simplify the interface layout or highlight key buttons. After obtaining the interface layout scheme, the interface can be updated so that the adjusted layout of interface elements better matches the user's operating habits and current needs.

[0105] In the embodiments of this application, the aforementioned interface element layout may include font size, button size and spacing, page layout density, color contrast, and voice prompt methods. The updated display interface can always maintain a presentation state that adapts to the user's cognitive abilities and operating habits, improving interface readability, operational comprehensibility, and overall user-friendliness.

[0106] In the embodiments of this application, when the determination result indicates that the user's operation is abnormal, an assistance strategy is determined and executed to assist the user in business processing.

[0107] In the embodiments of this application, based on the aforementioned difficulties in user operation, thresholds or rules can be set to determine whether abnormal user operation has occurred. This indicates that the updated interface element layout may not be effective, or the user cannot complete the business process relying on the on-screen prompts. In this case, an assistance strategy can be implemented to assist the user in processing the business process manually or via voice. The assistance strategy may include voice prompts, on-site manual assistance, or remote manual assistance, etc.

[0108] In the embodiments of this application, the user's operational status indicators on the self-service terminal device are monitored in real time to determine whether there is confusion or operational abnormality. When the judgment result or the aforementioned emotion recognition result meets the preset trigger conditions, corresponding strategies can be adopted. For example, when the judgment result indicates that the user's operation is abnormal, intelligent semantic interpretation can be used first to provide the user's current operation with immediate and easy-to-understand operation instructions and guidance. During this process, the user's operation status and emotion recognition results are continuously collected. If the user's operational abnormality, anxiety state lasting for more than 10 seconds, or repeated invalid operations more than twice are detected, it is determined that the user is still confused and the processing is escalated to initiate on-site or remote manual intervention.

[0109] In the embodiments of this application, for on-site intervention, the system determines the user's physical location and assesses the availability of staff at the location based on terminal device information. Availability is determined as follows: the system queries the staff status at the location, including whether they are on duty, available, or currently under workload; if available staff are available, the system automatically generates a task notification and assigns it to the nearest available staff member via intranet message or management system instructions; if no staff are available, the system sends a scheduling request to the location supervisor for confirmation and staff allocation. After on-site staff arrive, the system continuously monitors the user's operational status and multimodal emotion indicators, or determines whether the user has overcome confusion or anxiety based on information reported by staff. When the user's operation returns to normal, the mood index falls below a preset threshold (e.g., below 0.4), or staff confirm that the user has resolved the problem, the system terminates the on-site intervention process and records the event status. If on-site intervention is not feasible (e.g., no staff at the location or no available staff), the system notifies the back-end staff via a message queue.

[0110] In the embodiments of this application, the above steps can determine the situation of user operation difficulties or abnormal user operation by judging the validity of the operation status indicators. Furthermore, targeted assistance strategies can be provided to meet the needs of users in different situations, especially elderly users or users who are not familiar with self-service terminal equipment.

[0111] In embodiments of this application, the emotion recognition-driven business processing method further includes: in response to a user's usage request, determining the user group to which the user belongs; and if it is determined that the user group to which the user belongs is an older user group, switching the working mode to a business assistance mode.

[0112] In the embodiments of this application, when the user is an older user, the working mode can be adjusted to a business assistance mode. In the business assistance mode, the interface style, font size, button spacing, color contrast, and voice broadcast speed can be automatically adjusted. This makes the interface more intuitive, clear, and in line with the cognitive characteristics of older users, and supports switching between multiple interaction methods to adapt to different user needs.

[0113] In the embodiments of this application, by responding to the user's usage request, determining the user group to which the user belongs, and switching to business assistance mode when the user belongs to the older user group, the above steps can improve the operating experience of older users, reduce the difficulties and anxiety of older users in the operation process, optimize resource allocation, and reduce operating costs.

[0114] The deep learning models involved in the above steps can be trained in the following ways.

[0115] S1, Data Preparation Stage.

[0116] On a cloud server, anonymized user interaction logs from various terminals are collected to construct a massive pre-training dataset. This dataset contains multimodal sample pairs, such as (audio streams, video frame sequences, touch sequences), along with their corresponding emotion labels and operation result labels. Finally, the data is cleaned and labeled.

[0117] S2, Model Training and Fine-tuning Stage.

[0118] An advanced, publicly available pre-trained model is used as the base model, and the base model is fine-tuned using the previously prepared business dataset. Finally, model distillation is employed to transform the large model into a lightweight model that can run on the server.

[0119] S3, System Integration and Continuous Learning Phase.

[0120] The optimized model is deployed to the backend server. New data generated during operation is processed and sent to the training platform as incremental learning samples for periodic iterative fine-tuning of the model, thereby achieving continuous model optimization and system self-evolution.

[0121] Ultimately, the execution of the business processing method follows this intelligent closed loop: First, the user's identity is automatically identified via facial recognition or ID card scanning, and the interface is adaptively switched. Then, during the business processing, the user's operational behavior, voice, and facial expression data are captured in real time. Based on this data, emotion recognition is performed to generate dynamic emotion recognition results. Simultaneously, the user's operational status can be judged based on operational status indicators. Then, judgments are made according to preset rules: if confusion or abnormal user operation is detected, the semantic explanation module can be activated first to provide a simple explanation; if the problem persists or the emotional abnormality intensifies, the engine will prioritize dispatching staff for on-site assistance. If this fails, it will immediately escalate to the remote assistance module, where a human customer service representative will provide synchronous and voice guidance to complete the business. The interaction data during the process can be recorded and stored.

[0122] Based on the above-described emotion recognition-driven business processing method, embodiments of this application also provide an emotion recognition-driven business processing apparatus. The following will be combined with... Figure 7 The device is described in detail.

[0123] Figure 7 The diagram illustrates a structural block diagram of an emotion recognition-driven business processing apparatus according to an embodiment of this application.

[0124] like Figure 7 As shown, the emotion recognition-driven business processing device 700 in this embodiment includes a first acquisition module 710, a first recognition module 720, a matching module 730, and a first execution module 740.

[0125] The first acquisition module 710 is used to acquire the user's visual and voice data in response to being in a business assistance mode. In one embodiment, the acquisition module 710 can be used to perform the step S210 described above, which will not be repeated here.

[0126] The first recognition module 720 is used to perform emotion recognition based on visual and speech data collected within a first preset time period, and obtain an emotion recognition result. In one embodiment, the first recognition module 720 can be used to execute step S220 described above, which will not be repeated here.

[0127] The matching module 730 is used to perform assistance strategy matching based on the emotion recognition result to determine the target assistance strategy. In one embodiment, the matching module 730 can be used to perform step S230 described above, which will not be repeated here.

[0128] The first execution module 740 is used to execute the target assistance strategy to assist the user in business processing. In one embodiment, the execution module 740 can be used to execute the step S240 described above, which will not be repeated here.

[0129] According to an embodiment of this application, the matching module 730 may include a first matching submodule and a second matching submodule.

[0130] The first matching submodule is used to obtain the user's first emotional state based on multiple emotion recognition results determined within a second preset time period.

[0131] The second matching submodule is used to determine the target assistance strategy as a machine assistance strategy when the first emotional state is represented as a negative state.

[0132] According to an embodiment of this application, the execution module 740 may include a first execution submodule, a second execution submodule, and a third execution submodule.

[0133] The first execution submodule is used to collect the user's current operation data when the target assistance strategy is a machine assistance strategy.

[0134] The second execution submodule is used to process the current operation data using the intelligent agent to obtain operation instructions.

[0135] The third execution submodule is used to display operation instructions on the display interface based on the operation location related to the current operation data, so as to guide the user to perform business processing based on the operation instructions.

[0136] According to embodiments of this application, the second execution module may include a first execution unit and a second execution unit.

[0137] The first execution unit is used to generate prompt text based on current operation data, interface context, and user history.

[0138] The second execution unit is used to input the prompt text into the large language model included in the agent, so as to use the large language model to perform semantic understanding and dynamic interpretation of the user's operation steps and generate operation instructions.

[0139] According to an embodiment of this application, the emotion recognition-driven business processing apparatus 700 further includes a second recognition module and a strategy determination module.

[0140] The second recognition module is used to obtain the user's second emotional state based on multiple emotion recognition results determined within a third preset time period after the execution of the machine-assisted strategy.

[0141] The strategy determination module is used to determine the target assistance strategy as a human assistance strategy when the second emotional state is expressed as a negative state.

[0142] According to an embodiment of this application, the execution module 740 may further include a fourth execution submodule.

[0143] The fourth execution submodule is used to send a task notification to the branch where the self-service terminal belongs when the target assistance strategy is manual assistance, so that the staff of the branch can assist the user in handling business on-site.

[0144] According to an embodiment of this application, the execution module 740 may further include a fifth execution submodule.

[0145] The fifth execution submodule is used to send task notifications to the customer service center via message queue when the target assistance strategy is manual assistance, so that the staff of the customer service center can remotely assist users in handling business.

[0146] According to an embodiment of this application, the first identification module 720 may include a first identification submodule, a second identification submodule, and a third identification submodule.

[0147] The first recognition submodule is used to model the contextual relationship of the user's facial expressions based on the visual data collected within a first preset time period through a hierarchical structure and self-attention mechanism, so as to extract the user's facial muscle movements, expressions and micro-expression changes to obtain visual modal features.

[0148] The second recognition submodule is used to perform feature encoding on the speech data collected within the first preset time period to generate a vector representation containing the contextual speech relationship, thereby obtaining speech modal features.

[0149] The third recognition submodule is used to perform emotion recognition based on visual modal features and speech modal features to obtain emotion recognition results.

[0150] According to embodiments of this application, the third identification submodule may include a first identification unit, a second identification unit, and a third identification unit.

[0151] The first recognition unit is used to extract the temporal dependency between visual modal features and speech modal features through a multi-head attention mechanism to obtain associated features.

[0152] The second identification unit is used to normalize and map the associated features to obtain the probability distribution of emotional states.

[0153] The third recognition unit is used to obtain emotion recognition results based on the probability distribution of emotion states.

[0154] According to embodiments of this application, the emotion recognition-driven business processing device 700 further includes a second recognition module, a discrimination module, a scheme association module, and an update module.

[0155] The second identification module is used to collect user operation status indicators, which include at least one of the following: duration of stay on a single interface, number of clicks per unit time, number of consecutive invalid clicks, number of consecutive return operations, and menu level depth of the business operation path.

[0156] The discrimination module is used to determine the validity of an operation based on its operational status indicators and obtain the discrimination result.

[0157] The scheme association module is used to determine the interface layout scheme associated with the operation status index from the rule base when the judgment result indicates that the user operation is difficult.

[0158] The update module is used to update the layout of interface elements in the displayed interface based on the interface layout scheme.

[0159] According to an embodiment of this application, the emotion recognition-driven business processing device 700 further includes a second execution module.

[0160] The second execution module is used to determine and execute assistance strategies when the judgment result indicates that the user's operation is abnormal, so as to assist the user in business processing.

[0161] According to embodiments of this application, the emotion recognition-driven business processing apparatus 700 further includes a grouping module and a switching module.

[0162] The grouping module is used to determine the user group to which a user belongs in response to a user's usage request.

[0163] The switching module is used to switch the working mode to the business assistance mode when it is determined that the user belongs to the older user group.

[0164] According to embodiments of this application, any plurality of modules among the first acquisition module 710, the first identification module 720, the matching module 730, and the first execution module 740 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this application, at least one of the first acquisition module 710, the first identification module 720, the matching module 730, and the first execution module 740 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array, a programmable logic array, a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit, or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three implementation methods. Alternatively, at least one of the first acquisition module 710, the first identification module 720, the matching module 730, and the first execution module 740 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0165] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing an emotion recognition-driven business processing method according to an embodiment of this application.

[0166] like Figure 8 As shown, an electronic device 800 according to an embodiment of this application includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory 802 or a program loaded from a storage portion 808 into a random access memory 803. The processor 801 may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a dedicated microprocessor. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for executing different steps of the method flow according to an embodiment of this application.

[0167] Random access memory 803 stores various programs and data required for the operation of electronic device 800. Processor 801, read-only memory 802, and random access memory 803 are interconnected via bus 804. Processor 801 executes various steps of the method flow according to embodiments of this application by executing programs stored in read-only memory 802 and / or random access memory 803. It should be noted that the programs may also be stored in one or more memories other than read-only memory 802 and random access memory 803. Processor 801 may also execute various steps of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0168] According to embodiments of this application, the electronic device 800 may further include an input / output interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the input / output interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube, liquid crystal display, etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card, such as a local area network card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0169] Embodiments of this application also provide a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0170] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof. In embodiments of this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include the read-only memory 802 described above, and / or random access memory 803, and / or one or more memories other than read-only memory 802 and random access memory 803.

[0171] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this application.

[0172] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1209, and / or installed from the removable medium 1211. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0173] In embodiments of this application, the computer program can be downloaded and installed from a network via communication section 1209, and / or installed from removable medium 1211. When the computer program is executed by processor 1201, it performs the functions defined in the system of this application embodiment. According to embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0174] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0175] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0176] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A business processing method based on emotion recognition, applied to a self-service terminal, the method comprising: In response to being in business assistance mode, collect the user's visual and voice data; Emotion recognition is performed based on visual and speech data collected within the first preset time period to obtain emotion recognition results; Based on the emotion recognition results, assistance strategies are matched to determine the target assistance strategy; as well as The target assistance strategy is executed to assist the user in business processing.

2. The method according to claim 1, wherein, The step of matching assistance strategies based on the emotion recognition results to determine the target assistance strategy includes: Based on multiple emotion recognition results determined within a second preset time period, the user's first emotional state is obtained; and When the first emotional state is represented as a negative state, the target assistance strategy is determined to be a machine assistance strategy.

3. The method according to claim 2, wherein, The execution of the target assistance strategy to assist the user in business processing includes: When the target assistance strategy is the machine assistance strategy, the user's current operation data is collected; The agent processes the current operation data to obtain operation instructions; and Based on the operation location related to the current operation data, the operation instructions are displayed on the display interface to guide the user to perform business processing based on the operation instructions.

4. The method according to claim 3, wherein, The process of using an intelligent agent to process the current operation data to obtain operation description information includes: Based on the current operation data, interface context, and the user's historical behavior, generate prompt text; and The prompt text is input into the large language model included in the intelligent agent, so as to use the large language model to perform semantic understanding and dynamic interpretation of the user's operation steps to generate the operation instruction information.

5. The method according to claim 2, further comprising: Based on multiple emotion recognition results determined within a third preset time period after the execution of the machine-assisted strategy, the user's second emotional state is obtained. as well as When the second emotional state is expressed as a negative state, the target assistance strategy is determined to be a human assistance strategy.

6. The method according to claim 5, wherein, The execution of the target assistance strategy to assist the user in business processing includes: When the target assistance strategy is the manual assistance strategy, a task notification is sent to the branch where the self-service terminal belongs, so that the staff of the branch can assist the user in processing business on-site.

7. The method according to claim 5, wherein, The execution of the target assistance strategy to assist the user in business processing includes: When the target assistance strategy is the manual assistance strategy, a task notification is sent to the customer service center via a message queue so that the staff of the customer service center can remotely assist the user in handling business.

8. The method according to claim 1, wherein, The emotion recognition based on visual and speech data collected within a first preset time period, to obtain the emotion recognition result, includes: By using a hierarchical structure and self-attention mechanism, the contextual relationship of the user's facial expressions is modeled based on the visual data collected within the first preset time period, so as to extract the user's facial muscle movements, expressions and micro-expression changes, and obtain visual modal features. The speech data collected within the first preset time period is feature-encoded to generate a vector representation containing contextual speech relationships, thereby obtaining speech modal features; and Emotion recognition is performed based on the visual modal features and the speech modal features to obtain the emotion recognition result.

9. The method according to claim 8, wherein, The emotion recognition based on the visual modal features and the speech modal features, to obtain the emotion recognition result, includes: The temporal dependency between the visual modality features and the speech modality features is extracted using a multi-head attention mechanism to obtain associated features; The associated features are normalized and mapped to obtain the probability distribution of emotional states; and Based on the probability distribution of the emotional state, the emotion recognition result is obtained.

10. The method according to claim 1, further comprising: Collect user operation status indicators, which include at least one of the following: single interface dwell time, number of clicks per unit time, number of consecutive invalid clicks, number of consecutive return operations, and menu level depth of the business operation path; The effectiveness of the operation is determined based on the aforementioned operation status indicators, and the determination result is obtained. If the judgment result indicates that the user has difficulty operating, an interface layout scheme associated with the operation status indicator is determined from the rule base; as well as Based on the aforementioned interface layout scheme, update the layout of the interface elements displayed on the screen.

11. The method of claim 10, further comprising: If the determination result indicates that the user's operation is abnormal, an assistance strategy is determined and executed to assist the user in processing business.

12. The method according to claim 1, further comprising: In response to the user's usage request, determine the user group to which the user belongs; as well as If it is determined that the user belongs to the older user group, switch the working mode to the business assistance mode.

13. A business processing device based on emotion recognition, comprising: The data acquisition module is used to collect the user's visual and voice data in response to being in business assistance mode; The recognition module is used to perform emotion recognition based on visual and voice data collected within a first preset time period, and obtain the emotion recognition result. The matching module is used to perform assistance strategy matching based on the emotion recognition results in order to determine the target assistance strategy; as well as The execution module is used to execute the target assistance strategy in order to assist the user in business processing.

14. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 12.

16. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 12.

Images