Business handling progress output method, device, equipment and storage medium

Abstract

The application provides a service handling progress output method, device and equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: obtaining communication data between a customer service personnel and a user and a target service handled by the user; determining an exchange state of the customer service personnel and the user according to user-side audio and customer service-side audio; in response to the exchange state being a suspended exchange, identifying a user emotion corresponding to the user-side data by using a trained YOLO v5 emotion recognition model; in response to the user emotion comprising at least one first preset abnormal emotion, outputting first alarm information to the customer service personnel; and in response to the exchange state being continuously in the suspended state within a preset time after the first alarm information is outputted, obtaining a service handling progress of the target service and outputting the service handling progress to the user. The scheme of the application can avoid a long waiting state of the user, enable the user to know the service handling progress, and improve the experience of the user in remotely handling the service.

Description

Technical Field

[0001] This application relates to artificial intelligence, and more particularly to a method, apparatus, device, and storage medium for outputting business processing progress. Background Technology

[0002] Remote business processing is a new business processing model that allows users to handle business remotely via the internet without having to go to a designated location, offering convenience and speed. Therefore, remote business processing is gradually becoming the choice of more and more users.

[0003] Currently, during remote business processing, communication between users and customer service personnel primarily relies on audio and video. Users can only hear or see the customer service representative's voice, but cannot track the specific progress of their transactions. Therefore, when processing time is long, customer service personnel, focused on the task at hand, may neglect the user experience, leading to prolonged waiting times and a poor user experience. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for outputting business processing progress, in order to solve the problems of users being in a long waiting state, unable to know the progress of business processing, and having a poor user experience when handling business remotely.

[0005] According to a first aspect of this application, a method for outputting business processing progress is provided, comprising:

[0006] Acquire communication data between customer service personnel and users, as well as the target business processed by users; the communication data includes user-side data and customer service-side data; user-side data includes user-side audio or user-side audio and video; customer service-side data includes customer service-side audio.

[0007] The communication status between customer service personnel and users is determined based on the user-side audio and the customer service-side audio; the communication status is either in progress or paused.

[0008] In response to the communication state being paused, a trained YOLO v5 emotion recognition model is used to identify the user's emotion corresponding to the user-side data; the trained YOLO v5 recognition model is obtained by training a preset YOLO v5 emotion recognition model with at least one of the semantic emotion training dataset, the voice emotion training dataset, and the facial emotion training dataset.

[0009] In response to the user's emotions including at least one first preset abnormal emotion, a first alarm message is output to the customer service personnel; the first alarm message is used to remind the customer service personnel to soothe the user's emotions.

[0010] In response to the output of the first alarm message, the communication status remains in a paused state for a preset period of time, the business processing progress of the target business is obtained, and the business processing progress is output to the user.

[0011] According to a second aspect of this application, a business processing progress output device is provided, comprising:

[0012] The acquisition module is used to acquire communication data between customer service personnel and users, as well as the target business handled by users; the communication data includes user-side data and customer service-side data; user-side data includes user-side audio or user-side audio and video; customer service-side data includes customer service-side audio or customer service-side audio and video.

[0013] The determination module is used to determine the communication status between the customer service representative and the user based on the user-side audio and the customer service-side audio; the communication status is either in progress or paused.

[0014] The recognition module is used to identify the user's emotion corresponding to the user-side data in response to the communication state being paused. The trained YOLO v5 emotion recognition model is obtained by training a preset YOLO v5 emotion recognition model with at least one of the semantic emotion training dataset, the voice emotion training dataset, and the facial emotion training dataset.

[0015] The first output module is configured to output a first alarm message to customer service personnel in response to the user's emotion including at least one first preset abnormal emotion; the first alarm message is used to remind customer service personnel to soothe the user's emotions.

[0016] The second output module is used to respond to the fact that the communication state remains in a paused state for a preset time after the first alarm information is output, to obtain the business processing progress of the target business, and to output the business processing progress to the user.

[0017] According to a third aspect of this application, an electronic device is provided, comprising: a memory, a processor, and an output device;

[0018] The memory, the processor, and the output device are communicatively connected;

[0019] The memory stores computer-executed instructions;

[0020] The output device is used to output the first alarm information and the business processing progress;

[0021] The processor executes computer execution instructions stored in the memory to implement the method as described in the first aspect.

[0022] According to a fourth aspect of this application, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, which, when executed by a processor, are used to implement the method as described in the first aspect.

[0023] According to a fifth aspect of this application, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described in the first aspect.

[0024] The business processing progress output method, apparatus, device, and storage medium provided in this application acquire communication data between customer service personnel and users, as well as the target business being processed by the user. The communication data includes user-side data and customer service-side data. User-side data includes user-side audio or user-side audio-visual data; customer service-side data includes customer service-side audio. The communication status between the customer service personnel and the user is determined based on the user-side audio and customer service-side audio. The communication status is either in progress or paused. In response to the communication status being paused, a trained YOLO v5 emotion recognition model is used to identify the user's emotion corresponding to the user-side data. The trained YOLO v5 recognition model is obtained by training a preset YOLO v5 emotion recognition model using at least one of a semantic emotion training dataset, a voice emotion training dataset, and a facial emotion training dataset. In response to the user's emotion including at least one first preset abnormal emotion, a first alarm message is output to the customer service personnel. The first alarm message is used to remind the customer service personnel to soothe the user's emotions. In response to the output of the first alarm message, the communication status remains paused for a preset period of time, the business processing progress of the target business is acquired, and the business processing progress is output to the user. Because a trained YOLO v5 emotion recognition model is used to identify user emotions corresponding to user-side data when the communication state is paused, the user's emotions when the customer service representative is not interacting with the user can be obtained. Furthermore, if the user's emotions include at least one first preset abnormal emotion, a first alarm message is output to the customer service representative to remind them to soothe the user's emotions, thus preventing the user from being in a waiting state for an extended period. Simultaneously, since the communication state remains paused for a preset period after the first alarm message is output, it indicates that the customer service representative has not successfully received the first alarm message. The progress of the target service is then obtained and displayed to the user, allowing the user to be aware of the service progress. Therefore, the solution of this application can prevent users from being in a waiting state for an extended period, enabling users to be aware of the service progress and improving the user's experience of remotely handling business. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0026] Figure 1 This is a network architecture diagram corresponding to the application scenario of the business processing progress output method provided in the embodiments of this application;

[0027] Figure 2 This is a flowchart illustrating the business processing progress output method provided in Embodiment 1 of this application;

[0028] Figure 3 This is a flowchart illustrating the business processing progress output method provided in Embodiment 2 of this application;

[0029] Figure 4 This is a schematic diagram of the business processing progress output device provided in Embodiment 4 of this application;

[0030] Figure 5 This is a schematic diagram of the structure of an electronic device provided according to Embodiment 7 of this application.

[0031] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0032] The terms "first," "second," "third," etc., used in the specification, claims, and drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] The prior art involved in this application will be described in detail and analyzed below.

[0034] Remote service processing is a user-initiated method that involves conducting business online through voice or video calls with customer service personnel. It eliminates the need for users to travel to a specific location; as long as they have an internet connection, they can complete the process quickly and easily. However, because communication between users and customer service personnel is limited to online interactions, in voice communication, users can only track the progress of their application based on the audio. In video communication, users can only see a cartoon avatar or the actual customer service representative, without being able to track the processing progress. This can lead to situations where, for time-consuming transactions, such as those requiring extensive information retrieval, customer service personnel may neglect to engage with the user during the process, resulting in long waiting times and a poor user experience.

[0035] In summary, existing technologies present problems such as users spending long periods of time waiting, being unable to track the progress of their transactions, and experiencing a poor user experience when handling business remotely.

[0036] Therefore, when faced with the problems in existing technologies, the inventors, through creative research, discovered that customer service personnel, due to the need to search for information and verify data during the process of handling business, cannot focus all their attention on the user, and thus easily overlook the user's emotions. This leads to situations such as users being in a long waiting state, users' questions not being answered, and users experiencing abnormal emotions being left unattended. Therefore, under the premise that customer service personnel must focus most of their attention on handling business for users, in order to improve the user experience and avoid the aforementioned situations that degrade user experience, the system can identify the user's emotions after the customer service personnel pause their communication to start handling business. If the user experiences abnormal emotions, the system can remind the customer service personnel to soothe the user's emotions. Furthermore, if the customer service personnel are too focused on handling business to receive reminders or soothe the user's emotions, the system can obtain the business processing progress and output it to the user, thus avoiding users being in a long waiting state, soothing user emotions, and improving user experience.

[0037] Therefore, the inventors propose the technical solution of this application, which involves acquiring communication data between customer service personnel and users, and the target business handled by the user; the communication data includes user-side data and customer service-side data; user-side data includes user-side audio or user-side audio-visual; customer service-side data includes customer service-side audio; determining the communication status between customer service personnel and users based on user-side audio and customer service-side audio; the communication status is either in progress or paused; in response to a paused communication status, a trained YOLO v5 emotion recognition model is used to identify the user emotion corresponding to the user-side data; the trained YOLO v5 recognition model is obtained by training a preset YOLO v5 emotion recognition model using at least one of a semantic emotion training dataset, a voice emotion training dataset, and a facial emotion training dataset; in response to the user emotion including at least one first preset abnormal emotion, a first alarm message is output to the customer service personnel; the first alarm message is used to remind the customer service personnel to soothe the user's emotions; in response to a preset time after the output of the first alarm message, the communication status remains in a paused state, the business processing progress of the target business is acquired, and the business processing progress is output to the user. Because a trained YOLO v5 emotion recognition model is used to identify user emotions corresponding to user-side data when the communication state is paused, it is possible to obtain user emotions when customer service personnel are not interacting with the user. Furthermore, if the user's emotions include at least one pre-defined abnormal emotion, a first alarm message is output to the customer service personnel to remind them to soothe the user's emotions, thus preventing the user from being in a waiting state for an extended period. Simultaneously, since the communication state remains paused for a pre-defined period after the first alarm message is output, it indicates that the customer service personnel have not successfully received the first alarm message. The progress of the target business is then obtained and displayed to the user, allowing the user to be informed of the business processing progress. Therefore, the solution proposed in this application can prevent users from being in a waiting state for an extended period, enabling users to be informed of the business processing progress and improving the user's experience of remote business processing.

[0038] The business processing progress output method, apparatus, equipment, and storage medium provided in this application aim to solve the above-mentioned technical problems of the prior art. The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0039] The network architecture and application scenarios of the business processing progress output method provided in the embodiments of this application will be described below. When the following description refers to the accompanying drawings, unless otherwise indicated, the same data in different drawings represent the same or similar elements.

[0040] Figure 1 This is a network architecture diagram corresponding to the application scenario of the business processing progress output method provided in the embodiments of this application. For example... Figure 1 As shown in the figure, the network architecture corresponding to an application scenario provided in this application embodiment includes: an electronic device 10, a user-side device 11, and a customer service-side device 12. The electronic device 10, the user-side device 11, and the customer service-side device 12 are interconnected.

[0041] The electronic device 10 is pre-configured with a trained YOLO v5 emotion recognition model.

[0042] Electronic device 10 acquires communication data between customer service personnel and users, as well as the target business handled by users, through communication connections with user-side device 11 and customer service-side device 12. The communication data includes user-side data and customer service-side data; user-side data includes user-side audio or user-side audio and video; customer service-side data includes customer service-side audio.

[0043] Electronic device 10 determines the communication status between customer service personnel and users based on user-side audio and customer service-side audio. The communication status is either in progress or paused.

[0044] When the electronic device 10 responds to the communication state as "pause communication," it uses a trained YOLO v5 emotion recognition model to identify the user's emotion corresponding to the user-side data. The trained YOLO v5 recognition model is obtained by training a preset YOLO v5 emotion recognition model using at least one of the following datasets: semantic emotion training dataset, voice emotion training dataset, and facial emotion training dataset.

[0045] In response to a user's emotions, including at least one first preset abnormal emotion, the electronic device 10 outputs a first alarm message to customer service personnel via a communication connection with the customer service side device 12; the first alarm message is used to remind customer service personnel to soothe the user's emotions.

[0046] Within a preset time after the first alarm message is output, the electronic device 10 maintains a paused communication state, acquires the service processing progress of the target service, and outputs the service processing progress to the user through the communication connection with the user-side device 11.

[0047] In the application scenario of this application, the electronic device 10 can also be a customer service device 12 or a user device 11, and the methods or steps performed by the electronic device 10 can also be implemented by the customer service device 12 or the user device 11.

[0048] The embodiments of this application will now be described with reference to the accompanying drawings. The embodiments described below do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0049] Example 1

[0050] Figure 2 This is a flowchart illustrating the business processing progress output method provided in Embodiment 1 of this application. Figure 2 As shown, the executing entity of this application is a business processing progress output device, which is located in an electronic device. The business processing progress output method provided in this embodiment includes steps 201 to 205.

[0051] Step 201: Obtain communication data between customer service personnel and users, as well as the target business handled by users; communication data includes user-side data and customer service-side data; user-side data includes user-side audio or user-side audio and video; customer service-side data includes customer service-side audio.

[0052] In this embodiment, customer service personnel communicate with users through user-side devices and customer service-side devices. Therefore, both user-side and customer service-side devices can acquire the communication data between customer service personnel and users. Electronic devices can acquire communication data by communicating with at least one of the user-side and customer service-side devices. It is understood that user-side and customer service-side data are continuously generated during the communication process between customer service personnel and users; therefore, electronic devices can continuously and in real-time acquire the communication data between customer service personnel and users.

[0053] Communication data includes user-side data and customer service-side data. User-side data includes user-side audio or user-side audio-video. User-side audio-video includes both user-side audio and user-side video. Customer service-side data includes either customer service-side audio or customer service-side audio-video. Customer service-side audio-video includes both customer service-side audio and customer service-side video.

[0054] User-side audio refers to the audio input by the user during the interaction between the customer service representative and the user. For example, user-side audio may include what the user says to the customer service representative. User-side audio can be captured by the user-side device through its audio input device. For example, the user-side device may capture user-side audio through its microphone. User-side video refers to the video input by the user during the interaction between the customer service representative and the user. It can be captured by the user-side device through its video capture device. For example, user-side video may include the user's facial expressions and body language during the interaction with the customer service representative. User-side video is captured by the user-side device through its front-facing camera.

[0055] Customer service-side audio refers to the audio input by customer service personnel during interactions with users. For example, customer service-side audio could be what the customer service personnel say to the user. Customer service-side audio can be captured by the customer service-side device through its audio input device. For example, the customer service-side device can capture customer service-side audio through its microphone. Customer service-side video refers to the video input by customer service personnel during interactions with users, and can be captured by the user-side device through its video capture device. For example, customer service-side video can include the customer service personnel's facial expressions and body language during interactions with users. Customer service-side video can be captured by the customer service-side device through its front-facing camera, or it can be captured by a camera device communicatively connected to the customer service-side device.

[0056] The target service requested by a user refers to the service the user intends to complete when initiating a remote service request. The user can input the target service on their device when initiating the remote service request. Alternatively, the target service can be input or selected by customer service personnel on their device during communication with the user. Electronic devices can obtain the target service by communicating with either the user-side device or the customer-side device.

[0057] Step 202: Determine the communication status between the customer service representative and the user based on the user-side audio and the customer service-side audio; the communication status is either in progress or paused.

[0058] In this embodiment, it is understood that if a customer service representative and a user are communicating, at least one party will be speaking; if the communication between the customer service representative and the user is paused, neither the representative nor the user may be speaking, or only the user may be speaking. Therefore, it is possible to determine whether the user is speaking based on the user's audio, and whether the customer service representative is speaking based on the customer service representative's audio. Furthermore, the communication status between the customer service representative and the user can be determined based on the speaking activity of both parties. Specifically, if neither the customer service representative nor the user is speaking, and the duration of this silence exceeds a first preset duration, it can be determined that the communication between the customer service representative and the user has paused. Optionally, if the customer service representative is not speaking, and the duration of this silence exceeds a first preset duration, it can be determined that the communication between the customer service representative and the user has paused.

[0059] Specifically, the system can determine whether a customer service representative is speaking based on the decibel level of the audio from the customer service side, and whether a user is speaking based on the decibel level of the audio from the user side. For example, if the decibel level of the user-side audio is less than a first preset threshold, it is determined that the user is not speaking; if the decibel level of the user-side audio is greater than or equal to the first preset threshold, it is determined that the user is speaking. Similarly, if the decibel level of the customer service representative's audio is less than a second preset threshold, it is determined that the customer service representative is not speaking; if the decibel level of the customer service representative's audio is greater than or equal to the second preset threshold, it is determined that the customer service representative is speaking.

[0060] Step 203: In response to the communication state being paused, the trained YOLO v5 emotion recognition model is used to identify the user's emotion corresponding to the user-side data. The trained YOLO v5 recognition model is obtained by training the preset YOLO v5 emotion recognition model with at least one of the semantic emotion training dataset, the voice emotion training dataset, and the facial emotion training dataset.

[0061] In this embodiment, if the communication status is paused, it indicates that customer service personnel are processing business for the user, and may be focusing most of their attention on the business rather than the user. If the user asks a question or shows impatience at this time, the customer service personnel may not be able to answer the question or soothe the user's emotions in a timely manner. Therefore, when the customer service personnel stop communicating with the user and turn to operations such as verification, that is, when the communication status is paused, the electronic device continues to acquire user-side data and uses a trained YOLO v5 emotion recognition model to perform emotion recognition on the user-side data when the communication status is paused, thereby obtaining the user's emotion corresponding to the user-side data at the time of the pause.

[0062] Optionally, the electronic device may begin continuously capturing user-side data only when the communication state is paused, and may stop and continue waiting for the communication state to change back to paused as the communication state changes to active.

[0063] Optionally, in response to a paused communication state, the electronic device can obtain the duration of the paused communication state. If the duration of the paused communication state exceeds a preset pause duration, a trained YOLOv5 emotion recognition model is used to identify the user's emotion corresponding to the user-side data. The preset pause duration can be 30 seconds.

[0064] Step 204: In response to the user's emotions, including at least one first preset abnormal emotion, output a first alarm message to the customer service personnel; the first alarm message is used to remind the customer service personnel to soothe the user's emotions.

[0065] In this embodiment, the first preset abnormal emotion may include emotions such as complaining, anger, impatience, and confusion. If the user's emotion includes at least one of the first preset abnormal emotions, it is necessary to remind customer service personnel to soothe the user's emotions, and therefore, a first alarm message is output to the customer service personnel.

[0066] Specifically, the electronic device can communicate with the customer service device and instruct the customer service device to output the first alarm information on its interactive interface. The first alarm information can be output in the form of voice, text pop-up, image pop-up, etc., without limitation here.

[0067] Step 205: In response to the output of the first alarm information, the communication status remains in the paused state for a preset period of time, the business processing progress of the target business is obtained, and the business processing progress is output to the user.

[0068] In this embodiment, if the communication status remains in a paused state within a preset time after the electronic device outputs the first alarm information, it indicates that the customer service personnel have not calmed the user's emotions. Therefore, the electronic device can obtain the business processing progress of the target business and output the business processing progress to the user in order to calm the user's emotions.

[0069] For example, customer service personnel communicate with the business server through customer service-side devices to process business for users on the business server. Electronic devices can communicate with the business server to obtain the processing progress of the target business and send it to the user-side devices so that the user-side devices can output the processing progress to the user.

[0070] The business processing progress output method provided in this embodiment obtains communication data between customer service personnel and users, as well as the target business processed by the user. The communication data includes user-side data and customer service-side data. User-side data includes user-side audio or user-side audio-visual data; customer service-side data includes customer service-side audio. The communication status between customer service personnel and users is determined based on the user-side audio and customer service-side audio. The communication status is either in progress or paused. In response to a paused communication status, a trained YOLO v5 emotion recognition model is used to identify the user's emotion corresponding to the user-side data. The trained YOLO v5 recognition model is obtained by training a preset YOLO v5 emotion recognition model using at least one of a semantic emotion training dataset, a voice emotion training dataset, and a facial emotion training dataset. In response to the user's emotion including at least one first preset abnormal emotion, a first alarm message is output to the customer service personnel. The first alarm message is used to remind the customer service personnel to soothe the user's emotions. In response to a preset time after the first alarm message is output, the communication status remains paused, the business processing progress of the target business is obtained, and the business processing progress is output to the user. Because a trained YOLO v5 emotion recognition model is used to identify user emotions corresponding to user-side data when the communication state is paused, it is possible to obtain user emotions when customer service personnel are not interacting with the user. Furthermore, if the user's emotions include at least one pre-defined abnormal emotion, a first alarm message is output to the customer service personnel to remind them to soothe the user's emotions, thus preventing the user from being in a waiting state for an extended period. Simultaneously, since the communication state remains paused for a pre-defined period after the first alarm message is output, it indicates that the customer service personnel have not successfully received the first alarm message. The progress of the target business is then obtained and displayed to the user, allowing the user to be informed of the business processing progress. Therefore, the solution proposed in this application can prevent users from being in a waiting state for an extended period, enabling users to be informed of the business processing progress and improving the user's experience of remote business processing.

[0071] Optionally, step 202, "determine the communication status between customer service personnel and users based on user-side audio and customer service-side audio," can be further refined to include step 301 and step 304.

[0072] Step 301: Obtain the first duration when the decibel value of the user-side audio is less than the first preset threshold; the first duration has a first preset duration.

[0073] In this embodiment, the user-side audio and customer service-side audio may include audio sampling parameters. The electronic device can calculate the decibel values ​​of the user-side audio and customer service-side audio at various times based on the audio sampling parameters. For example, the audio sampling parameters include: the amplitude Prms value of the sound sampling point and the maximum sound amplitude Pref. The electronic device can calculate the decibel values ​​of the audio at various times using 20lg(Prms / Pref). Besides the example method, other methods can also be used to calculate the decibel values ​​of the user-side audio and customer service-side audio at various times; this embodiment does not limit this method.

[0074] In this embodiment, the electronic device finds a first duration from the user-side audio based on the decibel values ​​at various times. The decibel value at each time point within the first duration is less than a first preset threshold. The first preset threshold can be the minimum decibel value at which customer service personnel can clearly hear the user speaking.

[0075] Step 302: Determine the decibel value of the customer service-side audio during the second duration; the second duration has a second preset duration; the second duration is continuous with the first duration and is later than the first duration.

[0076] In this embodiment, the user-side audio and the customer service-side audio are two audio segments that run concurrently in time. After acquiring the first duration, the electronic device can determine the second duration in the customer service-side audio based on the first duration, and determine the communication status between the customer service representative and the user as paused communication based on the decibel values ​​at various moments in the second duration of the customer service-side audio.

[0077] Step 303: In response to the fact that the decibel value of the audio on the customer service side is less than the second preset threshold at each moment during the second duration, it is determined that the communication status between the customer service personnel and the user is paused.

[0078] In this embodiment, the second preset threshold can be the minimum decibel level at which a user can clearly hear the customer service representative speaking. The second preset threshold can be the same as the first preset threshold.

[0079] Step 304: In response to the fact that the decibel value of the audio on the customer service side is greater than or equal to the second preset threshold at any time during the second duration, determine that the communication status between the customer service personnel and the user is that they are communicating.

[0080] In this embodiment, if the decibel value of the customer service audio at each moment during the second duration is less than the second preset threshold, then the customer service representative is not speaking, the customer service representative pauses communication with the user, and the communication status between the customer service representative and the user is determined as paused communication. If the decibel value of the customer service audio at any moment during the second duration is greater than or equal to the second preset threshold, then the customer service representative is speaking to the user, the customer service representative pauses communication, and the communication status between the customer service representative and the user is determined as ongoing communication.

[0081] The business processing progress output method provided in this embodiment obtains a first duration in which the decibel value of the user-side audio is less than a first preset threshold; the first duration has a first preset length; determines the decibel value of the customer service-side audio in a second duration; the second duration has a second preset length; the second duration is continuous with the first duration and later than the first duration; in response to the decibel value of the customer service-side audio at each moment in the second duration being less than the second preset threshold, determines that the communication status between the customer service representative and the user is paused; in response to the decibel value of the customer service-side audio at any moment in the second duration being greater than or equal to the second preset threshold, determines that the communication status between the customer service representative and the user is ongoing. Because the second duration is continuous with the first duration and later than the first duration, and the decibel value of the customer service-side audio at each moment in the second duration is less than the second preset threshold, the communication status between the customer service representative and the user is determined to be paused, thus accurately determining the communication status between the customer service representative and the user.

[0082] Example 2

[0083] Figure 3 This is a flowchart illustrating the business processing progress output method provided in Embodiment 2 of this application. Figure 3 As shown, the business processing progress output method provided in this embodiment, based on any of the above embodiments, further refines step 205, "obtaining the business processing progress of the target business and outputting the business processing progress to the user", to include steps 401 to 403.

[0084] Step 401: Obtain relevant information on the processing of multiple task nodes for the target business.

[0085] In this embodiment, the target service may include multiple task nodes, and the number of task nodes included in different target services can be preset. Customer service personnel can access the service server through the customer service-side device to handle services for users. To handle services for users, customer service personnel need to complete multiple task nodes of the target service. The customer service-side device can record the multiple task nodes included in the target service, as well as the handling-related information for each of the multiple task nodes included in the target service. The handling-related information can include whether the service has been handled or not. Electronic devices can obtain the number of task nodes included in the target service, as well as the handling-related information for each task node, by communicating with the customer service-side device.

[0086] Step 402: Determine the processing progress based on the relevant information.

[0087] In this embodiment, the electronic device can determine the service processing progress based on the total number of task nodes included in the target service and the number of task nodes for which relevant information has been processed. For example, the ratio of the number of processed task nodes to the total number of task nodes included in the target service is used to determine the service processing progress. For instance, if the total number of task nodes included in the target service is 10 and the number of processed task nodes is 7, then the electronic device can determine the service processing progress as 7 / 10.

[0088] Step 403: Send the business processing progress to the user equipment so that the user equipment can output the business processing progress to the user.

[0089] In this embodiment, the electronic device sends the service processing progress to the user device, instructing the user device to output the service processing progress to the user. The user device can output the service processing progress to the user via voice, images, text, or other means.

[0090] The business processing progress output method provided in this embodiment obtains processing-related information from multiple task nodes of the target business; determines the business processing progress based on the processing-related information; and sends the business processing progress to the user device, so that the user device can output the business processing progress to the user. Since determining the business processing progress based on the processing-related information allows for rapid determination of the business processing progress, and sending the business processing progress to the user device allows the user to be informed of the business processing progress, alleviating any negative emotions and reassuring the user.

[0091] Optionally, the relevant information includes the processing status and average processing time; the processing status is completed or not completed; step 402 "determine the business processing progress based on the relevant information" is further refined into steps 501 to 502.

[0092] Step 501: Calculate the first sum of the average processing time for each task node.

[0093] In this embodiment, the relevant information includes processing status and average processing time. The processing status is either completed or incomplete. Completed means that the task node included in the target service has been completed, while incomplete means that the task node included in the target service has not yet been completed. The target service is considered completed only when all multiple task nodes included in the target service have been completed. The average processing time can be the average of the time consumed by customer service personnel when completing the task node multiple times.

[0094] In this embodiment, the electronic device can calculate a first sum of average processing time based on the average processing time of multiple task nodes included in the target service.

[0095] Step 502: Determine the business processing progress based on the first sum and the processing status of each task node.

[0096] In this embodiment, the electronic device can calculate a third sum of the average processing time for task nodes with an incomplete status, and then determine the business processing progress based on the first sum and the third sum. For example, the electronic device can calculate a first difference between the first sum and the third sum, and then determine the business processing progress by the ratio of the first difference to the first sum. For instance, if the first sum is 300 seconds and the third sum is 100 seconds, then the first difference is 200 seconds, and the business processing progress is 200 / 300 = 2 / 3.

[0097] The business processing progress output method provided in this embodiment includes processing-related information such as processing status and average processing time; the processing status is either completed or incomplete; a first sum of the average processing times of each task node is calculated; and the business processing progress is determined based on the first sum and the processing status of each task node. Because the business processing progress is determined by calculating the first sum of the average processing times of each task node and the processing status of each task node, the business processing progress can be calculated more accurately, thus providing a more accurate business processing progress output to the user.

[0098] Optionally, step 502, "determine the business processing progress based on the first sum and the processing status of each task node," can be further refined to include steps 601 to 602.

[0099] Step 601: Calculate the second sum of the average processing time of task nodes with a processing status of "completed".

[0100] In this embodiment, the electronic device sums up the average processing time of task nodes whose processing status is completed to obtain a second sum value.

[0101] Step 602: The ratio of the second sum to the first sum is determined as the business processing progress.

[0102] In this embodiment, the electronic device determines the business processing progress by the ratio of the second sum to the first sum.

[0103] The business processing progress output method provided in this embodiment calculates a second sum of the average processing time of task nodes with a processing status of "completed"; the ratio of the second sum to the first sum is then used to determine the business processing progress. Since the business processing progress is determined by calculating the second sum of the average processing time of task nodes with a processing status of "completed" and then using the ratio of the first sum to the second sum, the business processing progress can be determined more quickly and accurately through simple calculation.

[0104] Example 3

[0105] The business processing progress output method provided in Embodiment 3 of this application, based on any of the above embodiments, includes step 701 before step 203 "in response to the communication state being paused, the trained YOLO v5 emotion recognition model is used to identify the user emotion corresponding to the user-side data".

[0106] Step 701: Train the preset YOLO v5 emotion recognition model using at least one of the semantic emotion training dataset, the speech emotion training dataset, and the facial emotion training dataset; the semantic emotion training dataset includes at least one text vector corresponding to a first preset abnormal emotion; the speech emotion training dataset includes at least one speech spectrogram and speech corresponding to a first preset abnormal emotion; the facial emotion training dataset includes at least one facial image corresponding to a first preset abnormal emotion.

[0107] In this embodiment, the semantic emotion training dataset includes at least one text vector corresponding to a first preset abnormal emotion, such as text vectors corresponding to complaining words like "useless," "so long," or "still not done," or some profanity. Optionally, the semantic emotion training dataset also includes text vectors corresponding to user question words such as "excuse me," "is it," or "hello," so that after training, the preset YOLO v5 emotion recognition model can recognize user questions and thus remind customer service personnel to answer user questions.

[0108] The voice emotion training dataset includes at least one speech spectrogram and speech corresponding to a first preset abnormal emotion. The speech spectrogram corresponding to the first preset abnormal emotion is a spectrogram converted from audio containing the first preset abnormal emotion, and the speech corresponding to the first preset abnormal emotion is audio containing the first preset abnormal emotion.

[0109] The business processing progress output method provided in this embodiment trains a preset YOLO v5 emotion recognition model using at least one of a semantic emotion training dataset, a voice emotion training dataset, and a facial emotion training dataset. The semantic emotion training dataset includes text vectors corresponding to at least one first preset abnormal emotion; the voice emotion training dataset includes a speech spectrogram and speech corresponding to at least one first preset abnormal emotion; and the facial emotion training dataset includes a facial image corresponding to at least one first preset abnormal emotion. Because the preset YOLO v5 emotion recognition model is trained using at least one of these datasets, the trained YOLO v5 emotion recognition model can identify at least one first preset abnormal emotion from at least one of the user's audio and video.

[0110] Optionally, the user emotions corresponding to the user-side data include at least one of the following: user-side audio emotions, user-side video emotions; user-side audio and video include user-side audio and user-side video; the step 203 "using the trained YOLO v5 emotion recognition model to identify the user emotions corresponding to the user-side data" is further refined to include at least one of steps 801, 802 and 803.

[0111] Step 801: Filter the user-side audio and convert the filtered user-side audio into corresponding text vectors. Use the trained YOLO v5 emotion recognition model to identify the first emotion corresponding to the text vectors and determine the first emotion as the emotion of the user-side audio.

[0112] In this embodiment, the electronic device performs a filtering operation on the user-side audio. This filtering operation removes noise and interference from the user-side audio, converting it into a more accurate corresponding text vector. The electronic device then performs semantic recognition on the converted text vector to identify the first emotion corresponding to the text vector.

[0113] Step 802: Convert the user-side audio into a spectrogram, use the trained YOLO v5 emotion recognition model to identify the second emotion corresponding to the spectrogram, and determine the second emotion as the emotion of the user-side audio.

[0114] In this embodiment, the spectrogram converted from audio can reflect the user's emotions when speaking. Therefore, the electronic device uses a trained YOLO v5 emotion recognition model to identify the second emotion corresponding to the spectrogram.

[0115] Step 803: Extract interval frames from the user-side video according to a preset interval frame number, and identify the emotion of the user-side video based on the interval frames and the trained YOLO v5 emotion recognition model.

[0116] In this embodiment, the user-side video consists of consecutive image frames. The electronic device extracts interval frames from the user-side video according to a preset interval frame number, and uses a trained YOLO v5 emotion recognition model to identify the third emotion corresponding to each interval frame. The electronic device can then determine the third emotion as the emotion of the user-side video.

[0117] The business processing progress output method provided in this embodiment filters the user-side audio and converts the filtered audio into corresponding text vectors. A trained YOLO v5 emotion recognition model is then used to identify the first emotion corresponding to the text vectors, thus identifying the first emotion as the user-side audio emotion. The user-side audio is then converted into a spectrogram, and the trained YOLO v5 emotion recognition model is used to identify the second emotion corresponding to the spectrogram, thus identifying the second emotion as the user-side audio emotion. Interval frames are extracted from the user-side video at preset intervals, and the user-side video emotion is identified based on these interval frames and the trained YOLO v5 emotion recognition model. Since the trained YOLO v5 emotion recognition model identifies the first emotion corresponding to the text vectors and the second emotion corresponding to the spectrogram, and then identifies the user-side video emotion based on the interval frames and the trained YOLO v5 emotion recognition model, only one model is needed to identify user emotions from audio and video data. This improves the speed of user emotion identification, allows for faster detection of abnormal user emotions, and facilitates subsequent user reassurance.

[0118] Optionally, step 803, "Identifying user-side video emotions based on interval frames and the trained YOLO v5 emotion recognition model," is further refined to include steps 901 to 902.

[0119] Step 901: Use the trained YOLO v5 emotion recognition model to identify the third emotion corresponding to each interval frame.

[0120] Step 902: In response to the fact that the third emotion corresponding to a consecutive preset number of interval frames is the same, the third emotion corresponding to the consecutive preset number of interval frames is determined as the user-side video emotion.

[0121] In this embodiment, the preset number can be 5. If a preset number of consecutive images are all identified as the third emotion, then the third emotion corresponding to the preset number of consecutive interval frames is determined as the user-side video emotion. Here, by identifying the interval frames, the number of image frames that the trained YOLO v5 emotion recognition model needs to identify can be reduced. At the same time, in order to avoid errors in identifying a single image frame, which could lead to inaccurate identification, the third emotion is only determined as the user-side video emotion when a preset number of consecutive images are all identified as the third emotion. If the third emotions corresponding to a preset number of consecutive images are different, then to ensure accurate identification of the user-side video emotion, the different third emotions are not determined as the user-side video emotion.

[0122] The business processing progress output method provided in this embodiment identifies the third emotion corresponding to each interval frame by using a trained YOLO v5 emotion recognition model; in response to the third emotion corresponding to a consecutive preset number of interval frames being the same, the third emotion corresponding to the consecutive preset number of interval frames is determined as the emotion of the user-side video. Since the third emotion corresponding to the consecutive preset number of interval frames is determined as the emotion of the user-side video, the emotion of the user-side video can be identified more accurately from the user-side video.

[0123] Optionally, after step 204, which states "in response to the user's emotions including at least one first preset abnormal emotion, output the first alarm information to the customer service personnel", step 1001 is also included.

[0124] Step 1001: In response to the preset time after the first alarm information is output, the communication status remains in a paused state, and a second alarm information is output to the customer service personnel; the second alarm information has a more obvious output form compared to the first alarm information.

[0125] In this embodiment, if the electronic device remains in a paused state for a preset time after outputting the first alarm message, it indicates that the customer service personnel have not communicated with or reassured the user. This may be because the output format of the first alarm message is not clear enough. Therefore, a second alarm message is output to the customer service personnel. The second alarm message has a more obvious output format compared to the first alarm message. For example, if the first alarm message is output via audio, the second alarm message can have a louder volume than the first alarm message. Furthermore, the second alarm message can have more output formats than the first alarm message, such as a pop-up window on the customer service personnel's device while being output at a higher volume. If the first alarm message is output via a text pop-up window, the second alarm message can have a larger font size and a longer pop-up duration than the first alarm message. This embodiment does not limit these aspects.

[0126] The business processing progress output method provided in this embodiment outputs a second alarm message to customer service personnel by maintaining a paused communication state for a preset time after the first alarm message is output. The second alarm message has a more obvious output format than the first alarm message. Because the second alarm message has a more obvious output format than the first alarm message, it can remind customer service personnel to communicate with the user again, avoid the user not receiving feedback for a long time, and improve the user experience.

[0127] Optionally, after step 202 "determine the communication status between customer service personnel and users based on user-side audio and customer service-side audio", steps 1101 to 1102 are also included.

[0128] Step 1101: In response to the communication status being "in communication", the trained YOLO v5 emotion recognition model is used to identify the customer service emotion corresponding to the customer service data.

[0129] In this embodiment, if the communication status is "in progress," to improve user experience, the electronic device can also use a trained YOLO v5 emotion recognition model to identify the customer service emotions corresponding to the customer service data. Here, the trained YOLO v5 emotion recognition model identifies customer service data during the communication process between the customer service representative and the user. Customer service data includes customer service audio or customer service audio-visual content. Customer service audio-visual content includes both audio and video.

[0130] In this embodiment, the implementation method of using a trained YOLO v5 emotion recognition model to identify customer service emotions corresponding to customer service side data can be found in the implementation method of identifying user emotions corresponding to user side data, and will not be repeated here.

[0131] Step 1102: In response to customer service emotions including at least one second preset abnormal emotion, output a third alarm message to the customer service personnel; the third alarm message is used to remind the customer service personnel to maintain a service attitude; the content of the third alarm message is different from that of the first alarm message.

[0132] In this embodiment, the second preset abnormal emotion can include emotions such as impatience and anger. Correspondingly, during the training of the YOLO v5 emotion recognition model, the semantic emotion training dataset includes at least one text vector corresponding to the second preset abnormal emotion; the speech emotion training dataset includes at least one speech spectrogram and speech corresponding to the second preset abnormal emotion; and the facial emotion training dataset includes at least one facial image corresponding to the second preset abnormal emotion. The text vector corresponding to the second preset abnormal emotion can be complaining words such as "Why are you rushing me?", "What's the rush?", or "Are you blind?", or some profanity. Specific training methods are not detailed here.

[0133] In this embodiment, for customer service personnel, the trained YOLO v5 emotion recognition model can be used to identify customer service emotions. When a customer service representative has a verbal conflict or potential verbal conflict with a user during communication, or when the customer service representative's facial expression shows a negative emotional state, the model can remind the customer service representative, thereby enabling the customer service representative to provide smiling service and improve service quality.

[0134] The business processing progress output method provided in this embodiment identifies customer service emotions corresponding to customer service data using a trained YOLO v5 emotion recognition model in response to a communication status of "in progress". In response to customer service emotions including at least one second preset abnormal emotion, a third alarm message is output to the customer service personnel. The third alarm message serves as a reminder to the customer service personnel to maintain a good service attitude. The content of the third alarm message differs from that of the first alarm message. Because the trained YOLO v5 emotion recognition model is used to identify customer service emotions corresponding to customer service data during the communication process between the customer service personnel and the user, it can identify customer service emotions simultaneously with user emotions, reminding customer service personnel to maintain a good service attitude and improving user experience.

[0135] Example 4

[0136] Figure 4 This is a schematic diagram of the business processing progress output device provided according to Embodiment 4 of this application. Figure 4 As shown, the business processing progress output device 40 provided in this embodiment includes: an acquisition module 41, a determination module 42, an identification module 43, a first output module 44, and a second output module 45.

[0137] Module 41 is used to acquire communication data between customer service personnel and users and the target business handled by users; the communication data includes user-side data and customer service-side data; user-side data includes user-side audio or user-side audio and video; customer service-side data includes customer service-side audio or customer service-side audio and video.

[0138] Module 42 is used to determine the communication status between customer service personnel and users based on user-side audio and customer service-side audio; the communication status is either in progress or paused.

[0139] The recognition module 43 is used to recognize the user's emotion corresponding to the user-side data when the communication state is paused. The trained YOLO v5 emotion recognition model is obtained by training the preset YOLO v5 emotion recognition model with at least one of the semantic emotion training dataset, the voice emotion training dataset, and the facial emotion training dataset.

[0140] The first output module 44 is used to output a first alarm message to customer service personnel in response to user emotions, including at least one first preset abnormal emotion; the first alarm message is used to remind customer service personnel to soothe the user's emotions.

[0141] The second output module 45 is used to maintain the communication state in a paused state for a preset time after the first alarm information is output, obtain the business processing progress of the target business, and output the business processing progress to the user.

[0142] Optionally, module 42 is specifically used for:

[0143] The first duration is when the decibel value of the user-side audio is less than a first preset threshold; the first duration has a first preset length.

[0144] Determine the decibel value of the customer service-side audio during the second duration; the second duration has a second preset duration; the second duration is continuous with the first duration and is later than the first duration;

[0145] In response to the fact that the decibel value of the audio from the customer service side is less than the second preset threshold at each moment during the second duration, the communication status between the customer service personnel and the user is determined to be paused.

[0146] If the decibel value of the audio from the customer service side is greater than or equal to a second preset threshold at any point during the second duration, the communication status between the customer service representative and the user is determined to be in progress.

[0147] Optionally, the second output module 45 is specifically used for:

[0148] Obtain relevant information on multiple task nodes for processing the target business;

[0149] The processing progress will be determined based on the relevant information.

[0150] The service processing progress is sent to the user device so that the user device can output the service processing progress to the user.

[0151] Optionally, the relevant information includes processing status and average processing time; the processing status is either completed or incomplete; the second output module 45 is also specifically used for:

[0152] The processing progress is determined based on relevant information, including:

[0153] Calculate the first sum of the average processing times for each task node;

[0154] The processing progress is determined based on the first sum and the processing status of each task node.

[0155] Optionally, the second output module 45 is further used for:

[0156] Calculate the second sum of the average processing time for task nodes whose processing status is completed;

[0157] The ratio of the second sum to the first sum is used to determine the progress of the business process.

[0158] Optionally, the business processing progress output device 40 also includes a training module, which is used for:

[0159] The preset YOLO v5 emotion recognition model is trained using at least one of the semantic emotion training dataset, the speech emotion training dataset, and the facial emotion training dataset; the semantic emotion training dataset includes at least one text vector corresponding to a first preset abnormal emotion; the speech emotion training dataset includes at least one speech spectrogram and speech corresponding to a first preset abnormal emotion; and the facial emotion training dataset includes at least one facial image corresponding to a first preset abnormal emotion.

[0160] Optionally, the user sentiment corresponding to the user-side data includes at least one of the following: user-side audio sentiment, user-side video sentiment; user-side audio and video include user-side audio and user-side video; the recognition module 43 is specifically used for at least one of the following:

[0161] The user-side audio is filtered and converted into corresponding text vectors. The trained YOLO v5 emotion recognition model is used to identify the first emotion corresponding to the text vectors and the first emotion is determined as the emotion of the user-side audio.

[0162] The user-side audio is converted into a spectrogram, and the trained YOLO v5 emotion recognition model is used to identify the second emotion corresponding to the spectrogram. The second emotion is then identified as the emotion of the user-side audio.

[0163] Interval frames are extracted from the user-side video according to a preset interval frame number, and the emotions in the user-side video are identified based on the interval frames and the trained YOLO v5 emotion recognition model.

[0164] Optionally, the recognition module 43 is further used for:

[0165] A trained YOLO v5 emotion recognition model was used to identify the third emotion corresponding to each interval frame.

[0166] If the third emotion is the same for a consecutive preset number of interval frames, the third emotion corresponding to the consecutive preset number of interval frames is determined as the user-side video emotion.

[0167] Optionally, the business processing progress output device 40 further includes a third output module, which is used for:

[0168] In response to the output of the first alarm message, the communication status remains suspended for a preset period of time, and a second alarm message is output to customer service personnel; the second alarm message has a more obvious output format compared to the first alarm message.

[0169] Optionally, the business processing progress output device 40 also includes a fourth output module, and the identification module 43 is specifically used for:

[0170] In response to the communication status being "in communication", the trained YOLO v5 emotion recognition model is used to identify the customer service emotions corresponding to the customer service data.

[0171] The fourth output module is specifically used for:

[0172] In response to customer service emotions, including at least one second preset abnormal emotion, a third alarm message is output to the customer service personnel; the third alarm message is used to remind the customer service personnel to maintain a service attitude; the content of the third alarm message is different from that of the first alarm message.

[0173] The business processing progress output device provided in this embodiment can execute any of the business processing progress output methods provided in the above embodiments. The specific implementation and principle are similar, and will not be described again here.

[0174] Example 5

[0175] Figure 5 This is a schematic diagram of the structure of an electronic device according to Embodiment Seven of this application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes: a memory 51, a processor 52, and an output device 53.

[0176] The memory 51, processor 52, and output device 53 are communicatively connected;

[0177] Memory 51 stores instructions executed by the computer;

[0178] Output device 53 is used to output the first alarm information and the business processing progress;

[0179] The processor 52 executes the computer execution instructions stored in the memory 51 to implement the business processing progress output method provided in any of the above embodiments. The specific implementation method and principle are similar and will not be described again here.

[0180] Alternatively, the output device can be a transceiver.

[0181] The memory 51, processor 52, and output device 53 can communicate with each other via a bus. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0182] The memory 51 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk, etc.

[0183] Embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions. When executed by a processor, these instructions are used to implement the business processing progress output method provided in any of the above embodiments. The specific implementation and principle are similar and will not be repeated here. Exemplarily, the computer-readable storage medium can be a read-only memory (ROM), random access memory (RAM), magnetic tape, floppy disk, or optical data storage device, etc.

[0184] The embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the business processing progress output method provided in any of the above embodiments. The specific implementation method and principle are similar and will not be described again here.

[0185] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the module division in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules can be combined, or integrated into another system, or some features can be ignored or not executed.

[0186] Furthermore, unless otherwise specified, the functional modules in the various embodiments of this application can be integrated into one module, or each module can exist physically separately, or two or more modules can be integrated together. The integrated modules described above can be implemented in hardware or as software program modules.

[0187] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0188] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0189] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0190] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for outputting business processing progress, characterized in that, include: Acquire communication data between customer service personnel and users, as well as the target business processed by users; the communication data includes user-side data and customer service-side data; user-side data includes user-side audio or user-side audio and video; customer service-side data includes customer service-side audio. The communication status between customer service personnel and users is determined based on the user-side audio and the customer service-side audio; the communication status is either in progress or paused. In response to the communication state being paused, a trained YOLO v5 emotion recognition model is used to identify the user's emotion corresponding to the user-side data; the trained YOLO v5 recognition model is obtained by training a preset YOLO v5 emotion recognition model with at least one of the semantic emotion training dataset, the voice emotion training dataset, and the facial emotion training dataset. In response to the user's emotions, including at least one first preset abnormal emotion, a first alarm message is output to customer service personnel; The first alarm message is used to remind customer service personnel to calm the user down; Within a preset time after the first alarm message is output, the communication state remains in a paused state, the business processing progress of the target business is obtained, and the business processing progress is output to the user. The step of obtaining the processing progress of the target service and outputting the processing progress to the user includes: Obtain relevant information on multiple task nodes for processing the target business; The processing progress is determined based on the relevant processing information. The service processing progress is sent to the user equipment so that the user equipment outputs the service processing progress to the user. The relevant information includes the processing status and average processing time; the processing status is either completed or incomplete. Determining the processing progress of the business based on the relevant processing information includes: Calculate the first sum of the average processing times for each task node; The processing progress of the business is determined based on the first sum and the processing status of each task node; Determining the business processing progress based on the first sum and the processing status of each task node includes: Calculate the second sum of the average processing time of the task nodes whose processing status is completed; The ratio of the second sum to the first sum is determined as the progress of the business processing.

2. The method according to claim 1, characterized in that, The process of determining the communication status between customer service personnel and users based on user-side audio and customer service-side audio includes: The first duration is when the decibel value of the user-side audio is less than a first preset threshold; the first duration has a first preset length. Determine the decibel value of the customer service-side audio during the second duration; the second duration has a second preset duration; the second duration is continuous with the first duration and is later than the first duration; In response to the fact that the decibel value of the audio from the customer service side is less than the second preset threshold at each moment during the second duration, the communication status between the customer service personnel and the user is determined to be paused. If the decibel value of the audio from the customer service side is greater than or equal to a second preset threshold at any point during the second duration, the communication status between the customer service representative and the user is determined to be in progress.

3. The method according to claim 1, characterized in that, Before responding to the communication state as paused and using the trained YOLO v5 emotion recognition model to identify the user's emotion corresponding to the user-side data, the following steps are also included: The preset YOLO v5 emotion recognition model is trained using at least one of the semantic emotion training dataset, the speech emotion training dataset, and the facial emotion training dataset; the semantic emotion training dataset includes at least one text vector corresponding to a first preset abnormal emotion; the speech emotion training dataset includes at least one speech spectrogram and speech corresponding to a first preset abnormal emotion; and the facial emotion training dataset includes at least one facial image corresponding to a first preset abnormal emotion.

4. The method according to claim 3, characterized in that, The user emotions corresponding to the user-side data include at least one of the following: user-side audio emotions and user-side video emotions; the user-side audio and video include user-side audio and user-side video. The method of using a trained YOLO v5 emotion recognition model to identify user emotions corresponding to user-side data includes at least one of the following: The user-side audio is filtered and converted into corresponding text vectors. The trained YOLO v5 emotion recognition model is used to identify the first emotion corresponding to the text vectors and the first emotion is determined as the emotion of the user-side audio. The user-side audio is converted into a spectrogram, and the trained YOLO v5 emotion recognition model is used to identify the second emotion corresponding to the spectrogram. The second emotion is then determined as the emotion of the user-side audio. According to a preset interval frame number, interval frames are extracted from the user-side video, and the emotions in the user-side video are identified based on the interval frames and the trained YOLO v5 emotion recognition model.

5. The method according to claim 4, characterized in that, The step of identifying user-side video emotions based on the interval frames and the trained YOLO v5 emotion recognition model includes: A trained YOLO v5 emotion recognition model was used to identify the third emotion corresponding to each interval frame. If the third emotion is the same for a consecutive preset number of interval frames, the third emotion corresponding to the consecutive preset number of interval frames is determined as the user-side video emotion.

6. The method according to claim 1, characterized in that, In response to the user's emotion including at least one first preset abnormal emotion, after outputting a first alarm message to customer service personnel, the system further includes: In response to a preset time after the first alarm message is output, the communication state remains paused, and a second alarm message is output to the customer service personnel; the second alarm message has a more obvious output form than the first alarm message.

7. The method according to any one of claims 1-6, characterized in that, After determining the communication status between the customer service representative and the user based on the user-side audio and the customer service-side audio, the process also includes: In response to the communication status being "communication in progress", a trained YOLO v5 emotion recognition model is used to identify the customer service emotion corresponding to the customer service side data. In response to the customer service emotion including at least one second preset abnormal emotion, a third alarm message is output to the customer service personnel; the third alarm message is used to remind the customer service personnel to maintain a service attitude; the content of the third alarm message is different from that of the first alarm message.

8. A business processing progress output device, characterized in that, include: The acquisition module is used to acquire communication data between customer service personnel and users, as well as the target business handled by users; the communication data includes user-side data and customer service-side data; user-side data includes user-side audio or user-side audio and video; customer service-side data includes customer service-side audio or customer service-side audio and video. The determination module is used to determine the communication status between the customer service representative and the user based on the user-side audio and the customer service-side audio; the communication status is either in progress or paused. The recognition module is used to identify the user's emotion corresponding to the user-side data in response to the communication state being paused. The trained YOLO v5 emotion recognition model is obtained by training a preset YOLO v5 emotion recognition model with at least one of the semantic emotion training dataset, the voice emotion training dataset, and the facial emotion training dataset. The first output module is used to output a first alarm message to customer service personnel in response to the user's emotion, which includes at least one first preset abnormal emotion. The first alarm message is used to remind customer service personnel to calm the user down; The second output module is used to respond to the fact that the communication state remains in a paused state for a preset time after the first alarm information is output, to obtain the business processing progress of the target business, and to output the business processing progress to the user. The second output module is specifically used to obtain processing-related information of multiple task nodes for processing the target business; The processing progress is determined based on the relevant processing information. The service processing progress is sent to the user equipment so that the user equipment outputs the service processing progress to the user. The processing-related information includes processing status and average processing time; the processing status is either completed or incomplete; the second output module is also specifically used to calculate a first sum of the average processing time of each task node; The processing progress of the business is determined based on the first sum and the processing status of each task node; The second output module is further specifically used to calculate a second sum of the average processing time of the task nodes whose processing status is completed; The ratio of the second sum to the first sum is determined as the progress of the business processing.

9. An electronic device, characterized in that, include: Memory, processor, and output device; The memory, the processor, and the output device are communicatively connected; The memory stores computer-executed instructions; The output device is used to output the first alarm information and the business processing progress; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

11. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1-7.

Images