System and method for multimedia contact center interaction via audio-visual asynchronous channels

Abstract

A system and method for multimedia contact center interaction via an audio-visual asynchronous messaging channel is provided. In one embodiment, an organization may operate a website that includes information and is used to provide an entry point for communicating with a contact center. The communication channel may be directly integrated with the organization's website, which includes an editable audio-visual interface. A user may interact with the editable interface in a variety of ways, including sketching and recording on a web page to create a video form of an audio-visual asynchronous query on the web page. The interaction is analyzed and routed to an agent for processing. The agent may then connect to the user via the asynchronous channel.

Description

Background Art

[0001] The present invention generally relates to telecommunication systems and methods, and the integration of communication means within a website. More particularly, the present invention relates to customer engagement through a website.

[0002] Claiming priority

[0003] This article claims priority to U.S. patent application 16 / 420,258, filed on May 23, 2019, also entitled “SYSTEM AND METHOD FORMULTIMEDIA CONTACT CENTER INTERACTIONS VIA AN AUDIOVISUAL ASYNCHRONOUSCHANNEL.” Summary of the invention

[0004] A system and method for multimedia contact center interaction via an audio-visual asynchronous messaging channel is provided. In one embodiment, an organization may operate a website that includes information and is used to provide an entry point for communicating with a contact center. The communication channel may be directly integrated with the organization's website, which includes an editable audio-visual interface. A user may interact with the editable interface in a variety of ways, including sketching and recording on a web page to create a video form of an audio-visual asynchronous query on the web page. The interaction is analyzed and routed to an agent for processing. The agent may then connect to the user via the asynchronous channel.

[0005] In one embodiment, a method for transmitting interactive audio-visual asynchronous queries in a contact center system is provided, the method comprising: providing, by a web server, an editable audio-visual interface to a web-enabled user device associated with a first party; recording, by the web server, the interaction of the first party with the editable audio-visual interface; generating, by the web server, an asynchronous query for the contact center system based on the recorded interaction; processing, by an audio-visual analysis system, at least one of the audio and video of the asynchronous query; determining, by the audio-visual analysis system, an intent generated from the processed video; transmitting, by the audio-visual analysis system, the determined intent and the asynchronous query to a routing server; and transmitting, by the routing server, a signal for routing the asynchronous query to a second party, wherein the determined intent is used to determine a second-party recipient of the asynchronous query.

[0006] The editable audiovisual interface includes tools for enabling the first party to perform one or more of the following on the web page of the user device: drawing, pointing, writing text, and inputting speech.

[0007] The processing also includes determining an action performed by the first party during the asynchronous query. The processing may also include determining an object that is joined to a web page of the user device during the asynchronous query. The processing may also include determining a portion of the audio that needs to be spliced ​​to a corresponding video of the asynchronous query.

[0008] The second party may be a virtual agent. The second party may also respond to the recorded interaction through an editable audio-visual interface.

[0009] The intent includes routing parameters, wherein the routing parameters include at least one of: customer type, webpage type, intent type, user intent, keyword, sentiment, priority, and emotion.

[0010] The response includes: providing an editable audiovisual interface to the second party by the web server; recording, by the web server, the interaction of the second party with the editable audiovisual interface; recording, by the audiovisual analysis system, the second party interaction for storage by the contact center; and transmitting, by the audiovisual analysis system, the second party interaction to the first party via an asynchronous channel.

[0011] The intent determination includes: determining, by an audiovisual analysis system, frames in an asynchronous query indicating an action performed by a first party in a video of the query; generating, by the audiovisual analysis system, a plurality of audiovisual segments based on the query, separating the query from the determined frames indicating the action performed by the first party; processing, by the audiovisual analysis system, one of the generated audiovisual segments, wherein the processing includes using an action recognition application to determine an action performed during the generated audiovisual segment; processing, by the audiovisual analysis system, the generated audiovisual segment through an entity recognition application, the entity recognition application being used to determine interactive entities during the generated audiovisual segment; processing, by the audiovisual analysis system, audio of the query that needs to be spliced ​​to a corresponding action identified from the audiovisual segment; processing, by the audiovisual analysis system, the generated audiovisual segment through a speech recognition application, the speech recognition application being used to derive speech content from the generated audiovisual segment; generating, by the audiovisual analysis system, an audiovisual segment output consisting of the determined action, the interactive entities, and the speech content derived from the processed generated audiovisual segment; and determining, by the audiovisual analysis system, the first party intent based on the generated audiovisual segment output.

[0012] The action performed during the generated audiovisual segment is drawing by the first party in the queried video. The action performed during the generated audiovisual segment is typing text by the first party in the queried video.

[0013] The interactive entity is the web browser object in the web browser window.

[0014] The response includes an asynchronous automated response. The response may also include the following steps: providing an editable audiovisual interface to the first party by the web server; providing an interaction point for the first party by the web server; providing a recommendation window by the web server in response to the first party interacting with the interaction point; receiving a signal indicating a selection made by the first party from the recommendation window by the web server; determining by the recommendation engine whether an asynchronous automated response exists for the selection made by the first party from the recommendation window; and transmitting, by the web server, the asynchronous automated response to the end user if an asynchronous automated response exists for the selection made by the end user from the recommendation window, otherwise routing the interaction to the second party for processing.

[0015] The interaction point is associated with a web browser object shown on a web-enabled end-user device. The recommendation engine generates selections from the recommendation window based on a database of intents and entity classifications generated from other queries. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a diagram illustrating an embodiment of a contact center system.

[0017] Figure 2 is a diagram showing an embodiment of an audio-visual processing server.

[0018] Figure 3A is a diagram illustrating an embodiment of a web-connected user device displaying a web browser window.

[0019] Figure 3B is a diagram illustrating an embodiment of a web-connected user device displaying a web browser window.

[0020] Figure 4 is a flow diagram illustrating an embodiment of a process for an inbound asynchronous audiovisual channel using an editable audiovisual interface.

[0021] Figure 5 is a flow diagram illustrating an embodiment of a process for outbound asynchronous audiovisual channels using an editable audiovisual interface.

[0022] Figure 6 is a flow chart illustrating an embodiment of a process for audio-visual processing of asynchronous audio-visual channels.

[0023] Fig. 7A is a diagram illustrating an embodiment of a web-connected user device displaying a web browser window and an editable interface.

[0024] Figure 7B is a diagram illustrating an embodiment of a web-connected user device displaying a web browser window and an editable interface.

[0025] Figure 8 is a flow chart illustrating an embodiment of an asynchronous solution for providing recommendations through an asynchronous audio-visual channel.

[0026] Fig. 9A is a diagram illustrating an embodiment of a computing device.

[0027] Fig. 9B is a diagram illustrating an embodiment of a computing device. DETAILED DESCRIPTION

[0028] For the purpose of promoting understanding of the principles of the present invention, reference will now be made to the embodiments shown in the accompanying drawings, and specific language will be used to describe these embodiments. However, it should be understood that it is not intended to limit the scope of the present invention. As those skilled in the art involved in the present invention would normally expect, it is contemplated that any changes and further modifications in the described embodiments, as well as any further application of the principles of the present invention as described herein, are contemplated.

[0029] Including audio-visual media to enable communications between customers and commercial enterprises allows enterprises to convey brand stories, explain value propositions, and build relationships between customers and products. Generally speaking, commercial enterprises mainly use audio-visual technology for effective marketing, increasing sales, and showcasing their brand identity. However, enterprises can also deploy audio-visual technology in other channels. For example, in a contact center, dynamically created audio-visual media combined with image recognition and artificial intelligence can be used as an effective and efficient self-service channel. The creation of audio-visual media by the end users (i.e., customers) of the contact center can help agents provide accurate solutions to end users and strive to improve first call resolution rates.

[0030] Creating a new self-service channel in a contact center that allows users to generate audiovisual media can allow for solving some of the problems faced by customers using contact centers. There are currently many examples of "gaps" in contact centers that hinder the ability of customers to obtain the information they need from the contact center to solve customer problems. For example, if a customer makes an online purchase during non-business hours and needs clarification on the order, the customer may not want to wait until the next business day to clarify the order. In another example, customers may find it difficult to express the problems they face in the business through email. This may be especially true if the customer believes that the email will take a long time to generate and be answered by the enterprise. In another example, the customer may have a strong preference for not wanting to traverse the interactive voice response (IVR) system of the contact center and wait to interact with a skilled agent. In yet another example, the user may want to get immediate service during peak hours when all agents are busy.

[0031] The present disclosure describes an embodiment of an editable interface with an asynchronous audio-visual channel that customers can use to narrate their queries using sketching and recording tools. Such gestures are captured by video, for example, in sync with the accompanying audio for playback. In one embodiment, user interactions will be recorded as a single compressed video and sent to a contact center. These videos can be analyzed and meaningful information can be extracted for purposes such as providing accurate solutions to customers and selecting the most appropriate agent to handle.

[0032] Contact Center System

[0033] Figure 1 is a diagram illustrating an embodiment of a contact center system, generally indicated at 100. For example, Figure 1 A system for supporting a contact center when providing contact center services is shown. A contact center can be an internal facility of a business or enterprise for serving an enterprise when performing sales and service functions relative to products and services available through the enterprise. In another aspect, the contact center can be operated by a third-party service provider. In one embodiment, the contact center can be operated as a hybrid system, where some components of the contact center system are hosted at the contact center building and other components are remotely hosted (e.g., in a cloud-based environment). The contact center can be deployed on equipment dedicated to an enterprise or a third-party service provider, and / or in a remote computing environment, such as a private or public cloud environment with an infrastructure for supporting multiple contact centers for multiple enterprises. The various components of the contact center system can also be distributed in various geographic locations and computing environments, and are not necessarily contained in a single location, computing environment, or even computing device.

[0034] Components of the communication infrastructure, generally indicated at 100, include: multiple end-user devices 105A, 105B, 105C; a communication network 110; a switch / media gateway 115; a call controller 120; an IMR server 125; a routing server 130; a storage device 135; a statistics server 140; multiple proxy devices 145A, 145B, 145C including work pods 146A, 146B, 146C; a multimedia / social media server 150; a web server 155; an iXn server 160; a UCS 165; a recommendation server 170; and an audio-visual processing server 175.

[0035] In one embodiment, a contact center system manages resources (e.g., personnel, computers, telecommunications equipment, etc.) to enable the delivery of services via telephone or other communication mechanisms. Such services may vary depending on the type of contact center, and may range from customer service to help desk, emergency response, remote marketing, order taking, etc.

[0036] Customers, potential customers, or other users (collectively, customers or end-users) who desire to receive services from a contact center may initiate inbound communications (e.g., phone calls, emails, chats, etc.) to a contact center via end-user devices 105A, 105B, and 105C (collectively, 105). Each of the end-user devices 105 may be a conventional communication device in the art, such as a telephone, wireless telephone, smart phone, personal computer, electronic tablet, laptop, etc. (to name some non-limiting examples). A user operating the end-user device 105 may initiate, manage, and respond to phone calls, emails, chats, text messages, web browsing sessions, and other multimedia transactions. Although three end-user devices 105 are shown at 100 for simplicity, any number of end-user devices may be present.

[0037] Inbound and outbound communications from and to end-user device 105 may traverse network 110, depending on the type of device being used. Network 110 may include a communication network for telephone, cellular, and / or data services, and may also include a private or public switched telephone network (PSTN), a local area network (LAN), a private wide area network (WAN), and / or a public WAN such as the Internet (listing non-limiting examples). Network 110 may also include a wireless carrier network, including a code division multiple access (CDMA) network, a global system for mobile communications (GSM) network, or any wireless network / technology conventional in the art, including but not limited to 3G, 4G, LTE, etc.

[0038] In one embodiment, the contact center system includes a switch / media gateway 115 coupled to the network 110 for receiving and transmitting telephone calls between end users and the contact center. The switch / media gateway 115 may include a telephone switch or a communication switch configured to be used as a central switch for agent-level routing in the center. The switch may be a hardware switching system or a soft switch implemented via software. For example, the switch 115 may include an automatic call distributor, a private branch exchange (PBX), an IP-based software switch, and / or any other switch with dedicated hardware and software, which is configured to receive Internet-sourced interactions and / or telephone network-sourced interactions from customers, and route those interactions to, for example, agent phones or communication devices. In this example, the switch / media gateway establishes a voice path / connection (not shown) between a calling customer and an agent phone device by, for example, establishing a connection between the customer's phone device and the agent phone device.

[0039] In one embodiment, the switch is coupled to a call controller 120, which can be used, for example, as an adapter or interface between the switch and the rest of the routing, monitoring and other communication processing components of the contact center. The call controller 120 can be configured to handle PSTN calls, VoIP calls, etc. For example, the call controller 120 can be configured with computer telephony integration (CTI) software for engaging with the switch / media gateway and contact center equipment. In one embodiment, the call controller 120 may include a session initiation protocol (SIP) server for processing SIP calls. The call controller 120 can also extract data about customer interactions, such as the caller's phone number (e.g., automatic number identification (ANI) number), the customer's Internet Protocol (IP) address or email address, and communicate with other components of the system 100 when processing interactions.

[0040] In one embodiment, system 100 also includes interactive media response (IMR) server 125. IMR server 125 may also be referred to as self-service system, virtual assistant, etc. IMR server 125 may be similar to interactive voice response (IVR) server, but IMR server 125 is not limited to voice and may cover various media channels in addition. In the example of voice, IMR server 125 may be configured with IMR scripts for querying its needs to customers. For example, the contact center of a bank may inform customers via IMR scripts that if they wish to retrieve their account balances, "press 1". By continuing to interact with IMR server 125, customers may be able to complete services without speaking to agents. IMR server 125 may also ask open-ended questions, such as "how may I help you?", and customers may state or otherwise input the reason for contacting the contact center. Routing server 130 may use the response of customers to route calls or communications to appropriate contact center resources.

[0041] If the communication is to be routed to an agent, call controller 120 interacts with routing server (also referred to as orchestration server) 130 to find an appropriate agent for handling the interaction. The selection of an appropriate agent for routing the inbound interaction may be based on, for example, the routing policy employed by routing server 130, and further based on information about agent availability, skills, and other routing parameters provided, for example, by statistics server 140.

[0042] In one embodiment, the routing server 130 may query a customer database that stores information about existing customers, such as contact information, service level agreement (SLA) requirements, the nature of previous customer contacts, and the actions taken by the contact center to resolve any customer issues. The database may be, for example, Cassandra or any NoSQL database and may be stored in the mass storage device 135. The database may also be an SQL database and may be managed by any database management system, such as Oracle, IBM DB2, Microsoft SQL Server, Microsoft Access, PostgreSQL, etc. (to name some non-limiting examples). The routing server 130 may query the customer database for customer information via the ANI or query any other information collected by the IMR server 125.

[0043] Once an appropriate agent is identified as available to handle the communication, a connection may be established between the customer and the agent device 145A, 145B, and / or 145C (collectively 145) of the identified agent. Figure 1 , but any number of devices may be present. Information collected about the customer and / or customer history information may also be provided to the agent device to help the agent better serve the communication. In this regard, each agent device 145 may include a phone suitable for conventional telephone calls, VoIP calls, etc. The agent device 145 may also include a computer for communicating with one or more servers of the contact center and performing data processing associated with contact center operations, as well as for interacting with customers via voice and other multimedia communication mechanisms.

[0044] The contact center system 100 may also include a multimedia / social media server 150 for engaging in media interactions other than voice interactions with the end-user devices 105 and / or web servers 155. The media interactions may be related to, for example, email, voicemail (voicemail via email), chat, video, text messaging, web, social media, co-browsing, etc. The multimedia / social media server 150 may take the form of any IP router conventional in the art with dedicated hardware and software for receiving, processing, and forwarding multimedia events.

[0045] The web server 155 may include, for example, a social interaction site host of various known social interaction sites to which end users may subscribe, such as Facebook, Twitter, Instagram, etc., to name a few non-limiting examples. In one embodiment, although the web server 155 is depicted as part of the contact center system 100, the web server may also be provided by a third party and / or maintained outside the contact center premises. The web server 155 may also provide web pages for businesses being supported by the contact center system 100. End users may browse the web pages and obtain information about the products and services of the businesses. The web pages may also provide a mechanism for contacting the contact center via, for example, web chat, voice call, email, web real-time communication (WebRTC), etc.

[0046] In one embodiment, in addition to real-time interactions, deferred interactions / activities may also be routed to contact center agents. Deferred interactions or activities may include background work or work that can be performed offline, such as responses to emails, letters, attending training, or other activities that do not require real-time communication with customers. The interaction (iXn) server 160 interacts with the routing server 130 to select an appropriate agent to handle the activity. Once assigned to an agent, the activity may be pushed to the agent, or may appear in the agent's work warehouse 146A, 146B, 146C (collectively referred to as 146) as a task to be completed by the agent. The agent's work warehouse may be implemented via any data structure conventional in the art (such as a linked list, an array, etc.). In one embodiment, the work warehouse 146 may, for example, be maintained in a buffer memory of each agent device 145.

[0047] In one embodiment, the mass storage device 135 may store one or more databases related to agent data (e.g., agent profiles, schedules, etc.), customer data (e.g., customer profiles), interaction data (e.g., details of each interaction with a customer, including, but not limited to, the reason for the interaction, disposition data, wait time, processing time, etc.), etc. In another embodiment, some data (e.g., customer profile data) may be maintained in a customer relationship management (CRM) database hosted on the mass storage device 135 or elsewhere. The mass storage device 135 may take the form of a hard disk or disk array as is conventional in the art. In one embodiment, the mass storage device 135 may be accessible by multiple servers including the contact center 100. The contact center may consist of a municipality of mass storage devices 135 associated with a particular server including the contact center 100.

[0048] In one embodiment, the contact center system may include a universal contact server (UCS) 165 configured to retrieve information stored in a CRM database and direct the information to be stored in the CRM database. The UCS 165 may also be configured to facilitate maintaining customer preference history and interaction history, and to capture and store data regarding comments from agents, customer communication history, etc.

[0049] The contact center system may also include a recommendation server 170, which is configured with a recommendation engine to generate self-service solutions and suggestions for end users of the contact center system 100. The recommendation engine on the recommendation server 170 can be pre-trained based on information collected from existing enterprise FAQ pages and customer knowledge bases. The recommendation server 170 can also dynamically map each of the extracted question and answer groups to specific goods and / or services provided by the enterprise. The recommendation server can use data from the mass storage device 135, including agent data, customer data, and interaction data. The recommendation server can use data from other contact center systems (such as web servers 155), including information about enterprise goods and / or services. The recommendation server 170 can also learn user behavior based on actions. For example, based on the user's past purchases, web page traversals, wish lists, asynchronous audio-visual query videos, etc., the system can collect information about user behavior (e.g., user affinity for discounted products, user preference for a group of brands, etc.) and group users into various categories.

[0050] The contact center system may also include an audio-visual processing server 175 that is configured to process audio-visual media received by the contact center and convert the audio-visual media into other data that is usable by the contact center. For example, the audio-visual processing server 175 may receive a video input to the contact center, analyze the content of the input video, and provide the contact center with an output consisting of a text file describing certain actions that occurred during the video. In another embodiment, the output may also include intents collected from the video mapped to associated entities. Entities can be classified into broad categories so that intents received for an entity can be generalized to all entities belonging to that category. Figure 2 An embodiment of an audio-visual processing server is generally indicated in FIG. 8 and described in more detail below.

[0051] Figure 1The various servers may each include one or more processors that execute computer program instructions and interact with other system components for performing the various functions described herein. The computer program instructions are stored in a memory implemented using a standard memory device such as a random access memory (RAM). The computer program instructions may also be stored in other non-transitory computer readable media (such as, for example, a CD-ROM, a flash drive, etc.). Although the functionality of each server is described as being provided by a particular server, it should be appreciated by those skilled in the art that the functionality of the various servers may be combined or integrated into a single server, or the functionality of a particular server may be distributed across one or more other servers without departing from the scope of the embodiments of the present disclosure.

[0052] In one embodiment, the terms "interaction" and "communication" are used interchangeably and generally refer to any real-time and non-real-time interaction using any communication channel, including but not limited to phone calls (PSTN or VoIP calls), email, voicemail, video, chat, screen sharing, text messaging, social media messaging, WebRTC calls, etc.

[0053] Editable audio-visual interface

[0054] Figure 2 is a diagram illustrating an embodiment of a block diagram of an audio-visual processing server 175. The audio-visual processing server 175 may be part of the contact center 100 and may also be comprised of two systems: a video processing system 205 and a speech recognition system 210. The video processing system 205 may also be comprised of: a feature detection application 206, a motion detection application 207, an optical character recognition (OCR) application 208, and an entity detection application 209. In one embodiment, the feature detection application 206, the motion detection application 207, the OCR application 208, and the entity detection application 209 may be implemented as services accessible by the video processing system 205 or as separate systems working in conjunction with the video processing system 205.

[0055] In one embodiment, feature detection application 206 can be used to divide the incoming video file into individual video frames, groups of individual video frames, or shorter video clips. Feature detection application 206 can use scale-invariant feature transform (SIFT) to perform feature detection. SIFT can be deployed to take a video as input and divide it into frames. The number of frames into which the video is divided can be predetermined based on the number of frames set per minute of video. It starts to perform image processing using SIFT method, performing local feature extraction from these frames. Each frame is processed and compared with the previous and subsequent frames. In the presence of movement, the frames are constantly changing, and the method recognizes that there is a continuous action being performed. When there is a pause between each action, the frame remains the same, and the method divides the video into multiple video clips. This can be repeated in a loop to divide the video into the desired number of video clips based on the actions performed. After the video is divided into multiple video clips, the last frame of each video clip can be used to perform additional analysis on each video clip using action detection application 207 and optical character recognition (OCR) application 208. Entity detection application 209 can be used to detect interactive entities during the generated audio-visual clips. The entity may include a web browser object in a web browser window.The stitching system 211 may be used to identify audio portions related to video action and then stitch them together.

[0056] In one embodiment, the motion detection application 207 may be configured to detect and recognize motion depicted in an incoming video file. The motion detection application 207 may deploy appropriate methods (e.g., convolutional neural networks) to classify frames in the video, such as to determine the action being taken on a particular object in the video frame. For example, if a circle is drawn in the video of the incoming video file, the motion detection application 207 may be configured to detect that an action has occurred during the time period in which the circle was drawn. The motion detection application 207 may also be configured to identify any objects on the screen that are associated with the particular circle being drawn.

[0057] In one embodiment, the OCR application 208 can be configured to detect and recognize text typed during the incoming video file to determine the text written on the frame in the incoming video file. The OCR application 208 can deploy appropriate methods to classify the text in the video. For example, STN-OCR, a single semi-supervised deep neural network (DNN) consisting of a spatial transformer network (which can be used to detect text regions in frames) and a text recognition network (which recognizes the text content of the identified text regions), can be deployed in the OCR application 208. The OCR application 208 can provide text information synchronized with certain actions detected by the action detection application 207.

[0058] The speech recognition system 210 can be configured to detect and recognize speech patterns and transcribe spoken words into written text. The speech recognition system 210 can deploy appropriate methods to recognize spoken words in audio files. For example, natural language processing (NLP) and supervised machine learning (ML) can be deployed in the speech recognition system 210. The speech recognition system 210 can also generate text information synchronized with certain actions detected by the action detection application 207.

[0059] Figure 3A 305 is a diagram showing an embodiment of a web-connected user device generally indicated to display a web browser window. The web-connected user device 305 can be a personal computer, a mobile device, a tablet computer, a smart TV, a game console, or other device that can, for example, render HTML code to generate a text and graphic web page on a computing device. The web-connected user device 305 can include a screen or monitor for viewing the rendered web page. The web-connected user device 305 can display a web browser window 310. The web browser window 310 can be generated by a web or internet browser application or program (e.g., Internet Explorer or Google's Chrome) on the web-connected user device 305.

[0060] Web browser objects 315a, 315b, 315c (collectively 315) are elements seen in a web browser window 310. The web browser objects 315 may include content presented in the web browser window 310 by a web or internet browser application or program on a web-connected user device 305. Content displayed as web browser objects 315 may include, but is not limited to, text, graphics, tables, hyperlinks, and embedded objects. Although only three web browser objects 315 are shown for simplicity, the web browser window 310 may contain any number of web browser objects 315.

[0061] The web browser window 310 may also include an editable audio-visual interface engagement object 320 that may be used to engage an editable audio-visual interface 325 that facilitates capturing inquiries to be processed by the contact center (in Figure 3B ). The editable audio-visual interface engagement object 320 may be depicted as a button or clickable image shown in the web browser window 310. The location of the object 320 may be located at any reasonable location in the window 310. Figure 3A , which is generally shown in, for example, the lower right corner.

[0062] The editable audio-visual interface stitching object 320 may be provided by a platform or service that is different from the source of the content of the web browser window 310 (e.g., Genesys Web Engagement provided by Genesys Telecommunications Laboratories, Inc.). For example, a business enterprise (such as an online store) may have a website displayed in the web browser window 310 that contains web pages that display specific products for sale by the business enterprise. In one embodiment, the contact center may provide the editable audio-visual interface stitching object 320 as an overlay in the web browser window 310 through a web server 155 that is not controlled by the business enterprise. However, to the user of the web browser window 310, it may appear to be a single integrated web page.

[0063] Figure 3B is a diagram illustrating an embodiment of a web-connected user device, generally indicated, displaying a web browser window. Figure 3B The embodiments described in the foregoing may also include an editable audio-visual interface for a web browser window. The editable audio-visual interface 325 may be displayed in a variety of ways. In one embodiment, the interface 325 may be displayed as a separate window within the web browser of the user device 305. In another embodiment, it may be displayed as an overlay window on top of the web browser window 310. In another embodiment, it may be displayed as a separate application that can interact with the web browser window 310.

[0064] The editable audiovisual interface 325 may be displayed on the web browser window 310 after the user engages with the editable audiovisual interface engagement object 320. The editable audiovisual interface 325 may also be displayed on the web browser window 310 after the user is provided a link from a communication, such as an email or SMS text message, to engage the editable audiovisual interface 325. The editable audiovisual interface 325 may also be displayed on the web browser window 310 as directed by a remote system, such as a contact center, whereby an agent of the contact center may direct the engagement of the editable audiovisual interface 325.

[0065] In one embodiment, the editable audio-visual interface 325 can be configured to generate a multimedia-enabled and enriched video mail to be sent to a contact center. The editable audio-visual interface 325 can allow an end user to create an enriched video within a web browser window 310, thereby incorporating a web browser object 315 into the enriched video. Creating an enriched video within a web browser window 310 allows a user of the editable audio-visual interface 325 to identify or capture a query or question for a business enterprise at the point of origin of the question or problem. For example, if a customer has a question presented from a web page of the enterprise, the customer can seek help to answer the question directly from the web page. In another example, if a customer is shopping at an online retailer and has a question about a particular product offering on a web page, the customer can create an enriched video that specifically identifies the portion of the web page that poses the question, without having to switch to another medium (e.g., email or phone call) to find the answer.

[0066] The editable audio-visual interface 325 may include a record object 335 to initiate the creation of an enriched video. The record object 335 may be presented as text, a button, a graphic, or a known depiction. The record object 335 may be clicked or selected to initiate the creation of an enriched video. The record object 335 may be selected a second time to terminate the creation of an enriched video. The editable audio-visual interface 325 may include action objects 330a, 330b, and 330c (collectively 330). The action object 330, including the provided tools, may allow the user of the editable audio-visual interface 325 to speak, point, draw, comment, and zoom in on a portion of the web browser object 315. For example, once the recording begins, the user may begin speaking, zooming the frame, and writing notes if necessary. Such gestures may be captured synchronously with the accompanying audio, and together create audio-visual communication through the enriched video.

[0067] Figure 4 1 is a flow chart illustrating an embodiment of a process, generally indicated at 400, for an inbound asynchronous audiovisual channel using an editable audiovisual interface.

[0068] In operation 402, a party browsing the website, such as a customer, engages with the editable audiovisual interface 325. The customer may initiate engagement with the editable audiovisual interface 325 by clicking on the editable audiovisual interface engagement object 320 in the web browser window 310. For example, a customer may be browsing tables at an e-commerce furniture store. Prior to visiting the e-commerce furniture store, the customer has certain expectations regarding the color, price, and specifications of the table. In the event that these expectations are not found to be met on the website, the customer may attempt to initiate engagement with the editable audiovisual interface 325 to express the expectations of the table that the customer desires. The customer may want a black table with given specifications, but can only find brown tables. The customer may also initiate engagement with the editable audiovisual interface 325 by selecting a hyperlink presented on the web browser window 310, selecting a link provided through an email communication conducted via the web server 155, or through a link in a text or SMS communication on a mobile device via the multimedia / social media server 150. Control passes to operation 404 and the process continues.

[0069] In operation 404, the customer's interaction with the editable audio-visual interface 325 is recorded. The interaction may include taking actions on top of the web browser object 315 and in cooperation with the web browser object. The actions may include outlining, which includes drawing, adding text, annotating, underlining, highlighting, scrolling, pointing, zooming, etc. Referring to the table, after initiating the recording of the customer's interaction, the customer may select the action object 330 to draw freehand on the web browser object 315. In one embodiment, the recording may be initiated by the customer clicking on the recording object 335. The customer may draw a circle around the web browser object 315 of the table that is similar to the table specifications required by the customer. The customer may draw freehand on the web browser object 315. The customer may also type text ("I want this table in black") on the web browser object 315 of the table that is similar to the table specifications required by the customer. As part of creating the enhanced video, the customer may also initiate a voice recording. The customer may end recording the interaction performed through the editable audio-visual interface 325 by clicking (or selecting) the recording object 335 a second time. Control proceeds to operation 406 and the process continues.

[0070] In operation 406, the recorded situation is processed. For example, the customer interaction can be processed by a video and voice processing system. The video processing system can analyze the recorded customer interaction for key actions taken by the user during the session. The voice processing system can analyze the recorded customer interaction for verbal keywords during the session. The audio video splicing system 211 identifies which part of the audio needs to be associated / spliced ​​with which part of the action. The video processing system can recognize that the web browser object 315 with which the customer interacts includes a table. The voice processing system can recognize that the user has used the words "black" and "table". As described above, a suitable video and voice processing system can be included as part of the audio-visual processing server 175. In one embodiment, the customer interaction can be analyzed by the audio-visual processing server 175 using operation 600, as described in more detail below. Control proceeds to operation 408 and the process continues.

[0071] In operation 408, an intent may be generated. For example, a customer intent may be generated based on the recorded interaction. The provided video and voice processing system used at operation 406, together with the stitching system 211, may generate content information based on the recorded customer interaction. The processing system may generate appropriate output, such as a JSON object that may be utilized by other systems of the contact center 100. For example, the processing system may generate an intent that the customer is looking for a black table based on the processed video and voice from the recorded interaction. Additional information may be obtained, such as customer identification information (including customer contact information). Control proceeds to operation 410 and the process continues.

[0072] In operation 410, the interaction is routed. In one embodiment, the interaction can be routed to an agent for processing based on the derived customer intent. An appropriate routing device (such as routing server 130) can use the customer intent generated by operation 408 to determine the most appropriate agent to handle the response to the interaction. The agent can be an artificial agent, such as a chatbot directed by multimedia / social media server 150, an IMR device directed by IMR server 125, or a real-time agent using agent device 145. The routing device can determine that the chatbot is the agent that is most likely to provide a response that satisfies the customer's expectations in the customer intent. The chatbot may be able to provide a link in the chat session overlaid in the web browser window 310, thereby providing a link to a suggested table that is closer to satisfying the determined customer intent of the customer's search for a black table. In another embodiment, the routing device can also determine that a real-time agent will most likely provide a response that satisfies the customer's expectations. The customer interaction can be forwarded to the agent device 145 of the agent, including a recording of an enhanced video, and an output of a processing system with information of the determined customer intent. The agent may then handle the interaction using an appropriate communication channel such as web chat, video call, voice call, or email, etc. The agent may review the generated customer intent provided to the agent device 145 prior to viewing the video of the customer interaction.

[0073] In one embodiment, the interactions may be routed to a contact center analytics system for analysis and data storage based on the determined customer intent. The analytics system may be able to aggregate the determined customer intent and provide the business with market information about unmet customer expectations. The business enterprise may be able to use the aggregated customer intent to add, remove, or modify product offerings. For example, if the aggregated customer intent data indicates that multiple customers are looking for black tables, but the e-commerce furniture store does not currently offer black tables, the e-commerce furniture store may attempt to add black tables as part of its product offerings.

[0074] Figure 5 1 is a flow chart illustrating an embodiment of a process for an outbound asynchronous audiovisual channel for a contact center using an editable audiovisual interface, generally indicated at 500. Process 500 may occur in contact center 100.

[0075] Just as customers can use asynchronous audio-visual channels to express customer expectations to commercial enterprises through contact centers, agents at contact centers can also use asynchronous audio-visual channels to contact customers. Agents can use asynchronous audio-visual channels to respond to previously received recorded customer interactions or to communications received from customers through other contact center channels (such as voice calls, text messages, or web chats).

[0076] In operation 502, the agent receives the recorded customer interaction. In one embodiment, if the routing server 130 determines that the live agent will provide the best favorable solution opportunity, the routing device can transmit the recorded customer interaction for receipt by the work pod 146 of the agent device 145 for processing by the agent. However, even if the customer does not provide an incoming recorded customer interaction, the editable audio-visual interface from the asynchronous audio-visual channel can be used by the live agent to provide contact with the customer. Control passes to operation 504 and the process 500 continues.

[0077] In operation 504, the agent engages with the editable audio-visual interface. The agent can review the recorded interaction and edit the video using the editable audio-visual interface from the asynchronous audio-visual channel used by the customer. In one embodiment, the real-time agent can use the editable audio-visual interface engagement object 320 that is part of the agent device 145 to activate the editable audio-visual interface 325 from the asynchronous audio-visual channel. Once activated, the agent can interact with the editable audio-visual interface 325 in the manner described in operation 402 of operation 400. Control passes to operation 506 and the process continues.

[0078] In operation 506, the agent's interaction with the editable interface may be recorded. In one embodiment, the agent interaction may include taking an action on top of the web browser object 315 and cooperating with the web browser object, such as drawing or adding voice. The agent may initiate recording of the interaction by clicking on the record object 335. The agent may then select from the action object 330 to mark the web browser object 315. The agent may end the recording by clicking on the record object 335 a second time, thereby completing the creation of the enriched video. Control proceeds to operation 508 and the process continues.

[0079] In operation 508, the recorded agent interaction is routed to the customer, and process 500 ends. For example, the contact center may route the recorded agent interaction to the customer using appropriate means of the contact center (including the information collected from the customer), such as routing server 130, multimedia / social media server 150, web server 155, or other appropriate communication means. In one embodiment, the customer may receive the agent interaction as a link to the recorded agent interaction, which is embedded in an email, text message, or other appropriate communication application or channel. In one embodiment, the recorded agent interaction may be saved in an appropriate medium such as storage device 135. Storage device 135 may also store customer interactions and interaction data for use by the contact center for other appropriate uses for the business enterprise.

[0080] In one embodiment, the combined use of operations 400 and 500 allows a contact center to enable a full-duplex asynchronous audio-visual channel, thereby allowing customers and agents to asynchronously use recorded interactions to communicate through the contact center.

[0081] Figure 6 is a flow chart illustrating an embodiment of a process for audio-visual processing of asynchronous audio-visual channels, generally indicated at 600. Process 600 may occur in audio-visual processing server 200 or a similar audio-visual processing system.

[0082] In operation 602, a recording is received by the audiovisual processing system. For example, the audiovisual processing server 200 receives an audiovisual interface recording generated through the editable audiovisual interface 325. The recording may be generated as described for a customer-initiated contact in operation 400. When the processing system receives the recording, control passes to operation 604 and the process continues.

[0083] In operation 604, the record is divided into multiple video clips. In one embodiment, the division can be performed by a feature detection application 206 as a part of a video processing system 205. The feature detection application 206 can use scale invariant feature transform (SIFT) to segment the input video into frames. Each frame of the editable audio-visual interface record is processed and compared with the surrounding frames. The comparison is to search for movement or pause. For example, in the case of movement in the editable audio-visual interface record, such as when the user draws a freehand circle (only non-limiting example) on the screen, the frame of the video is constantly changing, and the method recognizes that there is a continuous action being performed. In another example, when there is a pause between actions, the frame remains the same, and the feature detection application 206 can record the frame corresponding to the place where there is a pause between actions in the video. After the feature detection application 206 completes the processing and recording of the frame corresponding to the place where there is a pause between actions in the video, the control of operation 600 proceeds to operation 606 and the process continues.

[0084] In operation 606, a plurality of shorter audiovisual segments are generated based on the editable audiovisual interface record. Figure 6 606a, 606b, and 606c (collectively 606). Any number of audio-visual segments may be generated, where N (606c) indicates this. The audio-visual segments are generated by dividing the editable audio-visual interface recording at the determined pauses in the action using analysis performed by the feature detection application 206 to determine pauses in the action from the editable audio-visual interface recording. Figure 6The generation of at least three unique segments according to the audiovisual interface record is depicted, but the number of generated segments is not set to three because 1 to n number of segments can be generated according to the audiovisual interface record. When the processing system receives the audiovisual interface record, control of operation 600 passes to operation 608 and the process continues.

[0085] In operations 608a, 608b, and 608c (collectively 608), action recognition is performed on each of the generated audio-visual segments from operation 606, respectively. Action recognition may be performed by an appropriate processing system such as an action detection application 207. The action detection application 207 may deploy an appropriate method to classify frames in the video, such as determining the action being taken on a specific object in the video frame. For example, if a circle is drawn in the video of the generated audio-visual segment, the action detection application 207 may be configured to detect that an action has occurred during the time period in which the circle is drawn. The action detection application 207 may also indicate the location in the frame of the video where the action is being performed. The OCR application 208 may be configured to detect and recognize text typed during the incoming video file to determine the text written on the frame in the incoming video file. Control of operation 600 proceeds to operation 610 and the process continues.

[0086] In operations 610a, 610b, and 610c (collectively, 610), entity recognition is performed in each of the generated audio-visual segments. Entity recognition may be performed by an appropriate processing system, such as the motion detection application 207. Using the identified locations in the frames of the video from operation 608 where the action is being performed, the motion detection application 207 may identify any objects on the screen associated with the detected action from operation 608. For example, if a circle is drawn around a picture of a table in the video of the generated audio-visual segment, the motion detection application 207 will attempt to identify the table in the video frame at operation 610. Control of operation 600 passes to operation 612 and the process continues.

[0087] In one embodiment, operations 608, 610, and 612 need not occur in the order shown at 600, but may occur in an order set by an administrator or selected by the contact center.

[0088] In operations 612a, 612b, and 612c (collectively 612), the splicing system 211 identifies the audio portions associated with the video action and splices these portions together. Speech recognition is performed for each generated audio-visual segment. Speech recognition can be performed by an appropriate processing system such as a speech recognition system 210. The speech recognition system 210 can be configured to detect and recognize speech patterns, and transcribe spoken words into written text. The speech recognition system 210 can deploy appropriate methods to recognize spoken words in audio files. For example, natural language processing (NLP) and supervised machine learning (ML) can be deployed in the speech recognition system 210. The speech recognition system 210 can generate text information synchronized with certain actions detected by the action detection application 214. Control of operation 600 proceeds to operation 614 and the process continues.

[0089] In operations 614a, 614b, and 614c (collectively 614), output data from the audio-visual processing of the audio-visual segment is generated. For example, the output data combines data generated from action recognition 608, entity recognition 610, and speech recognition 612 into output data. Control of operation 600 passes to operation 616 and the process continues.

[0090] In operation 616, customer intent is determined based on the output data of the audio-visual processing of the audio-visual clips from operation 614. For example, an appropriate processing system will correlate the information collected in process 600 to extract different parameters from the intent. The determined customer intent may include, for example, a desire for a specific product or service offering, a desire for a modification to a specific product or service offering, customer sentiment (i.e., is the customer angry?), the urgency of the question raised by the customer, and identification of the appropriate department for handling the response to the record. Routing parameters may also be generated to route to the appropriate agent for processing. In one embodiment, the determined customer intent may be expressed as a JSON object, a BSON object, a YAML format, etc.

[0091] Fig. 7A is a diagram illustrating an embodiment of a web-connected user device displaying a web browser window and an editable interface, as generally indicated. Fig. 7A An interface point 702 for a web browser object in a web browser window is included. The web object interface point 702 provides an interaction point for the editable audio-visual interface 325 to interact with the web browser object 315a. In one embodiment, the web object interface point 702 may be depicted as a visual representation, such as a circle shown on the web-connected user device 302, or may be a "virtual" circle that is not depicted on the web-connected user device 302. The web object interface point 702 allows a user to access additional information associated with the web browser object 315 through user selection.

[0092] Figure 7B is a diagram showing an embodiment of a web-connected user device generally indicated displaying a web browser window and an editable interface. In one embodiment, Figure 7B A pop-up FAQ window may also be included. After a user interacts with a web object interface point 702, a recommendation window 704 may appear on a web-connected user device 305. The recommendation window 704 may contain content generated by the recommendation engine 170 associated with content determined by information derived from other customers' records at the editable audiovisual interface 325. The recommendation window 704 may appear as a list of previously identified questions or FAQs associated with the interface point 702 linked to the web browser object 315. In one embodiment, the recommendation window may appear as a list of the most frequently asked questions for an entity, web page, or by a user, or any combination of these.

[0093] Figure 8 is a flow diagram illustrating an embodiment, generally indicated at 800, for providing a recommendation asynchronous solution via an asynchronous audio-visual channel.

[0094] In operation 802, the customer engages the editable audiovisual interface 325. The user may initiate the editable audiovisual interface 325 as previously described in operation 404 of process 400. The editable audiovisual interface 325 may then be displayed on the web-connected user device 302. Control passes to operation 804 and process 800 continues.

[0095] In operation 804, the editable audio-visual interface 325 provides an interaction point on the web browser object 315. The interaction point may use the interface as the disclosed web object interface point 702. The interaction point may be provided for each web browser object 315 or only for certain web browser objects 315. A visual indication may also be provided when an interface point 702 is selected. Control passes to operation 806 and the process continues.

[0096] In operation 806, when the user interacts with the interface point 702, a recommendation window appears. For example, the recommendation window may appear as a recommendation window 704, which may include a list of recommendations or previously asked questions. The contents of the list may be received from the recommendation server 170. The recommendation engine on the server 170 may be pre-populated with data from the mass storage device 135 (including agent data, customer data, and interaction data) to compile a list of known questions and answers about the goods and / or services provided by the enterprise. The recommendation engine may use information extracted from the customer's previous interactions using an asynchronous audio-visual channel. For example, the recommendation engine may use a determined intent (e.g., "The customer wants this table in black") to populate a question posed to the recommendation engine (e.g., "Does this table come in other colors?"). The recommendation engine may associate specific goods and / or services provided by the enterprise with specific previously asked questions about those specific goods and / or services. The recommendation engine may associate specific goods and / or services with specific web browser objects 315 indicating those goods and / or services, thereby associating specific previously asked questions with specific web browser objects 315. The recommendation window 704 may provide a specific number of recommendations associated with a particular web browser object 315, either by a predetermined number of recommendations or by a set threshold associated with the quality of the recommendations. Control passes to operation 808 and the process continues.

[0097] At operation 808, the user selects a recommendation from the recommendation window 704 at operation 808. The user selection may be made by, for example, clicking on a recommendation populated in the recommendation window 704. Control passes to operation 810 and the process continues.

[0098] In operation 810, it is determined whether the selected recommendation from operation 808 is associated with a known solution or a problem tag (alternatively referred to herein as a "recommendation option tag"). If it is determined that the selected recommendation is associated with a known solution, control passes to operation 814 and the process continues. If the selected recommendation from operation 808 is associated with a problem tag, control passes to operation 812 and the process continues.

[0099] A known solution may be defined as a definite answer to a recommendation posed as a question. For example, for the recommendation "Does this table come in other colors?", a recommendation with a known solution would respond to the question, such as "Yes, this table comes in white" or "No, this table does not come in other colors". A question label is associated with a recommendation that currently has no definite answer to the recommendation posed as a question. For example, for the recommendation "Does this table come in other sizes", if the recommendation engine has no known response to the question, it may simply have the question label "size" associated with the recommendation.

[0100] In operation 812, the question tag associated with the selected recommendation is associated with the current customer interaction. The question tag may be presented as a combination of different media. In this case, if the selected recommendation "I want to learn about other colors" is selected from the recommendation window 704, the "I want to learn about other colors" tag will be added to the editable audio-visual interface record. Operation 800 ends, allowing the customer using the editable audio-visual interface to continue recording through the editable audio-visual interface. In one embodiment, the interaction can be sent to a live agent for processing.

[0101] In operation 814, the customer is prompted with an option to view known solutions associated with the selected recommendation. For example, the known solutions may be provided to the customer in the form of text from the FAQ, a link to another portion of the website, a previous recording through an asynchronous audio-visual channel, or through other acceptable methods of conveying information to the customer. The known solutions may also provide the customer with a summary of its contents with an option to view the known solutions associated with the selected recommendation. Control passes to operation 816 and the process continues.

[0102] In operation 816, in response to the selected recommendation, it is determined whether the customer accepts the known solution provided to the customer. If the customer accepts the known solution, the known solution is presented to the customer at operation 818, and operation 800 ends. This indicates that the customer finds the known solution acceptable, and further information may not need to be provided to the customer. If the customer does not accept the known solution, the known solution may not be an acceptable response to the selected recommendation, and the recommendation engine may need to further refine the known solution associated with the selected recommendation. If the customer does not accept the known solution, control of operation 800 passes to operation 820, whereby a question tag is generated based on the selected recommendation and operation 800 continues back to operation 812.

[0103] Computer Systems

[0104] In one embodiment, each of the various servers, controls, switches, gateways, engines, and / or modules (collectively referred to as servers) in the figures is implemented via hardware or firmware (e.g., ASICs), as will be understood by those skilled in the art. Each of the various servers may be implemented on one or more computing devices (e.g., Fig. 9A , Fig. 9B) that executes computer program instructions and interacts with other system components for performing the various functions described herein. The computer program instructions are stored in a memory that can be implemented in a computing device using standard memory devices (such as RAM). The computer program instructions may also be stored in other non-transitory computer-readable media, such as CD-ROMs, flash drives, and the like. Those skilled in the art will recognize that a computing device may be implemented via firmware (e.g., an application specific integrated circuit), hardware, or a combination of software, firmware, and hardware. Those skilled in the art will also recognize that, without departing from the scope of the disclosed embodiments, the functions of various computing devices may be combined or integrated into a single computing device, or the functions of a particular computing device may be distributed across one or more other computing devices. A server may be a software module, which may also be referred to simply as a module. The set of modules in the contact center may include a server and other modules.

[0105] The various servers may be located on an on-site computing device at the same physical location as the contact center's agents, or may be located off-site (or in the cloud) at a geographically different location (e.g., in a remote data center) that is connected to the contact center via a network (such as the Internet). In addition, some of the servers may be located in an on-site computing device at the contact center, while other servers may be located in an off-site computing device, or servers providing redundant functionality may be provided via both on-site computing devices and off-site computing devices to provide greater fault tolerance. In some embodiments, functionality provided by servers located on off-site computing devices may be accessed and provided via a virtual private network (VPN) as if such servers were on-site, or functionality may be provided using software as a service (SaaS) to provide functionality over the Internet using various protocols, such as by exchanging data using data encoded in extensible markup language (XML) or JavaScript object notation (JSON).

[0106] Fig. 9A and Fig. 9B is a diagram illustrating an embodiment of a computing device, generally indicated at 900, that may be employed in embodiments of the present invention. Each computing device 900 includes a CPU 905 and a main memory unit 910. Fig. 9A As shown, the computing device 900 may also include a storage device 915, a removable media interface 920, a network interface 925, an input / output (I / O) controller 930, one or more display devices 935A, a keyboard 535B, and a pointing device 935C (e.g., a mouse). The storage device 915 may include, but is not limited to, a storage device for an operating system and software. Fig. 9BAs shown, each computing device 900 may also include additional optional elements, such as a memory port 940, a bridge 945, one or more additional input / output devices 935D, 935E, and a cache memory 950 that communicates with the CPU 905. The input / output devices 935A, 935B, 935C, 935D, and 935E may be collectively referred to herein as 535.

[0107] The CPU 905 is any logic circuit that responds to and processes instructions obtained from the main memory unit 910. It can be implemented, for example, as an integrated circuit, in the form of a microprocessor, microcontroller, or graphics processing unit, or in a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The main memory unit 910 can be one or more memory chips that can store data and allow the central processing unit 505 to directly access any storage location. Fig. 9A As shown, the central processing unit 905 communicates with the main memory 910 via the system bus 955. Fig. 9B As shown, the central processing unit 905 can also communicate directly with the main memory 910 via the memory port 940.

[0108] In one embodiment, CPU 905 may include multiple processors, and may provide functions for executing multiple instructions simultaneously or for executing one instruction simultaneously for more than one piece of data. In one embodiment, computing device 900 may include parallel processors with one or more cores. In one embodiment, computing device 900 includes a shared memory parallel device, which has multiple processors and / or multiple processor cores, thereby accessing all available memories as a single global address space. In another embodiment, computing device 900 is a distributed memory parallel device with multiple processors, and each processor only accesses local memory. Computing device 900 may have both shared certain memory and certain memory that can be accessed only by a specific processor or a subset of processors. CPU 905 may include a multi-core microprocessor, which combines two or more independent processors into a single package, for example, into a single integrated circuit (IC). For example, computing device 900 may include at least one CPU 905 and at least one graphics processing unit.

[0109] In one embodiment, CPU 905 provides single instruction multiple data (SIMD) functionality, e.g., executing a single instruction on multiple pieces of data simultaneously. In another embodiment, several processors in CPU 905 may provide functionality for executing multiple instructions (MIMD) on multiple pieces of data simultaneously. CPU 905 may also use any combination of SIMD cores and MIMD cores in a single device.

[0110] Fig. 9BAn embodiment is depicted in which the CPU 905 communicates directly with the cache memory 950 via a second bus (sometimes referred to as a backside bus). In other embodiments, the CPU 905 communicates with the cache memory 950 using a system bus 955. The cache memory 950 typically has a faster response time than the main memory 910. Fig. 9A As shown, the CPU 905 communicates with various I / O devices 935 via a local system bus 955. Various buses may be used as the local system bus 955, including but not limited to a Video Electronics Standards Association (VESA) local bus (VLB), an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express bus, or a NuBus. For embodiments in which the I / O device is a display device 935A, the CPU 905 may communicate with the display device 935A via an Advanced Graphics Port (AGP). Fig. 9B An embodiment of a computer 900 is depicted in which the CPU 905 communicates directly with an I / O device 935E. Fig. 9B Also depicted is an embodiment in which local bus and direct communications are mixed: CPU 905 communicates with I / O device 935D using local system bus 955 while communicating directly with I / O device 935E.

[0111] A variety of I / O devices 535 may be present in computing device 900. Input devices include one or more keyboards 935B, mice, trackpads, trackballs, microphones, and drawing tables, to name a few non-limiting examples. Output devices include video display devices 935A, speakers, and printers. Fig. 9A The illustrated I / O controller 930 may control one or more I / O devices such as, for example, a keyboard 935B and a pointing device 935C (eg, a mouse or optical pen).

[0112] Reference again Fig. 9A , the computing device 900 may support one or more removable media interfaces 520, such as a floppy disk drive, a CD-ROM drive, a DVD-ROM drive, a tape drive of various formats, a USB port, a secure digital or compact FLASH™ memory card port, or any other device suitable for reading data from a read-only medium or suitable for reading data from a read-write medium or writing data to a read-write medium. The I / O device 535 may be a bridge between the system bus 955 and the removable media interface 920.

[0113] Removable media interface 920 may be used, for example, to install software and programs. Computing device 900 may also include storage device 915, such as one or more hard drives or hard drive arrays, for storing an operating system and other related software and for storing application software programs. Optionally, removable media interface 920 may also be used as a storage device. For example, the operating system and software may be run from a self-bootable medium (e.g., a self-bootable CD).

[0114] In one embodiment, computing device 900 may include or be connected to multiple display devices 935A, and each display device may have the same or different types and / or forms. Therefore, any one of I / O device 535 and / or I / O controller 930 may include any type and / or form of suitable hardware, software or a combination of hardware and software to support, enable or provide computing device 900 to connect and use multiple display devices 935A. For example, computing device 900 may include any type and / or form of video adapter, video card, driver and / or library to engage, communicate, connect or otherwise use display device 935A. In one embodiment, video adapter may include multiple connectors to engage multiple display devices 935A. In another embodiment, computing device 900 may include multiple video adapters, wherein each video adapter is connected to one or more of display device 935A. In other embodiments, one or more of display device 935A may be provided by one or more other computing devices, connected to, for example, computing device 900 via a network. These embodiments may include any type of software designed and constructed to use a display device of another computing device as a second display device 935A of the computing device 900. One of ordinary skill in the art will recognize and appreciate the various ways and embodiments in which the computing device 900 may be configured to have multiple display devices 935A.

[0115] exist Fig. 9A and Fig. 9B An embodiment of a computing device generally indicated in can operate under the control of an operating system that controls the scheduling of tasks and access to system resources. The computing device 900 can run any operating system, any embedded operating system, any real-time operating system, any open source operating system, any proprietary operating system, any operating system for mobile computing devices, or any other operating system capable of running on a computing device and performing the operations described herein.

[0116] The computing device 900 may be any workstation, desktop computer, laptop or notebook computer, server machine, handheld computer, mobile phone or other portable telecommunication device, media playback device, gaming system, mobile computing device, or any other type and / or form of computing, telecommunication or media device capable of communication and having sufficient processor power and memory capacity to perform the operations described herein. In some embodiments, the computing device 900 may have a different processor, operating system, and input device consistent with the device.

[0117] In other embodiments, computing device 900 is a mobile device. Examples may include a Java-enabled cellular phone or personal digital assistant (PDA), a smart phone, a digital audio player, or a portable media player. In one embodiment, computing device 900 includes a combination of devices, such as a mobile phone combined with a digital audio player or a portable media player.

[0118] Computing device 900 can be one of a plurality of machines connected by a network, or it can include a plurality of machines connected in this way.Network environment can include one or more local machines, clients, client nodes, client machines, client computers, client devices, endpoints or endpoint nodes, which communicate with one or more remote machines (which can also be generally referred to as server machines or remote machines) via one or more networks.In one embodiment, the local machine has the ability of a server machine used as a client node seeking access to resources provided by a server machine, and as a server machine providing access to managed resources for other clients.Network can be a LAN or WAN link, a broadband connection, a wireless connection or any or all of the above combinations.Can use a variety of communication protocols to establish connection.In one embodiment, computing device 900 communicates with other computing devices 900 via any type and / or form of gateway or tunneling protocol (such as secure socket layer (SSL) or transport layer security (TLS)).Network interface can include built-in network adapter (such as network interface card), which is suitable for engaging computing device to any type of network capable of communicating and performing operations described herein.I / O device can be a bridge between system bus and external communication bus.

[0119] In one embodiment, the network environment can be a virtual network environment, wherein various components of the network are virtualized. For example, various machines can be virtual machines implemented as software-based computers running on physical machines. The virtual machines can share the same operating system. In other embodiments, different operating systems can be run on each virtual machine instance. In one embodiment, a "hypervisor" type of virtualization is implemented, wherein multiple virtual machines run on the same host physical machine, and each virtual machine acts as if it has its own dedicated box. Virtual machines can also run on different host physical machines.

[0120] Other types of virtualization are also contemplated, such as networks (eg, via software defined networking (SDN)). Functions (such as those of session border controllers and other types of functions) may also be virtualized, such as via network functions virtualization (NFV).

[0121] According to one or more disclosed embodiments, without departing from the spirit and scope of the disclosed embodiments, unless otherwise indicated, the various operations described above may be omitted, or additional operations may be included. In addition, without departing from the spirit and scope of the disclosed embodiments, unless otherwise indicated or implied, the order of operations may be modified or changed.

[0122] Many functional units of the system in the embodiments of the present invention can be implemented as code routines in computerized telephone equipment, computer servers and various workstations. As is well known, programmers are highly personalized and can implement similar functions by significantly different routines. In addition, the present invention can be applied to hardware systems that vary greatly. In addition, the hardware used to practice the present invention can be different in many aspects. Similarly, there are many other changes in the embodiments described herein, and these changes will fall within the spirit and scope of the disclosed embodiments in its described several aspects. The present invention is only limited by the scope of the following claims.

Claims

1. A method of processing an interactive audio-visual asynchronous query from a first party and routing the query to a second party for response in a contact center system, the method comprising: providing, by the web server, an editable audiovisual interface to a web-enabled user device associated with the first party; recording, by the web server, the first party's interaction with the editable audiovisual interface; generating, by the web server, an asynchronous query to the contact center system based on the recorded interaction, wherein the asynchronous query includes audio and video; processing the audio and the video of the asynchronous query by an audio-visual analysis system, wherein the processing includes identifying actions performed by the first party during the asynchronous query using at least one of: a convolutional neural network for classifying frames in the video, and an OCR application for classifying text in the video; generating, by the audiovisual analysis system, intent based on the processed audio and video; transmitting, by the audiovisual analysis system, the generated intent and the asynchronous query to a routing server, wherein the routing server selects a second recipient using the generated intent; as well as transmitting, by the routing server, a signal for routing the asynchronous query to the second party for response, The method further comprises: using the editable audiovisual interface to access an asynchronous audiovisual channel, outbound asynchronous audio-visual channels for contact centers using the editable audio-visual interface, and The inbound asynchronous audio-visual channel and the outbound asynchronous audio-visual channel can be combined to allow the contact center to enable a full-duplex asynchronous audio-visual channel, thereby allowing customers and agents to asynchronously use the recorded interactions to communicate through the contact center.

2. The method of claim 1 , wherein the editable audio-visual interface includes tools for enabling the first party to perform one or more of the following on a web page of the user device: drawing, pointing, writing text, and inputting voice. 3 . The method of claim 1 , wherein the processing further comprises determining objects that are engaged on a web page of the user device during the asynchronous query. 4 . The method of claim 1 , wherein the processing further comprises determining a portion of audio that needs to be spliced ​​to a corresponding video of the asynchronous query.

5. The method of claim 1, wherein the second party responds to the recorded interaction through the editable audio-visual interface. The method of claim 1 , wherein the second party comprises a virtual agent.

7. The method of claim 1, wherein the intent comprises routing parameters, wherein the routing parameters comprise at least one of: customer type, web page type, intent type, user intent, keyword, sentiment, priority, and emotion.

8. The method of claim 1, wherein the response by the second party comprises: providing, by the web server, an editable audio-visual interface to the second party; recording, by the web server, the interaction of the second party with the editable audiovisual interface; recording the second party interaction by the audiovisual analysis system for storage by the contact center; as well as The second party interaction is transmitted to the first party by the audio-visual analysis system via the asynchronous audio-visual channel.

9. The method according to claim 1, wherein the generating intention further comprises the following steps: classifying, by the audio-visual analysis system, frames in the asynchronous query, the frames indicating actions performed by the first party in a video of the query by applying a neural network; generating, by the audiovisual analysis system, a plurality of audiovisual segments based on the query, separating the query from the determined frames indicative of actions performed by the first party; processing, by the audiovisual analysis system, one of the generated audiovisual segments, wherein the processing includes using an action recognition application to determine an action performed during the generated audiovisual segment; processing the generated audio-visual segment by the audio-visual analysis system through an entity recognition application, the entity recognition application being operable to determine interacting entities during the generated audio-visual segment; processing the query by the audio-visual analysis system to splice the audio corresponding to the action identified from the audio-visual segment; processing the generated audio-visual segment by the audio-visual analysis system through a speech recognition application, the speech recognition application being used to derive speech content from the generated audio-visual segment; generating, by the audiovisual analysis system, an audiovisual segment output comprised of the determined actions, interacting entities, and speech content derived from the processed generated audiovisual segment; as well as A first-party intent is generated by the audio-visual analysis system based on the generated audio-visual segment output.

10. The method of claim 9, wherein the action performed during the generated audiovisual segment is drawing by the first party in the video of the query.

11. The method of claim 9, wherein the action performed during the generated audio-visual segment is typing text by the first party in the video of the query.

12. The method of claim 9, wherein the interactive entity is a web browser object in a web browser window.

13. The method of claim 9, wherein the first-party intent is expressed as a JSON object.

14. The method of claim 1, wherein the response comprises an asynchronous automated response.

15. The method of claim 14, wherein the responding further comprises the steps of: Providing, by the web server, an editable audio-visual interface to the first party; providing, by the web server, an interaction point for the first party; providing, by the web server, a recommendation window in response to the first party interacting with the interaction point; receiving, by the web server, a signal indicative of a selection by the first party from the recommendation window; determining, by a recommendation engine, whether an asynchronous automated response exists to the selection by the first party from the recommendation window; as well as If the asynchronous automated response exists for the selection made by the end-user from the recommendation window, the asynchronous automated response is transmitted by the web server to the end-user, otherwise the interaction is routed to a second party for processing.

16. The method of claim 15, wherein the interaction point is associated with a web browser object displayed on the end-user device.

17. The method of claim 15, wherein the recommendation engine generates the selections from the recommendation window based on a database of intents and entity classifications generated from other queries.

Images