Multi-mode touch screen interaction system, method and device for VoLTE (Voice over Long Term Evolution) call of mobile equipment

Abstract

The invention relates to the technical field of communication, and discloses a multi-mode touch screen interaction system, method and device for VoLTE communication of mobile equipment, and the key points of the technical scheme are that the system comprises a user equipment end which is used for initiating the VoLTE communication and collecting operation information; decoding the multimedia stream in the digital format, and restoring and displaying a user page; the SIP server is used for establishing call connection; the control middle station is used for receiving the operation information, forwarding the operation information to the cloud browser or the video coding server or the TTS service module for interactive processing, combining the coded video stream and audio stream into a multimedia stream, and sending the multimedia stream to the user equipment side; the cloud browser is used for simulating user operation, loading corresponding webpage content, performing page rendering and layout and generating a picture stream; the TTS service module is used for converting text information needing to be broadcasted in the webpage content into voice and generating an audio stream; and the video coding server is used for converting the picture stream into a video stream and coding the video stream and the audio stream into a digital format.

Description

Technical Field

[0001] The present invention relates to the field of call technology, and more specifically, to a multi-modal touch screen interaction system, method and device for VoLTE calls on a mobile device. Background Art

[0002] VoLTE (Voice over LTE) video calling is a technology that enables high-quality real-time video communication over a 4G LTE network. Users need to have a VoLTE-compatible smartphone or terminal device, and the mobile operator network must support VoLTE / 5G services. VoLTE / 5G services include:

[0003] Network connection: The user device is connected to the LTE / 5G network and a call is initiated.

[0004] Call establishment: The user selects the video call option through the calling application and then enters or selects the contact information of the call partner. The calling application initiates a call request to the operator's server.

[0005] Media negotiation: Once the connection is established, the devices on both ends of the call will perform media negotiation to determine parameters such as supported video codecs, resolutions, and frame rates to ensure that both devices can understand and present the video.

[0006] Media transmission: Video data is transmitted in digital format and sent to the other device via LTE / 5G network.

[0007] Video display: The receiving device receives the video data, decodes and presents it, and displays the video image on the screen.

[0008] Two-way communication: During a video call, both parties can watch each other's video at the same time and use the built-in microphone and speaker for voice communication. Since the essence of Volte calls is to transmit video streams, in the traditional implementation method, the interaction of Volte calls is mainly voice interaction and key interaction. After the call starts, the logic is: the call application provides options to switch to voice or keyboard mode. Users can use voice to communicate, or switch to keyboard mode and use the numeric keypad to communicate. The call application supports real-time voice recognition and converts voice into text, or users directly type numeric keypad messages. The other party receives and responds to the text message with voice or video stream.

[0009] Select video stream segments: In the call application, the server can select the video stream segments to be sent based on logic. These segments can be pre-recorded videos or screen sharing content, depending on the application's functions and settings. Send video segments: The selected video segments will be encoded and sent to the other party's device. The other party receives and displays: The receiving party's device receives the video segment and displays it on the screen.

[0010] Because video stream segments are returned to the client, the video segments are either captured by the camera, which is generally used for human agents, or fixed video images. Therefore, UI control interaction and control operations are not supported during the call, including click, double-click, and slide operations.

[0011] The existing solutions and technologies have the following problems: the input method only supports voice and numeric keys, but not clicks and slides. It cannot handle multi-modal interaction forms such as multimedia elements such as buttons, pictures, windows, and interactive bars; the returned images come from cameras or video splicing, and cannot provide a complete interactive product experience.

[0012] Call centers are important communication hubs between organizations and customers, providing services such as customer support, problem solving, and information delivery. Traditional VoLTE interactions do not support multimodal control interactions, but with the development of communication technology, multimodal interaction VoLTE has gradually become a trend in improving call center services, so it is necessary to further upgrade the technology of multimodal interaction in call centers. Summary of the invention

[0013] The purpose of the present invention is to provide a multi-modal touch screen interaction system, method and device for VoLTE calls on mobile devices, to achieve a complete product interaction experience, support page function display, and a complete interactive product experience of page jump; and support providing users with a real page operation experience.

[0014] The above technical objectives of the present invention are achieved through the following technical solutions: A multi-modal touch screen interaction system for VoLTE calls on a mobile device, comprising:

[0015] User equipment side, used to support users to initiate VoLTE calls, to collect operation information; to decode digital multimedia streams, and to restore and display user pages;

[0016] The SIP server is used to establish a call connection with the user equipment end,

[0017] The control center is used to receive operation information after the call connection is established, and forward it to the cloud browser or video encoding server or TTS service module for interactive processing, and to merge the encoded video stream and audio stream into a multimedia stream and send it to the user device end;

[0018] Cloud browser, used to simulate user operations according to operation information, load corresponding web page content, perform page rendering and layout, and take screenshots of the current page to generate image streams;

[0019] The TTS service module is used to convert the text information to be broadcasted in the webpage content into speech and generate an audio stream;

[0020] The video encoding server is used to convert the picture stream into a video stream, and compress and encode the video stream and audio stream into a digital format.

[0021] As a preferred technical solution of the present invention, the user equipment end has a VoLTE service, the mobile operator network where the user equipment end is located supports the VoLTE service, the user equipment end is installed with a call application supporting the VoLTE service, and the call application displays risk content to the user before providing the VoLTE service and obtains collection authority;

[0022] The user equipment end is provided with a decoder for restoring the multimedia stream in digital format into a user page.

[0023] As a preferred technical solution of the present invention, the SIP server is also used to establish a video channel and a signaling channel with the user equipment end; the Sip server is a server for implementing VoIP services, Volte services and real-time communication services, and is used to negotiate media parameters of both parties of the call, and transmit audio data, video data, control information and event information in real time; when establishing a call connection, the Sip server connects to the control center to request a call and initialize the call parameters.

[0024] As a preferred technical solution of the present invention, the video channel is used to transmit audio data and video data in real time, and encrypt the transmission content; the signaling channel is used to transmit user operation information, and encrypt the transmission content, and the operation information includes operation events and coordinate information.

[0025] As a preferred technical solution of the present invention, the control center receives user instructions from the client module and passes them to other modules for processing; the control center is also responsible for processing the user's context information to generate a suitable response; after receiving the call request sent by the SIP server, the control center initializes the call state according to the call parameters, allocates a unique session identifier for the call, and determines the service type required for the call;

[0026] When the control center determines that the call needs to interact with the cloud browser, it sends a request to the browser to obtain the entry information for accessing the web page.

[0027] As a preferred technical solution of the present invention, after the video encoding module completes encoding of the video stream, it returns to the control center, which transmits it to the SIP server, and then the SIP server transmits it to the user device of the recipient.

[0028] As a preferred technical solution of the present invention, the cloud browser module will load the corresponding web page content according to the user's operation information, and render and layout the page; the cloud browser is also linked to an interactive page module, which is an actual product business interface and is used to design and render the user page in the interaction;

[0029] The cloud browser is also used to calculate the text information that needs to be broadcast according to the web page content, and forward it to the TTS server for text-to-speech conversion to obtain an audio stream.

[0030] A multi-modal touch screen interaction method for a VoLTE call on a mobile device comprises the following steps:

[0031] Initiate a call request with the recipient and collect the operation information of the user page;

[0032] In response to the call request, establishing a call connection;

[0033] After the call connection is established, the operation information is received and the interactive logic processing is performed;

[0034] According to the operation information, simulate the user operation, load the corresponding web page content, perform page rendering and layout, take a screenshot of the current page, and generate an image stream;

[0035] Convert the text information to be broadcasted in the webpage content into voice and generate an audio stream;

[0036] Convert the picture stream into a video stream, compress and encode the video stream and audio stream into a digital format, and then merge them into a multimedia stream;

[0037] Send the multimedia stream to the call recipient, decode the multimedia stream, and restore the user page.

[0038] A multi-modal touch screen interaction device for VoLTE calls on a mobile device comprises: a processor and a memory, wherein the memory stores a computer program executable by the processor, and the processor implements the above method when executing the computer program.

[0039] In summary, the present invention has the following beneficial effects: it can break the input limitation and support click, slide and other interactive inputs in addition to traditional voice and key interaction; the video rendering method proposed in this solution can convert the previous overall display of video files into UI-based interactive logic and interactive experience, and can realize a complete product interactive experience in Volte, support page function display, and complete interactive product experience of page jump; support providing users with a real page operation experience;

[0040] Multimodal support enables call centers to better handle these diverse media. Providing multimodal interactions can enhance the user experience and help customers better understand and solve problems. This can increase customer satisfaction and loyalty. The use of video calling products in call centers helps improve user experience, problem solving efficiency and service quality, reduce costs, and meet the communication needs of different users. It is an indispensable part of modern call centers. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a system structure diagram of the present invention;

[0042] Figure 2 is a schematic diagram of coordinate information of the present invention;

[0043] Figure 3 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0044] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0045] like Figure 1 As shown, the present invention provides a multi-modal touch screen interaction system for VoLTE calls on mobile devices, including: a user device end, a SIP server, a control center, a cloud browser, a TTS service module, and a video encoding server.

[0046] User equipment side, used to support users to initiate VoLTE calls, to collect operation information; to decode digital multimedia streams, and to restore and display user pages;

[0047] Specifically, the user's device has VoLTE service, the mobile operator network where the user's device is located supports VoLTE service, and the user's device has installed a call application that supports VoLTE service. The call application displays risk content to the user before the VoLTE service is provided, fully informs the customer of the risks, and obtains the collection authority after obtaining the customer's authorization;

[0048] The user equipment end is provided with a decoder for restoring the multimedia stream in digital format into a user page.

[0049] SIP server, used to establish a call connection with the user's device;

[0050] The SIP server is also used to establish video channels and signaling channels with user equipment. The Sip server is a server that implements VoIP services, Volte services, and real-time communication services. It is used to negotiate the media parameters of the two parties of the call, such as audio and video codecs, to ensure proper media communication. It also uses real-time transmission protocols such as RTP to transmit audio data, video data, control information, and event information in real time. When establishing a call connection, the Sip server connects to the control center to request a call and initialize the call parameters.

[0051] Specifically, the video channel is used to transmit audio and video data in real time, and encrypt the transmitted content; such as sound and images. It is responsible for delivering real-time media content for users to watch and listen to. The video stream is encrypted during the transmission process to ensure data security.

[0052] The signaling channel is used to transmit user operation information and encrypt the transmission content. The operation information includes operation events and coordinate information. The signaling channel is mainly used to transmit non-video information, responsible for transmitting the control and event information of the communication session, such as click, double-click, slide and other information. The signaling is encrypted during the transmission process to ensure data security.

[0053] Regarding an embodiment of the coordinate information, Figure 2 As shown, the plane coordinate system used in the present invention takes the upper left corner as the origin and divides the plane into four quadrants. The origin is located in the upper left corner, and the x-axis extends to the right, with the positive direction pointing to the right. At the same time, the y-axis extends vertically downward, with the positive direction pointing downward. In this coordinate system, positive x values ​​are represented to the right of the origin, and positive y values ​​are represented below the origin. Negative x values ​​are represented to the left of the origin, and negative y values ​​are represented above the origin.

[0054] This coordinate system is widely used in many computer graphics and image processing applications because it corresponds closely to the screen coordinate system. Pixels on the screen are usually expressed in this coordinate system with the origin being the upper left corner of the screen, which allows users to easily describe and position elements, text, and objects in an image.

[0055] For fields such as computer graphics, drawing, interface design, and game development, this coordinate system provides a convenient way to specify positions and areas on the screen, making the layout and interaction of elements more intuitive and easy to control. In this coordinate system, the positive directions of x and y are consistent with the natural directions of daily observation, which makes it a common choice for practical applications.

[0056] The control center is the core of the entire system and is responsible for logic processing and information transfer. It receives user instructions from the client module and passes them to other modules for processing. The control center is also responsible for processing the user's context information in order to generate appropriate responses. It realizes real-time interaction with users through interfaces with other modules.

[0057] The control center is used to receive operation information after the call connection is established, and forward it to the cloud browser or video encoding server or TTS service module for interactive processing, and to merge the encoded video stream and audio stream into a multimedia stream and send it to the user device end;

[0058] The control center receives user instructions from the client module and passes them to other modules for processing. The control center is also responsible for processing the user's context information in order to generate appropriate responses. After receiving a call request from the SIP server, the control center initializes the call state according to the call parameters, assigns a unique session identifier to the call, and determines the service type required for the call.

[0059] When the control center determines that the call needs to interact with the cloud browser, it sends a request to the browser to obtain the entry information for accessing the web page.

[0060] The cloud browser module is a browser module running on the server. The cloud browser is used to simulate user operations based on operation information, load corresponding web page content, perform page rendering and layout, and take screenshots of the current page to generate image streams.

[0061] The cloud browser module will load the corresponding web page content according to the user's operation information, and render and layout the page; the cloud browser is also linked to the interactive page module, which is the actual product business interface and is used to design and render the user page in the interaction; the interactive page module is the actual product business interface of the present invention, which is responsible for the design and rendering of the product system. The interactive page module is specifically an interactive H5 page, which contains the user interface in the interaction, including buttons, input boxes, selection boxes, sliding boxes, windows and other elements.

[0062] The cloud browser is also used to calculate the text information that needs to be broadcast according to the web page content, and forward it to the TTS server for text-to-speech conversion to obtain an audio stream.

[0063] The TTS service module is used to convert the text information that needs to be broadcast in the webpage content into speech and generate an audio stream; the TTS module is a text-to-speech synthesis module, which is responsible for converting the text generated by the system into natural and fluent speech output. It uses speech synthesis technology and audio synthesis algorithms to convert text into audible speech signals.

[0064] The video encoding server is used to convert the image stream into a video stream, and compress and encode the video stream and audio stream into a digital format. After the video encoding module completes the encoding of the video stream, it returns to the control center, which is then transmitted to the SIP server, and then transmitted to the receiving user device by the SIP server.

[0065] The video encoding module is a key component for compressing and encoding video signals into a digital format so that video data can be effectively managed during storage, transmission, and playback. This process helps to reduce the size of video files and reduce transmission bandwidth requirements while maintaining sufficient video quality. The video encoding module needs to select a suitable codec, such as H.264, H.265, VP9, ​​etc. Different codecs have different compression algorithms and performance characteristics. The present invention selects the H.264 codec according to the requirements.

[0066] The present invention also provides a multi-modal touch screen interaction method for a VoLTE call on a mobile device, comprising the following steps:

[0067] S1. Initiate a call request with the recipient and collect operation information of the user page;

[0068] S2. In response to the call request, establish a call connection;

[0069] S3. After the call connection is established, the operation information is received and the interactive logic processing is performed;

[0070] S4. According to the operation information, simulate the user operation, load the corresponding web page content, perform page rendering and layout, and take a screenshot of the current page to generate a picture stream;

[0071] S5, converting the text information to be broadcasted in the webpage content into voice to generate an audio stream;

[0072] S6, converting the picture stream into a video stream, compressing and encoding the video stream and the audio stream into a digital format, and then merging them into a multimedia stream;

[0073] S7: Send the multimedia stream to the call recipient, decode the multimedia stream, and restore the displayed user page.

[0074] Corresponding to the above method, the present invention also provides a multi-modal touch screen interaction device for VoLTE calls on a mobile device, comprising: a processor and a memory, the memory storing a computer program executable by the processor, and the above method is implemented when the processor executes the computer program.

[0075] In the present invention, the control center is responsible for linking the video encoding module, the cloud browser module, and the TTS service module, passing the user's input to the cloud browser for cloud rendering, and taking screenshots at a certain frame rate (such as 25fps), and encoding the captured image stream into an H264 video stream through the video encoding module for video conversation. In the VoLTE communication process, the user clicks events and coordinates, the cloud browser simulates clicks, the cloud browser renders in the cloud, takes screenshots to generate image streams, encodes to generate video streams, and presents the screen on the mobile phone. This is a sophisticated and complex process, which is specifically divided into the following steps:

[0076] Sa1. When a user initiates a VoLTE call on a mobile phone, the mobile application will capture the user's click events and coordinate information. This information includes the location where the user clicked on the screen and the coordinates of the user's finger moving on the screen. These events and coordinates will be passed to the application for corresponding interactive operations during the call.

[0077] Sa2. Cloud browser simulates user click events. Cloud browser is a browser running on a remote server that can remotely control and simulate user operations. After receiving the user's click event and coordinate information, the cloud browser simulates the user's click action, just as if the user operated it directly on the local device. This step allows interactive operations on the remote server instead of relying on the processing power of the local device.

[0078] Sa3. Cloud browsers render in the cloud. In the cloud, cloud browsers load the corresponding web page content and render and layout the page based on the user's operations and click events. This process involves parsing and executing the HTML, CSS, and JavaScript code of the web page, as well as rendering and displaying the page elements. Cloud browsers render pages in the cloud instead of on local devices, which can achieve UI interactive effects.

[0079] Sa4. The cloud browser will take a screenshot and generate an image stream. After the page rendering is completed, the cloud browser will take a screenshot of the current page and convert the page content into a static image stream. This image stream contains all the elements and content on the page, which will be used for subsequent video stream encoding.

[0080] Sa5. The encoder converts the image stream into a video stream. Each frame in the image stream is encoded as a video frame, compressed and encoded using a video encoding algorithm (such as H.264). This process converts the image into a video ready for transmission in the call. The encoding of the video stream is intended to reduce the amount of data in order to save bandwidth and reduce latency when transmitting on the network.

[0081] Sa6. The encoded video stream will be sent to the mobile phone and displayed on the mobile phone screen. After receiving the encoded video stream, the VoLTE application on the mobile phone will decode and display it. The decoder will decode the video stream into image frames and restore it to the page content that the user can see on the mobile phone screen. This process allows users to see the web page content on the remote cloud browser in real time during the call, realizing real-time interaction and display of cloud pages.

[0082] Specifically, the present invention includes the process of initiating a session and the process of interactive experience, such as Figure 3 As shown,.

[0083] The process of initiating a session includes the user initiating a call request, SIP server, control center, cloud browser, video encoding server and mobile phone, covering the process of call establishment, media transmission and presentation. The specific steps are as follows:

[0084] Sb1. User initiates a call request:

[0085] The starting point of a call is that the user initiates a call request on the VoLTE mobile phone. The user can choose to dial a certain number or select a contact from the address book to make a call.

[0086] Sb2, SIP server connection control center:

[0087] The call request initiated by the user is transmitted to the SIP server through the LTE network. The SIP server is responsible for establishing and controlling the call. It connects to the control center, communicates with it, and initializes the call parameters.

[0088] Sb3, control the middle station to process the request:

[0089] After receiving the call request from the SIP server, the control center will initialize the call status according to the call parameters, assign a unique session identifier to the call, and determine the service type required for the call.

[0090] Sb4. Control the middle platform to request the entrance from the cloud browser:

[0091] When the control center learns that the call needs to interact with the cloud browser, it will send a request to the cloud browser to obtain the entry information for accessing the specified web page.

[0092] Sb5, open the homepage of the cloud browser:

[0093] After receiving the request from the control center, the cloud browser opens the specified homepage. This webpage is usually a URL containing an H5 (HTML5) page, which may contain rich multimedia elements and interactive components.

[0094] Sb6, interactive H5 page rendering home page:

[0095] The cloud browser loads and renders the specified H5 page and displays the page content in the browser window. This page may contain text, images, videos and other interactive elements, and users can perform various operations on the page.

[0096] Sb7. Cloud browser returns the image stream:

[0097] After the cloud browser successfully renders the homepage, it converts the page content into an image stream and encodes the visual information on the page into an image. These image streams will be sent back to the control center as part of the multimedia data.

[0098] Sb8, video encoding server processes image stream:

[0099] After receiving the image stream, the control center passes it to the video encoding server. The video encoding server is responsible for processing the image stream and compressing it into a video stream. In this process, each frame in the image stream will be encoded into a video frame, and video encoding algorithms such as H.264 will be used for compression and encoding to reduce the amount of data.

[0100] Sb9. Control the middle station to return the video stream to the SIP server:

[0101] The video stream generated by the video encoding server will be returned to the control center. This video stream contains the encoded image information, and the control center passes it to the SIP server.

[0102] Sb10, SIP server transmits the video interaction screen to the VoLTE mobile phone:

[0103] After receiving the video stream, the SIP server will pass it to the recipient's VoLTE mobile phone. The VoLTE application on the mobile phone receives the video stream, decodes and displays it. The decoder decodes the video stream into image frames and restores it to the page content that the user can see on the mobile phone screen.

[0104] Through this process, VoLTE video calls successfully achieve multimedia interaction between users. The entire process relies on the collaborative work of multiple components such as SIP server, control center, cloud browser, video encoding server, etc. Each link is an indispensable part of the process, ensuring the smoothness and quality of the call. This process of VoLTE video calls reflects the high complexity and integration of modern communication technology, providing users with high-quality, real-time multimedia communication experience.

[0105] The interactive experience process is that users conduct real-time audio and video interaction during a call. It includes user interactive operations, SIP servers, control centers, cloud browsers, video encoding servers, TTS servers, and various links on the mobile phone side, covering the entire process of call establishment, media transmission, and presentation. The specific steps are:

[0106] Sc1. User operation screen:

[0107] The starting point of a call is when the user performs screen operations on the VoLTE phone, such as tapping, sliding or other touch screen operations. These operations are usually used for interactive applications or pages, such as web browsing.

[0108] Sc2, SIP server signaling channel sends coordinates:

[0109] The user's screen operations are captured by the mobile phone and sent to the server through the SIP server's signaling channel. These operations usually include coordinate information to indicate the location where the user clicked or swiped.

[0110] Sc3, control the forwarding coordinates of the middle station:

[0111] After receiving the user's screen operation information, the SIP server passes it to the control center. The control center is the coordinator of the call, which analyzes the operation information and processes it as needed.

[0112] Sc4. Cloud browser simulates clicks:

[0113] When the control center learns that the screen operation needs to interact with the cloud browser, it will send a request to the cloud browser to simulate the user's click or slide operation. The cloud browser is a browser running on a remote server that can remotely control and simulate user operations.

[0114] Sc5, H5 interactive page rendering page:

[0115] The cloud browser loads and renders the specified H5 page according to the request of the control center. This page may contain text, images, videos and other multimedia elements, and the user's click operation will be responded to on this page.

[0116] Sc6, H5 interactive page image stream returns:

[0117] The cloud browser converts the rendered page content into image streams and sends them back to the control center. These image streams contain the visual content on the page, including the results of user operations.

[0118] Sc7. Video encoding server encodes video stream:

[0119] After receiving the image stream, the control center passes it to the video encoding server. The video encoding server is responsible for encoding the image stream into a video stream, usually using video encoding algorithms such as H.264 to compress and encode these images.

[0120] Sc8. Cloud browser calculation broadcast text:

[0121] The cloud browser is not only responsible for rendering the page, but also may calculate the text information to be broadcast based on the page content. This text information usually includes the text content on the page, such as news, articles or web pages.

[0122] Sc9, TTS server text to sound:

[0123] The calculated text information is then passed to the TTS server, which is responsible for converting the text into speech. It uses speech synthesis technology to convert text information into sound.

[0124] Sc10, control the middle platform merge video:

[0125] After receiving the encoded video stream and the audio stream generated by the TTS server, the control center merges them into a multimedia stream that contains video and audio information.

[0126] Sc11. Video stream returns to the VoLTE end:

[0127] The combined multimedia stream is returned to the SIP server, which transmits the stream to the recipient's VoLTE mobile phone through the LTE network. The VoLTE application on the mobile phone receives the multimedia stream, decodes it, and displays it.

[0128] Sc12, VoLTE mobile phone display control screen:

[0129] After the VoLTE mobile phone receives the multimedia stream, it decodes and presents the video and audio data. The user will see the operation results on the cloud browser page, including images, text and sound.

[0130] Through this process, VoLTE video calls enable multimedia interaction between users, allowing users to share the results of operations on the screen and communicate through text and voice messages. This process demonstrates the high complexity and integration of modern communication technology, providing users with a high-quality, real-time multimedia communication experience. In summary, this process integrates a variety of technologies and components, including SIP communication, cloud browser, video encoding, TTS synthesis, and multimedia presentation to achieve smooth and interactive VoLTE video calls.

[0131] The technical advantages of the multi-modal touch screen interaction solution for VoLTE calls on mobile devices of the present invention are:

[0132] It can break the input limitation. In addition to traditional voice and key interaction, it can support click, slide and other interactive inputs. The video rendering method proposed in this solution can convert the previous overall display of video files into UI-based interactive logic and interactive experience. It can realize the complete product interactive experience in Volte, support page function display, page jump and complete interactive product experience; it supports providing users with a real page operation experience.

[0133] Multimodal support enables call centers to better handle these diverse media. Providing multimodal interactions can enhance the user experience and help customers better understand and solve problems. This can increase customer satisfaction and loyalty. The use of video calling products in call centers helps improve user experience, problem solving efficiency and service quality, reduce costs, and meet the communication needs of different users. It is an indispensable part of modern call centers.

[0134] The commercial value of touch screen interaction in VoLTE (Voice over LTE) is reflected in the following aspects:

[0135] Improve user experience: Touch screen interaction provides a richer and more intuitive way of interaction for both parties on a call. Users can display, demonstrate and share information by touching the screen, thereby improving the efficiency and quality of the call. This intuitive way of interaction makes it easier for users to express their ideas and needs, thereby enhancing the practicality and pleasure of communication.

[0136] Improve customer support quality: Enterprises can provide higher-quality customer support services by enabling touch-screen interaction through VoLTE. Customers can demonstrate the problem, sales representatives can demonstrate product features in real time, and technical support can guide customers to solve the problem. This intuitive support method helps solve problems faster and improve customer satisfaction.

[0137] Promoting innovative business models: VoLTE’s touchscreen interaction brings innovative business opportunities to enterprises and service providers. Applications targeting specific fields can be developed, such as virtual reality, interactive advertising, and remote entertainment. These applications can provide personalized user experience and attract more users and advertisers.

[0138] Touchscreen interaction in VoLTE not only improves the call experience, but also brings innovation and business opportunities to enterprises and service providers. By providing a more intuitive interaction method, it can enhance user satisfaction, improve work efficiency, expand the market, and bring more business value to various industries and fields. This interaction method is expected to continue to develop in the future and give rise to new business models.

[0139] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A multi-modal touch screen interaction system for VoLTE calls on a mobile device, characterized by: include: User equipment side, used to support users to initiate VoLTE calls and collect operation information; Used to decode multimedia streams in digital format and restore and display user pages; SIP server is used to establish a call connection with the user's device. The control center is used to receive operation information after the call connection is established, and forward it to the cloud browser or video encoding server or TTS service module for interactive processing, and to merge the encoded video stream and audio stream into a multimedia stream and send it to the user device end; Cloud browser, used to simulate user operations according to operation information, load corresponding web page content, perform page rendering and layout, and take screenshots of the current page to generate image streams; The TTS service module is used to convert the text information to be broadcasted in the webpage content into speech and generate an audio stream; The video encoding server is used to convert the picture stream into a video stream, and compress and encode the video stream and audio stream into a digital format.

2. According to claim 1, a multi-modal touch screen interaction system for VoLTE calls on a mobile device is characterized by: The user equipment end has a VoLTE service, the mobile operator network where the user equipment end is located supports the VoLTE service, the user equipment end is installed with a call application that supports the VoLTE service, and the call application displays risky content to the user before providing the VoLTE service and obtains collection authority; The user equipment end is provided with a decoder for restoring the multimedia stream in digital format into a user page.

3. The multi-modal touch screen interaction system for VoLTE calls on a mobile device according to claim 2, characterized in that: The SIP server is also used to establish a video channel and a signaling channel with the user equipment end; The Sip server is a server that implements VoIP service, Volte service and real-time communication service, and is used to negotiate media parameters between the two parties of the call and transmit audio data, video data, control information and event information in real time; When establishing a call connection, the Sip server connects to the control center to request a call and initializes call parameters.

4. The multi-modal touch screen interaction system for VoLTE calls on a mobile device according to claim 3, characterized in that: The video channel is used to transmit audio data and video data in real time, and encrypt the transmission content; the signaling channel is used to transmit user operation information, and encrypt the transmission content, and the operation information includes operation events and coordinate information.

5. The multi-modal touch screen interaction system for VoLTE calls on a mobile device according to claim 4, characterized in that: The control center receives user instructions from the client module and passes them to other modules for processing; the control center is also responsible for processing the user's context information to generate an appropriate response; after receiving a call request from the SIP server, the control center initializes the call state according to the call parameters, assigns a unique session identifier to the call, and determines the service type required for the call; When the control center determines that the call needs to interact with the cloud browser, it sends a request to the browser to obtain the entry information for accessing the web page.

6. The multi-modal touch screen interaction system for VoLTE calls on a mobile device according to claim 5, characterized in that: After the video encoding module completes encoding of the video stream, it returns to the control center, which transmits it to the SIP server, and then the SIP server transmits it to the recipient's user device.

7. The multi-modal touch screen interaction system for VoLTE calls on a mobile device according to claim 6, characterized in that: The cloud browser module will load the corresponding web page content according to the user's operation information, and render and layout the page; the cloud browser is also linked to the interactive page module, which is the actual product business interface and is used to design and render the user page in the interaction; The cloud browser is also used to calculate the text information that needs to be broadcast according to the web page content, and forward it to the TTS server for text-to-speech conversion to obtain an audio stream.

8. A multi-modal touch screen interaction method for a VoLTE call on a mobile device, characterized by: The steps include: Initiate a call request with the recipient and collect the operation information of the user page; In response to the call request, establishing a call connection; After the call connection is established, the operation information is received and the interactive logic processing is performed; According to the operation information, simulate the user operation, load the corresponding web page content, perform page rendering and layout, take a screenshot of the current page, and generate an image stream; Convert the text information to be broadcasted in the webpage content into voice and generate an audio stream; Convert the picture stream into a video stream, compress and encode the video stream and audio stream into a digital format, and then merge them into a multimedia stream; Send the multimedia stream to the call recipient, decode the multimedia stream, and restore the user page.

9. A multi-modal touch screen interaction device for VoLTE calls on a mobile device, characterized by: include: A processor and a memory, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

Images