Methods, devices, equipment, and storage media for implementing XR animations during phone calls.

Abstract

This application provides a method, apparatus, device, and storage medium for implementing XR animation effects during a call, relating to the field of communication technology. It allows XR animation effects to be overlaid during VoLTE calls, improving the user experience. The method includes: obtaining a media resource stream of a VoLTE-based call transmitted from a first terminal to a second terminal from a core network element; further determining an XR animation effect request, which indicates the XR animation effect to be requested; requesting a corresponding target XR animation effect from an animation effect resource library based on the XR animation effect request; overlaying the target XR animation effect onto the media resource stream to obtain a composite media resource stream; and finally sending the composite media resource stream to the second terminal. This application can be used during VoLTE-based calls to solve the problem of not being able to implement XR animation effect interaction during calls.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a method, apparatus, device and storage medium for implementing XR motion effects during a call. Background Technology

[0002] With the rapid development of mobile communication network technology and the improvement of mobile terminal device performance, more and more mobile terminal devices are entering daily life and work. Users can use mobile terminal devices to complete various audio and video calls, and during these calls, operators can leverage Voice over Long Term Evolution (VoLTE) technology to provide users with a wider range of value-added services on top of basic multimedia calling services.

[0003] However, in existing technologies, the value-added services provided by operators are typically applied to the early media stage before audio and video calls. For the media stage during audio and video calls, there are currently no related value-added services to provide users with entertainment and interactive experiences. Therefore, the lack of engagement during audio and video calls leads to a poor user experience and low satisfaction. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for implementing extended reality (XR) animation effects during a call, which can overlay XR animation effects during a VoLTE-based call to improve the user experience.

[0005] In a first aspect, this application provides a method for implementing XR motion effects during a call, applied to an implementation device. The implementation device is connected to a core network element and to a motion effect resource library. The method includes: obtaining a media resource stream of a VoLTE-based call transmitted from a first terminal to a second terminal from a core network element; determining an XR motion effect request; the XR motion effect request is used to indicate the XR motion effect to be requested; requesting a corresponding target XR motion effect from the motion effect resource library according to the XR motion effect request; superimposing the target XR motion effect onto the media resource stream to obtain a composite media resource stream; and sending the composite media resource stream to the second terminal.

[0006] This application provides a method for implementing XR animation effects during a call. It obtains the media resource stream of a VoLTE-based call transmitted from a first terminal to a second terminal through core network elements, further determines the XR animation effect request, and then requests the corresponding target XR animation effect from the animation effect resource library based on the request. The target XR animation effect is then superimposed onto the media resource stream to obtain a composite media resource stream, which is finally sent to the second terminal. This application enables the superimposition of XR animation effects during VoLTE-based calls, overcoming the deficiency of existing technologies that lack value-added services during calls. It allows users to interact with XR animation effects during VoLTE calls, meeting diverse user needs and enhancing the fun and interactive experience of the call.

[0007] One possible implementation is that the XR motion effect application includes an instruction identifier from the first terminal and configuration information from the second terminal. Applying for the corresponding target XR motion effect from the motion effect resource library according to the XR motion effect application includes: querying the motion effect corresponding to the instruction identifier from the motion effect resource library according to the instruction identifier; and adjusting the quality of the XR motion effect according to the configuration information to obtain the target XR motion effect.

[0008] Another possible implementation is that the target XR motion effect includes: visual motion effect and / or audio motion effect; adjusting the quality of the XR motion effect according to the configuration information includes: if the target XR motion effect includes visual motion effect, adjusting the visual details of the XR motion effect according to the configuration information; the visual details are used to reflect the clarity of the visuals; if the target XR motion effect includes audio motion effect, adjusting the audio details of the XR motion effect according to the configuration information; the audio details are used to reflect the fidelity of the audio.

[0009] Another possible implementation involves overlaying the target XR motion effect onto the media resource stream to obtain a composite media resource stream, when the target XR motion effect includes both visual and audio motion effects. This includes: determining whether audio data of the user speaking exists in the media resource stream; if so, overlaying the visual motion effect onto the media resource stream to obtain a composite media resource stream; if not, overlaying both the visual and audio motion effects onto the media resource stream to obtain a composite media resource stream.

[0010] Another possible implementation involves overlaying the video animation onto the media resource stream, including: decoding the video animation and the media resource stream; aligning the time and pixel positions of the video animation and the media resource stream; and overlaying the pixels of the video animation and the media resource stream.

[0011] Another possible implementation is that, if the first terminal is the calling terminal, the second terminal is the called terminal; and if the first terminal is the called terminal, the second terminal is the calling terminal.

[0012] Another possible implementation method includes, before obtaining the media resource stream of the VoLTE-based call transmitted from the first terminal to the second terminal from the core network element, querying whether the first terminal and / or the second terminal support XR motion effects.

[0013] Secondly, this application provides an apparatus for implementing XR motion effects during a call. The apparatus includes: an acquisition module, a determination module, an application module, a synthesis module, and a transmission module. The acquisition module is used to acquire media resource streams of a VoLTE-based call transmitted from a first terminal to a second terminal from core network elements. The determination module is used to determine an XR motion effect application; the XR motion effect application indicates the XR motion effect to be applied for. The application module is used to apply for a corresponding target XR motion effect from a motion effect resource library based on the XR motion effect application. The synthesis module is used to superimpose the target XR motion effect onto the media resource stream to obtain a synthesized media resource stream. The transmission module is used to send the synthesized media resource stream to the second terminal.

[0014] One possible implementation is that the XR motion effect application includes an instruction identifier from the first terminal and configuration information from the second terminal. The application module is specifically used to: query the XR motion effect corresponding to the instruction identifier from the motion effect resource library according to the instruction identifier; and adjust the quality of the XR motion effect according to the configuration information to obtain the target XR motion effect.

[0015] Another possible implementation is that the target XR motion effect includes: visual motion effect and / or audio motion effect; the application module is specifically used to adjust the visual details of the XR motion effect according to the configuration information when the target XR motion effect includes visual motion effect; the visual details are used to reflect the clarity of the visual effect; when the target XR motion effect includes audio motion effect, the audio details of the XR motion effect are adjusted according to the configuration information; the audio details are used to reflect the fidelity of the audio.

[0016] Another possible implementation is that, when the target XR motion effect includes visual motion effects and audio motion effects, the compositing module is specifically used to determine whether there is audio data of the user speaking in the media resource stream; if so, the visual motion effect is superimposed on the media resource stream to obtain a composite media resource stream; if not, the visual motion effect and audio motion effect are superimposed on the media resource stream to obtain a composite media resource stream.

[0017] Another possible implementation is that the compositing module is specifically used to decode the visual effects and media resource streams; align the time and pixel positions of the visual effects and media resource streams; and overlay the pixels of the visual effects and media resource streams.

[0018] Another possible implementation is that, if the first terminal is the calling terminal, the second terminal is the called terminal; and if the first terminal is the called terminal, the second terminal is the calling terminal.

[0019] Another possible implementation includes a query module; the query module is used to query whether the first terminal and / or the second terminal support XR motion effects.

[0020] Thirdly, this application provides an electronic device comprising: a processor and a memory; the memory storing processor-executable instructions; when the processor is configured to execute the instructions, causing the electronic device to implement the method of the first aspect described above.

[0021] Fourthly, this application provides a computer-readable storage medium comprising: computer software instructions; which, when executed in an electronic device, cause the electronic device to implement the method described in the first aspect.

[0022] Fifthly, this application provides a computer program product that, when run on a computer, causes the computer to perform the steps of the relevant method described in the first aspect above, so as to implement the method of the first aspect above.

[0023] The beneficial effects of the second to fifth aspects mentioned above are described in the corresponding description of the first aspect and will not be repeated here. Attached Figure Description

[0024] Figure 1 A schematic diagram illustrating the application environment of the method for implementing XR animation effects during a call provided in this application;

[0025] Figure 2 An architectural diagram of a device for implementing XR motion effects during a call, provided in this application;

[0026] Figure 3 A flowchart illustrating a method for implementing XR animation effects during a phone call, as provided in this application;

[0027] Figure 4 A flowchart illustrating another method for implementing XR animation effects during a call, as provided in this application;

[0028] Figure 5 A flowchart illustrating another method for implementing XR animation effects during a call, as provided in this application;

[0029] Figure 6 A flowchart illustrating another method for implementing XR animation effects during a call, as provided in this application;

[0030] Figure 7 A flowchart illustrating another method for implementing XR animation effects during a call, as provided in this application;

[0031] Figure 8 A flowchart illustrating another method for implementing XR animation effects during a call, as provided in this application;

[0032] Figure 9 A schematic diagram of an application environment architecture is provided for this application;

[0033] Figure 10 A flowchart illustrating another method for implementing XR animation effects during a call, as provided in this application;

[0034] Figure 11 A flowchart illustrating another method for implementing XR animation effects during a call, as provided in this application;

[0035] Figure 12 A schematic diagram of the components of a device for implementing XR motion effects during a call, as provided in this application;

[0036] Figure 13 This is a schematic diagram of the composition of an electronic device provided in this application. Detailed Implementation

[0037] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] It should be noted that in the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0039] To facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish the same or similar items with essentially the same function and effect. Those skilled in the art can understand that the terms "first" and "second" are not intended to limit the quantity or execution order.

[0040] XR refers to the combination of real and virtual elements through computers to create an interactive virtual environment. It's a collective term for various technologies such as Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR). By integrating the visual interaction technologies of these three technologies, it provides users with a seamless "immersive" experience between the virtual and real worlds. With the rapid development of mobile communication network technology, XR technology is now widely used in over-the-top (OTT) communications and live internet broadcasting, attracting a large number of users through engaging interactivity. However, mobile phone calls, as the most important basic communication service, are limited by media technology and do not offer XR services during calls.

[0041] Currently, operators utilize VoLTE technology to provide users with basic multimedia calling services and a variety of value-added services during mobile calls. However, these value-added services are typically applied in the early media phase before the call, and there are currently no value-added services providing XR animation services for the media phase during the call. The basic multimedia call process provided by operators is as follows: the called party has activated the video ringback tone service and supports color vibration service. When the calling user calls the called party, the called party's ringback tone service platform plays a video media to both the calling and called parties before the call is connected, replacing the ordinary audio ringback tone service. After the call is connected, the video ringback tone playback stops, and both parties enter the call state. However, there are no value-added services that can provide users with entertainment and interaction during the call. Therefore, during VoLTE-based calls, the multimedia audio and video services currently provided by operators can no longer meet user needs, resulting in a poor user experience and low satisfaction.

[0042] In summary, there is an urgent need for a way to enable entertainment interaction between users during calls. Based on this, this application provides a method for implementing XR animation effects during calls. This method involves obtaining the media resource stream of a VoLTE-based call transmitted from a first terminal to a second terminal via a core network element, further determining an XR animation effect request, and then requesting a corresponding target XR animation effect from the animation effect resource library based on the request. The target XR animation effect is then superimposed onto the media resource stream to obtain a composite media resource stream, which is finally sent to the second terminal. This application enables the superimposition of XR animation effects during VoLTE-based calls, overcoming the deficiency of existing technologies that lack value-added services during calls. This allows users to interact using XR animation effects during VoLTE calls, meeting diverse user needs and enhancing the fun and interactive experience of the call.

[0043] The method for implementing XR animations during a call provided in this application can be applied to, for example... Figure 1 The application environment shown. For example... Figure 1 As shown, the application environment includes: an XR motion effect implementation device 101 (which can be simply referred to as the implementation device, or also as an XR call platform application server (AS)), a core network element 102, a first terminal 103, and a second terminal 104. The implementation device 101 is interconnected with the core network element 102, the first terminal 103 is interconnected with the core network element 102, and the second terminal 104 is interconnected with the core network element 102.

[0044] In some embodiments, the implementation device 101 may be a server cluster consisting of multiple servers, a single server, a computer, or a processor or processing chip in a server or computer, etc. This application does not limit the specific device form of the implementation device 101. Figure 1 The implementation device 101 is shown as a single server as an example.

[0045] In some embodiments, the core network element 102 may be a server cluster consisting of multiple servers, a single server, a computer, or a processor or processing chip in a server or computer, etc. This application does not limit the specific device form of the core network element 102. Figure 1 The example shown is a core network element 102, which is a single server.

[0046] In some embodiments, the first terminal 103 and the second terminal 104 described above can be devices with wireless transceiver capabilities, such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. The embodiments of the present invention do not limit the specific types of the first terminal 103 and the second terminal 104. Figure 1 The example shown is a smartphone terminal.

[0047] In some embodiments, the VoLTE-based call process between the first terminal and the second terminal both pass through the core network element. That is, the media resource stream of the call between the first terminal and the second terminal is sent to each other through the core network element. When the first terminal and the second terminal start a call, the implementing device can use signaling to obtain the media resource stream of the call between the first terminal and the second terminal from the core network element, further process the obtained media resource stream to obtain a composite media resource stream, and finally send the composite media resource stream to the second terminal.

[0048] Figure 2 This is an architectural diagram of the device for implementing XR animation effects during a call, provided in an embodiment of this application. Figure 2 As shown, the implementation device may include: an interaction module 201 and an XR motion synthesis module 202.

[0049] In some embodiments, the interaction module 201 is used to handle call signaling monitoring and media interaction with the core network element, that is, to interact with the core network element, acquire media resource streams, and send synthesized media resource streams. The XR motion effect synthesis module 202 is used to overlay XR motion effects onto the acquired media resource stream. In addition, it enables the connection between the device and the core network element, allowing it to receive user calls from the first terminal and / or the second terminal through the core network element.

[0050] Figure 3 This is a flowchart illustrating a method for implementing XR animation effects during a phone call, as provided in an embodiment of this application. Figure 3 As shown, the method for implementing XR animation during a call provided in this application can be implemented using the aforementioned device, and specifically includes the following steps:

[0051] S301, The device obtains the media resource stream of the VoLTE-based call transmitted from the first terminal to the second terminal from the core network element.

[0052] The media resource stream can be a video stream, an audio stream, or an audio-visual stream. As an example, the media resource stream is a real-time transport protocol (RTP) video stream.

[0053] In some embodiments, the implementation device is connected to a core network element. When the first terminal and the second terminal make a call through the core network element, the implementation device can obtain the media resource stream of the VoLTE-based call transmitted from the first terminal to the second terminal from the core network element.

[0054] For example, the implementation device can detect the call between the first terminal and the second terminal through the core network element. When the call between the first terminal and the second terminal is detected, the interaction module of the implementation device can establish a communication channel with the core network element. The media resource stream between the first terminal and the second terminal can flow through the implementation device so that the implementation device can process the media resource stream subsequently.

[0055] S302, Implement the device to determine the XR motion effect application.

[0056] Among them, XR motion effect application is used to indicate the XR motion effect to be applied for.

[0057] In some embodiments, after acquiring the media resource stream transmitted from the first terminal to the second terminal, the implementing device may further determine an XR motion effect request and apply for the corresponding XR motion effect to process the media resource stream.

[0058] For example, after acquiring the media resource stream, the implementation device can also obtain the instruction identifier of the first terminal and the configuration information of the second terminal through the core network element, and then determine the XR motion effect application based on the instruction identifier of the first terminal and the configuration information of the second terminal.

[0059] S303. The implementation device requests the corresponding target XR motion effect from the motion effect resource library according to the XR motion effect request.

[0060] In some embodiments, after determining an XR motion effect request, the implementing device can request the corresponding target XR motion effect from the motion effect resource library according to the XR motion effect request.

[0061] For example, the motion effect resource library and the implementation device are interconnected. The motion effect resource library stores multiple XR motion effects of different styles and types. The implementation device can send the XR motion effect request to the motion effect resource library, receive the XR motion effect in response from the motion effect resource library, and further process the XR motion effect to obtain the target XR motion effect.

[0062] S304. The device superimposes the target XR motion effect onto the media resource stream to obtain a composite media resource stream.

[0063] In some embodiments, after the implementing device obtains the corresponding target XR motion effect, the XR motion effect synthesis module in the implementing device can superimpose the target XR motion effect onto the media resource stream obtained above to obtain a synthesized media resource stream.

[0064] S305, The device sends the synthesized media resource stream to the second terminal.

[0065] In some embodiments, after the implementing device obtains the synthetic media resource stream, it can send the synthetic media resource stream to the second terminal through the core network element.

[0066] It should be understood that when the first terminal is the calling terminal, the second terminal is the called terminal; conversely, when the first terminal is the called terminal, the second terminal is the calling terminal. That is, the device can overlay XR effects onto the media resource stream transmitted from the calling terminal to the called terminal, and vice versa. Furthermore, the device can also process the media resource stream transmitted by either party during the interaction between the calling and called terminals and then send it to the other terminal for display, thereby enabling XR-based interaction between the two parties.

[0067] The technical solutions provided by the above embodiments bring at least the following beneficial effects. The method for implementing XR motion effects during a call provided in this application embodiment obtains the media resource stream of a VoLTE-based call transmitted from the first terminal to the second terminal through core network elements, further determines the XR motion effect request, and then requests the corresponding target XR motion effect from the motion effect resource library according to the XR motion effect request. The target XR motion effect is then superimposed onto the media resource stream to obtain a composite media resource stream, and finally the composite media resource stream is sent to the second terminal. This application embodiment realizes the superimposition of XR motion effects during a VoLTE-based call, making up for the deficiency of existing technologies in the absence of value-added services during calls. This allows users to interact using XR motion effects during VoLTE calls, meeting diverse user needs and improving the fun of the call process and the user's interactive experience.

[0068] The following describes in detail the method for implementing XR animation during a call provided in this application, with reference to specific embodiments and accompanying drawings.

[0069] like Figure 4 As shown, the method for implementing XR animation during a call provided in this application may specifically include the following steps:

[0070] S401, The device queries whether the first terminal and / or the second terminal support XR motion effects.

[0071] In some embodiments, before the first terminal and the second terminal make a call, the implementing device may query whether the first terminal and / or the second terminal support XR animation function, so as to determine whether the terminal configuration of the first terminal and / or the second terminal can support the display of XR animation.

[0072] For example, the implementing device can first query whether the first terminal and / or the second terminal have activated the XR animation service. If either the first terminal or the second terminal has activated the XR animation service, the implementing device can provide the corresponding XR animation service for the call process between the first terminal and the second terminal; otherwise, the implementing device cannot provide the corresponding service. Furthermore, the implementing device can also query the core network whether the first terminal and / or the second terminal support the XR animation function to avoid the situation where the XR animation cannot be displayed or is displayed incorrectly due to the terminal configuration of the first terminal and / or the second terminal not supporting XR animation.

[0073] S402, The device obtains the media resource stream of the VoLTE-based call transmitted from the first terminal to the second terminal from the core network element.

[0074] S403, Implement the device to determine the XR motion effect application.

[0075] Among them, XR motion effect application is used to indicate the XR motion effect to be applied for.

[0076] The relevant descriptions of S402-S403 above can be found in the descriptions of S301-S302 above, and will not be repeated here.

[0077] S404. The implementation device requests the corresponding target XR motion effect from the motion effect resource library according to the XR motion effect request.

[0078] In some embodiments, the XR motion effect application includes an instruction identifier from a first terminal and configuration information from a second terminal. The implementing device can apply for a corresponding target XR motion effect from the motion effect resource library based on the instruction identifier from the first terminal and the configuration information from the second terminal. Specifically, such as... Figure 5 As shown, S404 can be specifically implemented as S4041-S4042.

[0079] S4041. The implementation device queries the animation resource library for the XR animation corresponding to the instruction identifier based on the instruction identifier.

[0080] In some embodiments, the implementing device can obtain the instruction identifier of the first terminal through a core network element, process the instruction identifier to obtain the corresponding keyword, and then query the corresponding XR animation in the animation resource library based on the keyword. The instruction identifier represents a user operation of the first terminal, which includes, but is not limited to, button operations and search box input operations.

[0081] For example, multiple different buttons on the first terminal correspond to multiple different commonly used XR animations. The XR animation for each button can be set by the administrator according to actual needs; this application does not impose any restrictions on this. The implementing device can obtain the button operations of the first terminal user through core network elements, and then request the corresponding XR animation from the animation resource library based on the correspondence between the buttons and XR animations. The implementing device can also set an XR animation search box on the call interface of the first terminal, where the user can enter keywords. The implementing device can then request the corresponding XR animation from the animation resource library based on the user-entered keywords.

[0082] S4042. The implementation device adjusts the quality of the XR motion effect according to the configuration information to obtain the target XR motion effect.

[0083] In some embodiments, the implementing device obtains configuration information of a second terminal and adjusts the quality of the XR motion effect described above based on the configuration information to obtain a target XR motion effect. The target XR motion effect includes: visual motion effects and / or audio motion effects. Specifically, such as... Figure 6 As shown, S4042 can be specifically implemented as S4042a-S4042b.

[0084] S4042a. When the target XR motion effect includes screen motion effect, the device adjusts the screen details of the XR motion effect according to the configuration information.

[0085] Among them, image details are used to reflect the clarity of the image.

[0086] In some embodiments, when the target XR motion effect includes screen motion effects, the implementing device can adjust the screen details of the XR motion effect according to the configuration information of the second terminal. The screen details are used to reflect the clarity of the image.

[0087] For example, the implementation device can obtain the terminal configuration information pre-set by the second terminal user from the core network element, thereby determining the level of detail of the XR motion effect corresponding to the terminal configuration information. If the terminal configuration is higher, the XR motion effect corresponding to the terminal configuration will be clearer; if the terminal configuration is lower, the XR motion effect corresponding to the terminal configuration will be blurrier. The image details include, but are not limited to: image resolution, image bitrate, image encoding type, image frame rate, image pixels, and image size.

[0088] S4042b: When the target XR motion effect includes audio motion effects, the implementing device adjusts the audio details of the XR motion effect according to the configuration information.

[0089] Audio details are used to reflect the fidelity of the audio.

[0090] In some embodiments, where the target XR motion effect includes audio motion effects, the implementing device can adjust the audio details of the XR motion effect according to the configuration information of the second terminal. The audio details reflect the fidelity of the audio.

[0091] For example, the implementation device can obtain the terminal configuration information pre-set by the second terminal user from the core network element, thereby determining the audio detail level of the XR motion effect corresponding to the terminal configuration information. If the terminal configuration is higher, the audio fidelity of the XR motion effect corresponding to the terminal configuration is higher; if the terminal configuration is lower, the audio fidelity of the XR motion effect corresponding to the terminal configuration is lower. The audio details include, but are not limited to: audio distortion, audio frequency response, audio signal-to-noise ratio, audio dynamic range, audio transient response, audio stereo separation, and audio stereo balance.

[0092] S405. The device superimposes the target XR motion effect onto the media resource stream to obtain a composite media resource stream.

[0093] In some embodiments, the implementing device superimposes the obtained target XR motion effect onto the acquired media resource stream to obtain a composite media resource stream. Specifically, when the target XR motion effect includes both visual and audio motion effects, such as... Figure 7 As shown, S405 can be specifically implemented as S4051-S4053.

[0094] S4051, The implementation device determines whether there is audio data of a user speaking in the media resource stream.

[0095] In some embodiments, after the implementing device obtains the target XR motion effect, it can determine whether there is audio data of the user speaking in the acquired media resource stream, and then further determine whether to overlay the audio motion effect in the target XR motion effect onto the media resource stream.

[0096] For example, the implementation device can perform face recognition on user images in the acquired media video stream. Starting from the beginning frame of the media resource stream, it checks frame by frame whether there is a video frame sequence in the media resource stream that meets the user speech recognition conditions. It further determines whether there is audio data of user speech in the media resource stream, thereby determining whether to overlay the audio motion effect of the target XR motion effect onto the media resource stream. The user speech recognition condition is that multiple consecutive frames contain feature data corresponding to a preset set of facial mouth feature points. In addition, the specific feature points included in the set of facial mouth feature points can be set by the administrator according to the actual situation, and this application embodiment does not limit this.

[0097] S4052. If so, the device overlays the visual effects onto the media resource stream to obtain a composite media resource stream.

[0098] In some embodiments, after determining that audio data of user speech exists in the media resource stream, the implementing device can overlay the visual effects from the XR motion effects onto the media resource stream to obtain a composite media resource stream. Specifically, such as Figure 8 As shown, S4052 can be specifically implemented as S4052a-S4052c.

[0099] S4052a, The device decodes the screen animation and media resource stream.

[0100] In some embodiments, after determining that the visual effects in the XR motion effect are superimposed onto the media resource stream, the implementing device decapsulates and decodes the visual effects and the media resource stream to obtain the decoded visual effects and the media resource stream. The specific method by which the implementing device decodes the visual effects and the media resource stream can employ relevant decoding methods in the field of audio and video codec technology; only the visual effects and the media resource stream need to be decoded, and this application embodiment does not limit this approach.

[0101] S4052b, The device aligns the time and pixel position of the screen animation and media resource stream.

[0102] In some embodiments, XR motion effects can be overlaid at any position in the media resource stream, or at a specific position, such as: heart gestures, wearing glasses / hats, virtual backgrounds, fireworks, rabbit ears, etc. Therefore, after obtaining the decoded motion effects and the media resource stream, the implementing device can determine the overlay position of the motion effects based on the position of the user's image in the media resource stream. In addition, the implementing device can also determine the overlay time of the motion effects based on the time when the first terminal user triggers the XR motion effects, thereby aligning the time and pixel position of the decoded motion effects and the media resource stream according to the overlay position and overlay time.

[0103] S4052c, The device overlays pixels of visual effects and media resource streams.

[0104] In some embodiments, after aligning the time and pixel positions of the screen animation and the media resource stream, the implementing device can superimpose the pixels at corresponding positions of the screen animation and the media resource stream based on the image overlay algorithm in the prior art to obtain a composite media resource stream.

[0105] S4053. If not, the device overlays the visual and audio effects onto the media resource stream to obtain a composite media resource stream.

[0106] In some embodiments, after determining that there is no audio data of user speech in the media resource stream, the implementing device can overlay the visual and audio effects in the XR motion effects onto the media resource stream to obtain a composite media resource stream.

[0107] For example, the implementation device can superimpose the visual effects of the XR motion effects onto the media resource stream based on the methods described in S4052a-S4052c. Then, the implementation device processes the audio effects in the XR motion effects and the audio data in the media resource stream to ensure that the sampling rate, number of channels, and sampling precision of the audio effects and the audio data in the media resource stream are consistent. Finally, the implementation device superimposes the processed audio effects onto the media resource stream based on an audio mixing algorithm. The audio mixing algorithm can be any of the following algorithms: linear superposition algorithm, average weighting algorithm, normalization algorithm, adaptive weighted mixing algorithm, and automatic alignment algorithm, etc.

[0108] S406, The device sends the synthesized media resource stream to the second terminal.

[0109] The relevant description of S406 above can be found in the description of S305 above, and will not be repeated here.

[0110] It should be understood that during a call between a user on the first terminal and a user on the second terminal, the user on the first terminal can initiate one or more different XR motion effect requests. This means the user can trigger button operations on the terminal device multiple times or enter keywords in the search box multiple times. The implementing device can obtain a synthetic media resource stream for each motion effect request based on the aforementioned method. The implementing device can then send the latest synthetic media resource stream to the second terminal to replace the synthetic media resource stream corresponding to the previous motion effect request, thereby enabling the user to switch between different XR motion effects. This improves the user's interactive experience and entertainment effect during the call, and can attract users to subscribe to the XR motion effect service of this application. Furthermore, the XR motion effect implementation method of this application can also be applied to calls based on Voice over New Radio (VONR) technology.

[0111] Specifically, as an example, the effect of the first terminal user and the second terminal user triggering the XR motion effect operation twice in a row is shown in Table 1.

[0112] Table 1

[0113]

[0114] As shown in Table 1, 'a' represents the XR animation corresponding to the combined operation of the first and second terminal users; 'b' represents the XR animation corresponding to the operation of the first terminal user; and 'c' represents the XR animation corresponding to the operation of the second terminal user. Users of both the first and second terminals can select the display direction of the XR animation before applying for it. The display directions include: displaying the XR animation to the other party, displaying the XR animation to themselves, and displaying the XR animation to both parties.

[0115] When the first terminal and the second terminal request an XR effect only once, if the first terminal user chooses to show the XR effect to the other terminal user and the second terminal user chooses to show the XR effect to themselves, then the first terminal will not display any XR effect, and the second terminal will display an overlay of the XR effects corresponding to the actions of the first and second terminal users. If the first terminal user chooses to show the XR effect to themselves and the second terminal chooses to show the XR effect to the other terminal user, then the first terminal will display an overlay of the XR effects corresponding to the actions of the first and second terminal users, and the second terminal will not display any XR effect. If both the first and second terminal users choose to show the XR effect bidirectionally, then both the first and second terminals will display an overlay of the XR effects corresponding to the actions of the first and second terminal users.

[0116] When a first terminal and a second terminal make two consecutive XR animation request operations, if the first terminal user chooses to display the XR animation to the other terminal in both instances, and the second terminal user chooses to display the XR animation to the other terminal in the first instance and to display it bidirectionally in the second instance, then the first terminal displays the XR animation corresponding to the second terminal user's operation in both instances. The second terminal first displays the XR animation corresponding to the first terminal user's operation, and then displays an overlay of the XR animations corresponding to the operations of both the first and second terminals. If the first terminal user chooses to display the XR animation to themselves in both instances, and the second terminal user chooses to display it bidirectionally in the first instance and to display it to the other terminal in the second instance, then the first terminal displays an overlay of the XR animations corresponding to the operations of both the first and second terminals in both instances, and the second terminal only displays the XR animation corresponding to the first terminal user's operation. If the first terminal user chooses to display the XR animation bidirectionally in both instances, and the second terminal user chooses to display it to themselves in both instances, then the first terminal displays the XR animation corresponding to the first terminal user's operation in both instances, and the second terminal displays an overlay of the XR animations corresponding to the operations of both the first and second terminals in both instances. Each time a user makes an XR motion effect request, the device terminates the user's previous XR motion effect request and generates a composite media resource stream for display based on the latest XR motion effect request.

[0117] The technical solutions provided by the above embodiments bring at least the following beneficial effects. The method for implementing XR motion effects during a call provided in this application embodiment obtains the media resource stream of a VoLTE-based call transmitted from the first terminal to the second terminal through core network elements, further determines the XR motion effect request, and then requests the corresponding target XR motion effect from the motion effect resource library according to the XR motion effect request. The target XR motion effect is then superimposed onto the media resource stream to obtain a composite media resource stream, and finally the composite media resource stream is sent to the second terminal. This application embodiment realizes the superimposition of XR motion effects during a VoLTE-based call, making up for the deficiency of existing technologies in the absence of value-added services during calls. This allows users to interact using XR motion effects during VoLTE calls, meeting diverse user needs and improving the fun of the call process and the user's interactive experience.

[0118] Furthermore, this application can also adjust the quality of XR motion effects based on the instruction identifier of the first terminal and the configuration information of the second terminal, so that the adjusted XR motion effects, when superimposed with the media resource stream, meet the configuration requirements of the second terminal and the user's needs. This avoids situations where the user terminal's performance is affected when playing XR motion effects due to the user terminal's low configuration, thereby improving the adaptability of XR motion effects and ensuring a good user experience.

[0119] Figure 9 This is an application environment architecture diagram of an embodiment of the present application, including: XR call platform AS901, Internet Protocol Multimedia Subsystem (IMS) core network 902, calling terminal 903 and called terminal 904.

[0120] The XR calling platform AS901 (i.e., the implementation device) is interconnected with the IMS core network 902 (i.e., the core network element), the calling terminal 903 (i.e., the first terminal) is interconnected with the IMS core network 902, and the called terminal 904 (i.e., the second terminal) is interconnected with the IMS core network 902. The VoLTE-based call process between the calling terminal 903 and the called terminal 904 passes through the IMS core network 902; that is, the media resource stream of the call between the calling terminal 903 and the called terminal 904 is sent to the other party through the IMS core network 902. When the calling terminal 903 and the called terminal 904 start a call, the XR calling platform AS901 can use signaling to obtain the media resource stream of the call between the calling terminal 903 and the called terminal 904 from the IMS core network 902, further process the obtained media resource stream to obtain a composite media resource stream, and finally send the video stream + XR stream (i.e., the composite media resource stream) to the called terminal 904.

[0121] The following describes a specific embodiment of the method for implementing XR animation effects during a call according to this application. The specific implementation process of this method is as follows: Figure 10 As shown.

[0122] When the calling terminal (i.e., the first terminal) makes a video call to the called terminal (i.e., the second terminal), the core network queries whether the calling user has activated the AR animation service and sends all call signaling to the XR call platform (i.e., the implementation device). The XR call platform receives the call and checks whether the user has XR effects functionality. If so, it bridges the audio and video media during the call, and all voice streams of the calling and called parties pass through the platform (the call process between the first terminal and the second terminal passes through the core network elements, and the implementation device connects to the core network). The calling and called users conduct a video call, during which the calling user performs key operations on the terminal (i.e., the command identifier of the first terminal), such as 5#, which represents fireworks (key 5# corresponds to the fireworks XR animation). The XR call platform recognizes the user's key press content, triggers the XR effects function (i.e., confirms the XR animation request, obtains the target XR animation from the animation resource library), bridges the audio and video stream (i.e., the media resource stream) of the called terminal with the effects module (the target XR animation), and pushes the generated XR effects (synthesized media resource stream) to the called party. During a video call between the caller and the recipient, the recipient can see the effects corresponding to the buttons pressed.

[0123] The following describes a specific embodiment of the method for implementing XR animation effects during a call according to this application. The specific implementation process of this method is as follows: Figure 11 As shown.

[0124] The calling terminal (i.e., the first terminal), the call session control function (CSCF) module of the IMS core network, the XR call platform AS (i.e., the implementation device), and the XR motion effect synthesis module constitute the calling domain in this call process. A. After the calling terminal and the called terminal (i.e., the second terminal) establish a video call, 1. The calling terminal user clicks the button corresponding to the preset XR motion effect to perform a button operation (i.e., instruction identifier). 1-1. The XR call platform AS can obtain the media resource stream of the call between the calling terminal and the called terminal and the button operation of the calling terminal user through the CSCF module of the IMS core network. 2. The XR call platform AS parses the button operation as a request for XR motion effect function, initiates an XR motion effect request, and forwards the request information to the XR motion effect synthesis module. 3. The XR motion effect synthesis module responds to the XR motion effect request, obtains the target XR motion effect from the motion effect resource library, 4. The XR call platform AS overlays the target XR motion effect onto the media resource stream to obtain a synthesized media resource stream, and sends the synthesized media resource stream to the called terminal. During a call between the calling and called terminals, the called terminal can see the XR animation corresponding to the buttons pressed by the calling terminal.

[0125] It should be noted that the above Figure 9 , Figure 10 and Figure 11 The descriptions are all based on the scenario where the calling terminal initiates an XR motion effect request during a call between the calling and called terminals. This application embodiment also includes two other scenarios: the called terminal initiates an XR motion effect request during the call, or both the calling and called terminals initiate XR motion effect requests simultaneously. The detailed descriptions of these two scenarios refer to the description of the calling terminal initiating the motion effect request, and will not be repeated here. Specifically, when both the calling and called terminals initiate motion effect requests simultaneously, the implementing device can process the motion effect requests from the calling and called terminals separately before sending them to the other terminal. Furthermore, the call between the calling and called terminals can be an audio / video call or an audio call. If it is an audio call, the XR motion effect overlay only needs to overlay the corresponding audio motion effect.

[0126] In an exemplary embodiment, this device may also provide an apparatus for implementing XR animation effects during a call. The implementation of XR animation effects during a call may include one or more functional modules for implementing the method for implementing XR animation effects during a call as described in the above method embodiments.

[0127] For example, Figure 12 This is a schematic diagram of a device for implementing XR animation effects during a phone call, provided as an embodiment of this application. Figure 12 As shown, the device for implementing XR motion effects during the call includes: an acquisition module 1201, a determination module 1202, an application module 1203, a synthesis module 1204, and a sending module 1205.

[0128] The acquisition module 1201 is used to acquire the media resource stream of a VoLTE-based call transmitted from the first terminal to the second terminal from the core network elements. The determination module 1202 is used to determine an XR effect request, which indicates the XR effect to be requested. The request module 1203 is used to request the corresponding target XR effect from the effect resource library according to the XR effect request. The synthesis module 1204 is used to overlay the target XR effect onto the media resource stream to obtain a synthesized media resource stream. The sending module 1205 is used to send the synthesized media resource stream to the second terminal.

[0129] In some embodiments, the XR motion effect application includes an instruction identifier from the first terminal and configuration information from the second terminal. The application module 1203 is specifically used to query the XR motion effect corresponding to the instruction identifier from the motion effect resource library according to the instruction identifier, and adjust the quality of the XR motion effect according to the configuration information to obtain the target XR motion effect.

[0130] In other embodiments, the target XR motion effect includes: visual motion effect and / or audio motion effect. The synthesis module 1204 is specifically used to, when the target XR motion effect includes visual motion effect, adjust the visual details of the XR motion effect according to configuration information, wherein the visual details are used to reflect the clarity of the visual effect; and when the target XR motion effect includes audio motion effect, adjust the audio details of the XR motion effect according to configuration information, wherein the audio details are used to reflect the fidelity of the audio.

[0131] In other embodiments, when the target XR motion effect includes visual motion effect and audio motion effect, the compositing module 1204 is specifically used to determine whether there is audio data of user speech in the media resource stream. If so, the visual motion effect is superimposed on the media resource stream to obtain a composite media resource stream. If not, the visual motion effect and audio motion effect are superimposed on the media resource stream to obtain a composite media resource stream.

[0132] In other embodiments, the compositing module 1204 is specifically used to decode the screen animation and media resource stream, align the time and pixel positions of the screen animation and media resource stream, and superimpose the pixels of the screen animation and media resource stream.

[0133] In other embodiments, when the first terminal is the calling terminal, the second terminal is the called terminal; when the first terminal is the called terminal, the second terminal is the calling terminal.

[0134] In some other embodiments, a query module 1206 is also included, which is used to query whether the first terminal and / or the second terminal support XR motion effects.

[0135] In an exemplary embodiment, this application also provides an electronic device, which can be a device for implementing XR motion effects during a call in the above method embodiments. Figure 13 This is a schematic diagram of the structure of the device for implementing XR motion effects during a call, as provided in an embodiment of this application. Figure 13 As shown, the device for implementing XR motion effects during the call may include: a processor 1301 and a memory 1302; the memory 1302 stores instructions executable by the processor 1301; when the processor 1301 is configured to execute instructions, it causes the electronic device, network device, or manager to implement the method described in the foregoing method embodiments.

[0136] In an exemplary embodiment, this application also provides a computer-readable storage medium storing computer program instructions thereon; when the computer program instructions are executed by a computer, the computer causes the computer to implement the method described in the foregoing embodiments. The computer-readable storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0137] In an exemplary embodiment, this application also provides a computer program product that, when run on a computer, causes the computer to execute the aforementioned related method steps to implement the method for achieving XR motion effects during a call in the above embodiments.

[0138] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for implementing augmented reality (XR) animation effects during a phone call, characterized in that, The method is applied to an implementation device, which is connected to a core network element and to a motion effects resource library; the method includes: obtaining a media resource stream of a call based on VoLTE technology transmitted from a first terminal to a second terminal from the core network element; Determine an XR motion effect request; the XR motion effect request is used to indicate the XR motion effect to be requested; the XR motion effect request includes an instruction identifier from the first terminal and configuration information from the second terminal; According to the instruction identifier, the XR animation corresponding to the instruction identifier is queried from the animation resource library; the XR animation is obtained from the animation resource library based on the keywords obtained by processing the instruction identifier; the instruction identifier is a user operation of the first terminal; the user operation includes key operation and search box input operation; The XR motion effects include: visual motion effects and / or audio motion effects; adjusting the quality of the XR motion effects according to the configuration information includes: When the XR animation includes the screen animation, the screen details of the XR animation are adjusted according to the configuration information; the screen details are used to reflect the clarity of the screen; the configuration information is the terminal configuration information pre-set by the user of the second terminal obtained from the core network element; whether the configuration information can support the display of XR animation is determined based on the XR animation function support status of the second terminal queried by the core network element; the higher the configuration of the configuration information, the clearer the screen of the XR animation; the lower the configuration of the configuration information, the blurrier the screen of the XR animation. When the XR motion effect includes the audio motion effect, the audio details of the XR motion effect are adjusted according to the configuration information; the audio details are used to reflect the fidelity of the audio; the higher the configuration information, the higher the audio fidelity of the XR motion effect; the lower the configuration information, the lower the audio fidelity of the XR motion effect. The XR motion effects are superimposed onto the media resource stream to obtain a composite media resource stream; The synthesized media resource stream is sent to the second terminal.

2. The method according to claim 1, characterized in that, When the XR motion effect includes both visual and audio motion effects, the step of overlaying the XR motion effect onto the media resource stream to obtain a composite media resource stream includes: Determine whether audio data of the user speaking exists in the media resource stream; If so, the video effects are overlaid onto the media resource stream to obtain the composite media resource stream; If not, the visual effects and audio effects are superimposed onto the media resource stream to obtain the composite media resource stream.

3. The method according to claim 2, characterized in that, The step of overlaying the video animation onto the media resource stream includes: Decode the aforementioned visual effects and the aforementioned media resource stream; Align the time and pixel position of the visual effects and the media resource stream; The video animation and the pixels of the media resource stream are superimposed.

4. The method according to any one of claims 1 to 3, characterized in that, If the first terminal is the calling terminal, the second terminal is the called terminal; If the first terminal is the called terminal, then the second terminal is the calling terminal.

5. The method according to claim 1, characterized in that, Before obtaining the media resource stream of the VoLTE-based call transmitted from the first terminal to the second terminal from the core network element, the method further includes: Check whether the first terminal and / or the second terminal support the XR animation function.

6. A device for implementing XR motion effects during a phone call, characterized in that, The device includes: an acquisition module, a determination module, an application module, a synthesis module, and a sending module; The acquisition module is used to acquire, from the core network elements, the media resource stream of the call based on VoLTE technology transmitted from the first terminal to the second terminal; The determining module is used to determine an XR motion effect application; the XR motion effect application is used to indicate the XR motion effect to be applied for; the XR motion effect application includes an instruction identifier from the first terminal and configuration information from the second terminal; The application module is specifically used to: query the animation resource library for XR animation corresponding to the instruction identifier based on the instruction identifier; the XR animation is obtained from the animation resource library based on keywords obtained by processing the instruction identifier; the instruction identifier is a user operation of the first terminal; the user operation includes key operation and search box input operation; The XR animation includes visual animation and / or audio animation. Specifically, the application module is used to adjust the visual details of the XR animation based on the configuration information when the XR animation includes visual animation. The visual details reflect the clarity of the visuals. The configuration information is the user-preset terminal configuration information of the second terminal obtained from the core network element. Whether the configuration information supports the display of XR animation is determined based on the XR animation function support status of the second terminal queried by the core network element. The higher the configuration of the configuration information, the clearer the visuals of the XR animation; the lower the configuration of the configuration information, the blurrier the visuals of the XR animation. When the XR animation includes audio animation, the module adjusts the audio details of the XR animation based on the configuration information. The audio details reflect the fidelity of the audio. The higher the configuration of the configuration information, the higher the audio fidelity of the XR animation; the lower the configuration of the configuration information, the lower the audio fidelity of the XR animation. The compositing module is used to overlay the XR motion effects onto the media resource stream to obtain a composite media resource stream; The sending module is used to send the synthesized media resource stream to the second terminal.

7. The apparatus according to claim 6, characterized in that, When the XR motion effect includes the visual motion effect and the audio motion effect, the compositing module is specifically used to determine whether there is audio data of user speech in the media resource stream; if so, the visual motion effect is superimposed on the media resource stream to obtain the composite media resource stream; if not, the visual motion effect and the audio motion effect are superimposed on the media resource stream to obtain the composite media resource stream.

8. The apparatus according to claim 7, characterized in that, The compositing module is specifically used to decode the visual effects and the media resource stream; align the time and pixel positions of the visual effects and the media resource stream; and overlay the pixels of the visual effects and the media resource stream.

9. The apparatus according to any one of claims 6 to 8, characterized in that, When the first terminal is the calling terminal, the second terminal is the called terminal; when the first terminal is the called terminal, the second terminal is the calling terminal.

10. The apparatus according to claim 6, characterized in that, It also includes a query module; the query module is used to query whether the first terminal and / or the second terminal support the XR motion effect function.

11. An electronic device, characterized in that, The electronic device includes: a processor and a memory; The memory stores instructions that the processor can execute; When the processor is configured to execute the instructions, the electronic device performs the method as described in any one of claims 1-5.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes: computer software instructions; When the computer software instructions are executed in an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-5.

Images