Method and system for realizing AR (Augmented Reality) function of enhanced call, and computer equipment

Abstract

The invention relates to an AR (Augmented Reality) function implementation method and system for enhancing a call, and computer equipment. The method comprises the steps of obtaining an AR video sent to a second terminal in response to an AR call opening request sent by a first applet; performing video coding on the AR video to obtain a coded video stream; sending the coded video stream to the first chip; and the first chip sends the coded video stream to the second terminal through a data transmission channel in an enhanced call, and the second terminal decodes the coded video stream and displays an AR video obtained by decoding. By adopting the method, the AR function can be realized in the enhanced call.

Description

Technical Field

[0001] This application relates to the field of wireless communication technology, and in particular to a method, system, computer device, computer-readable storage medium, and computer program product for implementing AR functionality to enhance calls. Background Technology

[0002] In traditional communication technologies, enhanced calls and augmented reality calls (AR calls) represent two different directions of communication evolution. Enhanced calls achieve real-time sharing of multimedia data based on operator networks (such as IMS (IP Multimedia Subsystem)). Augmented reality calls integrate augmented reality technology into real-time calls, essentially a real-time fusion of virtual and real scenarios.

[0003] In existing technologies, augmented calling allows users to share multimedia content such as pictures, videos, locations, and files during voice calls, enhancing the information interaction capabilities of traditional calls. However, augmented calling is still limited to 2D screen interaction and cannot achieve core AR calling functions such as spatial awareness, 3D virtual object overlay, and environmental interaction. Therefore, current augmented calling cannot realize the AR functionality of AR calls. Summary of the Invention

[0004] Therefore, it is necessary to address the above-mentioned technical problems by providing a method, system, computer device, computer-readable storage medium, and computer program product for implementing AR functionality in augmented calls.

[0005] In a first aspect, this application provides a method for implementing AR functionality to enhance calls, applied to a first application in a first terminal, wherein the first terminal further includes a first mini-program and a first chip, and the method includes:

[0006] In response to the AR call activation request sent by the first mini-program, obtain the AR video sent to the second terminal;

[0007] The AR video is encoded to obtain an encoded video stream;

[0008] The encoded video stream is sent to the first chip; the first chip sends the encoded video stream to the second terminal through the data transmission channel in the enhanced call; the second terminal decodes the encoded video stream and displays the decoded AR video.

[0009] In one embodiment, the first application is configured with a first enhanced calling SDK and a first AR SDK; the method further includes:

[0010] The AR call activation request sent by the first mini-program is sent to the first AR SDK through the first augmented call SDK in the first application; the first AR SDK determines the AR video to be sent to the second terminal and passes it to the first augmented call SDK;

[0011] The AR video is encoded using the first enhanced call SDK to obtain the encoded video stream, and the encoded video stream is sent to the first chip.

[0012] In one embodiment, after the first enhanced call SDK receives the AR video sent by the first AR SDK, the method further includes:

[0013] Send the AR call activation request sent by the first mini-program to the first chip;

[0014] The first chip sends the AR call activation request to the second terminal to notify the second terminal that an AR call will be initiated.

[0015] In one embodiment, the method further includes:

[0016] The system receives an AR call shutdown command from the second terminal transmitted by the first chip, and shuts down the AR call function of the first terminal.

[0017] Secondly, this application also provides a method for implementing AR functionality to enhance calls, applied to a second application in a second terminal, wherein the second terminal further includes a second mini-program and a second chip, the method comprising:

[0018] Receive the encoded video stream sent by the first terminal and transmitted by the second chip;

[0019] The encoded video stream is decoded to obtain the decoded AR video;

[0020] The AR video is displayed.

[0021] In one embodiment, the method further includes:

[0022] Receive the AR call closing command triggered by the user from the second mini-program;

[0023] The AR call shutdown command is transmitted to the second chip, which then sends the AR call shutdown command to the first terminal.

[0024] Thirdly, this application also provides an AR function implementation system for enhancing calls, including a first terminal and a second terminal. The first terminal includes a first application, a first mini-program and a first chip, and the second terminal includes a second application, a second mini-program and a second chip.

[0025] The first mini-program is used to send an AR call activation request to the first application in response to the user's AR call activation operation;

[0026] The first application is configured to, in response to an AR call activation request sent by the first mini-program, acquire an AR video sent to the second terminal; encode the AR video to obtain an encoded video stream; and send the encoded video stream to the first chip.

[0027] The first chip is used to send the encoded video stream to the second chip of the second terminal through an enhanced data transmission channel during a call;

[0028] The second chip is used to transmit the encoded video stream to the second application.

[0029] The second application is used to decode the encoded video stream transmitted by the second chip and display the decoded AR video.

[0030] Fourthly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0031] In response to the AR call activation request sent by the first mini-program, obtain the AR video sent to the second terminal;

[0032] The AR video is encoded to obtain an encoded video stream;

[0033] The encoded video stream is sent to the first chip; the first chip sends the encoded video stream to the second terminal through the data transmission channel in the enhanced call; the second terminal decodes the encoded video stream and displays the decoded AR video.

[0034] Fifthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0035] In response to the AR call activation request sent by the first mini-program, obtain the AR video sent to the second terminal;

[0036] The AR video is encoded to obtain an encoded video stream;

[0037] The encoded video stream is sent to the first chip; the first chip sends the encoded video stream to the second terminal through the data transmission channel in the enhanced call; the second terminal decodes the encoded video stream and displays the decoded AR video.

[0038] Sixthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0039] In response to the AR call activation request sent by the first mini-program, obtain the AR video sent to the second terminal;

[0040] The AR video is encoded to obtain an encoded video stream;

[0041] The encoded video stream is sent to the first chip; the first chip sends the encoded video stream to the second terminal through the data transmission channel in the enhanced call; the second terminal decodes the encoded video stream and displays the decoded AR video.

[0042] The aforementioned method, system, computer device, computer-readable storage medium, and computer program product for implementing AR functionality in enhanced calls involve a first application in a first terminal responding to an AR call initiation request sent by a first applet. The first application retrieves the AR video sent to the second terminal, encodes the AR video to obtain an encoded video stream, and sends the encoded video stream to a first chip. The first chip then transmits the encoded video stream to the second terminal via the data transmission channel in the enhanced call. The second terminal decodes the encoded video stream and displays the decoded AR video. This method proposes an enhanced call AR functionality design scheme based on a DC channel, using the DC channel to transmit the AR video generated by the enhanced call. After generating the AR video, the enhanced call encodes the AR video into a video stream and sends it to the second terminal via the DC channel. Upon receiving the encoded video stream, the second terminal decodes it to generate the AR video and displays it locally. Compared to existing video calls based on IMS networks, this proposed enhanced call AR functionality design scheme based on a DC channel provides a new data transmission scheme for AR video calls, enabling AR functionality within enhanced calls. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is an application environment diagram of an AR function implementation method for enhancing calls in one embodiment;

[0045] Figure 2 This is a schematic diagram of the zero-layer architecture of an AR function design scheme for enhanced calling based on the DC channel in one embodiment.

[0046] Figure 3 This is a flowchart illustrating a method for implementing AR functionality to enhance calls in one embodiment;

[0047] Figure 4 This is a flowchart illustrating a method for implementing AR functionality to enhance calls in another embodiment;

[0048] Figure 5 This is a zero-level timing flowchart of an AR function design scheme for enhanced calling based on a DC channel in one embodiment;

[0049] Figure 6 This is a one-layer UML class diagram of an AR function design scheme for enhanced calling based on the DC channel in one embodiment;

[0050] Figure 7 This is a block diagram of a system for implementing AR functionality to enhance calls, as shown in one embodiment.

[0051] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0053] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The terms "comprising" and "having," and any variations thereof, as used in this application, are intended to cover non-exclusive inclusion.

[0054] The AR function implementation method for enhanced calls provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, the first terminal 101 communicates with the second terminal 102 via a network. Both the first terminal 101 and the second terminal 102 contain modules at three levels: an application (APP), a mini-program, and a chip. In a specific application, the user initiates an AR call request through the first mini-program on the first terminal 102. The mini-program sends this AR call request to the first application. The first application receives the AR video sent to the second terminal 102, encodes the AR video to obtain an encoded video stream, and sends the encoded video stream to the first chip. The first chip then sends the encoded video stream to the second terminal 102 through the data transmission channel (DataChannel, DC) in the enhanced call. The second terminal 102 receives the encoded video stream, decodes it, and displays the decoded AR video. This achieves the AR function in the enhanced call. The first terminal 101 and the second terminal 102 can be, but are not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. Head-mounted devices can include virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses.

[0055] This application, in its zero-layer design of the enhanced call AR function design scheme based on the DC channel, divides the overall code architecture from top to bottom into three levels: the mini-program level, the APP level (corresponding to the application), and the AIDL level. AIDL (Android Interface Definition Language) is an interface definition language used in the Android system for cross-process communication. Its core function is to allow components running in different processes to interact through predefined interfaces. The main functions of each level are as follows:

[0056] Mini Program Layer: Contains the mini program module for enhanced calling. The mini program module communicates with the APP layer through the JsApi interface (DC mini program and enhanced calling interface definition).

[0057] At the APP level: This includes the Enhanced Call SDK module and the AR SDK module. The Enhanced Call SDK is responsible for communication with the upper-level mini-program layer and with the IMS DC AIDL in the lower-level AIDL layer. It also encodes the video stream generated by the AR SDK and decodes the video stream received by the IMS DC. The AR SDK module is responsible for providing AR rendering capabilities to the Enhanced Call SDK.

[0058] AIDL layer (corresponding chip): contains the IMS DC AIDL module. This layer mainly implements the communication function with the chip's IMS DCService.

[0059] This application addresses the current situation where augmented calls cannot implement AR functionality by proposing a design scheme for augmented call AR functionality based on the FaceMesh framework. This scheme makes specific modifications to the FaceMesh framework and integrates the modified FaceMesh framework into augmented calls. It combines the ARCall AIDL interface provided by the terminal chip, the ARCall OEM AIDL interface provided by the terminal manufacturer, and the mini-program JS interface to realize AR functionality in augmented calls.

[0060] In one exemplary embodiment, such as Figure 3 As shown, a method for implementing AR functionality to enhance calls is provided, and this method is applied to... Figure 1 Taking the first application in the first terminal 101 as an example, the following steps are included:

[0061] Step S310: In response to the AR call activation request sent by the first mini-program, obtain the AR video sent to the second terminal.

[0062] AR, or Augmented Reality, is a technology that overlays and merges virtual scenes or information with the real physical environment, presenting them interactively to the user and creating a shared space between the virtual and real worlds. AR Call (Augmented Reality Call) is an interactive method that integrates augmented reality technology into real-time calls. By overlaying virtual elements (such as 3D avatars, special effects, real-time translation, etc.) into video calls, it enhances the immersiveness and functionality of communication. AR videos contain content that blends virtual and real elements.

[0063] In this step, the user of the first terminal initiates an AR call through a first mini-program on the first terminal. Responding to the user's trigger, the first mini-program sends an AR call initiation request to the first application, notifying the application to activate the AR call. Upon receiving the AR call initiation request from the first mini-program, the first application can capture a real-time video stream via its camera, perform AR element overlay processing, and create an AR video.

[0064] Step S320: Perform video encoding on the AR video to obtain an encoded video stream.

[0065] Specifically, after obtaining the AR video, the first application further encodes the AR video to compress it into a transmittable format, thus obtaining an encoded video stream.

[0066] For example, an encoded video stream can be obtained by inputting AR video into an encoder for encoding.

[0067] In step S330, the encoded video stream is sent to the first chip; the first chip sends the encoded video stream to the second terminal through the data transmission channel in the enhanced call; the second terminal decodes the encoded video stream and displays the decoded AR video.

[0068] Specifically, after encoding the AR video, the first application sends the encoded video stream to the second terminal, thereby realizing the AR function. More specifically, the first application first sends the encoded video stream to the first chip of the first terminal, which then sends the encoded video stream to the second terminal through the data transmission channel (DC channel) in the enhanced call. After receiving the encoded video stream, the second terminal decodes the encoded video stream and displays the decoded AR video.

[0069] In the aforementioned method for implementing AR functionality in enhanced calls, the first application in the first terminal responds to an AR call initiation request sent by the first mini-program, acquires the AR video sent to the second terminal, encodes the AR video to obtain an encoded video stream, and sends the encoded video stream to the first chip. The first chip then sends the encoded video stream to the second terminal through the data transmission channel in the enhanced call. The second terminal decodes the encoded video stream and displays the decoded AR video. This method proposes an enhanced call AR functionality design scheme based on a DC channel, using the DC channel to transmit the AR video generated by the enhanced call. After generating the AR video, the enhanced call encodes the AR video into a video stream and sends it to the second terminal through the DC channel. Upon receiving the encoded video stream, the second terminal decodes it to generate the AR video and displays it locally. Compared with existing video calls based on IMS networks, this proposed enhanced call AR functionality design scheme based on a DC channel provides a new data transmission scheme for AR video calls, enabling AR functionality in enhanced calls.

[0070] In an exemplary embodiment, the first application is equipped with a first enhanced call SDK and a first AR SDK; the method further includes: sending an AR call activation request sent by the first mini-program to the first AR SDK through the first enhanced call SDK in the first application; the first AR SDK determines the AR video to be sent to the second terminal and passes it to the first enhanced call SDK; the first enhanced call SDK encodes the AR video to obtain an encoded video stream and sends the encoded video stream to the first chip.

[0071] An SDK (Software Development Kit) is a set of tools that helps developers create applications for a specific platform, system, or service. An SDK is a pre-packaged collection of tools, libraries, documentation, and code examples designed to simplify programming in a specific development environment. It typically includes: an application programming interface (API), development tools (such as compilers and debuggers), documentation and tutorials, sample code, and the necessary runtime environment.

[0072] In this solution, the first application includes a first enhanced call SDK and a first AR SDK. The first enhanced call SDK is responsible for communication with the first mini-program and with the first chip, and also encodes the video stream generated by the first AR SDK. The first AR SDK provides AR rendering capabilities to the first enhanced call SDK.

[0073] It is understandable that when the first terminal acts as a receiver of AR video, the first enhanced call SDK will be used to decode the encoded video stream sent by the first chip. In other words, for any given terminal, the functionality of its enhanced call SDK and AR SDK depends on the terminal's current role. When the terminal acts as a sender of AR video, its enhanced call SDK encodes the video stream generated by the AR SDK, and the AR SDK provides AR rendering capabilities to the enhanced call SDK. When the terminal acts as a receiver of AR video, its enhanced call SDK decodes the encoded video stream transmitted by the first chip; in this case, the AR SDK does not need to participate in the processing. Similarly, the functions of the chip and the mini-program within the terminal, as well as the interactions between the mini-program, the enhanced call SDK, the AR SDK, and the chip, will also implement different functions depending on the terminal's role.

[0074] In this embodiment, the first terminal acts as the sender for initiating AR calls and sending AR videos. Therefore, the first augmented call SDK in the first application communicates with the first mini-program, receives the AR call initiation request sent by the first mini-program, and sends the request to the first AR SDK. In response to the AR call initiation request, the first AR SDK renders the video captured by the first terminal using its AR rendering capabilities, obtains the AR video, and returns it to the first augmented call SDK. The first augmented call SDK encodes the AR video sent by the first AR SDK to obtain an encoded video stream, and sends the encoded video stream to the first chip, which then transmits it to the second terminal.

[0075] In this embodiment, by setting a first enhanced call SDK and a first AR SDK in the first application, the AR video acquisition and encoding functions are realized, thereby sending the encoded video stream of the AR video to the second terminal, realizing the AR function of the first terminal and the second terminal in enhanced call.

[0076] In an exemplary embodiment, after the first enhanced call SDK receives the AR video sent by the first AR SDK, it further includes: sending the AR call activation request sent by the first mini-program to the first chip; and sending the AR call activation request to the second terminal through the first chip to notify the second terminal that an AR call will be conducted.

[0077] Specifically, after receiving the AR video sent by the first AR SDK in the first application, the first enhanced call SDK can also send an AR call initiation request to the second terminal to notify the second terminal that an AR call will be conducted. More specifically, the first enhanced call SDK can first send the AR call initiation request sent by the first mini-program to the first chip, and then the first chip will send the AR call initiation request to the second terminal to notify the second terminal that an AR call will be conducted.

[0078] In some embodiments, the first enhanced call SDK in the first application can send the encoded video stream to the second terminal via the first chip after receiving the AR call acceptance message returned by the second terminal.

[0079] In this embodiment, after the first augmented call SDK in the first application receives the AR video sent by the first AR SDK, before sending the encoded video stream to the second terminal, it sends an AR call start request to the second terminal to notify the second terminal that an AR call will be conducted, so that the user of the second terminal can confirm. In some embodiments, the interface of the second terminal can also be switched to AR mode to ensure stable AR interaction between the first terminal and the second terminal.

[0080] In one exemplary embodiment, the method further includes: receiving an AR call shutdown command sent by a second terminal from a first chip, and disabling the AR call function of the first terminal.

[0081] In its implementation, the second terminal also comprises three layers: an application, a mini-program, and a chip, referred to as the second application, the second mini-program, and the second chip. When the user of the second terminal closes the AR call, the second mini-program notifies the second application, specifically the second enhanced calling SDK within the second application, to close the AR call. The second enhanced calling SDK then transmits the AR call closure command to the second chip, which in turn sends it to the first terminal. Specifically, the AR call closure command first reaches the first chip in the first terminal, and then the first chip transmits it to the first application, specifically the first enhanced calling SDK within the first application. The first enhanced calling SDK, in response to the AR call closure command, closes the AR call function of the first terminal.

[0082] In this embodiment, the AR call function of the first terminal is turned off by receiving the AR call shutdown command from the second terminal through the first chip, thereby realizing the on-demand opening and closing operation of AR call between the first terminal and the second terminal.

[0083] The architecture of the second terminal is the same as that of the first terminal, also including applications, mini-programs, and chips. The applications also include an enhanced call SDK and an AR SDK. In an exemplary embodiment, such as... Figure 4 As shown, a method for implementing AR functionality to enhance calls is also provided, which can be applied to... Figure 1 The following steps are used as an example to illustrate the process of using the second application in the second terminal 102:

[0084] Step S410: Receive the encoded video stream sent by the first terminal and transmitted by the second chip;

[0085] Step S420: Decode the encoded video stream to obtain the decoded AR video;

[0086] Step S430: Display the AR video.

[0087] Specifically, when the encoded video stream sent by the first terminal arrives at the second terminal, it first reaches the second chip in the second terminal. After receiving the encoded video stream sent by the first chip of the first terminal through the DC channel, the second chip passes the encoded video stream to the second enhanced call SDK. The second enhanced call SDK decodes the encoded video stream, for example, by using a decoder, to obtain the decoded AR video, and then displays the decoded AR video on the second terminal.

[0088] It is understood that in this embodiment, the second terminal acts as the receiver of AR video. Therefore, the second enhanced call SDK is used to decode the encoded video stream sent by the second chip. If the second terminal acts as the sender of AR video, that is, as the first terminal, the components of the second terminal will perform the same operations as the first terminal. In other words, the first terminal and the second terminal in this application are named to distinguish between the two terminals. In actual application, the two terminals can perform the same operations.

[0089] This method proposes an enhanced call AR function design scheme based on the DC channel. The second terminal decodes the received encoded video stream to generate AR video and displays it on the local terminal. Compared with existing video calls based on IMS networks, this proposed enhanced call AR function design scheme based on the DC channel provides a new data transmission scheme for AR video calls, enabling AR functionality within enhanced calls.

[0090] In one exemplary embodiment, the method further includes: receiving an AR call shutdown command triggered by a user transmitted by a second mini-program; transmitting the AR call shutdown command to a second chip, which then sends the AR call shutdown command to a first terminal.

[0091] Specifically, when the user of the second terminal closes the AR call, the second applet notifies the second application, specifically the second augmented call SDK in the second application, to close the AR call. The second augmented call SDK then transmits the AR call closure instruction to the second chip, which in turn sends it to the first terminal to notify the first terminal to close the AR call function.

[0092] In this embodiment, the user's AR call shutdown command transmitted by the second mini-program is sent to the first terminal through the second chip via the second application, thereby realizing the on-demand opening and closing operation of AR calls between the first terminal and the second terminal.

[0093] This application can be applied to the implementation of AR functions in enhanced calls, providing a new implementation scheme to meet users' AR function requirements in video calls. To facilitate understanding of the embodiments of this application by those skilled in the art, the following will further describe this solution.

[0094] like Figure 2 As shown, the zero-layer code architecture design for this solution mainly includes a three-layer design: the mini-program layer, the APP layer, and the AIDL layer. The mini-program layer contains the mini-program module, the APP layer contains the enhanced call SDK module and the ARSDK module, and the AIDL layer contains the IMS DC AIDL module.

[0095] The zero-level code timing design for implementing this solution mainly includes: a mini-program module, an enhanced call SDK module, an ARSDK module, and an IMS DC AIDL module. The interaction and communication flow between these modules is as follows: Figure 5 As shown, the overall calling logic is as follows:

[0096] 1. UEA (equivalent to the first terminal) users can start AR calls through the first mini-program.

[0097] 2. The first mini-program notifies the first enhanced call SDK to enable AR calling.

[0098] 3. The first enhanced call SDK notifies the first AR SDK to enable AR calling.

[0099] 4. The first AR SDK transmits AR video to the first augmented call SDK.

[0100] 5. After receiving the AR video, the first augmented call SDK transmits an AR call activation command to the first chip.

[0101] 6. The first chip transmits the AR call activation command to the UEB (equivalent to the second terminal) through the DC channel.

[0102] 7. The second chip sends an AR call activation command to the second enhanced call SDK.

[0103] 8. The first enhanced call SDK encodes AR video.

[0104] 9. The first enhanced call SDK sends an encoded video stream through the first chip.

[0105] 10. The first enhanced call SDK sends an encoded video stream to the UEB via the DC channel.

[0106] 11. The UEB receives and decodes the encoded video stream. Specifically, the second chip receives the encoded video stream and passes it to the second enhanced call SDK, which then decodes it.

[0107] 12. UEB displays AR video.

[0108] 13. UEB users should disable AR calling.

[0109] 14. The second mini-program notifies the second enhanced call SDK to turn off AR calling.

[0110] 15. The second enhanced call SDK notifies the second chip to send a shutdown command.

[0111] 16. The second chip sends a shutdown command to the first chip of UEA through the DC channel.

[0112] 17. The first chip transmits a shutdown command to the first augmented call SDK to shut down the AR call.

[0113] The single-layer code UML class design for implementing this solution mainly includes: a mini-program process module, a main process module, an AR function module, a video stream encoding / decoding module, and an IMS DC chip interface module, such as... Figure 6 As shown, the main functions of each module are as follows:

[0114] The mini-program process module is responsible for distributing and transmitting the interactive commands sent by the mini-program, and finally transmitting the commands to the main process according to the type of command.

[0115] The main process module is primarily responsible for managing the various interface modules and converting the received AR commands into specific logical operations.

[0116] The AR SDK module is primarily responsible for generating AR image data and rendering it into the main process module.

[0117] The video stream encoding / decoding module is mainly responsible for encoding the generated AR video stream and decoding the received AR video stream.

[0118] The IMS DC chip interface module is mainly responsible for sending and receiving AR video data streams.

[0119] The functions of each submodule included in each module are as follows:

[0120] (1) Mini Program Process

[0121] IJsEventDispatchers: A public event dispatching interface that defines a public event dispatching functionality interface.

[0122] AREventDispatchers: AR event dispatcher class. This class implements a public event dispatcher interface to dispatch AR-related mini-program control events.

[0123] IARMiniUsecase: The AR transaction management interface in the mini-program process, which defines AR-related event methods.

[0124] ARMiniUsecase: The AR transaction management class in the mini-program process. This class implements the AR transaction management interface and is mainly responsible for the preprocessing of AR control-related logic in the mini-program process.

[0125] IMiniToParentManager: The cross-process communication interface between the mini-program process and the main process, defining the methods that cross-process communication classes should implement.

[0126] MiniToParentManager: A cross-process communication interface class between the mini-program process and the main process. It implements the cross-process communication interface and is mainly responsible for cross-process communication between the mini-program and the main process, passing the control logic that the AR mini-program needs to execute to the main process.

[0127] (2) Main process

[0128] AppService: The mini-program command server in the main process. After receiving commands from the mini-program process, it executes the relevant logic operations in the main process.

[0129] IAppServiceEventDispatcher: The AppService event dispatch interface, responsible for defining public dispatch interfaces.

[0130] ARServiceEventDispatcher: The main program's AR event dispatcher class, responsible for distributing AR commands received by AppService.

[0131] IARUsecase: The main process AR transaction management interface, which defines the logical methods related to AR events.

[0132] ARUsecase: The main process AR event management class, responsible for decomposing AR command-related logic and calling relevant AR function classes to execute corresponding logical operations.

[0133] (3) IMS DC chip interface module

[0134] IMS DC AIDL: Responsible for cross-process communication of DC services.

[0135] IIMsDCManager: The interface definition for DC business management class, defining the interfaces related to DC.

[0136] IMSDCManager: A DC business management class that encapsulates the IMS DC AIDL interface and provides a unified DC function interface to the main process.

[0137] (4) Video stream encoding / decoding module

[0138] IVideoEnDeCodeManager: Definition of video stream encoding and decoding interface functions.

[0139] videoEnDecodeManager: A video stream encoding / decoding class responsible for encoding and decoding AR-generated videos.

[0140] (5) AR Function Module

[0141] AR SDK: AR functionality SDK, responsible for implementing AR functions.

[0142] IARSDKManager: The interface definition for DC business management classes, defining the interfaces related to DC.

[0143] ARSDKManager: The AR SDK management class, responsible for encapsulating AR function interfaces and providing AR function methods to ARUsecase.

[0144] This method proposes an augmented call AR function design scheme based on a DC channel, which uses the DC channel to transmit the AR video generated by the augmented call. After generating the AR video, the augmented call encodes the AR video into a video stream and sends it to the second terminal through the DC channel. The second terminal, upon receiving the encoded video stream, decodes it to generate the AR video and displays it locally. Compared with existing video calls based on IMS networks, this proposed DC channel-based augmented call AR function design scheme provides a new data transmission solution for AR video calls, enabling AR functionality within augmented calls.

[0145] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0146] Based on the same inventive concept, this application also provides an AR function implementation system for enhancing calls, used to implement the aforementioned AR function implementation method for enhanced calls. The solution provided by this system is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more AR function implementation system embodiments for enhanced calls provided below can be found in the limitations of the AR function implementation method for enhanced calls described above, and will not be repeated here.

[0147] In one exemplary embodiment, such as Figure 7 As shown, an AR function implementation system for enhancing calls is provided, including: a first terminal 710 and a second terminal 720. The first terminal includes a first application, a first mini-program and a first chip, and the second terminal includes a second application, a second mini-program and a second chip.

[0148] The first mini-program is used to respond to the user's AR call activation operation by sending an AR call activation request to the first application.

[0149] The first application is used to respond to the AR call activation request sent by the first mini-program, obtain the AR video sent to the second terminal 720; encode the AR video to obtain an encoded video stream; and send the encoded video stream to the first chip.

[0150] The first chip is used to send the encoded video stream to the second chip of the second terminal 720 through the data transmission channel in the enhanced call;

[0151] The second chip is used to transmit the encoded video stream to the second application;

[0152] The second application is used to decode the encoded video stream transmitted by the second chip and display the decoded AR video.

[0153] In some embodiments, the first application is further configured to send the AR call activation request sent by the first mini-program to the first AR SDK through the first enhanced call SDK in the first application; the first AR SDK is further configured to determine the AR video to be sent to the second terminal and pass it to the first enhanced call SDK; the first application is further configured to encode the AR video through the first enhanced call SDK to obtain an encoded video stream and send the encoded video stream to the first chip.

[0154] In some embodiments, the first application is further configured to send the AR call activation request sent by the first mini-program to the first chip; and send the AR call activation request to the second terminal through the first chip to notify the second terminal that an AR call will be made.

[0155] In some embodiments, the first application is further configured to receive an AR call shutdown command sent by the second terminal 720 via the first chip, thereby disabling the AR call function of the first terminal 710.

[0156] In some embodiments, the second application is further configured to receive an AR call shutdown command triggered by a user and transmitted by the second applet; transmit the AR call shutdown command to the second chip, and the second chip sends the AR call shutdown command to the first terminal 710.

[0157] The various modules in the AR function implementation system for enhanced calling described above can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0158] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements an AR (Augmented Reality) function to enhance calls. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0159] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0160] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0161] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0162] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0163] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0164] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0165] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0166] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for implementing AR functionality to enhance phone calls, characterized in that, A first application is applied to a first terminal, wherein the first terminal also includes a first mini-program and a first chip, the method comprising: In response to the AR call activation request sent by the first mini-program, obtain the AR video sent to the second terminal; The AR video is encoded to obtain an encoded video stream; The encoded video stream is sent to the first chip; the first chip sends the encoded video stream to the second terminal through the data transmission channel in the enhanced call; the second terminal decodes the encoded video stream and displays the decoded AR video.

2. The method according to claim 1, characterized in that, The first application is equipped with a first enhanced call SDK and a first AR SDK; the method further includes: The AR call activation request sent by the first mini-program is sent to the first AR SDK through the first augmented call SDK in the first application; the first AR SDK determines the AR video to be sent to the second terminal and passes it to the first augmented call SDK; The AR video is encoded using the first enhanced call SDK to obtain the encoded video stream, and the encoded video stream is sent to the first chip.

3. The method according to claim 2, characterized in that, After the first enhanced call SDK receives the AR video sent by the first AR SDK, it also includes: Send the AR call activation request sent by the first mini-program to the first chip; The first chip sends the AR call activation request to the second terminal to notify the second terminal that an AR call will be initiated.

4. The method according to claim 1, characterized in that, The method further includes: The system receives an AR call shutdown command from the second terminal transmitted by the first chip, and shuts down the AR call function of the first terminal.

5. A method for implementing AR functionality to enhance phone calls, characterized in that, A second application is applied to a second terminal, the second terminal also having a second applet and a second chip, the method comprising: Receive the encoded video stream sent by the first terminal and transmitted by the second chip; The encoded video stream is decoded to obtain the decoded AR video; The AR video is displayed.

6. The method according to claim 5, characterized in that, The method further includes: Receive the AR call closing command triggered by the user from the second mini-program; The AR call shutdown command is transmitted to the second chip, which then sends the AR call shutdown command to the first terminal.

7. A system for implementing AR functionality to enhance phone calls, characterized in that, The system includes a first terminal and a second terminal. The first terminal includes a first application, a first mini-program, and a first chip. The second terminal includes a second application, a second mini-program, and a second chip. The first mini-program is used to send an AR call activation request to the first application in response to the user's AR call activation operation; The first application is configured to, in response to an AR call activation request sent by the first mini-program, acquire the AR video sent to the second terminal; and encode the AR video to obtain an encoded video stream. Send the encoded video stream to the first chip; The first chip is used to send the encoded video stream to the second chip of the second terminal through an enhanced data transmission channel during a call; The second chip is used to transmit the encoded video stream to the second application. The second application is used to decode the encoded video stream transmitted by the second chip and display the decoded AR video.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images