Multimedia type-based communication method, apparatus, device, and storage medium

Abstract

The application relates to a multimedia type-based communication method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: querying each media type and a data transmission sequence between each media type according to user information and service information; finding a resource transmission threshold of each media type in a communication process according to each media type and the data transmission sequence; and controlling data of each media type to communicate through the resource transmission threshold of each media type. The method can enable terminal to synchronously display data of each media type, and improve the authenticity of data received by the terminal. In addition, each service end does not need to be greatly adjusted based on the data transmission sequence, which helps to improve the service deployment range and efficiency.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a communication method, apparatus, computer equipment, storage medium, and computer program product based on multimedia. Background Technology

[0002] With the development of communication technology, multimedia communication services such as AR / VR and holography have emerged. These multimedia communication services involve different modal media streams, including audio, video, and haptic feedback. Current technologies utilize real-time communication service network architectures to access and control these multimedia communication services.

[0003] However, during the transmission of data through the real-time communication service network architecture, the data of different modalities lacks synchronization, resulting in poor authenticity of the data received by the user at the receiving end. Summary of the Invention

[0004] Therefore, it is necessary to provide a multimedia-based communication method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the synchronization of data during communication, in order to address the aforementioned technical problems.

[0005] In a first aspect, this application provides a communication method based on a multimedia type, the method comprising:

[0006] Based on user information and business information, query the data transmission order between each media type and each of the aforementioned media types;

[0007] According to each of the media types and the data transmission order, find the resource transmission threshold of each of the media types in the communication process;

[0008] By controlling the communication of data for each media type through the resource transmission threshold of each media type.

[0009] In one embodiment, the step of querying the data transmission order between each media type based on user information and business information includes:

[0010] Identify the online terminals indicated by user and business information;

[0011] In the data warehouse, retrieve the media types applicable to the online terminals;

[0012] The data transmission order among the various media types is determined according to the preset data transmission order.

[0013] In one embodiment, retrieving the media types applicable to the online terminals includes:

[0014] Determine the terminal type and media types supported by the online terminals; the terminal type is classified according to the information collection method.

[0015] Based on the terminal type and the media types supported by the online terminals, retrieve the media types applicable to the online terminals.

[0016] In one embodiment, the communication process includes a data transmission process; the step of finding the resource transmission threshold for each media type in the communication process according to each media type and the data transmission order includes:

[0017] According to the data transmission order, determine the time difference threshold for each of the media types during the transmission process;

[0018] Based on the time difference threshold, a resource transmission threshold for controlling the synchronous transmission of data for each of the aforementioned media types is determined.

[0019] In one embodiment, controlling data communication for each media type by using resource transmission thresholds for each media type includes:

[0020] By using the resource transmission threshold of each media type, media computing resources of each media type are scheduled to obtain media computing resources of each media type.

[0021] The media computing resources are used to process the data of each media type, so that the data of each media type is synchronously transmitted to the terminal in the communication process.

[0022] In one embodiment, controlling data communication for each media type by using resource transmission thresholds for each media type includes:

[0023] By using the resource transmission thresholds for each media type, resource scheduling is performed on the bearer network to obtain the transmission resources for each media type.

[0024] Data of each media type is transmitted through the transmission resources, so that the data of each media type is synchronously transmitted to the terminal in the communication process.

[0025] In one embodiment, before querying the data transmission order between each media type based on user information and business information, the method further includes:

[0026] Receive a registration request sent by a multimedia user for service information; the registration request includes the user information to be registered by the multimedia user, the terminal to be registered belonging to the user information to be registered, the terminal type to be registered according to the data collection method, and the media types to be registered supported by the terminal to be registered;

[0027] When the registration request is approved, the terminal to be registered, the type of the terminal to be registered, and the type of the media to be registered are recorded;

[0028] The business contract information containing the business information and the identifier of the user to be registered is fed back to the sending end of the registration request.

[0029] Secondly, this application also provides a multimedia-based communication device, the device comprising:

[0030] The data warehouse module is used to query the data transmission order between each media type based on user information and business information;

[0031] The threshold management module is used to find the resource transmission threshold of each media type in the communication process according to each media type and the data transmission order.

[0032] The threshold management module is used to control the communication of data of each media type by using the resource transmission threshold of each media type.

[0033] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the multimedia-based communication steps described in any of the above embodiments.

[0034] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of multimedia-based communication in any of the above embodiments.

[0035] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of multimedia-based communication in any of the above embodiments.

[0036] The aforementioned communication method, apparatus, computer equipment, storage medium, and computer program product based on multimedia types query the data transmission order between each media type according to user information and business information, so as to adaptively adjust the data transmission order of each media type for each user's business transaction; according to each media type and the data transmission order, find the resource transmission threshold of each media type in the communication process, so that the resource transmission threshold in the communication process is set for the media type and the data transmission order; control the communication of data of each media type through the resource transmission threshold of each media type, and restrict the data transmission process of each media type, so that the data of each media type can reach the terminal synchronously, and the terminal can display the data of each media type synchronously, so that the data received by the terminal has good authenticity. Moreover, this process is implemented during transmission, and each business terminal can share the basic modules of the multimedia communication business architecture, so that each business terminal does not need to make significant adjustments based on the data transmission order, which helps to improve the scope and efficiency of business deployment. Attached Figure Description

[0037] Figure 1 This is an application environment diagram of a multimedia-based communication method in one embodiment;

[0038] Figure 2 This is a flowchart illustrating a multimedia-based communication method in one embodiment;

[0039] Figure 3 This is a flowchart illustrating the steps for determining the data transmission order in one embodiment;

[0040] Figure 4 This is a diagram of a communication network architecture supporting multimodal synchronization threshold coordination in another embodiment;

[0041] Figure 5 This is a flowchart illustrating communication via a call in one embodiment;

[0042] Figure 6 This is a schematic diagram of the registration process for a multimedia-based communication method in one embodiment;

[0043] Figure 7 This is a structural block diagram of a multimedia-based communication device in one embodiment;

[0044] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0046] The multimedia-based communication method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server.

[0047] Server 104 is a server in a multimodal communication architecture, used to control data transmission, network transmission, and the allocation of computing resources to control the data transmission process of various media types. Optionally, server 104 can be a core network device in 5G, 6G, or other communication networks.

[0048] Optionally, server 104 can be a real server cluster, a virtual module, or a distributed server. Optionally, terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc.

[0049] In one embodiment, such as Figure 2 As shown, a communication method based on multimedia types is provided, which can be applied to... Figure 1 Taking server 104 as an example, the following steps are included:

[0050] Step 202: Query the data transmission order between different media types based on user information and business information.

[0051] User information is used to identify the user during the communication process; user information can be a username, user ID, etc., and can also be terminal information determined based on the username or user ID. Service information is used to identify the communication service the user is engaging in during the communication process; service information involves a specific business logic within the communication process, which involves multiple media types and corresponding data transmission sequences; service information can be a service ID, etc. Optionally, both user information and service messages are contained within the call request sent by the user through the terminal.

[0052] Media type is used to characterize the type of media resource. During communication, the media type can change dynamically, and it is obtained by querying user information and business information. Optionally, media type includes, but is not limited to, image type and audio type. Different media types have different transmission efficiencies during communication, therefore, the data transmission order needs to be set accordingly.

[0053] Optionally, data of multiple media types falls under the category of multimodal data. Specifically, multimodal data refers to a data set with different characteristics from different sensors on a terminal. Data of multiple media types can include sensor data such as images, videos, text, and audio. Optionally, multimodal data can be communication data transmitted or processed by immersive devices such as augmented reality devices and holographic communication devices, which may have sensors for images, videos, audio, etc.

[0054] The data delivery order is used to characterize the sequence in which data of each media type is transmitted to the terminal during the communication process, so that the data of each media type is ultimately displayed synchronously on the terminal. Optionally, querying user information can adjust the media type and data delivery order to suit different users, while querying business information can adjust the media type and data delivery order to suit different business selections by the user.

[0055] In one embodiment, the session control synchronization threshold query module initiates a multimodal synchronization threshold query request to the data warehouse module, and the query message carries the username and business ID.

[0056] Correspondingly, the system queries the data transmission order between different media types based on user information and business information. This includes: the data warehouse module determining the user's terminal information based on the username and business ID, and retrieving the media types and data transmission order involved in the current call using the terminal information. Optionally, a call may involve two media types: audio media and haptic media, with the audio media arriving before the haptic media.

[0057] In one embodiment, querying the data transmission order between different media types based on user information and business information includes: querying the services available to the user based on user information; and selecting the data transmission order between different media types for the user based on the business information among the available services.

[0058] Step 204: According to each media type and data transmission order, find the resource transmission threshold of each media type in the communication process.

[0059] Resource transmission thresholds are used to characterize the degree of restriction on resources for each media type during data transmission. Resource transmission thresholds are set for different media types to control the data transmission efficiency of each media type. Since media type and data transmission order dynamically change based on user and business information, the resource transmission threshold also dynamically changes during each communication process.

[0060] The resource transmission threshold is positively correlated with the data transmission order. In the data transmission order, the resource transmission threshold of the media type that arrives at the terminal first is lower than that of the media type that arrives later. This is to ensure that the complete data arriving at the terminal is approximately synchronous by adjusting the data transmission thresholds for different media types, thus reproducing the real-world scenario. Optionally, two media types are involved: audio media and tactile media. The data transmission order is such that the audio media arrives before the tactile media. In this case, the resource transmission threshold for the audio media is higher than that for the tactile media.

[0061] In one embodiment, the resource transmission threshold for each media type during the communication process is determined according to each media type and the data transmission order, including: generating a multimodal synchronization threshold strategy to ensure the data transmission order according to each media type and the data transmission order; the multimodal synchronization threshold strategy is used to determine the resource transmission threshold for each media type during the communication process.

[0062] Optionally, the media type and data delivery order are sent from the data warehouse to the session control module, which then forwards the media type and data delivery order to the Quality of Service (QoS) rule engine. The QoS rule engine then generates a multimodal synchronization threshold policy, which determines the resource transmission threshold for each media type.

[0063] In one implementation, the resource transmission threshold for each media type during the communication process is determined according to the media type and the data transmission order, including: determining the time difference threshold for each media type during the transmission process according to the data transmission order; and determining the resource transmission threshold for controlling the synchronous transmission of data for each media type based on the time difference threshold.

[0064] Time difference thresholds are used to determine the range of time differences between data of different media types during transmission. If the time difference between different media types during data transmission falls within the time difference threshold, it can be determined that the data of each media type is transmitted synchronously. Synchronous transmission means that data of different media types are transmitted within the same time period. The transmission process is part of the communication process and is used to control data transmission during communication.

[0065] Optionally, when representing service A, both audio and tactile media need to be transmitted. The audio media arrives before the tactile media, and the tactile media must arrive within 12 milliseconds after the audio media appears. Therefore, 12 milliseconds is the time difference threshold. When representing service B, both video and tactile media need to be transmitted. The video media arrives before the tactile media, and the tactile media must arrive within 30 milliseconds after the video media appears. Therefore, 30 milliseconds is the time difference threshold.

[0066] In scenarios where data of different media types are transmitted simultaneously, a time difference threshold is set for each media type during transmission according to the data transmission order. In order to ensure that the time difference of data of each media type during transmission is within the time difference threshold, a resource transmission threshold is set based on the time difference threshold so that the data of each media type is transmitted to the terminal within the time difference threshold, thereby realizing the synchronous transmission of data of different media types.

[0067] Step 206: Control the communication of data for each media type by using the resource transmission threshold for each media type.

[0068] In one embodiment, data communication for each media type is controlled by using resource transmission thresholds for each media type. This includes: limiting data transmission for each media type based on its resource transmission thresholds to obtain restricted data for each media type; and using the restricted data for each media type to enable communication between different terminals. Based on this, the data communication process for multimedia types encompasses both data querying and transmission processes. For server 104, the data query process determines the data transmission order between media types, while the transmission control process involves resource scheduling.

[0069] In one implementation, the communication of data for each media type is controlled by using the resource transmission threshold for each media type, including: scheduling media computing resources for each media type based on the resource transmission threshold for each media type to obtain media computing resources for each media type; and processing the data of each media type separately using the media computing resources so that the data of each media type is synchronously transmitted to the terminal in the communication process.

[0070] Media computing resources are computer resources used to store or process data of various media types. The resource transmission threshold is positively correlated with the size of the media computing resources. This allows for control over the data processing of different media types by adjusting the size of the media computing resources, thereby ensuring the synchronous transmission of data across all media types. Optionally, a higher resource transmission threshold for a particular media type will allocate more media computing resources for processing that type of data. Optionally, media computing resources can be third-party platforms or other media providing resources, including but not limited to media used for generating or processing images, media used for generating or processing audio, and media used for generating or processing tactile data.

[0071] In one optional implementation, media computing resources for each media type are scheduled based on resource transmission thresholds for each media type to obtain media computing resources for each media type. This includes: receiving a multimodal synchronization threshold policy sent by the Quality of Service (QoS) rule engine through the session control module; the session control module sending the multimodal synchronization threshold policy to the media computing resource scheduling module for computing resource scheduling to obtain media computing resources for each media type. Therefore, for media computing resources outside the core network to which the server belongs, a dedicated session control module is used for connection to ensure data security.

[0072] By using the resource transmission thresholds for each media type, media computing resources are scheduled for each media type, so that the data processed by different media computing resources are different. This allows media computing resources to be adaptively adjusted for each media type, thereby controlling the communication process from the perspective of data processing, so as to achieve synchronous data transmission for each media type.

[0073] In one implementation, controlling the communication of data for each media type by using resource transmission thresholds for each media type includes: scheduling resources on the bearer network by using resource transmission thresholds for each media type to obtain transmission resources for each media type; and transmitting data for each media type separately using the transmission resources so that the data for each media type is synchronously transmitted to the terminal during the communication process.

[0074] The bearer network is the network within the communication service architecture, used to control the data transmission process for various media types. Optionally, different multimedia terminals can use the same bearer network to determine the resources for data transmission of different media types.

[0075] Transmission resources are the computer resources in a communication service architecture used for transmitting data of different media types. The size of the transmission resources controls the data transmission process for different media types. Optionally, if the resource transmission threshold for a certain media type is higher, more transmission resources are allocated for data transmission of that media type. Optionally, transmission resources include, but are not limited to, resources such as links and bandwidth used in the communication process.

[0076] In one feasible implementation, resource scheduling of the bearer network is performed based on resource transmission thresholds for each media type to obtain transmission resources for each media type. This includes sending the resource transmission thresholds to the bearer network via a Quality of Service (QoS) rule engine, enabling the bearer network to perform resource scheduling and obtain transmission resources for each media type. Therefore, as part of the communication service architecture, the bearer network can schedule its transmission resources more quickly without interacting with the session control module, efficiently controlling the data transmission process of each media type and facilitating the synchronous transmission of data across media types.

[0077] By using the resource transmission thresholds for each media type, transmission resources for each media type are scheduled, resulting in different amounts of data transmitted by different transmission resources. This allows transmission resources to be adaptively adjusted for each media type, controlling the communication process from the perspective of data transmission, so as to achieve synchronous data transmission for each media type.

[0078] In the aforementioned multimedia-type-based communication method, the data transmission order between different media types is queried based on user and business information. This allows for adaptive adjustment of the data transmission order for each media type for each user's specific service request. Furthermore, based on the media type and data transmission order, the resource transmission threshold for each media type during the communication process is determined, ensuring that the threshold is set specifically for the media type and data transmission order. By controlling the communication of data for each media type using these thresholds, the data transmission process for each media type is restricted, ensuring that the data from each media type arrives at the terminal synchronously. This allows the terminal to display the data synchronously, resulting in high data fidelity. Moreover, this process is implemented during transmission, allowing each service terminal to share the basic modules of the multimedia communication service architecture. This eliminates the need for significant adjustments based on the data transmission order, improving the scope and efficiency of service deployment.

[0079] In one embodiment, such as Figure 3 As shown, the data transmission order between different media types can be queried based on user information and business information, including:

[0080] Step 302: Determine the online terminals indicated by user information and business information.

[0081] An online terminal is a terminal connected to the bearer network, selected based on two factors: user information and service information. For different user information or different service information, the online terminal will adaptively change to adjust the media types and data transmission order during each communication. Optionally, the online terminal can be a data sensor, microphone, camera, or other device.

[0082] In one implementation, determining the online terminal indicated by user information and service information includes: mapping the user information and service information to obtain the mapped terminal; and identifying the mapped terminal in the online state as the online terminal indicated by the user information and service information. Therefore, the mapping process is performed within the communication service network architecture, and its data processing is relatively fast.

[0083] Step 304: In the data warehouse, retrieve the media types applicable to the online terminals.

[0084] The data warehouse stores information about each terminal, and can be based on this information and the media types applicable to each terminal. Optionally, the data warehouse stores two categories: online terminals and offline terminals. Offline terminals include damaged terminals and terminals that are not in use.

[0085] The media types applicable to the online terminals are those that the online terminals can use. By searching for the media types applicable to the online terminals, the media types involved in the communication process can be accurately identified, thus reducing the amount of data involved in the communication process.

[0086] In one implementation, retrieving the media types applicable to the online terminals includes: determining the terminal type of the online terminals and the media types supported by the online terminals; the terminal type is classified according to the information collection method; and retrieving the media types applicable to the online terminals based on the terminal type and the media types supported by the online terminals.

[0087] Terminal types are categorized according to information collection methods. Information collection methods characterize the information input methods of online terminals. Optionally, information collection methods include audio collection methods, video collection methods, tactile data collection methods, etc. Optionally, audio collection methods are used to identify devices such as microphones, video collection methods are used to identify devices such as cameras, and tactile data collection methods involve devices such as pressure sensors. Optionally, a terminal may have multiple information collection methods, and thus belong to multiple terminal types. Searches can be performed separately for each terminal type to more accurately determine the terminal type.

[0088] The media types supported by the online terminals are designed based on the media types that the online terminals can output. The media types supported by the online terminals are used to characterize the information output methods of the online terminals. Optionally, the media types supported by the online terminals can be adaptively adjusted according to the status of the online terminals.

[0089] In one feasible implementation, based on the terminal type and the media types supported by the online terminal, the applicable media types for the online terminal are retrieved. This includes: using the terminal type and the media types supported by the online terminal as key fields in the retrieval process; and obtaining the applicable media types for the online terminal based on the media types corresponding to the key fields. Thus, based on the terminal type and the media types supported by the online terminal, the media types jointly indicated by these two elements are determined; the media types jointly indicated by these two elements belong to the intersection of media types; and the applicable media types for the online terminal are then determined through the intersection of media types, accurately identifying the applicable media types for the online terminal.

[0090] From the perspective of information input from online terminals, the terminal type reflects the information collection method of online terminals. From the perspective of information output from online terminals, the media types supported by online terminals reflect the media types that online terminals can output. Based on this, more accurate retrieval can be performed by considering both the information input and output perspectives of online terminals, so as to make the applicable media types of online terminals more accurate.

[0091] Step 306: Determine the data transmission order among different media types according to the preset data transmission order.

[0092] The preset data delivery order is a sequence set based on user information and business information. Optionally, the preset data delivery order can be generated for a specific business after a user has signed up for it. Optionally, the preset data delivery order is designed based on online and offline terminals.

[0093] In one implementation, determining the data transmission order among various media types according to a preset data transmission order includes: determining the position of each media type within the preset data transmission order; and determining the data transmission order among the media types based on the position. The position represents the sequential order in which data is transmitted between the media types. Optionally, the position can be represented by an identifier such as a position number. This allows for a narrowing of the range of media types applicable to the online terminals, enabling a more accurate determination of the data transmission order among the various media types.

[0094] In this embodiment, user information and service information are used to determine the online terminals, thereby accurately identifying the terminals that can communicate during the current service execution. Then, from the candidate media types indicated by the service information, a search is performed for media types applicable to the online terminals. This allows the data processing and data transmission processes during communication to be adaptively adjusted for the online terminals, accurately determining the media type used in the current communication. Furthermore, according to a preset data transmission order, the data transmission order between each media type is accurately determined. Thus, based on user information and service information, the data transmission order between each media type is accurately determined, ensuring that data from each media type ultimately arrives at the terminal synchronously, enabling the terminal to display the data from each media type synchronously.

[0095] Based on this, the system meets the business requirement of coordinating multimodal synchronization thresholds according to business logic and media stream type when multimedia communication users are simultaneously transmitting multimodal media streams. It enables the coordination of multimodal synchronization threshold strategies for multimedia communication users, providing synchronized transmission of multimodal media streams for multimedia communication services and meeting users' needs for a realistic experience.

[0096] The shared server 104, a fundamental module of the multimedia communication service architecture, utilizes the multimedia communication infrastructure of the operator's network to control the data transmission order of different media types. This eliminates the need for significant adjustments to the data transmission order at each service end connected to the communication system, thereby improving the scope and efficiency of service deployment. It also provides users with collaborative multimodal synchronization thresholds in scenarios where multimodal media streams are transmitted simultaneously, enhancing the user experience. Furthermore, the data transmitted by a single multimedia communication service architecture can be provided to multiple service ends for data processing, reducing the data processing steps involved for multiple service ends, lowering service deployment costs, and maximizing the scope and efficiency of service deployment.

[0097] In a specific embodiment, such as Figure 4 As shown, the aforementioned communication method based on multimedia types constitutes a business control process. On one hand, a synchronization threshold query module is added to the session control, responsible for querying and interacting with the synchronization threshold strategy for multimodal media streams in business requests. On the other hand, the data warehouse is responsible for storing and managing user business data, business media types, media appearance order, user-connected terminals / terminal types / media types, and other data. Furthermore, a synchronization threshold management module is added to the quality service rule engine, responsible for managing the synchronization thresholds and generating strategies for multimodal media streams. In addition, the business control process also involves media computing resource scheduling, used to schedule media computing resources.

[0098] Specifically, the session control synchronization threshold query module is responsible for querying the synchronization threshold policy for multimodal media streams in business requests. Upon receiving a user business request, it initiates a multimodal threshold query request to the data warehouse based on the business ID and username to query the media types and their order of appearance involved in this business request. It then sends the media types and their order of appearance to the service quality rule engine's synchronization threshold management module to query the synchronization threshold policy for this business request. Finally, it sends the synchronization threshold policy to the media computing resource scheduling system to allocate media resources for this business request.

[0099] Data warehouse: Responsible for storing and managing user business data, business media types, media appearance order, user-connected terminals / terminal types / media types, etc. When a user registers, it records the user's connected terminals, terminal types, and media types.

[0100] The Service Quality Rule Engine Synchronization Threshold Management Module is responsible for managing synchronization thresholds and generating policies for multimodal media streams. It receives the media type and media appearance order of the current service request from the Session Control Synchronization Threshold Query Module, generates the synchronization threshold policy for this service request, and returns the policy to the Session Control Synchronization Threshold Query Module. Finally, it sends the synchronization threshold policy to the bearer network.

[0101] Optionally, the aforementioned online terminals are determined from n multimedia terminals, and the media computing power is also set for each business terminal. Each business terminal belongs to the case of distributed media access, and each business terminal adopts distributed computing.

[0102] In a specific embodiment, such as Figure 5 As shown, step 202 corresponds to Figure 5 Steps three, four, and five; step 202 corresponds to Figure 5 Steps six and seven; step 202 corresponds to Figure 5 Steps eight and nine in the process.

[0103] Step 1: The user initiates a call request.

[0104] Step 2: The session control synchronization threshold query module initiates a multimodal synchronization threshold query request to the data warehouse, with the query message carrying the username and business ID.

[0105] Step 3: The data warehouse retrieves the media type and data transmission order involved in the current call based on the username, business ID, and the terminal type and media type of the user's online terminal (e.g., if a call involves audio and haptic media, the audio media arrives before the haptic media).

[0106] Step 4: The data warehouse returns the media type and media appearance order to the session control synchronization threshold query module.

[0107] Step 5: The session control synchronization threshold query module initiates a multimodal synchronization threshold policy query request to the service quality rule engine synchronization threshold management module. The query message carries information such as media type and media appearance order.

[0108] Step Six: The Service Quality Rule Engine Synchronization Threshold Management Module generates a multimodal synchronization threshold strategy for this business request based on the media type and the order in which the media appears.

[0109] Step 7: The Service Quality Rule Engine Synchronization Threshold Management Module returns the multimodal synchronization threshold strategy to the Session Control Synchronization Threshold Query Module.

[0110] Step 8: The session control synchronization threshold query module sends the synchronization threshold policy to the media computing resource scheduling system in order to allocate media resources for this service.

[0111] Step 9: The Service Quality Rule Engine Synchronization Threshold Management Module sends the multimodal synchronization threshold policy to the bearer network to allocate network resources for this service.

[0112] In one embodiment, before querying the data transmission order between media types based on user information and business information, the method further includes: receiving a registration request sent by a multimedia user for business information; the registration request includes the multimedia user's user information to be registered, the terminal to be registered belonging to the user information to be registered, the terminal type to be registered according to the data collection method, and the media types to be registered supported by the terminal to be registered; when the registration request is approved, the terminal to be registered, the terminal type to be registered, and the media type to be registered are recorded; and the business subscription information containing business information and the user identifier to be registered is fed back to the sender of the registration request.

[0113] The user information to be registered refers to the user information during the registration process; the terminal to be registered refers to the terminal during the registration process; the terminal type to be registered refers to the terminal type categorized for the terminal to be registered; and the media type to be registered refers to the media type categorized for the terminal to be registered. The user information to be registered, the terminal to be registered, the terminal type to be registered, and the media type to be registered are, in order, settings for multimedia-based communication processes during registration. Optionally, a successful registration request indicates successful authentication.

[0114] The business contract information indicates that the registration request has been approved, and that the user information to be registered, the terminal to be registered, the terminal type to be registered, and the media type to be registered have all been recorded in the data warehouse. Optionally, after the terminal to be registered completes the contract, it can be called a registered terminal. Optionally, if the registered terminal is determined based on business information and user information, and the registered terminal is connected to the bearer network, then the registered terminal is the online terminal indicated by the business information and user information.

[0115] Therefore, for multimedia-based communication methods, the user registration process is modified, and the corresponding registration structure is fed back through business contract information. In this case, only certain fields in the registration request are adjusted, rather than setting the registration request separately. This method has relatively high processing efficiency and still enables the multimedia-based communication process to be executed accurately.

[0116] In a specific embodiment, such as Figure 6 As shown, it includes:

[0117] Step 1: User 1 initiates a registration request. The registration request message carries information such as the username, the terminal belonging to the user, the terminal type, and the media type.

[0118] Step 2: The session control module forwards the registration request to the data warehouse.

[0119] Step 3: The data warehouse authenticates and authorizes users. After successful authentication, it records the user's online terminals, terminal types, and media types.

[0120] Step 4: After successful authentication, the data warehouse returns information such as the user's business contract data.

[0121] Step 5: The session control notifies the user that registration was successful.

[0122] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0123] Based on the same inventive concept, this application also provides a multimedia-type communication device for implementing the multimedia-type communication method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more multimedia-type communication device embodiments provided below can be found in the limitations of the multimedia-type communication method described above, and will not be repeated here.

[0124] In one embodiment, such as Figure 7As shown, a multimedia-based communication device is provided, comprising:

[0125] Data warehouse module 702 is used to query the data transmission order between each media type based on user information and business information;

[0126] The threshold management module 704 is used to find the resource transmission threshold of each media type in the communication process according to each media type and the data transmission order.

[0127] The threshold management module 704 is used to control the communication of data of each media type by means of the resource transmission threshold of each media type.

[0128] In one embodiment, the data warehouse module 702 is used for:

[0129] Identify the online terminals indicated by user and business information;

[0130] In the data warehouse, retrieve the media types applicable to the online terminals;

[0131] The data transmission order among the various media types is determined according to the preset data transmission order.

[0132] In one embodiment, the data warehouse module 702 is used for:

[0133] Determine the terminal type and media types supported by the online terminals; the terminal type is classified according to the information collection method.

[0134] Based on the terminal type and the media types supported by the online terminals, retrieve the media types applicable to the online terminals.

[0135] In one embodiment, the communication process includes a data transmission process; the threshold management module 704 includes a synchronization threshold management module 704, which is used for:

[0136] Based on the data transmission order, determine the time difference threshold for each media type during transmission;

[0137] Based on the time difference threshold, a resource transmission threshold for controlling the synchronous transmission of data for each of the aforementioned media types is determined.

[0138] In one embodiment, the synchronization threshold management module 704 is used to:

[0139] By using the resource transmission threshold of each media type, media computing resources of each media type are scheduled to obtain media computing resources of each media type.

[0140] The media computing resources are used to process the data of each media type, so that the data of each media type is synchronously transmitted to the terminal in the communication process.

[0141] In one embodiment, the synchronization threshold management module 704 is used to:

[0142] By using the resource transmission thresholds for each media type, resource scheduling is performed on the bearer network to obtain the transmission resources for each media type.

[0143] Data of each media type is transmitted through the transmission resources, so that the data of each media type is synchronously transmitted to the terminal in the communication process.

[0144] In one embodiment, the apparatus further includes a media stream filtering strategy module, which is used to:

[0145] Receive a registration request sent by a multimedia user for service information; the registration request includes the user information to be registered by the multimedia user, the terminal to be registered belonging to the user information to be registered, the terminal type to be registered according to the data collection method, and the media types to be registered supported by the terminal to be registered;

[0146] When the registration request is approved, the terminal to be registered, the type of the terminal to be registered, and the type of the media to be registered are recorded;

[0147] The business contract information containing the business information and the identifier of the user to be registered is fed back to the sending end of the registration request.

[0148] The modules in the aforementioned multimedia-based communication device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0149] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores and manages user business data, business media types, media appearance order, and user-connected terminals / terminal types / media types. When a user registers, the system records the user's connected terminals, terminal types, and media types. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a multimedia-based communication method.

[0150] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0151] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0152] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0153] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0154] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant laws, regulations and standards.

[0155] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0156] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0157] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A communication method based on multimedia types, characterized in that, The method includes: Identify the online terminals indicated by user information and business information; retrieve the applicable media types for the online terminals from the data warehouse; determine the data transmission order among the media types according to a preset data transmission order; the data transmission order indicates the order in which data of each media type reaches the terminal; According to each media type and the data transmission order, find the resource transmission threshold for each media type during the communication process; the resource transmission threshold limits the data transmission efficiency of each media type; the resource transmission threshold is generated by QoS based on media type and data transmission order; By controlling the communication of data for each media type through the resource transmission threshold of each media type.

2. The method according to claim 1, characterized in that, The retrieval of media types applicable to the online terminals includes: Determine the terminal type and media types supported by the online terminals; the terminal type is classified according to the information collection method. Based on the terminal type and the media types supported by the online terminals, retrieve the media types applicable to the online terminals.

3. The method according to claim 1, characterized in that, The communication process includes a data transmission process; the step of finding the resource transmission threshold for each media type in the communication process according to each media type and the data transmission order includes: According to the data transmission order, determine the time difference threshold for each of the media types during the transmission process; Based on the time difference threshold, a resource transmission threshold for controlling the synchronous transmission of data for each of the aforementioned media types is determined.

4. The method according to claim 1, characterized in that, The step of controlling data communication for each media type by using resource transmission thresholds for each media type includes: By using the resource transmission threshold of each media type, media computing resources of each media type are scheduled to obtain media computing resources of each media type. The media computing resources are used to process the data of each media type, so that the data of each media type is synchronously transmitted to the terminal in the communication process.

5. The method according to claim 1, characterized in that, The step of controlling data communication for each media type by using resource transmission thresholds for each media type includes: By using the resource transmission thresholds for each media type, resource scheduling is performed on the bearer network to obtain the transmission resources for each media type. Data of each media type is transmitted through the transmission resources, so that the data of each media type is synchronously transmitted to the terminal in the communication process.

6. The method according to claim 1, characterized in that, Before determining the online terminal indicated by user information and business information, the process also includes: Receive a registration request sent by a multimedia user for service information; the registration request includes the user information to be registered by the multimedia user, the terminal to be registered belonging to the user information to be registered, the terminal type to be registered according to the data collection method, and the media types to be registered supported by the terminal to be registered; When the registration request is approved, the terminal to be registered, the type of the terminal to be registered, and the type of the media to be registered are recorded; The business contract information containing the business information and the identifier of the user to be registered is fed back to the sending end of the registration request.

7. The method according to claim 1, characterized in that, The data warehouse stores data corresponding to both online and offline terminals.

8. A communication device based on multimedia, characterized in that, The device includes: The data warehouse module is used to identify online terminals indicated by user information and business information; in the data warehouse, it retrieves various media types applicable to the online terminals; and determines the data transmission order among the various media types according to a preset data transmission order; the data transmission order indicates the order in which data of each media type reaches the terminal. The threshold management module is used to find the resource transmission threshold of each media type in the communication process according to each media type and the data transmission order; the resource transmission threshold limits the data transmission efficiency of each media type; the resource transmission threshold is generated by QoS based on media type and data transmission order; The threshold management module is used to control the communication of data of each media type by using the resource transmission threshold of each media type.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images