Multi-modal converged communication system and method

Abstract

The invention provides a multi-mode converged communication system and method, and relates to the technical field of communication. The system comprises a first network used for receiving multi-modal data reported by a terminal in a target emergency scene, the multi-modal data comprising data of at least one data type; the first network element is used for acquiring the multi-modal data and network data related to target emergency scene scheduling from the first network, and the network data comprises at least one of a terminal identifier, a user identifier and position information; and the second network element is used for acquiring the multi-modal data and the network data from the first network element, and generating scheduling information corresponding to the target emergency scene according to the multi-modal data and the network data. According to the scheme of the invention, fusion access of multi-modal data can be realized.

Description

Technical Field

[0001] This application relates to the field of communication technology, specifically to a multimodal fusion communication system and method. Background Technology

[0002] Unified communications systems refer to systems that integrate various types of services, such as telephone, conferencing, monitoring, intercom, and messaging, based on traditional communication technologies, to provide comprehensive converged communication services. Unified communications systems can be applied to numerous industries, including public safety, smart parks, transportation and logistics, healthcare, and energy.

[0003] Currently, converged communication systems are mostly used in emergency command scenarios in many industries such as public safety, smart parks, transportation and logistics, medical care, and energy. The converged communication system is connected to the monitoring and command center screen, which displays audio, video, and image information collected by field terminals. Commanders can remotely monitor the command center screen, analyze the monitoring data to formulate dispatch plans, and issue dispatch instructions through telephone, conference, group messages, and other means.

[0004] However, existing converged communication systems require different protocols to acquire data from different modalities. The lack of protocol uniformity makes data acquisition inconvenient, and it also requires a lot of overhead to achieve the fusion of various modal data. Summary of the Invention

[0005] At least one embodiment of this application provides a multimodal fusion communication system and method to solve the above-mentioned technical problems.

[0006] To solve the above-mentioned technical problems, this application is implemented as follows:

[0007] In a first aspect, embodiments of this application provide a multimodal fusion communication system, including:

[0008] A first network is used to receive multimodal data reported by terminals in a target emergency scenario, wherein the multimodal data includes data of at least one data type;

[0009] The first network element is used to obtain the multimodal data and network data related to the scheduling of the target emergency scenario from the first network. The network data includes at least one of terminal identifier, user identifier, and location information.

[0010] The second network element is used to obtain the multimodal data and the network data from the first network element, and generate scheduling information corresponding to the target emergency scenario based on the multimodal data and the network data.

[0011] Optionally, in the multimodal converged communication system, the first network includes an IP multimedia subsystem supporting a data channel DC, and the first network is specifically used for:

[0012] The DC of the IP multimedia subsystem receives multimodal data reported by terminals in the target emergency scenario.

[0013] Optionally, in the aforementioned multimodal fusion communication system, the first network is specifically used for:

[0014] Receive a session establishment request sent by a terminal in the target emergency scenario, wherein the session establishment request carries the data type of multimodal data;

[0015] A DC for the multimodal data transmission is established according to the session establishment request, and a session establishment response is sent to the terminal, the session establishment response carrying the data type of the multimodal data;

[0016] The multimodal data is received through the DC.

[0017] Optionally, in the multimodal fusion communication system, the first network is further used for:

[0018] Receive a first subscription message sent by the first network element, wherein the first subscription message is used to subscribe to the multimodal data and the network data;

[0019] A first notification message is sent to the first network element, the first notification message carrying the multimodal data and the network data.

[0020] Optionally, in the aforementioned multimodal fusion communication system, the first network element is further configured to:

[0021] Send a first subscription message to the first network, the first subscription message being used to subscribe to the multimodal data and the network data;

[0022] Receive a first notification message sent by the first network, the first notification message carrying the multimodal data and the network data.

[0023] Optionally, in the aforementioned multimodal fusion communication system, the first network element is further configured to:

[0024] Receive a second subscription message sent by the second network element, the second subscription message being used to subscribe to the multimodal data and the network data;

[0025] A second notification message is sent to the second network element, the second notification message carrying the multimodal data and the network data.

[0026] Optionally, in the aforementioned multimodal fusion communication system, the second network element is further configured to:

[0027] Send a second subscription message to the first network element, the second subscription message being used to subscribe to the multimodal data and the network data;

[0028] The system receives a second notification message sent by the first network element, the second notification message carrying the multimodal data and the network data.

[0029] Optionally, in the aforementioned multimodal fusion communication system, the first network element is further configured to:

[0030] Send a third subscription message to the second network element, wherein the third subscription message is used to subscribe to the scheduling information corresponding to the target emergency scenario;

[0031] Receive a third notification message sent by the second network element, the third notification message carrying the scheduling information;

[0032] Based on the scheduling information, a scheduling instruction is sent. Optionally, in the multimodal fusion communication system, the second network element is further configured to:

[0033] Receive a third subscription message sent by the first network element, wherein the third subscription message is used to subscribe to the scheduling information corresponding to the target emergency scenario;

[0034] A third notification message is sent to the first network element, the third notification message carrying the scheduling information.

[0035] Secondly, embodiments of this application provide a multimodal fusion communication method applied to a first network, the method comprising:

[0036] The system receives multimodal data reported by terminals in a target emergency scenario and generates network data related to the scheduling of the target emergency scenario; wherein the multimodal data includes data of at least one data type; and the network data includes at least one of terminal identifier, user identifier, and location information.

[0037] The multimodal data and the network data are sent to the first network element.

[0038] Optionally, in the multimodal converged communication method, the first network includes an IP multimedia subsystem supporting DC;

[0039] Receive multimodal data reported by terminals in the target emergency scenario, including:

[0040] The DC of the IP multimedia subsystem receives multimodal data reported by terminals in the target emergency scenario.

[0041] Optionally, the multimodal fusion communication method, wherein receiving multimodal data reported by a terminal in the target emergency scenario includes:

[0042] Receive a session establishment request sent by a terminal in the target emergency scenario, wherein the session establishment request carries the data type of multimodal data;

[0043] A DC for the multimodal data transmission is established according to the session establishment request, and a session establishment response is sent to the terminal, the session establishment response carrying the data type of the multimodal data;

[0044] The multimodal data is received through the DC.

[0045] Optionally, in the multimodal fusion communication method, sending the multimodal data and the network data to the first network element includes:

[0046] Receive a first subscription message sent by the first network element, wherein the first subscription message is used to subscribe to the multimodal data and the network data;

[0047] A first notification message is sent to the first network element, the first notification message carrying the multimodal data and the network data.

[0048] Thirdly, embodiments of this application provide a multimodal fusion communication method applied to a first network element, the method comprising:

[0049] The system acquires multimodal data reported by terminals in the target emergency scenario through a first network, as well as network data related to the scheduling of the target emergency scenario; wherein the multimodal data includes data of at least one data type; and the network data includes at least one of terminal identifier, user identifier, and location information.

[0050] The multimodal data and the network data are sent to the second network element.

[0051] Optionally, in the multimodal converged communication method, the first network includes an IP multimedia subsystem supporting DC;

[0052] Through the first network, acquire multimodal data reported by terminals in the target emergency scenario, including:

[0053] The multimodal data reported by the terminal in the target emergency scenario is obtained through the DC of the IP multimedia subsystem.

[0054] Optionally, the multimodal fusion communication method, wherein acquiring multimodal data reported by terminals in the target emergency scenario and acquiring network data related to the scheduling of the target emergency scenario through a first network includes:

[0055] Send a first subscription message to a first network, the first subscription message being used to subscribe to the multimodal data and the network data;

[0056] Receive a first notification message sent by the first network, the first notification message carrying the multimodal data and the network data.

[0057] Optionally, in the multimodal fusion communication method, sending the multimodal data and the network data to the second network element includes:

[0058] Receive a second subscription message sent by a second network element, the second subscription message being used to subscribe to the multimodal data and the network data;

[0059] A second notification message is sent to the second network element, the second notification message carrying the multimodal data and the network data.

[0060] Optionally, the multimodal fusion communication method further includes:

[0061] Send a third subscription message to the second network element, wherein the third subscription message is used to subscribe to the scheduling information corresponding to the target emergency scenario;

[0062] Receive a third notification message sent by the second network element, the third notification message carrying the scheduling information;

[0063] Based on the scheduling information, a scheduling instruction is sent.

[0064] Fourthly, embodiments of this application provide a multimodal fusion communication method applied to a second network element, the method comprising:

[0065] The first network element acquires multimodal data reported by terminals in the target emergency scenario, as well as network data related to the scheduling of the target emergency scenario; wherein, the multimodal data includes data of at least one data type; and the network data includes at least one of terminal identifier, user identifier, and location information.

[0066] Based on the multimodal data and the network data, scheduling information corresponding to the target emergency scenario is generated.

[0067] Optionally, the multimodal fusion communication method, wherein acquiring multimodal data reported by terminals in the target emergency scenario and acquiring network data related to the scheduling of the target emergency scenario through a first network element includes:

[0068] Send a second subscription message to the first network element, the second subscription message being used to subscribe to the multimodal data and the network data;

[0069] The system receives a second notification message sent by the first network element, the second notification message carrying the multimodal data and the network data.

[0070] Optionally, the multimodal fusion communication method further includes:

[0071] Receive a third subscription message sent by the first network element, wherein the third subscription message is used to subscribe to the scheduling information corresponding to the target emergency scenario;

[0072] A third notification message is sent to the first network element, the third notification message carrying the scheduling information.

[0073] Compared with existing technologies, the multimodal converged communication system and method provided in this application embodiment includes: a first network for receiving multimodal data reported by terminals in a target emergency scenario, the multimodal data including data of at least one data type; a first network element for acquiring the multimodal data and network data related to the scheduling of the target emergency scenario from the first network, the network data including at least one of terminal identifier, user identifier, and location information; and a second network element for acquiring the multimodal data and the network data from the first network element, and generating scheduling information corresponding to the target emergency scenario based on the multimodal data and the network data. This enables the converged access of multimodal data, reduces overhead costs, and improves the efficiency and accuracy of emergency response; and the generation of scheduling information corresponding to the target emergency scenario based on the multimodal data enables intelligent command and dispatch. Attached Figure Description

[0074] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0075] Figure 1 This is a schematic diagram of the architecture of the multimodal fusion communication system described in the embodiments of this application;

[0076] Figure 2This is one of the interactive schematic diagrams of the multimodal fusion communication system described in the embodiments of this application;

[0077] Figure 3 This is a second interactive schematic diagram of the multimodal fusion communication system described in the embodiments of this application;

[0078] Figure 4 This is the third interactive schematic diagram of the multimodal fusion communication system described in the embodiments of this application;

[0079] Figure 5 This is the fourth interactive schematic diagram of the multimodal fusion communication system described in the embodiments of this application;

[0080] Figure 6 This is a flowchart illustrating the multimodal fusion communication method according to one embodiment of this application;

[0081] Figure 7 This is a flowchart illustrating another embodiment of the multimodal fusion communication method described in this application;

[0082] Figure 8 This is a flowchart illustrating a multimodal fusion communication method according to another embodiment of this application. Detailed Implementation

[0083] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, without limiting the number of objects; for example, the first object can be one or more. Furthermore, "or" in this application indicates at least one of the connected objects. For example, "A or B" covers three scenarios: Scenario 1: including A but not B; Scenario 2: including B but not A; Scenario 3: including both A and B. The character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0084] Please refer to Figure 1 This application provides a multimodal fusion communication system, including:

[0085] A first network is used to receive multimodal data reported by terminals in a target emergency scenario, wherein the multimodal data includes data of at least one data type;

[0086] The first network element is used to obtain the multimodal data and network data related to the scheduling of the target emergency scenario from the first network. The network data includes at least one of terminal identifier, user identifier, and location information.

[0087] The second network element is used to obtain the multimodal data and the network data from the first network element, and generate scheduling information corresponding to the target emergency scenario based on the multimodal data and the network data.

[0088] In this embodiment of the application, the first network is used to receive multimodal data reported by at least one terminal in the target emergency scenario, that is, the first network is used to receive multimodal data reported by one terminal in the target emergency scenario, or the first network is used to receive multimodal data reported by each of the multiple terminals in the target emergency scenario, or the first network is used to receive single-modal data reported by each of the multiple terminals in the target emergency scenario.

[0089] Multimodal data can include data of one data type, or it can include data of multiple different data types. These data types include, but are not limited to: audio, video, text, images, and files.

[0090] Network data includes, but is not limited to:

[0091] Terminal identification, including but not limited to: International Mobile Equipment Identity (IMEI);

[0092] User identifiers for individual users include, but are not limited to: SubscriptionPermanent Identifier (SUPI) and Generic Public Subscription Identifier (GPSI);

[0093] User identifiers for vertical industries or enterprise users include, but are not limited to: Single Network Slice Selection Assistance Information (S-NSSAI), Data Network Name (DNN), 5G Virtual Network Group identifier, Closed Access Group (CAG) identifier, group identifier, and application identifier.

[0094] Location information refers to the location information corresponding to the terminal, including but not limited to: IP address, port number, Tracking Area Identity (TAI), and Cell Identity (Cell ID). Dispatch information includes but is not limited to: personnel, resources, equipment, and traffic.

[0095] Terminals can be for individual users, vertical industry users, or enterprise users, used to collect multimodal data on site, such as monitoring terminals, video conferencing terminals, mobile terminals, and applications (APPs).

[0096] Since the first network can be applied to a multimodal converged communication system, the first network can also be called a converged communication network. The first network is used to access terminals for individual users, vertical industry users or enterprise users, receive multimodal data reported by the terminals, and is also used for routing and exchanging multimodal data.

[0097] It should be noted that the second network element is used for intelligent processing of multimodal data and network data to generate scheduling information. Specifically, intelligent processing includes at least one of the following: data collection; image recognition; content recognition; data cleaning; data analysis; data classification; data association; and intelligent decision-making. The second network element can perform the above intelligent processing based on AI; therefore, this second network element can also be called an AI processing center.

[0098] In one implementation, optionally, the first network includes an IP Multimedia Subsystem (IMS) supporting a Data Channel (DC), and the first network is specifically used for:

[0099] The IMS receives multimodal data reported by terminals in the target emergency scenario through its DC.

[0100] In this embodiment of the application, multimodal data fusion access is achieved by supporting IMS for DC.

[0101] Optionally, a DC in the IMS can receive multimodal data reported by terminals in the target emergency scenario, which includes data of at least one data type.

[0102] Optionally, IMS can receive multimodal data reported by terminals in the target emergency scenario through multiple DCs, where each DC corresponds to at least one data type.

[0103] In one implementation, optionally, the first network is specifically used for:

[0104] Receive a session establishment request sent by a terminal in the target emergency scenario, wherein the session establishment request carries the data type of multimodal data;

[0105] A DC for the multimodal data transmission is established according to the session establishment request, and a session establishment response is sent to the terminal, the session establishment response carrying the data type of the multimodal data;

[0106] The multimodal data is received through the DC.

[0107] Figure 2 This is one of the interactive schematic diagrams of the multimodal fusion communication system described in an embodiment of this application. For example... Figure 2 As shown, the method includes the following steps:

[0108] 201. The terminal sends a session establishment request to the first network, which carries the data type of multimodal data.

[0109] 202. The first network receives a session establishment request, establishes a session and a DC, and sends a session establishment response to the terminal. The session establishment response carries the data type of multimodal data.

[0110] In this embodiment, the terminal sends a SIP (Session Initiation Protocol) INVITE / reINVITE request to the first network, and carries the data type of the multimodal data in the SDP (Session Description Protocol) offer. The first network completes the session and DC establishment, sends a 200 OK message to the terminal, and carries the data type of the successfully negotiated multimodal data in the SDP answer. Further, the terminal reports the multimodal data to the first network through the DC between the terminal and the first network.

[0111] In one implementation, optionally, the first network is further configured to:

[0112] Receive a first subscription message sent by the first network element, wherein the first subscription message is used to subscribe to the multimodal data and the network data;

[0113] A first notification message is sent to the first network element, the first notification message carrying the multimodal data and the network data.

[0114] In one implementation, optionally, the first network element is further configured to:

[0115] Send a first subscription message to the first network, the first subscription message being used to subscribe to the multimodal data and the network data;

[0116] Receive a first notification message sent by the first network, the first notification message carrying the multimodal data and the network data.

[0117] Figure 3 This is a second interactive schematic diagram of the multimodal fusion communication system described in an embodiment of this application. For example... Figure 3 As shown, the method includes the following steps:

[0118] 301. The first network element sends a first subscription message to the first network. This first subscription message is used to subscribe to multimodal data and network data.

[0119] 302. The first network receives the first subscription message, and when it receives multimodal data and generates network data, it sends a first notification message to the first network element. The first notification message carries multimodal data and network data.

[0120] In one implementation, optionally, the first network element is further configured to:

[0121] Receive a second subscription message sent by the second network element, the second subscription message being used to subscribe to the multimodal data and the network data;

[0122] A second notification message is sent to the second network element, the second notification message carrying the multimodal data and the network data.

[0123] In one implementation, optionally, the second network element is further used for:

[0124] Send a second subscription message to the first network element, the second subscription message being used to subscribe to the multimodal data and the network data;

[0125] The system receives a second notification message sent by the first network element, the second notification message carrying the multimodal data and the network data.

[0126] Figure 4 This is the third interactive schematic diagram of the multimodal fusion communication system described in the embodiments of this application. For example... Figure 4 As shown, the method includes the following steps:

[0127] 401. The second network element sends a second subscription message to the first network element. This second subscription message is used to subscribe to multimodal data and network data.

[0128] 402. The first network element receives the second subscription message and, upon acquiring multimodal data and network data, sends a second notification message to the second network element, the second notification message carrying multimodal data and network data.

[0129] In one implementation, optionally, the first network element is further configured to:

[0130] Send a third subscription message to the second network element, wherein the third subscription message is used to subscribe to the scheduling information corresponding to the target emergency scenario;

[0131] Receive a third notification message sent by the second network element, the third notification message carrying the scheduling information;

[0132] Based on the scheduling information, a scheduling instruction is sent.

[0133] It should be noted that the first network element is not only used to send multimodal data and network data to the second network element, but also to receive scheduling information generated by the second network element and send scheduling instructions according to the scheduling information.

[0134] Specifically, the first network element is used to make command and dispatch decisions based on dispatch information or to assist decision-makers in selecting the optimal dispatch information.

[0135] Since the first network element can acquire multimodal data of the target emergency scenario and obtain scheduling information generated by the second network element, and send scheduling instructions based on the scheduling information, the first network element can be called a multimodal scheduling center.

[0136] In one implementation, optionally, the second network element is further used for:

[0137] Receive a third subscription message sent by the first network element, wherein the third subscription message is used to subscribe to the scheduling information corresponding to the target emergency scenario;

[0138] A third notification message is sent to the first network element, the third notification message carrying the scheduling information.

[0139] Figure 5 This is the fourth interactive schematic diagram of the multimodal fusion communication system described in the embodiments of this application. Figure 5 As shown, the method includes the following steps:

[0140] 501. The first network element sends a third subscription message to the second network element. This third subscription message is used to subscribe to the scheduling information corresponding to the target emergency scenario. Optionally, the third subscription message includes, but is not limited to: intelligent processing capability identifier, time period, region, and identifier of the terminal in the target emergency scenario. Further, the intelligent processing capability identifier includes, but is not limited to: load status, congestion status, and performance status. The region can be the geographical location of the target emergency scenario; the time period is related to the time when the terminal in the target emergency scenario reports multimodal data. It should be noted that the third subscription information can subscribe to the load status, congestion status, and performance status corresponding to the region where the target emergency scenario is located within the time period.

[0141] 502. When the second network element receives the third subscription message and obtains the scheduling information corresponding to the target emergency scenario, it sends a third notification message to the first network element, which carries the scheduling information.

[0142] Please refer to Figure 6 This application provides a multimodal fusion communication method applied to a first network.

[0143] Furthermore, the above method includes the following steps:

[0144] 601, receiving multimodal data reported by a terminal in a target emergency scenario, and generating network data related to the scheduling of the target emergency scenario; wherein, the multimodal data includes data of at least one data type; the network data includes at least one of terminal identifier, user identifier, and location information;

[0145] 602, send the multimodal data and the network data to the first network element.

[0146] In one implementation, optionally, the first network includes an IMS that supports DC;

[0147] Receive multimodal data reported by terminals in the target emergency scenario, including:

[0148] The IMS's DC receives multimodal data reported by terminals in the target emergency scenario. Optionally, in one implementation, receiving the multimodal data reported by terminals in the target emergency scenario includes:

[0149] Receive a session establishment request sent by a terminal in the target emergency scenario, wherein the session establishment request carries the data type of multimodal data;

[0150] A DC for the multimodal data transmission is established according to the session establishment request, and a session establishment response is sent to the terminal, the session establishment response carrying the data type of the multimodal data;

[0151] The multimodal data is received through the DC.

[0152] In one implementation, optionally, sending the multimodal data and the network data to the first network element includes:

[0153] Receive a first subscription message sent by the first network element, wherein the first subscription message is used to subscribe to the multimodal data and the network data;

[0154] A first notification message is sent to the first network element, the first notification message carrying the multimodal data and the network data.

[0155] Please refer to Figure 7 This application provides a multimodal fusion communication method applied to a first network element.

[0156] Furthermore, the above method includes the following steps:

[0157] 701. Through the first network, acquire multimodal data reported by terminals in the target emergency scenario, and acquire network data related to the scheduling of the target emergency scenario; wherein, the multimodal data includes data of at least one data type; the network data includes at least one of terminal identifier, user identifier, and location information;

[0158] 702, send the multimodal data and the network data to the second network element.

[0159] In one implementation, optionally, the first network includes an IMS that supports DC;

[0160] Through the first network, acquire multimodal data reported by terminals in the target emergency scenario, including:

[0161] The IMS's DC acquires multimodal data reported by terminals in the target emergency scenario.

[0162] In one implementation, optionally, the system acquires multimodal data reported by terminals in the target emergency scenario and network data related to the scheduling of the target emergency scenario through a first network, including:

[0163] Send a first subscription message to a first network, the first subscription message being used to subscribe to the multimodal data and the network data;

[0164] Receive a first notification message sent by the first network, the first notification message carrying the multimodal data and the network data.

[0165] In one implementation, optionally, sending the multimodal data and the network data to the second network element includes:

[0166] Receive a second subscription message sent by the second network element, the second subscription message being used to subscribe to the multimodal data and the network data;

[0167] A second notification message is sent to the second network element, the second notification message carrying the multimodal data and the network data.

[0168] In one embodiment, optionally, the above method further includes:

[0169] Send a third subscription message to the second network element, wherein the third subscription message is used to subscribe to the scheduling information corresponding to the target emergency scenario;

[0170] Receive a third notification message sent by the second network element, the third notification message carrying the scheduling information;

[0171] Based on the scheduling information, a scheduling instruction is sent.

[0172] Please refer to Figure 8 This application provides a multimodal fusion communication method applied to a second network element.

[0173] Furthermore, the above method includes the following steps:

[0174] 801, through the first network element, acquire multimodal data reported by terminals in the target emergency scenario, and acquire network data related to the scheduling of the target emergency scenario; wherein, the multimodal data includes data of at least one data type; the network data includes at least one of terminal identifier, user identifier, and location information;

[0175] 802. Based on the multimodal data and the network data, generate scheduling information corresponding to the target emergency scenario.

[0176] In one implementation method, optionally, the first network element acquires multimodal data reported by terminals in the target emergency scenario, and acquires network data related to the scheduling of the target emergency scenario, including:

[0177] Send a second subscription message to the first network element, the second subscription message being used to subscribe to the multimodal data and the network data;

[0178] The system receives a second notification message sent by the first network element, the second notification message carrying the multimodal data and the network data.

[0179] In one embodiment, optionally, the above method further includes:

[0180] Receive a third subscription message sent by the first network element, wherein the third subscription message is used to subscribe to the scheduling information corresponding to the target emergency scenario;

[0181] A third notification message is sent to the first network element, the third notification message carrying the scheduling information.

[0182] In some embodiments, the first subscription message is a Subscribe message, and correspondingly, the first notification message is a Notify message.

[0183] In some embodiments, the second subscription message is a Subscribe message, and correspondingly, the second notification message is a Notify message.

[0184] In some embodiments, the third subscription message is a Subscribe message, and correspondingly, the third notification message is a Notify message.

[0185] In summary, this application provides a multimodal converged communication system and method. By supporting DC IMS, it enables real-time reporting of multimodal data for target emergency scenarios targeting individual users, vertical industry users, or enterprise users. It can better achieve convergence and interconnection with mobile 2G / 3G / 4G / 5G networks and fixed voice networks. Through a second network element, the multimodal data is analyzed and processed to generate scheduling information, helping decision-makers select the optimal solution and achieve intelligent scheduling, thereby improving the efficiency and accuracy of emergency response.

[0186] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0187] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0188] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A multi-modal converged communication system, characterized by, The method comprises: a first network, configured to receive multi-modal data reported by a terminal in a target emergency scenario, the multi-modal data comprising data of at least one data type; a first network element, configured to obtain the multi-modal data and network data related to dispatch of the target emergency scenario from the first network, the network data comprising at least one of a terminal identifier, a user identifier, and location information; a second network element, configured to obtain the multi-modal data and the network data from the first network element, and generate dispatch information corresponding to the target emergency scenario according to the multi-modal data and the network data.

2. The multi-modal fused communication system of claim 1, wherein, The first network comprises an IP multimedia subsystem supporting a data channel (DC), and the first network is specifically configured to: receive, through the DC of the IP multimedia subsystem, the multi-modal data reported by the terminal in the target emergency scenario.

3. The multi-modal fused communication system of claim 1 or 2, wherein, The first network is specifically configured to: receive a session establishment request sent by the terminal in the target emergency scenario, the session establishment request carrying a data type of the multi-modal data; establish a DC for transmission of the multi-modal data according to the session establishment request, and send a session establishment response to the terminal, the session establishment response carrying the data type of the multi-modal data; receive the multi-modal data through the DC.

4. The multi-modal fused communication system of claim 1, wherein, The first network is further configured to: receive a first subscription message sent by the first network element, the first subscription message being used to subscribe to the multi-modal data and the network data; send a first notification message to the first network element, the first notification message carrying the multi-modal data and the network data.

5. The multi-modal fused communication system of claim 1, wherein, The first network element is further configured to: send a first subscription message to the first network, the first subscription message being used to subscribe to the multi-modal data and the network data; receive a first notification message sent by the first network, the first notification message carrying the multi-modal data and the network data.

6. The multi-modal fused communication system of claim 1, wherein, The first network element is further configured to: receive a second subscription message sent by the second network element, the second subscription message being used to subscribe to the multi-modal data and the network data; send a second notification message to the second network element, the second notification message carrying the multi-modal data and the network data.

7. The multimodal fused communication system of claim 1, wherein, The second network element is further configured to: send a second subscription message to the first network element, the second subscription message being used to subscribe to the multi-modal data and the network data; receive a second notification message sent by the first network element, the second notification message carrying the multi-modal data and the network data.

8. The multi-modal fused communication system of claim 1, wherein, The first network element is further configured to: send a third subscription message to the second network element, the third subscription message being used to subscribe to dispatch information corresponding to the target emergency scenario; receive a third notification message sent by the second network element, the third notification message carrying the dispatch information; send a dispatch instruction according to the dispatch information.

9. The multi-modal fused communication system of claim 1, wherein, The second network element is further configured to: receive a third subscription message sent by the first network element, the third subscription message being used to subscribe to dispatch information corresponding to the target emergency scenario; send a third notification message to the first network element, the third notification message carrying the dispatch information.

10. A multi-modal converged communication method, characterized by, The method applied to the first network comprises: Receiving multi-modal data reported by a terminal in a target emergency scenario, and generating network data related to dispatch of the target emergency scenario; wherein the multi-modal data comprises data of at least one data type; and the network data comprises at least one of a terminal identifier, a user identifier, and location information; Sending the multi-modal data and the network data to a first network element.

11. The multimodal fused communication of claim 10, wherein, The first network comprises an IP multimedia subsystem supporting DC. Receiving multi-modal data reported by a terminal in a target emergency scenario, and generating network data related to dispatch of the target emergency scenario; wherein the multi-modal data comprises data of at least one data type; and the network data comprises at least one of a terminal identifier, a user identifier, and location information; Receiving multi-modal data reported by a terminal in a target emergency scenario through DC of the IP multimedia subsystem.

12. The multimodal fused communication of claim 10 or 11, wherein, Receiving multi-modal data reported by a terminal in a target emergency scenario, and generating network data related to dispatch of the target emergency scenario; wherein the multi-modal data comprises data of at least one data type; and the network data comprises at least one of a terminal identifier, a user identifier, and location information; Receiving a session establishment request sent by a terminal in a target emergency scenario, the session establishment request carrying a data type of multi-modal data; Establishing DC for transmission of the multi-modal data according to the session establishment request, and sending a session establishment response to the terminal, the session establishment response carrying the data type of the multi-modal data; Receiving the multi-modal data through the DC.

13. The multimodal fused communication of claim 10, wherein, Sending the multi-modal data and the network data to a first network element, comprising: Receiving a first subscription message sent by the first network element, the first subscription message being used for subscribing to the multi-modal data and the network data; Sending a first notification message to the first network element, the first notification message carrying the multi-modal data and the network data.

14. A multi-modal converged communication method, characterized by, Applied to a first network element, the method comprises: Obtaining multi-modal data reported by a terminal in a target emergency scenario through a first network, and obtaining network data related to dispatch of the target emergency scenario; wherein the multi-modal data comprises data of at least one data type; and the network data comprises at least one of a terminal identifier, a user identifier, and location information; Sending the multi-modal data and the network data to a second network element.

15. The multi-modal fused communication method of claim 14, wherein, The first network comprises an IP multimedia subsystem supporting DC. Obtaining multi-modal data reported by a terminal in a target emergency scenario through a first network, comprising: Obtaining multi-modal data reported by a terminal in a target emergency scenario through DC of the IP multimedia subsystem.

16. The multi-modal fused communication method of claim 14, wherein, Obtaining multi-modal data reported by a terminal in a target emergency scenario through a first network, and obtaining network data related to dispatch of the target emergency scenario, comprising: Sending a first subscription message to the first network, the first subscription message being used for subscribing to the multi-modal data and the network data; Receiving a first notification message sent by the first network, the first notification message carrying the multi-modal data and the network data.

17. The multimodal fused communication method of claim 14, wherein, Sending the multi-modal data and the network data to a second network element, comprising: Receiving a second subscription message sent by the second network element, the second subscription message being used for subscribing to the multi-modal data and the network data; Sending a second notification message to the second network element, the second notification message carrying the multi-modal data and the network data.

18. The multimodal fused communication method of claim 14, wherein, Further comprising: Sending a third subscription message to the second network element, the third subscription message being used for subscribing to dispatch information corresponding to the target emergency scenario; receive a third notification message sent by the second network element, the third notification message carrying the scheduling information; send a scheduling instruction according to the scheduling information.

19. A multi-modal converged communication method, characterized by, The method is applied to a second network element, and the method comprises: obtaining, by a first network element, multi-modal data reported by a terminal in a target emergency scenario, and obtaining network data related to scheduling of the target emergency scenario; wherein the multi-modal data comprises data of at least one data type; and the network data comprises at least one of a terminal identifier, a user identifier, and location information; generating, according to the multi-modal data and the network data, scheduling information corresponding to the target emergency scenario.

20. The multi-modal fused communication method of claim 19, wherein, obtaining, by a first network element, multi-modal data reported by a terminal in a target emergency scenario, and obtaining network data related to scheduling of the target emergency scenario, comprises: sending a second subscription message to the first network element, the second subscription message being used to subscribe to the multi-modal data and the network data; receiving a second notification message sent by the first network element, the second notification message carrying the multi-modal data and the network data.

21. The multimodal fused communication method of claim 19, wherein, Further comprising: receiving a third subscription message sent by the first network element, the third subscription message being used to subscribe to scheduling information corresponding to the target emergency scenario; sending a third notification message to the first network element, the third notification message carrying the scheduling information.

Images