Information processing method, information transmission method, terminal, system, equipment and medium

Abstract

The invention provides an information processing method, an information transmission method, a terminal, a system, equipment and a medium, and the information processing method comprises the steps: carrying out the feature extraction of collected first multi-dimensional image data, and obtaining the feature information of the first multi-dimensional image data; converting the feature information into first behavior instruction data, wherein the first behavior instruction data is used for indicating behavior information of a first user to which the first user terminal belongs; the first behavior instruction data is transmitted to a second user terminal through a dedicated channel, the dedicated channel is a data transmission channel established in advance based on the real-time communication technology, and the second user terminal is used for carrying out information reconstruction according to the first behavior instruction data to obtain reconstructed first multi-dimensional image data.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to an information processing method, an information transmission method, a terminal, a system, a device and a medium. Background Art

[0002] With the development of the fifth generation mobile communication technology (5G) and the sixth generation wireless systems (6G) networks, various new video communication methods have emerged, including virtual reality (VR), augmented reality (AR), and holographic projection.

[0003] Although the uplink and downlink rates of 5G and 6G wireless networks are getting faster and faster, and the bandwidth is getting larger and larger. But at the same time, the data transmitted by various applications is also getting larger and larger. Taking holographic video communication as an example, for life-size or 70-inch TV screen size, the dynamic transmission data volume of a single data stream is about 1.9Tbps, and the required delay is 1ms-5ms. If applied to multi-person conference scenarios, the amount of data will increase exponentially, and the existing wireless bandwidth cannot meet the requirements, and the bandwidth utilization rate is low. Summary of the invention

[0004] The present application provides an information processing method, an information transmission method, a terminal, a system, a device and a medium, which can reduce bandwidth occupancy and improve bandwidth utilization.

[0005] An embodiment of the present application provides an information processing method, which is applied to a first user terminal, and the method includes: performing feature extraction on collected first multidimensional image data to obtain feature information of the first multidimensional image data; converting the feature information into first behavior instruction data, wherein the first behavior instruction data is used to indicate behavior information of a first user to which the first user terminal belongs; transmitting the first behavior instruction data to a second user terminal via a dedicated channel, wherein the dedicated channel is a data transmission channel established in advance based on real-time communication technology, and the second user terminal is used to reconstruct information according to the first behavior instruction data to obtain reconstructed first multidimensional image data.

[0006] An embodiment of the present application provides an information processing method, which is applied to a second user terminal, and the method includes: receiving first behavior instruction data of a first user terminal through a dedicated channel; wherein the dedicated channel is a data transmission channel established in advance based on real-time communication technology, and the first behavior instruction data is behavior instruction data obtained by the first user terminal performing feature extraction on collected first multi-dimensional image data and converting the feature information of the extracted first multi-dimensional image data, and the first behavior instruction data is used to indicate the behavior information of the first user to which the first user terminal belongs; information reconstruction is performed according to the first behavior instruction data to obtain reconstructed first multi-dimensional image data.

[0007] An embodiment of the present application provides an information transmission method, which is applied to a communication system, and the method includes: receiving first behavior instruction data of a first user terminal, the first behavior instruction data is behavior instruction data obtained by the first user terminal performing feature extraction on collected first multidimensional image data and converting feature information of the extracted first multidimensional image data, and the first behavior instruction data is used to indicate behavior information of a first user to which the first user terminal belongs; transmitting the first behavior instruction data to a second user terminal through a dedicated channel, the dedicated channel is a data transmission channel established in advance based on real-time communication technology, and the second user terminal is used to reconstruct information according to the first behavior instruction data to obtain reconstructed first multidimensional image data.

[0008] An embodiment of the present application provides a first user terminal, including: an extraction module, used to extract features from collected first multidimensional image data to obtain feature information of the first multidimensional image data; a conversion module, used to convert the feature information into first behavior instruction data, wherein the first behavior instruction data is used to indicate behavior information of a first user to which the first user terminal belongs; a transmission module, used to transmit the first behavior instruction data to a second user terminal through a dedicated channel, wherein the dedicated channel is a data transmission channel established in advance based on real-time communication technology, and the second user terminal is used to reconstruct information according to the first behavior instruction data to obtain reconstructed first multidimensional image data.

[0009] An embodiment of the present application provides a second user terminal, including: a receiving module, used to receive first behavior instruction data of a first user terminal through a dedicated channel; wherein the dedicated channel is a data transmission channel established in advance based on real-time communication technology, and the first behavior instruction data is behavior instruction data obtained by the first user terminal performing feature extraction on collected first multi-dimensional image data and converting the feature information of the extracted first multi-dimensional image data, and the first behavior instruction data is used to indicate behavior information of a first user to which the first user terminal belongs; a reconstruction module, used to reconstruct information according to the first behavior instruction data to obtain reconstructed first multi-dimensional image data.

[0010] An embodiment of the present application provides a communication system, including: a receiving module, used to receive first behavior instruction data of a first user terminal, the first behavior instruction data is the behavior instruction data obtained by the first user terminal performing feature extraction on collected first multidimensional image data and converting the feature information of the extracted first multidimensional image data, and the first behavior instruction data is used to indicate the behavior information of a first user to which the first user terminal belongs; a transmission module, used to transmit the first behavior instruction data to a second user terminal through a dedicated channel, the dedicated channel is a data transmission channel pre-established based on real-time communication technology, and the second user terminal is used to reconstruct information according to the first behavior instruction data to obtain reconstructed first multidimensional image data.

[0011] An embodiment of the present application provides a network device, including: one or more processors; a memory, on which one or more programs are stored. When the one or more programs are executed by one or more processors, the one or more processors implement any one of the information processing methods in the embodiments of the present application.

[0012] An embodiment of the present application provides a storage medium, which stores a computer program. When the computer program is executed by a processor, any one of the information processing methods in the embodiments of the present application is implemented.

[0013] According to the information processing method of the embodiment of the present application, the first user terminal can extract features from the collected first multidimensional image data, convert the feature information of the extracted first multidimensional image data into first behavior instruction data, and transmit the first behavior instruction data to the second user terminal through a dedicated channel, wherein the dedicated channel is a data transmission channel pre-established based on real-time communication technology, and the second user terminal can reconstruct information based on the first behavior instruction data to obtain the reconstructed first multidimensional image data. According to the method, converting the multidimensional image data into behavior instruction data before transmitting it is conducive to reducing the traffic volume of data transmission, thereby reducing the bandwidth occupation and improving the bandwidth utilization; and real-time communication technology is a real-time communication technology, and the data transmission channel established based on real-time communication technology is a dedicated channel, which allows the user terminal to establish a point-to-point connection with the communication network without the help of an intermediate medium. Therefore, transmitting the first behavior instruction data to the second user terminal through a dedicated channel can achieve efficient data transmission.

[0014] With regard to the above embodiments and other aspects of the present application and their implementation, further description is provided in the accompanying drawings, detailed description and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A flowchart of an information processing method applied to a first user terminal provided in an embodiment of the present application is shown.

[0016] Figure 2 A schematic diagram showing the principles of holographic image acquisition and generation according to an embodiment of the present application.

[0017] Figure 3 A schematic diagram showing the principle of holographic image reconstruction according to an embodiment of the present application.

[0018] Figure 4 An information processing method applied to a second user terminal according to an embodiment of the present application is shown.

[0019] Figure 5 An information processing method applied to a communication system according to an embodiment of the present application is shown.

[0020] Figure 6 A schematic diagram showing the architecture of a communication system according to an exemplary embodiment of the present application.

[0021] Figure 7 A detailed schematic diagram of the architecture of a communication system according to an exemplary embodiment of the present application is shown.

[0022] Figure 8 A flowchart showing an information processing method of a first user terminal according to an exemplary embodiment of the present application.

[0023] Fig. 9A flowchart showing an information processing method of a second user terminal according to an exemplary embodiment of the present application.

[0024] Fig.10 A schematic diagram showing the structure of a first user terminal according to an embodiment of the present application is shown.

[0025] Fig.11 A schematic diagram showing the structure of a second user terminal according to an embodiment of the present application is shown.

[0026] Fig.12 A schematic diagram showing the structure of a communication system according to an embodiment of the present application is shown.

[0027] Fig.13 A structural diagram showing an exemplary hardware architecture of a computing device capable of implementing the method and apparatus according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present application more clear, the embodiments of the present application will be described in detail below with reference to the accompanying drawings. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other arbitrarily without conflict.

[0029] With the development of 5G and 6G networks, various new video communication methods have emerged, including VR, AR, and holographic projection. Among them, VR is a computer simulation system that can create and experience a virtual world. It uses a computer to generate a simulated environment and uses human binocular parallax to achieve a three-dimensional dynamic vision and entity behavior system simulation of multi-source information fusion, so that users can immerse themselves in the environment. AR is a technology that calculates the position and angle of camera images in real time and adds corresponding images. It is a new technology that "seamlessly" integrates real-world information and virtual-world information. The goal is to put virtual images on the real world on the screen. Holographic technology is a technology that uses the principle of interference to record and reproduce the three-dimensional image of an object by diffraction. Holographic technology can record all the information of the amplitude and phase of the light wave of an object and reproduce it, so that a stereoscopic image that is exactly the same as the original object can be obtained. The stereoscopic image is used to observe the reproduced virtual image of the hologram from different angles, and different sides of the object can be seen, with an inspection effect and a sense of depth of field.

[0030] Holography is different from ordinary photography. In ordinary photography, the light emitted by an object or the light emitted and refracted by the surface of an object can be imaged through the objective lens, and the light intensity can be recorded on the photosensitive film, and then the flat image of the object can be displayed on the photographic paper. Holography is a lens-free imaging method that uses the principle of light interference to record all the information of the object being photographed - amplitude and phase - on a holographic dry plate, so it is called holography. When a holographic photograph is reproduced, the object seen is three-dimensional and lifelike.

[0031] Digital holography is a process of recording the interference fringes produced by the coherent superposition of the light waves reflected by the object and the reference light waves through a charge coupled device (CCD) and then reconstructing the object in a computer. This process is called digital reconstruction. Since this technology can record not only the amplitude information of the object, but also the phase information of the object, it can reflect the original object more realistically.

[0032] Although the uplink and downlink rates of 5G and 6G wireless networks are getting faster and faster, and the bandwidth is getting larger and larger, at the same time, the data transmitted by various applications is also getting larger and larger. Taking holographic video communication as an example, for life-size or 70-inch TV screen size, the dynamic transmission data volume of a single data stream is about 1.9Tbps, and the required delay is 1ms-5ms. If it is applied to multi-person conference scenarios, the data volume will increase exponentially, and the existing wireless bandwidth cannot meet the requirements.

[0033] In the related art, video content can be compressed to reduce the data size of the video content. However, on the one hand, this method is usually only applicable to two-dimensional images, and is not applicable to the compression of three-dimensional images such as VR, AR, and holograms. In the related art, no technology has been found to achieve real-time holographic communication; on the other hand, due to the differences in the terminals used by the communicating parties, the use of video compression is insufficient when adapting to different types of terminals. For example, when a mobile phone communicates with a dedicated large-screen device, the picture captured on the mobile phone will become blurred when presented on the dedicated large-screen device.

[0034] Based on this, the embodiments of the present application provide an information processing method, an information transmission method, a terminal, a system, a device and a medium, which are used to reduce bandwidth occupancy and improve bandwidth utilization.

[0035] Figure 1 A flowchart of an information processing method applied to a first user terminal provided in an embodiment of the present application.

[0036] First, refer to Figure 1 , an embodiment of the present application provides an information processing method, which can be applied to a first user terminal, and the method includes:

[0037] S110, performing feature extraction on the collected first multi-dimensional image data to obtain feature information of the first multi-dimensional image data.

[0038] In this step, the multidimensional image data includes at least one of a multidimensional image and corresponding audio data. The dimension of the multidimensional is greater than or equal to 2. For example, the multidimensional image is a two-dimensional image, a three-dimensional image, or a higher-dimensional image. When the multidimensional image is a three-dimensional image, it can be at least one of VR image data, VR video data, AR image data, AR video data, holographic image data, and holographic video data.

[0039] In this step, exemplarily, when the first multidimensional image data includes a multidimensional image, the corresponding feature information includes but is not limited to: at least one of features such as color, shape, texture, depth, etc.; when the first multidimensional image data includes corresponding audio data, the corresponding feature information includes but is not limited to: at least one of tone and intonation.

[0040] The following takes the multi-dimensional image data as holographic image data as an example to illustrate the principles of collecting and generating holographic image data. Figure 2 A schematic diagram of the principles of holographic image acquisition and generation provided in an embodiment of the present application.

[0041] like Figure 2 As shown in the figure, the data acquisition device for holographic images is a laser. The laser emits a laser beam, which is split into two by a beam splitter. One beam is directed to the scene being photographed, called the object beam; the other beam is directly directed to the photosensitive film, i.e., the holographic dry plate, called the reference beam. When the beam is reflected by the object, its reflected beam is also directed to the film, and the holographic photography process is completed.

[0042] pass Figure 2 It can be seen that when collecting digital holographic images, the object light wave emitted from the object surface interferes with the reference light wave on the CCD target surface, the CCD records the light intensity distribution, and sends it to the computer for storage, forming a digital matrix, namely a digital hologram.

[0043] In this step, the first user terminal is a terminal belonging to the first user, and the first user's terminal performs a data acquisition operation, such as a digital holographic image and related sensor data.

[0044] S120: Convert the characteristic information into first behavior instruction data, where the first behavior instruction data is used to indicate behavior information of a first user to which the first user terminal belongs.

[0045] Exemplarily, the first behavior instruction data includes, for example: body rotates 15 degrees clockwise, head tilts up 45 degrees, etc. In this step, the feature information of the first multi-dimensional image data can be converted into the behavior instruction data of the corresponding user to perform behavior recognition on the first user. Behavior recognition is the process of extracting action features, representing features, and recognizing actions on a video sequence (e.g., multiple frames of images) containing user actions, and belongs to the category of semantic understanding of multi-dimensional image data.

[0046] S130, transmitting the first behavior instruction data to the second user terminal through a dedicated channel, which is a data transmission channel pre-established based on real-time communication technology. The second user terminal is used to reconstruct information according to the first behavior instruction data to obtain reconstructed first multi-dimensional image data. The reconstructed first multi-dimensional image data is used to display the behavior information of the first user on the second user terminal.

[0047] In this step, the dedicated channel can be a data transmission channel established in advance based on real-time communication technology (Web Real-Time Communication, WebRTC). The dedicated channel is different from a conventional audio and video channel. For example, the audio and video channel can be a channel established based on a streaming media protocol. The streaming media protocol is generally divided into a protocol for transmitting control messages and a protocol for transmitting media data, wherein the transmission media data message is generally used to transmit audio and video media data, such as the Real-time Transport Protocol (RTP), the Real-time Transport Control Protocol (RTCP), the Secure Real-time Transport Protocol (SRTP) and other protocols.

[0048] According to the information processing method of the embodiment of the present application, the first user terminal can perform feature extraction on the collected first multi-dimensional image data, convert the feature information of the extracted first multi-dimensional image data into first behavior instruction data, and transmit the first behavior instruction data to the second user terminal through a dedicated channel, wherein the dedicated channel is a data transmission channel pre-established based on real-time communication technology, and the second user terminal can reconstruct information according to the first behavior instruction data to obtain the reconstructed first multi-dimensional image data. According to the method, converting the multi-dimensional image data into behavior instruction data before transmitting it is conducive to reducing the traffic volume of data transmission, thereby reducing the bandwidth occupation and improving the bandwidth utilization; and the data transmission channel established based on real-time communication technology, i.e., the dedicated channel, allows the user terminal to establish a point-to-point connection with the communication network without the aid of an intermediate medium, therefore, transmitting the first behavior instruction data to the second user terminal through a dedicated channel can achieve efficient data transmission.

[0049] In this step, after receiving the first behavior instruction data, the second user terminal may perform an information reconstruction operation (ie, information reconstruction) according to the behavior instruction data, and present the reconstructed multi-dimensional image data.

[0050] The following takes the multi-dimensional image data as holographic image data as an example to illustrate the reconstruction principle of the holographic image.

[0051] Figure 3 A schematic diagram of the principle of holographic image reconstruction provided in an embodiment of the present application.

[0052] In practical applications, since the phase and amplitude information of light is recorded on the holographic film (negative), the diffracted light waves of a linearly recorded sinusoidal hologram can generally give two images under the irradiation of coherent lasers, namely the original image (also called initial image) and the conjugate image. The reproduced image has a strong three-dimensional sense and a real visual effect. Each part of the hologram records the light information of each point on the object, so in principle each part of it can reproduce the entire image of the original object. Through multiple exposures, multiple different images can be recorded on the same negative, and they can be displayed separately without interfering with each other.

[0053] like Figure 3 As shown, the projection device can project the holographic image at different angles onto a holographic projection film (transparent film), use the diffraction principle to reproduce the object's light wave information, and display the image to form a holographic projection imaging, thereby realizing the three-dimensional image reproduction of the object.

[0054] According to the information processing method of the embodiment of the present application, the first user terminal can extract features from the collected first multi-dimensional image data, convert the feature information of the extracted first multi-dimensional image data into first behavior instruction data, and transmit the first behavior instruction data to the second user terminal through a dedicated channel. The dedicated channel is a data transmission channel pre-established based on real-time communication technology. The second user terminal can reconstruct information based on the first behavior instruction data to obtain the reconstructed first multi-dimensional image data. According to this method, converting the multi-dimensional image data into behavior instruction data before transmitting it is conducive to reducing the traffic volume of data transmission, thereby reducing the bandwidth occupation and improving the bandwidth utilization; and the data transmission channel established based on real-time communication technology, i.e., the dedicated channel, allows the user terminal to establish a point-to-point connection with the communication network without the aid of an intermediate medium. Therefore, transmitting the first behavior instruction data to the second user terminal through a dedicated channel can achieve efficient data transmission.

[0055] According to this method, the information that the user terminal needs to transmit can be abstracted and described (feature extraction and transformation) to obtain the corresponding behavior instruction data, and the converted behavior instruction data can be transmitted to the user terminal of the communication counterpart through a dedicated channel in the network. The user terminal of the communication counterpart then reconstructs the information based on the behavior instruction data, and converts the received behavior instruction into multi-dimensional image data through information reconstruction. That is to say, in this method, when the first user terminal communicates with the second user terminal, it no longer transmits the collected original data itself, but transmits the abstracted behavior instruction data, which is conducive to greatly reducing the data transmission business volume. Taking holographic communication as an example, in actual scenarios, using the method provided by the embodiment of the present application, due to the significant reduction in the data transmission business volume, the bandwidth occupancy is reduced by more than 80%, which greatly improves the bandwidth utilization.

[0056] In some embodiments, the first multidimensional image data includes multidimensional image data, and the feature information is a feature of the multidimensional image data; the above step S110 may specifically include: preprocessing the multidimensional image data collected by the image sensor, the preprocessing including at least one of the following: image denoising, image alignment, posture correction and motion compensation; performing feature extraction on the preprocessed multidimensional image data to obtain multidimensional image data features.

[0057] In this embodiment, the first user terminal performs data preprocessing, such as image denoising, image alignment, posture correction, motion compensation, etc., to improve the quality and availability of data.

[0058] In some embodiments, the step of converting the feature information into the first behavior instruction data in step S120 may specifically include: matching the feature information with a semantic knowledge base to obtain semantic information corresponding to the feature information, the semantic information including descriptive information of the entity behavior; converting the descriptive information of the entity behavior into corresponding first behavior instruction data.

[0059] Exemplarily, the semantic knowledge base may include knowledge in various fields. If the multidimensional image data is holographic image data, the semantic knowledge base includes indications in the holographic field, such as at least one of entities, relationships, events, etc. Among them, an event is a knowledge fragment that can be triggered by an action or a state change.

[0060] Exemplarily, the semantic knowledge base may include semantic information of multiple multidimensional image data. The feature information of the first multidimensional image data is calculated to have a similarity with the feature information of all multidimensional image data in the semantic knowledge base to obtain image data with the highest similarity. The semantic information of the multidimensional image data with the highest similarity is used as the corresponding semantic information matched to the feature information of the first multidimensional image data.

[0061] In this embodiment, the first user terminal matches and analyzes the extracted feature information with the semantic knowledge base to obtain semantic information, and then semantically represents the semantic information to obtain first behavior instruction data. After identifying the multidimensional image data as the corresponding behavior instruction data, the amount of data transmission can be greatly reduced.

[0062] In some embodiments, the dedicated channel includes: a first dedicated channel between the first user terminal and the first communication network, and a second dedicated channel between the second user terminal and the second communication network.

[0063] In this embodiment, before the step of transmitting the first behavior instruction data to the second user terminal through the dedicated channel in step S130, the method further includes the following steps:

[0064] S11, sending a first session request to a first communication network corresponding to a first user terminal, wherein the first session request includes at least: first dedicated channel media information and service type indication information of the first user terminal, and the first communication network is used to: forward the first dedicated channel media information and service type indication information to the second user terminal through a second communication network corresponding to the second user terminal;

[0065] S12, receiving, through the first communication network, a session response from the second user terminal forwarded by the second communication network; wherein the session response is a session response generated and sent to the second communication network by the second user terminal after receiving the first dedicated channel media information and the service type indication information, and the session response includes the second dedicated channel media information of the second user terminal; the second communication network is further used to: establish a second dedicated channel with the second user terminal according to the received second dedicated channel media information;

[0066] S13: In response to the session response, establish a first dedicated channel with the first communication network according to the first dedicated channel media information.

[0067] In the above steps S11-S13, the dedicated channel is composed of a first dedicated channel, a first communication network, a second communication network and a second dedicated channel. When establishing the first dedicated channel and the second dedicated channel, it is necessary to indicate one's own media information to the participating parties, such as the user terminal of the communication counterpart, and to use the service type indication information to indicate a specific type of call service. For example, when the service type indication information is a holographic communication media mark, it is used to indicate that the session request is used to implement the holographic communication service.

[0068] In some embodiments, in step S130, the first behavior instruction data is transmitted to the second user terminal through a dedicated channel, which may specifically include: sending the first behavior instruction data to the first communication network through the first dedicated channel; wherein the first communication network is used to send the first behavior instruction data to the second communication network, and the second communication network is used to send the first behavior instruction data to the second user terminal through the second dedicated channel.

[0069] In this embodiment, the first behavior instruction data is transmitted through a pre-established dedicated channel (including a first dedicated channel and a second dedicated channel). The dedicated channel is different from the audio and video channel, and can realize a point-to-point connection between the user terminal and the communication network. The first behavior instruction data is transmitted to the second user terminal through the dedicated channel, which can realize efficient data transmission.

[0070] In some embodiments, the first session request further includes: first reconstruction capability information, where the first reconstruction capability information is used to indicate the information reconstruction capability of the first user terminal.

[0071] In this embodiment, the first communication network is also used to: send a first session request to a first application server; wherein the first application server is used to select corresponding first model data for the first user terminal based on the reconstruction capability information of the first user terminal, and instruct the first communication network to send a second session request to the second communication network, the second session request including: first dedicated channel media information and service type indication information; and, before sending the session response to the second user terminal, add an identifier of the first model data in the session response.

[0072] The second communication network is also used to: send a second session request to the second application server corresponding to the second user terminal, the second application server is used to select the corresponding second model data for the second user terminal according to the stored reconstruction capability information of the second user terminal, and instruct the second communication network to send a third session request to the second user terminal, the third session request includes: the first dedicated channel media information, the service type indication information and the identifier of the second model data.

[0073] The second user terminal is also used to: after downloading the second model data through the second dedicated channel, perform feature extraction on the collected second multidimensional image data through the second model data and convert the feature information of the extracted second multidimensional image data into second behavior instruction data, and send the second behavior instruction data to the second communication network through the second dedicated channel.

[0074] In this embodiment, after the above-mentioned step S13, the method also includes: downloading first model data through a first dedicated channel; receiving second behavior instruction data from the first communication network through the first dedicated channel, the second behavior instruction data being data transmitted from the second communication network to the second communication network; reconstructing information of the second behavior instruction data according to the first model data to obtain reconstructed second multidimensional image data.

[0075] In this embodiment, the first application server can select a corresponding model package for the user terminal based on the information reconstruction capability of the user terminal, so that the user terminal can download the model package through a dedicated channel, and use the downloaded model package to reconstruct the received behavior instruction data to obtain reconstructed multi-dimensional image data.

[0076] Through the information processing method of the embodiment of the present application, the first user terminal can extract features from the collected first multi-dimensional image data, convert the feature information of the extracted first multi-dimensional image data into first behavior instruction data, and transmit the first behavior instruction data to the second user terminal through a dedicated channel. The dedicated channel is a data transmission channel pre-established based on real-time communication technology. The second user terminal can reconstruct information based on the first behavior instruction data to obtain the reconstructed first multi-dimensional image data. According to this method, converting the multi-dimensional image data into behavior instruction data before transmitting it is conducive to reducing the traffic volume of data transmission, thereby reducing the bandwidth occupation and improving the bandwidth utilization; and the data transmission channel established based on real-time communication technology, i.e., the dedicated channel, allows the user terminal to establish a point-to-point connection with the communication network without the help of an intermediate medium. Therefore, transmitting the first behavior instruction data to the second user terminal through a dedicated channel can achieve efficient data transmission.

[0077] Figure 4 An information processing method applied to a second user terminal provided in an embodiment of the present application.

[0078] Second, refer to Figure 4 , an embodiment of the present application provides an information processing method, which can be applied to a second user terminal, and the method includes:

[0079] S410, receiving first behavior instruction data of the first user terminal through a dedicated channel; wherein the dedicated channel is a data transmission channel pre-established based on real-time communication technology, and the first behavior instruction data is behavior instruction data obtained by the first user terminal performing feature extraction on the collected first multi-dimensional image data and converting the feature information of the extracted first multi-dimensional image data, and the first behavior instruction data is used to indicate the behavior information of the first user to which the first user terminal belongs.

[0080] S420, reconstructing information according to the first line of instruction data to obtain reconstructed first multi-dimensional image data.

[0081] According to the information processing method of the embodiment of the present application, the second user terminal can receive the first behavior instruction data of the first user terminal through a dedicated channel, and the first behavior instruction data is the behavior instruction data of the first user terminal performing feature extraction on the collected first multidimensional image data, converting the feature information of the extracted first multidimensional image data into the first behavior instruction data, and transmitting the first behavior instruction data to the second user terminal through the dedicated channel, and the dedicated channel is a data transmission channel established in advance based on real-time communication technology, and the second user terminal can reconstruct the information according to the first behavior instruction data to obtain the reconstructed first multidimensional image data.

[0082] According to the information processing method of this embodiment, multi-dimensional image data can be converted into behavior instruction data before transmission, which is beneficial to reducing the traffic volume of data transmission, thereby reducing the occupancy of bandwidth and improving bandwidth utilization; and, the data transmission channel established based on real-time communication technology, i.e., the dedicated channel, allows the user terminal to establish a point-to-point connection with the communication network without the aid of an intermediate medium. Therefore, the first behavior instruction data can be transmitted to the second user terminal through the dedicated channel, thereby realizing efficient data transmission.

[0083] In some embodiments, the dedicated channel includes: a first dedicated channel between the first user terminal and the first communication network, and a second dedicated channel between the second user terminal and the second communication network.

[0084] The first communication network is used to: send the received first session request of the first user terminal to the first application server, the first application server is used to select the corresponding first model data for the first user terminal according to the reconstruction capability information of the first user terminal, and instruct the first communication network to send a second session request to the second communication network, wherein the second session request includes: first dedicated channel media information and service type indication information;

[0085] The second communication network is used to: send a second session request to a second application server corresponding to the second user terminal, the second application server is used to select corresponding second model data for the second user terminal according to the stored reconstruction capability information of the second user terminal, and instruct the second communication network to send a third session request to the second user terminal, the third session request includes: first dedicated channel media information, service type indication information and an identifier of the second model data.

[0086] In this embodiment, before step S410, the information processing method also includes: receiving a third session request sent by the second communication network; sending a session response to the second communication network in response to the third session request, wherein the session response includes second dedicated channel media information of the second user terminal; and establishing a second dedicated channel with the second communication network through the second dedicated channel media information.

[0087] In this embodiment, the second user terminal can obtain the first dedicated channel media information, service type indication information and the identifier of the second model data carried in the third session request through the third session request, and determine the service type of this communication according to the information carried in the third session request, and download the corresponding model package through the identifier of the second model data; and, it can also carry its own dedicated channel media information, i.e., the second dedicated channel media information, through the session response, so as to communicate and negotiate with the first user terminal through the session response.

[0088] In some embodiments, step S420 may specifically include: downloading second model data through a second dedicated channel; and reconstructing information of the first behavior instruction data through the second model data to obtain reconstructed first multi-dimensional image data.

[0089] In this embodiment, the second model data is downloaded through the second dedicated channel, and the first behavior instruction data is processed through the second model data to obtain the reconstructed first reconstruction data. The first dedicated channel and the second dedicated channel are different from the audio and video channels. The dedicated channel can realize the point-to-point connection between the corresponding user terminal and the communication network. The first behavior instruction data is transmitted to the second user terminal through the dedicated channel, which can realize efficient data transmission.

[0090] In some embodiments, the first user terminal is also used to: in response to a session response, establish a first dedicated channel with the first communication network according to the first dedicated channel media information, and download the first model data through the first dedicated channel; after downloading the second model data through the second dedicated channel in the above steps, the processing method also includes: extracting features from the collected second multidimensional image data, and converting the extracted feature information of the second multidimensional image data into second behavior instruction data; sending the second behavior instruction data to the second communication network through the second dedicated channel; wherein the second communication network is used to transmit the second behavior instruction data to the first communication network, so that the first communication network transmits the second behavior instruction data to the first user terminal through the first dedicated channel; the first user terminal is used to: reconstruct the second behavior instruction data according to the first model data to obtain reconstructed second multidimensional image data.

[0091] In this embodiment, the second user terminal can also perform feature extraction and conversion on the collected second multidimensional image data to obtain second behavior instruction data, and transmit the second behavior instruction data to the first user terminal through a dedicated channel, so that the first user terminal can reconstruct the information of the second behavior instruction data according to the model package corresponding to its own reconstruction capability to obtain reconstructed second multidimensional image data.

[0092] According to the information processing method of the embodiment of the present application, the second user terminal can receive the first behavior instruction data of the first user terminal through a dedicated channel, and the first behavior instruction data is the behavior instruction data of the first user terminal performing feature extraction on the collected first multidimensional image data, converting the feature information of the extracted first multidimensional image data into the first behavior instruction data, and transmitting the first behavior instruction data to the second user terminal through the dedicated channel, and the dedicated channel is a data transmission channel established in advance based on real-time communication technology, and the second user terminal can reconstruct the information according to the first behavior instruction data to obtain the reconstructed first multidimensional image data.

[0093] According to the information processing method of this embodiment, multi-dimensional image data can be converted into behavior instruction data before transmission, which is beneficial to reducing the traffic volume of data transmission, thereby reducing the occupancy of bandwidth and improving bandwidth utilization; and, the data transmission channel established based on real-time communication technology, i.e., the dedicated channel, allows the user terminal to establish a point-to-point connection with the communication network without the aid of an intermediate medium. Therefore, the first behavior instruction data can be transmitted to the second user terminal through the dedicated channel, thereby realizing efficient data transmission.

[0094] Figure 5 An information processing method applied to a communication system provided in an embodiment of the present application.

[0095] Thirdly, refer to Figure 5 , an embodiment of the present application provides an information transmission method, which can be applied to a communication system, and the method includes:

[0096] S510, receiving first behavior instruction data from a first user terminal, where the first behavior instruction data is obtained by the first user terminal performing feature extraction on collected first multi-dimensional image data and converting feature information of the extracted first multi-dimensional image data, and the first behavior instruction data is used to indicate behavior information of a first user to which the first user terminal belongs;

[0097] S520, transmitting the first behavior instruction data to the second user terminal through a dedicated channel, where the dedicated channel is a data transmission channel pre-established based on real-time communication technology, and the second user terminal is used to reconstruct information according to the first behavior instruction data to obtain reconstructed first multi-dimensional image data.

[0098] Through the information transmission method of the embodiment of the present application, the communication system can receive first behavior instruction data from a first user terminal, and transmit the first behavior instruction data to a second user terminal through a dedicated channel, the first behavior instruction data being behavior instruction data of the first user terminal performing feature extraction on the collected first multidimensional image data, converting the feature information of the extracted first multidimensional image data into first behavior instruction data, and transmitting the first behavior instruction data to the second user terminal through a dedicated channel, and the dedicated channel is a data transmission channel established in advance based on real-time communication technology, and the second user terminal can reconstruct information according to the first behavior instruction data to obtain reconstructed first multidimensional image data.

[0099] In some embodiments, the dedicated channel includes a first dedicated channel and a second dedicated channel; the communication system includes at least a first communication network and a second communication network; before receiving the first behavior instruction data of the first user terminal, the method further includes the following steps:

[0100] S21, receiving a first session request of a first user terminal through a first communication network, wherein the first session request at least includes: first dedicated channel media information and service type indication information of the first user terminal; S22, sending the first dedicated channel media information and service type indication information to a second communication network through the first communication network, where the second communication network is a communication network corresponding to an identifier of the second user terminal; S23, sending the first dedicated channel media information and service type indication information to the second user terminal through the second communication network; S24, receiving and forwarding a session response of the second user terminal to the first communication network through the second communication network, wherein the session response includes second dedicated channel media information of the second user terminal; S25, sending second dedicated channel media information to the first user terminal through the first communication network, and establishing a first dedicated channel with the first user terminal according to the first dedicated channel media information; S26, establishing a second dedicated channel with the second user terminal through the second communication network using the second dedicated channel media information.

[0101] In this embodiment, a first dedicated channel is established between the first user terminal and the first communication network, and a second dedicated channel is established between the second user terminal and the second communication network. The first dedicated channel and the second dedicated channel are different from the audio and video channels, and a point-to-point connection between the user terminal and the communication network can be achieved. The first behavior instruction data is transmitted to the second user terminal through the dedicated channel, so that efficient data transmission can be achieved.

[0102] In some embodiments, the communication system also includes: a first application server and a second application server; the above step S22 may specifically include: sending a first session request to the first application server through the first communication network, the first session request also including reconstruction capability information of the first user terminal; the first application server selects the corresponding first model data for the first user terminal according to the reconstruction capability information of the first user terminal, and instructs the first communication network to send a second session request to the second communication network, the second session request including: first dedicated channel media information and service type indication information.

[0103] In this embodiment, the above step S23 may specifically include: sending a second session request to a second application server through a second communication network; the second application server selects corresponding second model data for the second user terminal based on the stored reconstruction capability information of the second user terminal, and instructs the second communication network to send a third session request to the second user terminal, wherein the third session request includes: first dedicated channel media information, service type indication information, and an identifier of the second model data.

[0104] In this embodiment, the above step S25 may specifically include: adding an identifier of the first model data in a session response through the first communication network; and sending the session response carrying the identifier of the first model data and the second dedicated channel media information to the first user terminal.

[0105] In this embodiment, the first application server is used to select corresponding model data for the first user terminal based on the reconstruction capability information of the first user terminal, and the second application server is used to select corresponding model data for the second user terminal based on the reconstruction capability information of the second user terminal, so as to be used for subsequent information reconstruction of the behavior instruction data received from the communication counterpart.

[0106] In some embodiments, the step of transmitting the first behavior instruction data to the second user terminal through a dedicated channel in step S520 may specifically include: sending the first behavior instruction data received through the first dedicated channel to the second communication network; in the second communication network, sending the first behavior instruction data to the terminal device through the second dedicated channel.

[0107] In this embodiment, the dedicated channel is composed of a first dedicated channel, a first communication network, a second communication network and a second dedicated channel, and efficient transmission of behavior instruction data is achieved based on the dedicated channel.

[0108] In some embodiments, after the above-mentioned step S26, the method further includes: receiving second behavior instruction data from a second user terminal through a second dedicated channel in a second communication network, wherein the second behavior instruction data is behavior instruction data obtained by the second user terminal through second model data by performing feature extraction on the collected second multidimensional image data and converting the feature information of the extracted second multidimensional image data; transmitting the second behavior instruction data to the first communication network through the second communication network; and sending the second behavior instruction data to the first user terminal through the first dedicated channel in the first communication network, and the first user terminal is used to: reconstruct the second behavior instruction data according to the first model data to obtain reconstructed second multidimensional image data.

[0109] In this embodiment, the second user terminal can extract and convert features of the collected second multidimensional image data through a dedicated channel to obtain second behavior instruction data, and transmit the second behavior instruction data to the first user terminal through the dedicated channel, so that the first user terminal can reconstruct information on the second behavior instruction data according to a model package corresponding to its own reconstruction capability to obtain reconstructed second multidimensional image data.

[0110] Figure 6 Schematic diagram of the architecture of the communication system of the exemplary embodiment of the present application. Take the communication system corresponding to the terminal on the communication side as an example. Figure 6As shown, the communication system may include: a communication network 610 and an application service 620, and the application scenario of the communication system also includes: an information reconstruction terminal 1 (a first user terminal) and an information reconstruction terminal 2 (a second user terminal).

[0111] The communication network 610 is used to support the negotiation of a newly added dedicated channel (which is different from the audio and video channel), and provides authentication for terminal access, addressing between terminals and between terminals and application servers, and the transmission of behavior instruction data; the application service 620 is used to add model management functions, including the storage, addition, update, and deletion of information models and reconstruction models. In the process of establishing communication, it supports the negotiation of a newly added dedicated channel, the selection of corresponding model packages for different types of terminals, and the forwarding of one-to-many behavior instructions.

[0112] exist Figure 6 In the embodiment, the first user terminal and the second user terminal may use the same communication network. In an actual application scenario, the first user terminal and the second user terminal may also use different communication networks.

[0113] Combine the following Figure 6 , describing the specific process of the information processing method and information transmission method of the embodiment of the present application.

[0114] In some embodiments, the method comprises the steps of:

[0115] S701: A first user terminal sends a first session request to a communication network of a first user.

[0116] Among them, the first session request includes dedicated channel media information of the first user terminal, a specific type of call service indication, and capability characteristics supporting information reconstruction, etc. The specific type of call service indication is service type indication information, such as holographic communication service, VR communication service, AR communication service, etc.

[0117] S702: The first communication network receives a first session request, and reports the first session request to a first application server according to a specific type of call service indication in the first session request.

[0118] The first application server is a server corresponding to the application service of the first user, and the type of the application service corresponds to the business type.

[0119] S703: The first application server selects corresponding model data for the first user terminal by analyzing the information reconstruction capability of the first user terminal in the first session request.

[0120] Exemplarily, the information reconstruction capability includes at least one of the following: imaging capability, GPU capability, screen size, etc. The model data is used to subsequently instruct the first user terminal to use the model data for information generation and information reconstruction.

[0121] S704: The first application server instructs the first communication network to continue the call, which carries dedicated channel media information and a specific type of call service indication.

[0122] In this step, the first application server may instruct the first communication network to continue the call through a second session request, where the second session request carries dedicated channel media information and a specific type of call service indication, that is, a capability feature of information reconstruction without the first user terminal.

[0123] S705: The first communication network sends a session request to the second communication network according to the identification information of the second user terminal.

[0124] In this step, the identification information of the second user terminal may be a user number, for example including but not limited to at least one of the International Mobile Subscriber Identity (IMSI) and the Mobile Station International Subscriber Directory Number (MSISDN) of the first user. The first communication network may be connected to the network corresponding to the identification information through the identification information of the second user terminal.

[0125] S706: The second communication network receives the session request and reports session information to the second application server according to the specific type of call service indication in the session request.

[0126] S707: The second application server selects corresponding model data for the second user terminal according to the information reconstruction capability of the second user terminal when it is registered, so as to instruct the second user terminal to use the model data for information generation and information reconstruction.

[0127] S708, the second application server instructs the second communication network to continue the call, which carries the dedicated channel media information, the specific type of call service indication, and the identifier of the second model data.

[0128] In this step, the second application server may instruct the second communication network to continue the call through a third session request, where the third session request carries a specific type of call service indication and a model data identifier required for information generation and information reconstruction.

[0129] S709: The second communication network sends a session request to the second user terminal according to the identification information of the second user terminal.

[0130] In this step, the identification information of the second user terminal may be a user number, for example including but not limited to at least one of the IMSI and MSISDN of the second user.

[0131] S710, the second user terminal determines that the information generation and reconstruction function is to be used this time, and performs initialization of terminal-related capabilities based on the information in the session request, such as dedicated channel media information, a specific type of call service indication, and a model data identifier required for information generation and information reconstruction.

[0132] In this step, the initialization may include, for example, the initialization of the information generation function and the initialization of the information reconstruction function.

[0133] S711, the second user terminal returns a session response to the second communication network, which carries local dedicated channel media information.

[0134] S712: The second communication network forwards the session response to the first communication network.

[0135] S713: The first communication network inserts a model data identifier required for information generation and information reconstruction used by the first user terminal into the session response, and forwards the session response to the first user terminal.

[0136] S714, after receiving the session response, the first user terminal establishes a dedicated channel with the first communication network, and downloads the specified model data through the dedicated channel.

[0137] S715, after confirming that the first user has received the session response, the second user terminal establishes a dedicated channel with the second communication network and downloads the specified model data through the dedicated channel.

[0138] S716, after the second user terminal answers the call, it uses the terminal hardware to collect information, generates information using the downloaded model data, and sends the second user's behavior instruction data to the second communication network through a dedicated channel.

[0139] S717: The second communication network forwards the behavior instruction data of the second user to the first communication network through a dedicated channel.

[0140] S718: The first communication network forwards the behavior instruction data of the second user to the first user terminal through a dedicated channel.

[0141] S719: The first user terminal reconstructs the behavior instruction data of the second user by using the downloaded model data, and presents the reconstructed data on the first user terminal.

[0142] S720, the first user uses the terminal hardware to collect information, generates information using the downloaded model data, and sends the first user's behavior instruction data to the first communication network through a dedicated channel.

[0143] S721, the first communication network forwards the behavior instruction data of the first user to the second communication network through a dedicated channel.

[0144] S722: The second communication network forwards the behavior instruction data of the first user to the second user terminal through a dedicated channel.

[0145] S723: The second user terminal reconstructs the behavior instruction data of the first user by using the downloaded model data, and presents it to the second user.

[0146] Through the above steps S701-S723, real-time communication of three-dimensional information can be completed based on the information reconstruction method.

[0147] Figure 7 A detailed schematic diagram of the architecture of a communication system according to an exemplary embodiment of the present application. Figure 7 and Figure 6 The same or equivalent sections are followed by the same reference numerals. Figure 7 As shown, in the first communication system, the first communication network 610 includes: a session edge controller (Session Border Controller, SBC) 611, a call session control function (InterrogationCall Session Control Function, CSCF) node 612, a multimedia system application server 613 and a service interaction platform 614.

[0148] Among them, the session edge controller 611 can serve as an access module in the Internet Protocol Multimedia System (IP Multimedia System, IMS), responsible for providing access services to information reconstruction terminals such as the first user terminal, the second user terminal, etc., and can support the establishment of dedicated channels and the forwarding of behavior instruction data.

[0149] The call session control function node 612 can be used as a session control and routing module in the IMS, providing an interrogation call session control function (I-CSCF) and a serving call session control function (S-CSCF). Among them, the main functions of the I-CSCF include at least one of the following: assigning a serving call session control function (S-CSCF) to the user when the user registers, and being responsible for sending the session to the S-CSCF registered by the user or a designated S-CSCF in the business process; the main functions of the S-CSCF include at least one of the following: providing registration and authentication for mobile phones and AR devices; providing session initialization protocol (SIP) session management, including session establishment, session refresh, session release, session maintenance, emergency calls and message management, etc.; providing service triggering function to trigger the call to the corresponding multimedia system application server. The multimedia system application server can be, for example: Voice over Long-Term Evolution Application Server (VoLTE AS).

[0150] The multimedia system application server 613 can be, for example, a VoLTE AS. As an application service in the IMS, the multimedia system application can provide basic supplementary services, call continuity and other functions. In an embodiment of the present application, the VoLTE AS reports the call event to the service interaction platform, receives the control instruction issued by the service interaction platform, and implements the management of the dedicated data channel, such as at least one of creation, modification and deletion.

[0151] The business interaction platform 614 provides a unified service portal for applications and terminals. The specific functions include at least one of the following:

[0152] Provide subscription for user registration events or call events; provide automatic addressing for different services; provide control capability for in-call signaling, including but not limited to connecting calls, rejecting calls, playing audio to terminals, etc.; provide control capability for in-call media, including but not limited to anchoring the terminal audio and video media to the media plane, anchoring the terminal's dedicated data channel to the media plane, and management functions for audio and video media channels and dedicated data channels, such as at least one of creation, modification, and deletion; provide forwarding control capability for behavioral instruction data.

[0153] like Figure 7As shown, the communication system may also include: a Home Subscriber Server (HSS) 615, which may be responsible for storing user authentication information, specific information of contracted users, dynamic information of contracted users, etc., and cooperating with CSCF to complete registration and authentication of information reconstruction terminals; providing storage services and transparent data storage for VoLTE AS, etc.

[0154] In some embodiments, the first communication network includes: a first session edge controller, a first call session control function node, a first multimedia system application server, and a first business interaction platform connected in sequence, wherein the first session edge controller is connected to a first user terminal, and the first business interaction platform is connected to the first application server; the second communication network includes: a second session edge controller, a second call session control function node, a second multimedia system application server, and a second business interaction platform connected in sequence, wherein the second session edge controller is connected to a second user terminal, and the second business interaction platform is connected to the second application server; and wherein the first call session control function node is connected to the second multimedia system application server for communication transmission from the first communication network to the second communication network; and the second call session control function node is connected to the first multimedia system application server for communication transmission from the second communication network to the second communication network.

[0155] In this embodiment, in the communication network, the first communication network implements the functions of the communication network through the first session edge controller, the first call session control function node, the first multimedia system application server, and the first service interaction platform.

[0156] Figure 8 A flowchart of an information processing method of a first user terminal provided by an exemplary embodiment of the present application.

[0157] like Figure 8 As shown, the information processing method includes the following steps.

[0158] S801, data acquisition.

[0159] In this step, the first user terminal obtains data transmitted by holographic communication, such as holographic images, related sensor data, etc., through hardware facilities.

[0160] S802, data preprocessing.

[0161] In this step, the first user terminal preprocesses the acquired data, including image denoising, image alignment, posture correction, motion compensation and other processing, to improve the quality and availability of the data.

[0162] S803, feature extraction.

[0163] In this step, the first user terminal extracts feature information in the holographic image from the preprocessed data, such as color, shape, texture, depth and other features.

[0164] S804, semantic analysis.

[0165] In this step, the first user terminal matches and analyzes the extracted feature information with the semantic knowledge base to obtain semantic information. The semantic knowledge base may include knowledge in the holographic field, such as entities, relationships, events, etc.

[0166] S805, semantic representation.

[0167] In this step, the first user terminal represents and stores the semantic information obtained by the analysis to obtain corresponding behavior instruction data, for example, blink the left eye three times, each blink lasting 50 milliseconds, etc.

[0168] S806, data transmission.

[0169] In this step, the first user terminal establishes a session with the communication network and transmits the information data completed with the semantic representation, that is, the behavior instruction data obtained by the semantic representation, to a remote device, such as the second user terminal.

[0170] Through the above steps S801-S806, any user terminal can convert the acquired data for holographic communication into corresponding behavior instruction data, and transmit the behavior instruction data to the second user terminal to realize information reconstruction in the second user terminal.

[0171] Fig. 9 A flowchart of an information processing method of a second user terminal provided by an exemplary embodiment of the present application.

[0172] like Fig. 9 As shown, the information processing method includes the following steps.

[0173] S901, information presentation.

[0174] In this step, the second user terminal may establish a session with the communication network to receive the behavior instruction data transmitted in the communication network, that is, the behavior information data represented by semantics.

[0175] S902, semantic analysis.

[0176] In this step, the second user terminal matches and analyzes the received information data with the semantic knowledge base to obtain the semantic information corresponding to the behavioral instruction to be presented, for example, blink the left eye 3 times, each blink lasting 50 milliseconds.

[0177] S903, data transmission.

[0178] In this step, the second user terminal loads the corresponding data model according to the semantic information, applies the semantic information to the existing scene, and uses the terminal's own hardware capabilities to present information such as sound and images on the second user terminal.

[0179] Through the above steps S901-S903, the second user terminal can receive the first behavior instruction data of the first user terminal through the dedicated channel, and reconstruct information according to the first behavior instruction data to obtain reconstructed first multi-dimensional image data.

[0180] The following takes the implementation of holographic communication in an IMS network, where the first user terminal and the second user terminal can use the same IMS network as an example, to describe the process of two user terminals of the same type supporting holographic communication capabilities using the method of an embodiment of the present application to interact. It should be understood that the first user terminal and the second user terminal can use the same IMS network or different IMS networks, and the processing flow corresponding to using the same IMS network or different IMS networks is the same. In some embodiments, the method includes the following steps:

[0181] S1001, a first user terminal sends an INVITE request to an SBC, which carries dedicated channel media information of the first user terminal, a service indication of holographic communication, and terminal capability information.

[0182] Exemplarily, the holographic communication service indication may be implemented by the following method:

[0183] A holographic communication media label is added to the media line (m=line) of the application layer control protocol (Session Description Protocol, SDP) of the multimedia session.

[0184] m=application 10000UDP / DTLS / SCTP webrtc-datachannel

[0185] a=dcmap:1002label=”holo”

[0186] In WebRTC, point-to-point message communication is achieved through the data channel using the User Datagram Protocol (UDP), the Datagram Transport Layer Security (DTLS), or the Stream Control Transmission Protocol (SCTP).

[0187] Each media level section begins with an "m=" line and continues to the next media level section or the end of the entire session.

[0188] Exemplarily, the terminal capabilities include: one or more of: graphics processing unit (GPU) capabilities, three-dimensional (3D) rendering capabilities, screen size, resolution, multi-dimensional acquisition capabilities, etc.

[0189] S1002, SBC forwards the INVITE request to S-CSCF.

[0190] S1003, S-CSCF forwards the INVITE request to the VoLTE AS.

[0191] S1004, the VoLTE AS reports the call event to the service interaction platform, which carries the dedicated channel media information of the first user terminal, the holographic communication service indication, and the terminal capability information.

[0192] S1005, the service interaction platform reports the call event to the information reconstruction application server corresponding to the first user terminal according to the holographic communication service instruction.

[0193] S1006: After receiving the request, the information reconstruction application server corresponding to the first user terminal selects model package 1 for the first user terminal according to the terminal capability information of the first user terminal.

[0194] Table 1 below shows an example of the specific content of model package 1.

[0195] Table 1 Specific contents of the model package

[0196]

[0197] Exemplarily, a model package such as model package 1 includes: data analysis model, data presentation model, semantic model, behavior data model, etc., and the corresponding information is recorded in the system. As an example, the data analysis model is used to parse the corresponding behavior instruction data to obtain the description information of the entity behavior; the semantic model is used to determine the semantic information corresponding to the description information of the entity behavior; the behavior data model can be used to convert the corresponding semantic information into the feature information of the corresponding behavior; the data presentation model is used to present the corresponding behavior with the feature information.

[0198] S1007, the application server corresponding to the first user terminal instructs the service interaction platform to continue the call and carries the dedicated channel media information and holographic communication service indication of the first user terminal.

[0199] S1008: The service interaction platform instructs the VoLTE AS to continue the call.

[0200] S1009, the VoLTE AS forwards the INVITE request to the I / S-CSCF.

[0201] S1010: The I / S-CSCF forwards the INVITE session request to the VoLTE AS of the second user terminal according to the identification information of the second user terminal.

[0202] S1011, the VoLTE AS corresponding to the second user terminal reports a call event to the service interaction platform, which carries a holographic communication service indication and terminal capability information.

[0203] S1012, the service interaction platform corresponding to the second user terminal reports the call event to the information reconstruction application server corresponding to the second user terminal according to the holographic communication service indication.

[0204] S1013, after receiving the request, the information reconstruction application server corresponding to the second user terminal selects model package 1 for the second user terminal according to the terminal capability information of the second user terminal during registration.

[0205] Table 2 below shows an example of the specific content of model package 1.

[0206]

[0207] It can be seen from Table 1 and Table 2 that the first user terminal and the second user terminal have the same terminal capability information, and therefore the corresponding model packages are also the same.

[0208] In Table 2, model package 1 includes data analysis model, data presentation model, semantic model, behavioral data model, etc., and the corresponding information is recorded in the system.

[0209] S1014: The application server corresponding to the second user terminal instructs the service interaction platform to continue the call.

[0210] S1015: The service interaction platform instructs the VoLTE AS to continue the call.

[0211] S1016, VoLTE AS forwards the INVITE request to I / S-CSCF, which carries the holographic communication service indication, media description and model package identification information.

[0212] For example, the holo tag carried in SDP indicates a holographic communication service, and the model package identifier is carried in package.

[0213] m=application 10000UDP / DTLS / SCTP webrtc-datachannel

[0214] a=dcmap:1002label=”holo”

[0215] a=package:1

[0216] S1017, the I / S-CSCF sends an INVITE request to the SBC registered with the second user terminal.

[0217] S1018: The SBC registered by the second user terminal forwards the INVITE request to the second user terminal.

[0218] S1019, when the second user terminal receives the INVITE call request, it performs preparations for holographic communication according to the holographic communication service indication in the call request, including but not limited to initializing the presentation device, graphics processing unit (GPU) function, etc.

[0219] S1020: The second user terminal returns a 183 response to the SBC corresponding to the second user terminal.

[0220] S1021, the SBC corresponding to the second user terminal forwards the 183 response to the S-CSCF.

[0221] S1022, the S-CSCF forwards the response to the VoLTE AS, and the VoLTE AS interacts with the media function (MRF) to create a dedicated channel.

[0222] S1023, the second user terminal downloads the data of model package 1 through a dedicated channel.

[0223] S1024, the VoLTE AS corresponding to the second user terminal sends a 183 response to the I / S-CSCF, which carries the media and network information (IP address, port number, etc.) of the second user terminal.

[0224] S1025, the I / S-CSCF forwards the 183 response to the VoLTE AS corresponding to the first user terminal.

[0225] S1026, the VoLTE AS corresponding to the first user terminal inserts the model identification information corresponding to the first user in the 183 response and forwards it to the I / S-CSCF. For example, a=package attribute is added to the m line in the SDP to indicate the information of the model package.

[0226] m=application 10000UDP / DTLS / SCTP webrtc-datachannel

[0227] a=dcmap:1002lable=”holo”

[0228] a=package:1

[0229] S1027, the I / S-CSCF forwards the 18x response to the SBC where the first user terminal is registered.

[0230] S1028, the SBC forwards the 18x response to the first user terminal.

[0231] S1029, the first user terminal uses the media information in the 18x response to establish a connection with the first user application server, and downloads the data of the model package 1 through a dedicated channel.

[0232] S1030: The first user terminal and the second user terminal complete a subsequent IMS precondition process.

[0233] The Precondition process includes: reserving network resources before a session is established.

[0234] S1031, after the second user terminal answers the call, the second user terminal returns a 200OK response to the first user terminal through the IMS network, and the first user terminal establishes a media path with the second user terminal.

[0235] In some embodiments, the user data may be transferred through the application server or not. After the first user terminal receives the user data of the second user terminal, it uses the model package 1 to generate the holographic presentation content; after the second user terminal receives the user data of the second user, it uses the model package 1 to generate the holographic presentation content.

[0236] Through the above steps, the first user terminal and the second user terminal complete the holographic communication process.

[0237] The following takes the implementation of holographic communication in an IMS network, where the first user terminal and the second user terminal can use the same IMS network as an example, to describe the process of two different types of user terminals supporting holographic communication capabilities interacting using the method of an embodiment of the present application. It should be understood that the first user terminal and the second user terminal can use the same IMS network or different IMS networks, and the processing flow corresponding to using the same IMS network or different IMS networks is the same. In some embodiments, the method includes the following steps:

[0238] S1101, a first user terminal sends an INVITE request to an SBC, which carries dedicated channel media information of the first user terminal, a service indication of holographic communication, and terminal capability information.

[0239] In this step, the terminal capability information may reflect the characteristics of the corresponding user terminal.

[0240] Exemplarily, the holographic communication service indication may be implemented by the following method:

[0241] Add a holographic communication media label in the media line (m=line) of the SDP.

[0242] m=application 10000UDP / DTLS / SCTP webrtc-datachannel

[0243] a=dcmap:1002label=”holo”

[0244] The specific content of the holographic communication media mark can be customized according to actual needs, and the embodiments of the present application do not make specific limitations.

[0245] S1102, SBC forwards the INVITE request to S-CSCF.

[0246] S1103, the S-CSCF forwards the INVITE request to the VoLTE AS.

[0247] S1104, the VoLTE AS reports the call event to the service interaction platform, which carries the holographic communication service indication and terminal characteristics.

[0248] S1105, the service interaction platform reports the call event to the information reconstruction application server corresponding to the first user terminal according to the holographic communication service indication.

[0249] S1106: After receiving the request, the information reconstruction application server corresponding to the first user terminal selects model package 1 for the first user terminal according to the terminal capability information of the first user terminal.

[0250] The specific content of model package 1 can be referred to the above Table 1, which will not be repeated here.

[0251] S1107: The application server corresponding to the first user terminal instructs the service interaction platform to continue the call.

[0252] S1108: The service interaction platform instructs the VoLTE AS to continue the call.

[0253] S1109, the VoLTE AS forwards the INVITE request to the I / S-CSCF.

[0254] S1110. The I / S-CSCF forwards the INVITE session request to the VoLTE AS corresponding to the second user terminal according to the identification information of the second user terminal.

[0255] S1111, the VoLTE AS corresponding to the second user terminal reports a call event to the service interaction platform, which carries a holographic communication service indication and terminal characteristics.

[0256] S1112, the service interaction platform corresponding to the second user terminal reports the call event to the information reconstruction application server corresponding to the second user terminal according to the holographic communication service indication.

[0257] S1113, after receiving the request, the information reconstruction application server corresponding to the second user terminal selects model package 2 for the second user terminal according to the terminal capability information of the second user during registration.

[0258] Table 3 below shows an example of the specific content of model package 3.

[0259] Table 3 Specific contents of the model package

[0260]

[0261] Model package 2 includes data analysis model, data presentation model, semantic model, behavioral data model, etc., and records the corresponding information in the system.

[0262] S1114. The application server corresponding to the second user terminal instructs the service interaction platform to continue the call and carries the dedicated channel media information and holographic communication service indication of the first user terminal.

[0263] S1115. The service interaction platform instructs the VoLTE AS to continue the call.

[0264] S1116. The VoLTE AS forwards the INVITE request to the I / S-CSCF, which carries the holographic communication service indication, media description, and model package identification information.

[0265] For example, the holo tag carried in SDP indicates a holographic communication service, and the model package identifier is carried in package.

[0266] m=application 10000UDP / DTLS / SCTP webrtc-datachannel

[0267] a=dcmap:1002label=”holo”

[0268] a=package:2

[0269] Among them, package:2 indicates that the model package is model package 2.

[0270] S1117, the I / S-CSCF sends an INVITE request to the SBC registered with the second user terminal.

[0271] S1118: The SBC registered by the second user terminal forwards the INVITE request to the second user terminal.

[0272] S1119, when the second user terminal receives the INVITE call request, it performs preparations for holographic communication according to the holographic communication service indication in the call request, including but not limited to initializing the presentation device, graphics processing unit (GPU) function, etc.

[0273] S1120: The second user terminal returns a 183 response to the SBC corresponding to the second user terminal.

[0274] S1121. The SBC corresponding to the second user terminal forwards the 183 response to the S-CSCF.

[0275] S1122, the S-CSCF forwards the response to the VoLTE AS, and the VoLTE AS interacts with the media function (MRF) to create a dedicated channel with the second user terminal.

[0276] S1123, the second user terminal downloads the data of model package 2 through a dedicated channel.

[0277] S1124, the second user terminal sends a 183 response to the I / S-CSCF corresponding to the VoLTE AS, which carries the media and network information (IP address, port number, etc.) of the second user terminal.

[0278] S1125, the I / S-CSCF forwards the 183 response to the VoLTE AS corresponding to the first user terminal.

[0279] S1126, the VoLTE AS corresponding to the first user terminal inserts the model identification information corresponding to the first user in the 183 response and forwards it to the I / S-CSCF. For example, a=package attribute is added to the m line in the SDP to indicate the information of the model package.

[0280] m=application 10000UDP / DTLS / SCTP webrtc-datachannel

[0281] a=dcmap:1002lable=”holo”

[0282] a=package:1

[0283] S1127, the I / S-CSCF forwards the 18x response to the SBC where the first user terminal is registered.

[0284] S1128, the SBC forwards the 18x response to the first user terminal.

[0285] S1129, the first user terminal uses the media information in the 18x response to establish a connection with the first user application server and downloads the data of model package 1 through a dedicated channel.

[0286] S1130: The first user terminal and the second user terminal complete a subsequent IMS precondition process.

[0287] S1131, after the second user answers the call, the second user terminal returns a 200OK response to the first user terminal through the IMS network, and the first user terminal establishes a media path with the second user terminal.

[0288] In some embodiments, the user data may be transferred through the application server or not. After the first user terminal receives the user data of the second user terminal, it uses model package 1 to generate holographic presentation content; after the second user terminal receives the user data of the second user, it uses model package 2 to generate holographic presentation content.

[0289] Through the above steps, the first user terminal and the second user terminal complete the holographic communication process.

[0290] The following takes the implementation of holographic communication in an IMS network, where the first user terminal and the second user terminal can use the same IMS network as an example, to describe the process of interaction between a user terminal with holographic communication capability and a user terminal without holographic communication capability. It should be understood that the first user terminal and the second user terminal can use the same IMS network or different IMS networks, and the processing flow corresponding to using the same IMS network or different IMS networks is the same. In some embodiments, the method includes the following steps:

[0291] S1201, the first user terminal sends an INVITE request to the SBC, which includes the dedicated channel media information of the first user terminal, the service indication of the holographic communication and the terminal capability information.

[0292] Exemplarily, the holographic communication service indication may be implemented by the following method:

[0293] Add a holographic communication media label in the media line (m=line) of the SDP.

[0294] m=application 10000UDP / DTLS / SCTP webrtc-datachannel

[0295] a=dcmap:1002label=”holo”

[0296] Exemplarily, the terminal capabilities include: one or more of: graphics processing unit (GPU) capabilities, three-dimensional (3D) rendering capabilities, screen size, resolution, multi-dimensional acquisition capabilities, etc.

[0297] S1202, SBC forwards the INVITE request to S-CSCF;

[0298] S1203, S-CSCF forwards the INVITE request to the VoLTE AS;

[0299] S1204, the VoLTE AS reports the call event to the service interaction platform, which carries the dedicated channel media information of the first user terminal, the holographic communication service indication, and the terminal capability information.

[0300] S1205, the service interaction platform reports the call event to the information reconstruction application server corresponding to the first user terminal according to the holographic communication service indication;

[0301] S1206: After receiving the request, the information reconstruction application server corresponding to the first user terminal selects model package 1 for the first user terminal according to the terminal capability information of the first user.

[0302] S1207: The application server corresponding to the first user terminal instructs the service interaction platform to continue the call.

[0303] S1208: The service interaction platform instructs the VoLTE AS to continue the call.

[0304] S1209, the VoLTE AS forwards the INVITE request to the I / S-CSCF.

[0305] S1210: The I / S-CSCF forwards the INVITE session request to the VoLTE AS corresponding to the second user terminal according to the identification information of the second user terminal.

[0306] S1211, the VoLTE AS corresponding to the second user terminal reports a call event to the service interaction platform, which carries a holographic communication service indication and terminal capability information.

[0307] S1212, the service interaction platform corresponding to the second user terminal reports the call event to the information reconstruction application server corresponding to the second user terminal according to the holographic communication service indication.

[0308] S1213, after receiving the request, the information reconstruction application server corresponding to the second user terminal determines that the terminal characteristics of the second user do not support holographic communication, and instructs the service interaction platform to delete the dedicated channel and continue the call.

[0309] S1214: The service interaction platform instructs the VoLTE AS to delete the dedicated channel and then continue the call.

[0310] S1215, the VoLTE AS forwards the INVITE request to the I / S-CSCF, which does not carry the holographic communication service indication, media description, and model package identification information. For example, the holographic communication corresponding information is deleted in the SDP.

[0311] The deleted holographic communication corresponding information includes the following:

[0312] m=application 10000UDP / DTLS / SCTP webrtc-datachannel

[0313] a=dcmap:1002label=”holo”

[0314] a=package:2".

[0315] S1216, the I / S-CSCF sends an INVITE request to the SBC registered with the second user terminal.

[0316] S1217: The SBC registered by the second user terminal forwards the INVITE request to the second user terminal.

[0317] S1218: When receiving the INVITE call request, the second user terminal prepares resources according to the basic call of non-holographic communication.

[0318] S1219: The second user terminal returns a 183 response to the SBC corresponding to the second user terminal.

[0319] S1220: The SBC corresponding to the second user terminal forwards the 183 response to the S-CSCF.

[0320] S1221, S-CSCF forwards the response to the VoLTE AS.

[0321] S1222, the VoLTE AS corresponding to the second user terminal sends a 183 response to the I / S-CSCF, and sets the port number of the holographic media to 0, indicating that it is unavailable.

[0322] m=application 0UDP / DTLS / SCTP webrtc-datachannel

[0323] S1223, the I / S-CSCF forwards the 183 response to the VoLTE AS corresponding to the first user terminal.

[0324] S1224, the VoLTE AS corresponding to the first user terminal determines that the second user does not support holographic communication, and does not insert the model identification information corresponding to the first user in the 183 response and forward it to the I / S-CSCF.

[0325] S1225, the I / S-CSCF forwards the 18x response to the SBC where the first user terminal is registered.

[0326] S1226. The SBC forwards the 18x response to the first user terminal.

[0327] S1227: The first user terminal and the second user terminal complete a subsequent IMS precondition process.

[0328] S1228, after the second user answers the call, the second user terminal returns a 200OK response to the first user terminal through the IMS network, and the first user terminal establishes a common audio and video media path with the second user terminal.

[0329] Through the above steps, the ordinary audio and video call process between the user terminal with holographic communication capability and the user terminal without holographic communication capability is completed.

[0330] Fig.10 A schematic diagram of the structure of a first user terminal provided in an embodiment of the present application.

[0331] Fourthly, refer to Fig.10 , the embodiment of the present application provides a first user terminal 1000, including:

[0332] An extraction module 1010 is used to perform feature extraction on the collected first multi-dimensional image data to obtain feature information of the first multi-dimensional image data;

[0333] A conversion module 1020, configured to convert the characteristic information into first behavior instruction data, where the first behavior instruction data is used to indicate behavior information of a first user to which the first user terminal belongs;

[0334] The transmission module 1030 is used to transmit the first behavior instruction data to the second user terminal through a dedicated channel. The dedicated channel is a data transmission channel established in advance based on real-time communication technology. The second user terminal is used to reconstruct information according to the first behavior instruction data to obtain reconstructed first multi-dimensional image data.

[0335] In some embodiments, the first multidimensional image data includes multidimensional image data, and the feature information is a feature of the multidimensional image data; the extraction module 1010 is specifically used to: preprocess the multidimensional image data collected by the image sensor, the preprocessing including at least one of the following: image denoising, image alignment, posture correction and motion compensation; perform feature extraction on the preprocessed multidimensional image data to obtain multidimensional image data features.

[0336] In some embodiments, the conversion module 1020 is specifically used to: match the feature information with the semantic knowledge base to obtain semantic information corresponding to the feature information, the semantic information including description information of the entity behavior; and convert the description information of the entity behavior into corresponding first behavior instruction data.

[0337] In some embodiments, the dedicated channel includes: a first dedicated channel between the first user terminal and the first communication network, and a second dedicated channel between the second user terminal and the second communication network; the first user terminal 1000 also includes: a sending module, which is used to send a first session request to the first communication network corresponding to the first user terminal before transmitting the first behavior instruction data to the second user terminal through the dedicated channel, wherein the first session request at least includes: first dedicated channel media information and service type indication information of the first user terminal, and the first communication network is used to: forward the first dedicated channel media information and service type indication information to the second user terminal through the second communication network corresponding to the second user terminal; a receiving module, which is used to receive a session response from the second user terminal forwarded by the second communication network through the first communication network; wherein the session response is a session response generated and sent to the second communication network by the second user terminal after receiving the first dedicated channel media information and the service type indication information, and the session response includes the second dedicated channel media information of the second user terminal; the second communication network is also used to: establish a second dedicated channel with the second user terminal according to the received second dedicated channel media information; an establishing module, which is used to establish the first dedicated channel with the first communication network according to the first dedicated channel media information in response to the session response.

[0338] In some embodiments, the transmission module 1030 is specifically used to: send first behavior instruction data to a first communication network through a first dedicated channel; wherein the first communication network is used to send the first behavior instruction data to a second communication network, and the second communication network is used to send the first behavior instruction data to a second user terminal through a second dedicated channel.

[0339] In some embodiments, the first session request also includes: first reconstruction capability information, the first reconstruction capability information is used to indicate the information reconstruction capability of the first user terminal; the first communication network is also used to: send the first session request to the first application server; wherein the first application server is used to select the corresponding first model data for the first user terminal according to the reconstruction capability information of the first user terminal, and instruct the first communication network to send a second session request to the second communication network, the second session request includes: first dedicated channel media information and service type indication information; and, before sending the session response to the second user terminal, add the identifier of the first model data to the session response; the second communication network is also used to: send the second session request to the first application server; wherein the first application server is used to select the corresponding first model data for the first user terminal according to the reconstruction capability information of the first user terminal, and instruct the first communication network to send the second session request to the second communication network. The second application server corresponding to the second user terminal is used to select the corresponding second model data for the second user terminal according to the stored reconstruction capability information of the second user terminal, and instruct the second communication network to send a third session request to the second user terminal, wherein the third session request includes: first dedicated channel media information, service type indication information and an identifier of the second model data; the second user terminal is also used to: after downloading the second model data through the second dedicated channel, perform feature extraction on the collected second multidimensional image data through the second model data and convert the feature information of the extracted second multidimensional image data into second behavior instruction data, and send the second behavior instruction data to the second communication network through the second dedicated channel.

[0340] The first user terminal 1000 also includes: a download module, which is used to download the first model data through the first dedicated channel after establishing the first dedicated channel with the first communication network according to the first dedicated channel media information; a receiving module, which is also used to receive second behavior instruction data from the first communication network through the first dedicated channel, and the second behavior instruction data is data transmitted from the second communication network to the second communication network; an information reconstruction module, which is used to reconstruct the second behavior instruction data according to the first model data to obtain reconstructed second multidimensional image data.

[0341] It should be clear that the present invention is not limited to the specific configurations and processes described in the above embodiments and shown in the figures. For the convenience and brevity of description, a detailed description of the known methods is omitted here, and the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, which will not be repeated here.

[0342] Fig.11 A schematic diagram of the structure of a second user terminal provided in an embodiment of the present application.

[0343] Fifth, refer to Fig.11 , the embodiment of the present application provides a second user terminal 1100, including:

[0344] The receiving module 1110 is used to receive first behavior instruction data of a first user terminal through a dedicated channel; wherein the dedicated channel is a data transmission channel established in advance based on real-time communication technology, and the first behavior instruction data is obtained by the first user terminal performing feature extraction on the collected first multi-dimensional image data and converting the feature information of the extracted first multi-dimensional image data, and the first behavior instruction data is used to indicate the behavior information of the first user to which the first user terminal belongs;

[0345] The reconstruction module 1120 is used to reconstruct information according to the first line of instruction data to obtain reconstructed first multi-dimensional image data.

[0346] In some embodiments, the dedicated channel includes: a first dedicated channel between a first user terminal and a first communication network, and a second dedicated channel between a second user terminal and a second communication network; wherein the first communication network is used to: send a first session request received from the first user terminal to a first application server, the first application server is used to select corresponding first model data for the first user terminal according to the reconstruction capability information of the first user terminal, and instruct the first communication network to send a second session request to the second communication network, the second session request includes: first dedicated channel media information and service type indication information; the second communication network is used to: send a second session request to a second application server corresponding to the second user terminal, the second application server is used to select corresponding second model data for the second user terminal according to the stored reconstruction capability information of the second user terminal, and instruct the second communication network to send a third session request to the second user terminal, the third session request includes: first dedicated channel media information, service type indication information and an identifier of the second model data.

[0347] Before receiving the first behavior instruction data of the first user terminal through the dedicated channel, the method further includes:

[0348] The receiving module 1110 is also used to receive a third session request sent by the second communication network; the second user terminal 1100 also includes: a sending module, used to respond to the third session request and send a session response to the second communication network, wherein the session response includes the second dedicated channel media information of the second user terminal; and an establishing module, used to establish a second dedicated channel with the second communication network through the second dedicated channel media information.

[0349] In some embodiments, the reconstruction module 1120 is specifically used to: download the second model data through the second dedicated channel; and reconstruct the first behavior instruction data through the second model data to obtain the reconstructed first multi-dimensional image data.

[0350] In some embodiments, the first user terminal is also used to: in response to a session response, establish a first dedicated channel with the first communication network according to the first dedicated channel media information, and download the first model data through the first dedicated channel; the second user terminal 1100 also includes: an extraction module, which is used to extract features from the collected second multidimensional image data after downloading the second model data through the second dedicated channel, and convert the extracted feature information of the second multidimensional image data into second behavior instruction data; a sending module, which is used to send the second behavior instruction data to the second communication network through the second dedicated channel; wherein the second communication network is used to transmit the second behavior instruction data to the first communication network, so that the first communication network transmits the second behavior instruction data to the first user terminal through the first dedicated channel; the first user terminal is used to: reconstruct the second behavior instruction data according to the first model data to obtain reconstructed second multidimensional image data.

[0351] It should be clear that the present invention is not limited to the specific configurations and processes described in the above embodiments and shown in the figures. For the convenience and brevity of description, a detailed description of the known methods is omitted here, and the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, which will not be repeated here.

[0352] Fig.12 A schematic diagram of the structure of a communication system provided in an embodiment of the present application.

[0353] Sixth, refer to Fig.12 , an embodiment of the present application provides a communication system 1200, including:

[0354] The receiving module 1210 is used to receive first behavior instruction data of a first user terminal, where the first behavior instruction data is behavior instruction data obtained by the first user terminal performing feature extraction on the collected first multi-dimensional image data and converting the feature information of the extracted first multi-dimensional image data, and the first behavior instruction data is used to indicate the behavior information of the first user to which the first user terminal belongs; the transmitting module 1220 is used to transmit the first behavior instruction data to the second user terminal through a dedicated channel, where the dedicated channel is a data transmission channel established in advance based on real-time communication technology, and the second user terminal is used to reconstruct information according to the first behavior instruction data to obtain reconstructed first multi-dimensional image data.

[0355] In some embodiments, the dedicated channel includes a first dedicated channel and a second dedicated channel; the communication system includes at least a first communication network and a second communication network; the transmission module 1220 is also used to: before receiving the first behavior instruction data of the first user terminal, receive a first session request of the first user terminal through the first communication network, wherein the first session request at least includes: first dedicated channel media information and service type indication information of the first user terminal; send the first dedicated channel media information and service type indication information to the second communication network through the first communication network, and the second communication network is a communication network corresponding to the identifier of the second user terminal; send the first dedicated channel media information and service type indication information to the second user terminal through the second communication network; receive and forward the session response of the second user terminal to the first communication network through the second communication network, wherein the session response includes the second dedicated channel media information of the second user terminal; send the second dedicated channel media information to the first user terminal through the first communication network, and establish a first dedicated channel with the first user terminal according to the first dedicated channel media information; establish a second dedicated channel with the second user terminal through the second communication network using the second dedicated channel media information.

[0356] In some embodiments, the communication system also includes: a first application server and a second application server; when the transmission module 1220 is used to send the first dedicated channel media information and service type indication information to the second communication network through the first communication network, it is specifically used to: send a first session request to the first application server through the first communication network, and the first session request also includes reconstruction capability information of the first user terminal; the first application server selects the corresponding first model data for the first user terminal according to the reconstruction capability information of the first user terminal, and instructs the first communication network to send a second session request to the second communication network, and the second session request includes: the first dedicated channel media information and service type indication information.

[0357] When the transmission module 1220 is used to send the first dedicated channel media information and the service type indication information to the second user terminal through the second communication network, it is specifically used to: send a second session request to the second application server through the second communication network; the second application server selects the corresponding second model data for the second user terminal based on the stored reconstruction capability information of the second user terminal, and instructs the second communication network to send a third session request to the second user terminal, wherein the third session request includes: the first dedicated channel media information, the service type indication information and the identifier of the second model data.

[0358] When the transmission module 1220 is used to send the second dedicated channel media information to the first user terminal through the first communication network, it is specifically used to: add the identifier of the first model data in the session response through the first communication network; and send the session response carrying the identifier of the first model data and the second dedicated channel media information to the first user terminal.

[0359] In some embodiments, when the transmission module 1220 is used to transmit the first behavior instruction data to the second user terminal through a dedicated channel, it is specifically used to: send the first behavior instruction data received through the first dedicated channel to the second communication network; in the second communication network, send the first behavior instruction data to the terminal device through the second dedicated channel.

[0360] In some embodiments, after being used to establish a second dedicated channel with a second user terminal via a second communication network using second dedicated channel media information, the transmission module 1220 is further specifically used to: receive second behavior instruction data from the second user terminal via the second dedicated channel in the second communication network, wherein the second behavior instruction data is behavior instruction data obtained by the second user terminal extracting features from the collected second multidimensional image data through second model data and converting feature information of the extracted second multidimensional image data; transmit the second behavior instruction data to the first communication network via the second communication network; and send the second behavior instruction data to the first user terminal via the first dedicated channel in the first communication network, and the first user terminal is used to: reconstruct the second behavior instruction data according to the first model data to obtain reconstructed second multidimensional image data.

[0361] In some embodiments, the first communication network includes: a first session edge controller, a first call session control function node, a first multimedia system application server, and a first business interaction platform connected in sequence, wherein the first session edge controller is connected to a first user terminal, and the first business interaction platform is connected to the first application server; the second communication network includes: a second session edge controller, a second call session control function node, a second multimedia system application server, and a second business interaction platform connected in sequence, wherein the second session edge controller is connected to a second user terminal, and the second business interaction platform is connected to the second application server; and wherein the first call session control function node is connected to the second multimedia system application server for communication transmission from the first communication network to the second communication network; and the second call session control function node is connected to the first multimedia system application server for communication transmission from the second communication network to the second communication network.

[0362] It should be clear that the present invention is not limited to the specific configurations and processes described in the above embodiments and shown in the figures. For the convenience and brevity of description, a detailed description of the known methods is omitted here, and the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, which will not be repeated here.

[0363] In a seventh aspect, an embodiment of the present application also provides a network device.

[0364] Reference Fig.13 The electronic device includes: at least one processor 1301; at least one memory 1302, and one or more I / O interfaces 1303, connected between the processor 1301 and the memory 1302; wherein the memory 1302 stores one or more computer programs that can be executed by at least one processor 1301, and the one or more computer programs are executed by at least one processor 1301 so that the at least one processor 1301 can execute the above-mentioned data processing method.

[0365] Among them, the processor is a device with data processing capabilities, including but not limited to the central processing unit (CPU); the memory is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); the I / O interface (read-write interface) is connected between the processor and the memory, and can realize information exchange between the memory and the processor, including but not limited to the data bus (Bus), etc.

[0366] The present application also provides a computer-readable storage medium on which a computer program is stored, wherein the computer program implements the above-mentioned information processing method when executed by the processor / processing core. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0367] Those skilled in the art will appreciate that all or some of the steps, systems, and functional modules / units in the apparatus disclosed above may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0368] The method of the embodiment of the present application can be widely used in the fields of assisted manufacturing, assisted education, AR / VR video conferencing, digital twins, etc., and has broad application prospects.

[0369] In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed by several physical components in cooperation.

[0370] Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit (CPU), a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-temporary medium) and a communication medium (or temporary medium). As known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH) or other disk storage; read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical disk storage; magnetic cassettes, magnetic tapes, disk storage or other magnetic storage; any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0371] The present application has disclosed example embodiments, and although specific terms are used, they are only used and should only be interpreted as general illustrative meanings, and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly indicated, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, it will be understood by those skilled in the art that various changes in form and detail may be made without departing from the scope of the present application as set forth in the appended claims.

Claims

1. An information processing method, characterized in that: Applied to a first user terminal, the method includes: Performing feature extraction on the collected first multi-dimensional image data to obtain feature information of the first multi-dimensional image data; converting the characteristic information into first behavior instruction data, where the first behavior instruction data is used to indicate behavior information of a first user to which the first user terminal belongs; The first behavior instruction data is transmitted to the second user terminal through a dedicated channel, which is a data transmission channel pre-established based on real-time communication technology. The second user terminal is used to reconstruct information according to the first behavior instruction data to obtain reconstructed first multi-dimensional image data.

2. The method according to claim 1, characterized in that The first multidimensional image data includes multidimensional image data, and the feature information is a feature of the multidimensional image data; the feature extraction of the collected first multidimensional image data to obtain the feature information of the first multidimensional image data includes: Preprocessing the multidimensional image data collected by the image sensor, wherein the preprocessing includes at least one of the following: image denoising, image alignment, posture correction and motion compensation; Feature extraction is performed on the preprocessed multidimensional image data to obtain multidimensional image data features.

3. The method according to claim 1, characterized in that The converting the characteristic information into first behavior instruction data includes: Matching the feature information with a semantic knowledge base to obtain semantic information corresponding to the feature information, wherein the semantic information includes description information of entity behavior; The description information of the entity behavior is converted into corresponding first behavior instruction data.

4. The method according to claim 1, characterized in that: The dedicated channel includes: a first dedicated channel between the first user terminal and the first communication network, and a second dedicated channel between the second user terminal and the second communication network; before transmitting the first behavior instruction data to the second user terminal through the dedicated channel, the method further includes: Sending a first session request to a first communication network corresponding to the first user terminal, wherein the first session request includes at least: first dedicated channel media information and service type indication information of the first user terminal, and the first communication network is used to: forward the first dedicated channel media information and the service type indication information to the second user terminal through a second communication network corresponding to the second user terminal; Receiving, through the first communication network, a session response from the second user terminal forwarded by the second communication network; wherein the session response is a session response generated by the second user terminal and sent to the second communication network after receiving the first dedicated channel media information and the service type indication information, and the session response includes the second dedicated channel media information of the second user terminal; the second communication network is also used to: establish a second dedicated channel with the second user terminal according to the received second dedicated channel media information; In response to the session response, a first dedicated channel is established with the first communication network according to the first dedicated channel media information.

5. The method according to claim 4, characterized in that The transmitting the first behavior instruction data to the second user terminal through a dedicated channel includes: Sending the first behavior instruction data to the first communication network through the first dedicated channel; The first communication network is used to send the first behavior instruction data to the second communication network, and the second communication network is used to send the first behavior instruction data to the second user terminal through the second dedicated channel.

6. The method according to claim 4, characterized in that The first session request also includes: first reconstruction capability information, where the first reconstruction capability information is used to indicate the information reconstruction capability of the first user terminal; The first communication network is further used to: send the first session request to the first application server; wherein the first application server is used to select corresponding first model data for the first user terminal according to the reconstruction capability information of the first user terminal, and instruct the first communication network to send a second session request to the second communication network, wherein the second session request includes: the first dedicated channel media information and service type indication information; and before sending the session response to the second user terminal, add an identifier of the first model data in the session response; The second communication network is further used to: send the second session request to the second application server corresponding to the second user terminal, the second application server is used to select the corresponding second model data for the second user terminal according to the stored reconstruction capability information of the second user terminal, and instruct the second communication network to send a third session request to the second user terminal, wherein the third session request includes: the first dedicated channel media information, the service type indication information and the identifier of the second model data; The second user terminal is further used to: after downloading the second model data through the second dedicated channel, extract features from the collected second multidimensional image data through the second model data, convert the extracted feature information of the second multidimensional image data into second behavior instruction data, and send the second behavior instruction data to the second communication network through the second dedicated channel; After establishing the first dedicated channel with the first communication network according to the first dedicated channel media information, the method further includes: downloading the first model data through the first dedicated channel; receiving the second behavior instruction data from the first communication network through the first dedicated channel, where the second behavior instruction data is data transmitted from the second communication network to the second communication network; The second behavior instruction data is reconstructed according to the first model data to obtain reconstructed second multi-dimensional image data.

7. An information processing method, characterized in that: Applied to a second user terminal, the method includes: receiving first behavior instruction data of a first user terminal through a dedicated channel; wherein the dedicated channel is a data transmission channel established in advance based on real-time communication technology, and the first behavior instruction data is behavior instruction data obtained by the first user terminal performing feature extraction on the collected first multi-dimensional image data and converting the feature information of the extracted first multi-dimensional image data, and the first behavior instruction data is used to indicate behavior information of a first user to which the first user terminal belongs; Information is reconstructed according to the first line of instruction data to obtain reconstructed first multi-dimensional image data.

8. The method according to claim 7, characterized in that The dedicated channel includes: a first dedicated channel between the first user terminal and the first communication network, and a second dedicated channel between the second user terminal and the second communication network; wherein the first communication network is used to: send the received first session request of the first user terminal to the first application server, the first application server is used to select the corresponding first model data for the first user terminal according to the reconstruction capability information of the first user terminal, and instruct the first communication network to send a second session request to the second communication network, the second session request includes: the first dedicated channel media information and service type indication information; The second communication network is used to: send the second session request to the second application server corresponding to the second user terminal, the second application server is used to select the corresponding second model data for the second user terminal according to the stored reconstruction capability information of the second user terminal, and instruct the second communication network to send a third session request to the second user terminal, wherein the third session request includes: first dedicated channel media information, service type indication information and an identifier of the second model data; Before receiving the first behavior instruction data of the first user terminal through the dedicated channel, the method further includes: receiving a third session request sent by the second communication network; In response to the third session request, sending a session response to the second communication network, wherein the session response includes the second dedicated channel media information of the second user terminal; The second dedicated channel is established with the second communication network through the second dedicated channel media information.

9. The method according to claim 8, characterized in that The information reconstruction is performed according to the first behavior instruction data to obtain the reconstructed first multi-dimensional image data, including: downloading the second model data through the second dedicated channel; The first behavior instruction data is reconstructed using the second model data to obtain reconstructed first multi-dimensional image data.

10. The method according to claim 8, characterized in that The first user terminal is further configured to: in response to the session response, establish the first dedicated channel with the first communication network according to the first dedicated channel media information, and download the first model data through the first dedicated channel; After downloading the second model data through the second dedicated channel, the method further includes: Extracting features from the collected second multi-dimensional image data, and converting the extracted feature information of the second multi-dimensional image data into second behavior instruction data; Sending the second behavior instruction data to the second communication network through the second dedicated channel; Among them, the second communication network is used to transmit the second behavior instruction data to the first communication network, so that the first communication network transmits the second behavior instruction data to the first user terminal through the first dedicated channel; the first user terminal is used to: reconstruct the information of the second behavior instruction data according to the first model data to obtain the reconstructed second multidimensional image data.

11. An information transmission method, characterized in that: Applied to a communication system, the method comprises: receiving first behavior instruction data from a first user terminal, where the first behavior instruction data is obtained by the first user terminal performing feature extraction on the collected first multidimensional image data and converting the feature information of the extracted first multidimensional image data, and the first behavior instruction data is used to indicate behavior information of a first user to which the first user terminal belongs; The first behavior instruction data is transmitted to the second user terminal through a dedicated channel, and the dedicated channel is a data transmission channel pre-established based on real-time communication technology. The second user terminal is used to reconstruct information according to the first behavior instruction data to obtain reconstructed first multi-dimensional image data.

12. The method according to claim 11, characterized in that The dedicated channel includes a first dedicated channel and a second dedicated channel; the communication system includes at least a first communication network and a second communication network; Before receiving the first behavior instruction data of the first user terminal, the method further includes: Receiving a first session request of the first user terminal through the first communication network, wherein the first session request at least includes: first dedicated channel media information and service type indication information of the first user terminal; Sending the first dedicated channel media information and the service type indication information to a second communication network through the first communication network, where the second communication network is a communication network corresponding to the identifier of the second user terminal; sending the first dedicated channel media information and service type indication information to the second user terminal through the second communication network; receiving and forwarding a session response of the second user terminal to the first communication network through the second communication network, wherein the session response includes second dedicated channel media information of the second user terminal; sending the second dedicated channel media information to the first user terminal through the first communication network, and establishing a first dedicated channel with the first user terminal according to the first dedicated channel media information; A second dedicated channel is established with the second user terminal through the second communication network using the second dedicated channel media information.

13. The method according to claim 12, characterized in that The communication system further includes: a first application server and a second application server; and sending the first dedicated channel media information and the service type indication information to the second communication network through the first communication network includes: Sending the first session request to the first application server through the first communication network, wherein the first session request also includes reconstruction capability information of the first user terminal; The first application server selects corresponding first model data for the first user terminal according to the reconstruction capability information of the first user terminal, and instructs the first communication network to send a second session request to the second communication network, wherein the second session request includes: the first dedicated channel media information and service type indication information; The sending the first dedicated channel media information and service type indication information to the second user terminal through the second communication network includes: sending the second session request to the second application server through the second communication network; The second application server selects corresponding second model data for the second user terminal according to the stored reconstruction capability information of the second user terminal, and instructs the second communication network to send a third session request to the second user terminal, wherein the third session request includes: the first dedicated channel media information, the service type indication information, and an identifier of the second model data; The sending the second dedicated channel media information to the first user terminal through the first communication network includes: adding an identifier of the first model data to the session response through the first communication network; A session response carrying the identifier of the first model data and the media information of the second dedicated channel is sent to the first user terminal.

14. The method according to claim 12, characterized in that The transmitting the first behavior instruction data to the second user terminal through a dedicated channel includes: sending the first behavior instruction data received through the first dedicated channel to the second communication network; In the second communication network, the first behavior instruction data is sent to the terminal device through the second dedicated channel.

15. The method according to claim 13, characterized in that After establishing the second dedicated channel with the second user terminal through the second communication network using the second dedicated channel media information, the method further includes: receiving, in the second communication network, second behavior instruction data from the second user terminal through the second dedicated channel, wherein the second behavior instruction data is behavior instruction data obtained by the second user terminal performing feature extraction on the collected second multidimensional image data through the second model data and converting feature information of the extracted second multidimensional image data; transmitting the second behavior instruction data to the first communication network via the second communication network; In the first communication network, the second behavior instruction data is sent to the first user terminal through the first dedicated channel, and the first user terminal is used to: reconstruct information of the second behavior instruction data according to the first model data to obtain reconstructed second multi-dimensional image data.

16. The method according to claim 13, characterized in that The first communication network includes: a first session edge controller, a first call session control function node, a first multimedia system application server, and a first service interaction platform connected in sequence, wherein the first session edge controller is connected to the first user terminal, and the first service interaction platform is connected to the first application server; The second communication network includes: a second session edge controller, a second call session control function node, a second multimedia system application server, and a second service interaction platform connected in sequence, wherein the second session edge controller is connected to the second user terminal, and the second service interaction platform is connected to the second application server; And wherein, the first call session control function node is connected to the second multimedia system application server for communication transmission from the first communication network to the second communication network; and the second call session control function node is connected to the first multimedia system application server for communication transmission from the second communication network to the second communication network.

17. A first user terminal, characterized in that: The first user terminal comprises: An extraction module, used for performing feature extraction on the collected first multi-dimensional image data to obtain feature information of the first multi-dimensional image data; a conversion module, configured to convert the characteristic information into first behavior instruction data, where the first behavior instruction data is used to indicate behavior information of a first user to which the first user terminal belongs; A transmission module is used to transmit the first behavior instruction data to a second user terminal through a dedicated channel, where the dedicated channel is a data transmission channel established in advance based on real-time communication technology, and the second user terminal is used to reconstruct information according to the first behavior instruction data to obtain reconstructed first multi-dimensional image data.

18. A second user terminal, characterized in that: The second user terminal comprises: a receiving module, configured to receive first behavior instruction data of a first user terminal through a dedicated channel; wherein the dedicated channel is a data transmission channel established in advance based on real-time communication technology, and the first behavior instruction data is obtained by the first user terminal performing feature extraction on the collected first multi-dimensional image data and converting the feature information of the extracted first multi-dimensional image data, and the first behavior instruction data is used to indicate the behavior information of the first user to which the first user terminal belongs; A reconstruction module is used to reconstruct information according to the first line instruction data to obtain reconstructed first multi-dimensional image data.

19. A communication system, characterized in that: include: a receiving module, configured to receive first behavior instruction data from a first user terminal, wherein the first behavior instruction data is obtained by the first user terminal performing feature extraction on the collected first multi-dimensional image data and converting the feature information of the extracted first multi-dimensional image data, and the first behavior instruction data is used to indicate behavior information of a first user to which the first user terminal belongs; A transmission module is used to transmit the first behavior instruction data to a second user terminal through a dedicated channel, where the dedicated channel is a data transmission channel pre-established based on real-time communication technology, and the second user terminal is used to reconstruct information according to the first behavior instruction data to obtain reconstructed first multi-dimensional image data.

20. A network device comprising: one or more processors; A memory having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-6 or the method of any one of claims 7-10.

21. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 or the method according to any one of claims 7 to 10 is implemented.

Images