Backward compatible 3d messaging

Abstract

An example server device for communicating media data, the example server device comprising: a memory configured to store media data; and a processing system implemented in circuitry and configured to: receive a set of three-dimensional (3D) media data being transmitted to a client device; transcoding the set of 3D media data into a set of one or more images; and transmitting the set of one or more images to the client device. A server device may receive a message destined for a client device from a source client device, wherein the message includes 3D media data. The message may be a multimedia message service (MMS) message including 3D media data. The client device may initially provide data representing rendering capabilities of the client device to the server device such that the server device may determine that the client device cannot render the 3D media data.

Description

[0001] This application claims priority to U.S. Patent Application No. 18 / 784,524, filed July 25, 2024, and U.S. Provisional Application No. 63 / 516,328, filed July 28, 2023, the entire contents of each of which are incorporated herein by reference. U.S. Patent Application No. 18 / 784,524, filed July 25, 2024, claims the benefit of U.S. Provisional Application No. 63 / 516,328, filed July 28, 2023. Technical Field

[0002] This disclosure relates to the transmission of encoded 3D media data. Background Technology

[0003] Media data can be packaged for transmission or storage. Media data can be assembled into media files that conform to any of various standards, such as the International Organization for Standardization (ISO) Basic Media File Format and its extensions, such as AVC. Summary of the Invention

[0004] Generally, this disclosure describes techniques for transcoding 3D media data into 2D image data to support backward compatibility with devices that do not support rendering 3D media data. That is, 3D media data can be exchanged, for example, in Multimedia Messaging Service (MMS) messages. Many devices cannot process 3D media data as part of an MMS message, but can process 2D image data (e.g., a single image, image sequences such as GIFs, or video sequences) as part of an MMS message. Therefore, according to the techniques of this disclosure, a server device (e.g., a multimedia messaging server) can transcode the 3D media data of an MMS message into an image or image sequence, and then transmit the transcoded MMS message to a device that cannot render the 3D media data of the MMS message.

[0005] In one example, a method for transmitting media data includes: receiving a set of three-dimensional (3D) media data being transmitted to a client device by a server device; transcoding the set of 3D media data into a set of one or more images by the server device; and transmitting the set of one or more images to the client device by the server device.

[0006] In another example, a server device for transmitting media data includes: a memory configured to store media data; and a processing system implemented in circuitry and configured to: receive a set of three-dimensional (3D) media data being transmitted to a client device; transcode the set of 3D media data into a set of one or more images; and transmit the set of one or more images to the client device.

[0007] In another example, a server device for transmitting media data includes: components for receiving a set of three-dimensional (3D) media data being transmitted to a client device; components for transcoding the set of 3D media data into a set of one or more images; and components for transmitting the set of one or more images to the client device.

[0008] In another example, a computer-readable storage medium having instructions stored thereon, which, when executed, cause a processor of a server device to: receive a set of three-dimensional (3D) media data being transmitted to a client device; transcode the set of 3D media data into a set of one or more images; and transmit the set of one or more images to the client device.

[0009] Details of one or more examples are set forth in the accompanying drawings and the following description. Other features, objects, and advantages will be apparent from these descriptions and drawings, and from the claims. Attached Figure Description

[0010] Figure 1 This is a block diagram illustrating an example system for implementing a technology for streaming media data over a network.

[0011] Figure 2 This is a block diagram illustrating the elements of the example video file.

[0012] Figure 3 This is a flowchart illustrating an example method for transcoding 3D media data according to the technology of this disclosure.

[0013] Figure 4 This is a block diagram illustrating a set of example devices capable of exchanging messages including 3D media data according to the technology of this disclosure.

[0014] Figure 5 This is a flowchart illustrating an example process of transmitting transcoded image data to a client device according to the technology of this disclosure. Detailed Implementation

[0015] Generally, this disclosure describes techniques related to exchanging 3D media data between two devices when one of the devices is unable to render 3D media data. The 3D media data can be, for example, augmented reality (AR), virtual reality (VR), mixed reality (MR), or other such extended reality (XR) data, or can be exchanged, for example, via a multimedia messaging service (MMS) as part of an AR, VR, MR, or XR session. In the following description, XR is discussed in general, and AR, VR, and MR represent examples of XR.

[0016] In some cases, a user with an XR-enabled device (e.g., a cellular phone, smartphone, tablet, etc.) may want to transmit a 3D representation of an object to another user with a device that does not support XR. For example, two users might be planning to buy a new piece of furniture and want to see how it will look in their existing space. Therefore, one user could find a new piece of furniture and capture an image of it, and their device could convert the image into a 3D representation, allowing the other device to render a virtual representation of the furniture within an image of the existing space.

[0017] However, when another device cannot render the 3D representation, according to the techniques disclosed herein, an intermediate device (such as an edge server in a 5G or other radio access network (RAN)) can transcode the 3D representation into an image or a series of images. In some cases, the edge server can determine that the other device cannot render the 3D representation. Therefore, in response to receiving the 3D representation, for example, in an MMS message or via an RTP session, the edge server can transcode the 3D representation into an image or series of images (e.g., a GIF or video data set) that can be processed by the other device. The edge server can then transmit the transcoded data (2D media data) to the other device.

[0018] In some examples, the receiving device may request specific movements of the virtual camera along the camera path, such as pitch, yaw, roll, and / or movement along the X, Y, and / or Z dimensions. In some examples, the edge server may be configured with a set of predefined virtual camera movements for the camera path. In either case, the edge server may move the virtual camera along the camera path and capture images along the camera path, for example, at a set of predefined locations along the path or at a specific frame rate, such as 30fps, 60fps, 120fps, etc.

[0019] The communication session between the two devices can be, for example, an RTP session, a multimedia message transmitted according to a multimedia messaging service (MMS), an XR session, or other communication sessions.

[0020] In this manner, the techniques disclosed herein can be used to provide 3D object data to devices that cannot render 3D object data. Therefore, a device can participate in a messaging session with another device and can receive 3D object data from another device as part of the messaging session, even if the 3D object data cannot be rendered locally. Thus, these techniques allow backward compatibility of 3D object data messaging with devices that cannot render 3D object data.

[0021] Figure 1This is a block diagram illustrating an example system 10 for implementing techniques for streaming media data over a network. In this example, system 10 includes a content preparation device 20, a server device 60, and a client device 40. Client device 40 and server device 60 are communicatively connected via a network 74, which may include the Internet. In some examples, content preparation device 20 and server device 60 may also be connected via network 74 or another network, or may be directly communicatively connected.

[0022] Content preparation device 20 represents a device that captures and prepares media data to be transmitted to client device 40. Client device 40, in turn, represents a device that receives the media data and presents it to a user of client device 40. In some examples, content preparation device 20 may also include the components shown in client device 40, and similarly, client device 40 may include the components shown in content preparation device 20. For example, content preparation device 20 and client device 40 may each represent a corresponding user equipment (UE) device, such as a mobile phone, tablet computer, or other communication device. Content preparation device 20 and client device 40 may participate in a two-way communication session that may include the exchange of audio, video, image, and / or three-dimensional (3D) media data.

[0023] In some cases, devices such as client device 40 may not be able to receive and process 3D media data received via a specific communication protocol, such as Multimedia Messaging Service (MMS). For example, content preparation device 20 may be configured to capture 3D media data, which may be a 3D object to be rendered by a destination device. That is, 3D media data can typically represent a set of vertices and edges that define the shape of a 3D object, as well as various properties indicating how the 3D object should be rendered (e.g., color, reflectivity, and other such information). Client device 40 may not be able to render the 3D media data.

[0024] According to the technology disclosed herein, a server device 60, positioned between a content preparation device 20 and a client device 40, can be configured to receive 3D media data from the content preparation device 20 and transcode the 3D media data into a set of image data. The image data may include still images, a series of images (e.g., as a GIF), or video data. In some examples, the server device 60 may render a video or image sequence based on predefined virtual camera movement around a 3D object. In some examples, the client device 40 may request one or more movements of the virtual camera, and the server device 60 may generate and transmit image or video data based on the requested movements of the virtual camera by the client device 40.

[0025] exist Figure 1In the example, content preparation device 20 includes an audio source 22 and a 3D media source 24. The audio source 22 may include, for example, a microphone that generates electrical signals representing captured audio data to be encoded by the audio encoder 26. Alternatively, the audio source 22 may include: a storage medium storing previously recorded audio data; an audio data generator, such as a computerized synthesizer; or any other audio data source. The 3D media source 24 may include a still image or video camera that captures one or more images to be used to generate 3D media data to be encoded by the 3D media encoder 28, a storage medium encoding previously recorded 3D media data, a 3D media data generation unit (such as a computer graphics source or game engine), or any other 3D media data source.

[0026] The raw data may include analog or digital data. Analog data may be digitized before being encoded by audio encoder 26 and / or 3D media encoder 28. While the speaker is speaking, audio source 22 may acquire audio data from the speaker, and 3D media source 24 may simultaneously acquire 3D media data of the speaker. In other examples, audio source 22 may include a computer-readable storage medium containing stored audio data, and 3D media source 24 may include a computer-readable storage medium containing stored 3D media data. Thus, the techniques described in this disclosure can be applied to live, streaming, real-time audio and 3D media data, or to archived, pre-recorded audio and 3D media data.

[0027] An audio frame corresponding to a 3D media frame is typically an audio frame containing audio data, which is simultaneously captured (or generated) by audio source 22 and 3D media data captured (or generated) by 3D media source 24 and contained within the 3D media frame. For example, when a speaker typically generates audio data by speaking, audio source 22 captures the audio data, and 3D media source 24 simultaneously (i.e., while audio source 22 is capturing audio data) captures the speaker's 3D media data. Therefore, an audio frame can temporally correspond to one or more specific 3D media frames. Thus, an audio frame corresponding to a 3D media frame typically corresponds to the following scenario: in which audio data and 3D media data are captured simultaneously, and in this scenario, the audio frame and the 3D media frame respectively include the simultaneously captured audio data and 3D media data.

[0028] In some examples, the audio encoder 26 may encode a timestamp into each encoded audio frame, where the timestamp indicates the time when the audio data for the encoded audio frame was recorded; similarly, the 3D media encoder 28 may encode a timestamp into each encoded 3D media frame, where the timestamp indicates the time when the 3D media data for the encoded 3D media frame was recorded. In such examples, the audio frame corresponding to the 3D media frame may include: an audio frame including a timestamp, and a 3D media frame including the same timestamp. The content preparation device 20 may include an internal clock, which the audio encoder 26 and / or the 3D media encoder 28 may use to generate timestamps, or the audio source 22 and the 3D media source 24 may use the internal clock to associate the audio and 3D media data with timestamps, respectively.

[0029] In some examples, audio source 22 may transmit data corresponding to the time when the audio data was recorded to audio encoder 26, and 3D media source 24 may transmit data corresponding to the time when the 3D media data was recorded to 3D media encoder 28. In some examples, audio encoder 26 may encode sequence identifiers into the encoded audio data to indicate the relative time order of the encoded audio data, rather than indicating the absolute time when the audio data was recorded; similarly, 3D media encoder 28 may use sequence identifiers to indicate the relative time order of the encoded 3D media data. Similarly, in some examples, sequence identifiers may be mapped or otherwise associated with timestamps.

[0030] Audio encoder 26 typically produces encoded audio data streams, while 3D media encoder 28 produces encoded 3D media data streams. Each individual data stream (whether audio or 3D media) can be referred to as a primary stream. A primary stream is a single, digitally decoded (possibly compressed) component of a media presentation. For example, a decoded 3D media or audio portion of a media presentation can be a primary stream. Primary streams can be converted into packed primary streams (PES) before being encapsulated within a 3D media file. Within the same media presentation, stream IDs can be used to distinguish PES packets belonging to one primary stream from those belonging to another. The basic unit of data in a primary stream is the packed primary stream (PES) packet. Therefore, decoded 3D media data typically corresponds to a primary 3D media stream. Similarly, audio data corresponds to one or more corresponding primary streams.

[0031] exist Figure 1In one example, the encapsulation unit 30 of the content preparation device 20 receives a base stream of decoded 3D media data from the 3D media encoder 28 and a base stream of decoded audio data from the audio encoder 26. In some examples, the 3D media encoder 28 and the audio encoder 26 may each include a packer for forming PES packets based on the encoded data. In other examples, the 3D media encoder 28 and the audio encoder 26 may each interface with a corresponding packer for forming PES packets based on the encoded data. In other examples, the encapsulation unit 30 may include a packer for forming PES packets based on the encoded audio and 3D media data.

[0032] The 3D media encoder 28 can encode 3D media data of multimedia content in various ways to produce different representations of the multimedia content at various bit rates and by utilizing various characteristics such as pixel resolution, frame rate, compliance with various decoding standards, compliance with various profiles and / or profile levels used for various decoding standards, representation with one or more views (e.g., for two-dimensional or three-dimensional playback), or other such characteristics.

[0033] Server device 60 includes a 3D-to-image transcoding unit 66, a Multimedia Messaging Service (MMS) sending unit 70, and a network interface 72. In some examples, server device 60 may include multiple network interfaces. Furthermore, any or all of the features of server device 60 may be implemented on other devices in the content delivery network, such as routers, bridges, proxy devices, switches, or other devices. In some examples, intermediate devices in the content delivery network may cache data for multimedia content 64 and include components that substantially conform to the components of server device 60. Generally, network interface 72 is configured to transmit and receive data via network 74.

[0034] According to the technology disclosed herein, server device 60 can receive data from client device 40 instructing client device 40 on its rendering capabilities for multimedia messages (e.g., based on RTP, MMS, etc.). For example, client device 40 may instruct server device 60 that it cannot render 3D media data.

[0035] Therefore, when server device 60 receives multimedia content 64 including 3D media data, 3D-to-image transcoding unit 66 can transcode the 3D media data into a collection of one or more images, such as a single image, a series of images, GIFs, video data, etc. Server device 60 can generate image sequences or image sequences encoded with High Efficiency Image File (HEIF), animated GIFs, etc. In some examples, server device 60 can form an alpha channel / image associated with each rendered image to achieve simulation of the AR experience. That is, client device 40 can present an image sequence on top of a background image or image sequence captured by client device 40, so that the alpha channel portion does not obstruct the background image or image sequence.

[0036] In some examples, content preparation device 20 or client device 40 may provide a requested or recommended camera path along which images of 3D media data should be rendered to generate an image sequence. The camera path may include an initial pose and a set of additional pose and time samples, where each pose / time sample pair provides the camera's pose and the time offset at which that pose should be activated. Alternatively, the camera path may include the camera's initial pose and a description of the path, including the shape of each segment of the path and the speed at which the camera moves. In some examples, server device 60 may select from a set of pre-configured camera paths for rendering the image sequence.

[0037] MMS sending unit 70 is configured to deliver media data to client device 40 via network 74 according to MMS. In other examples, MMS sending unit 70 may additionally or alternatively implement, for example, Real-time Transport Protocol (RTP), RTP Control Protocol (RTCP), Real-time Streaming Protocol (RTSP), Session Initiation Protocol (SIP), and / or Session Description Protocol (SDP). MMS sending unit 70 may transmit media data via network interface 72.

[0038] The MMS sending unit 70 of server device 60 can transmit media data (e.g., media data packets) to client device 40 according to MMS. Server device 60 and client device 40 can exchange control data indicating, for example, reception statistics of client device 40, so that server device 60 can perform congestion control or otherwise diagnose and resolve transmission failures.

[0039] Server device 60 can receive multimedia messages from content preparation device 20, wherein the multimedia messages are formatted as multi-part MIME messages. The first part of the message contains a GL Transmit Format (glTF) file or a GL Transmit Format Binary (glB) file including image data. Another part of the message may contain recommended camera information. Alternatively, the recommended camera information may be included as part of the glTF file, for example, as an animation element. 3D-to-image transcoding unit 66 can form transcoded messages from the received messages, which include a MIME portion indicating an image / image sequence.

[0040] In this way, server device 60 can implement a technique for adding support for 3D content in multimedia messaging and addressing devices that do not support 3D content in multimedia messages. The 3D content can be rendered by server device 60 (which may be a multimedia server), allowing server device 60 to transcode the 3D content into images or image sequences supported by client device 40.

[0041] Network interface 54 can receive selected media and provide it to MMS receiving unit 52, which in turn can provide the media data to decapsulation unit 50. Decapsulation unit 50 can decapsulate the elements of the 3D media file into component parts to retrieve encoded data, and transmit the encoded data to audio decoder 46 or image decoder 48. Audio decoder 46 decodes the encoded audio data and transmits the decoded audio data to audio output unit 42, while image decoder 48 decodes the encoded image data and transmits the decoded image data (which may include still images, a series of images (e.g., GIF files), and / or video data) to image output unit 44.

[0042] The 3D media encoder 28, image decoder 48, audio encoder 26, audio decoder 46, encapsulation unit 30, MMS receiving unit 52, and decapsulation unit 50 can each be implemented as any of a variety of suitable processing circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic circuit systems, software, hardware, firmware, or any combination thereof. Each of the 3D media encoder 28 and image decoder 48 can be included in one or more encoders or decoders, and either the video encoder or the video decoder can be integrated as part of a combined media encoder / decoder (CODEC). Similarly, each of the audio encoder 26 and audio decoder 46 can be included in one or more encoders or decoders, and either the audio encoder or the audio decoder can be integrated as part of a combined CODEC. The device including 3D media encoder 28, image decoder 48, audio encoder 26, audio decoder 46, encapsulation unit 30, MMS receiver unit 52 and / or decapsulation unit 50 may include integrated circuits, microprocessors and / or wireless communication devices, such as cellular phones.

[0043] Client device 40, server device 60, and / or content preparation device 20 may be configured to operate according to the techniques of this disclosure. For illustrative purposes, these techniques are described with respect to client device 40 and server device 60. However, it should be understood that content preparation device 20 may also be configured to perform these techniques as an alternative to (or other than) server device 60.

[0044] Encapsulation unit 30 can form NAL units, which include a header identifying the program to which the NAL unit belongs and a payload, such as audio data, video data, or data describing the transport or program stream corresponding to the NAL unit. For example, in H.264 / AVC, a NAL unit includes a 1-byte header and a payload of varying size. NAL units whose payloads include video data can include video data at various granularities. For example, a NAL unit can include video data blocks, multiple blocks, video data slices, or entire frames of video data. Encapsulation unit 30 can receive encoded video data in PES packet format with elementary streams from video encoder 28. Encapsulation unit 30 can associate each elementary stream with its corresponding program.

[0045] The encapsulation unit 30 can also assemble access units based on multiple NAL units. Generally, an access unit may include one or more NAL units representing a video data frame, and the corresponding audio data (when the audio data is available). Access units typically include all NAL units for a single output time instance, such as all audio and video data for a single time instance. For example, if each view has a frame rate of 20 frames per second (fps), each time instance may correspond to a time interval of 0.05 seconds. During this time interval, specific frames for all views with the same access unit (same time instance) can be rendered simultaneously. In one example, an access unit may include a decoded image within a time instance, which can be rendered as the primary decoded image.

[0046] Therefore, an access unit may include all audio, 3D media, and / or video frames of a common time instance, such as all views corresponding to time X. This disclosure also refers to the encoded picture of a particular view as a "view component." That is, a view component may include pictures (or frames) encoded for a particular view at a particular time. Therefore, an access unit can be defined as including all view components of a common time instance. The decoding order of the access units need not be the same as the output or display order.

[0047] Figure 2 This is a block diagram illustrating the elements of an example video file 150. The video file 150 can be included in a message transmitted according to a messaging service, such as Multimedia Messaging Service (MMS). Therefore, the video file 150 can be included in an MMS message. Alternatively or additionally, the video file 150 can be included in one or more Multimedia Message Body (MMBP) portions of a multimedia message, which can be an MMS message or other messaging protocol message.

[0048] As described above, video files, based on the ISO basic media file format and its extensions, store data in a series of objects called "boxes". Figure 2 In the example, video file 150 includes a file type (FTYP) box 152, a movie (MOOV) box 154, a segment index (sidx) box 162, a movie clip (MOOF) box 164, and a movie clip random access (MFRA) box 166. Although Figure 2 The example shown is a video file, but it should be understood that other media files may include other types of media data (e.g., audio data, timed text data, etc.) constructed similarly to those in video file 150, depending on the ISO Basic Media File Format and its extensions.

[0049] The File Type (FTYP) box 152 generally describes the file type of the video file 150. The File Type box 152 may include data identifying the specifications for the best use of the video file 150. The File Type box 152 may optionally be placed before the MOOV box 154, the Movie Clip box 164, and / or the MFRA box 166.

[0050] exist Figure 2 In the example, the MOOV box 154 includes a Movie Header (MVHD) box 156, a Track (TRAK) box 158, and one or more Movie Extension (MVEX) boxes 160. Typically, the MVHD box 156 describes general characteristics of the video file 150. For example, the MVHD box 156 may include data describing when the video file 150 was initially created, when the video file 150 was last modified, the timestamp of the video file 150, the playback duration of the video file 150, or other data generally describing the video file 150.

[0051] TRAK box 158 may include track data of video file 150. TRAK box 158 may include a Track Header (TKHD) box that describes the characteristics of the track corresponding to TRAK box 158. In some examples, TRAK box 158 may include decoded video images, while in other examples, the decoded video images of the track may be included in movie clip 164, which may be referenced by data from TRAK box 158 and / or sidx box 162.

[0052] In some examples, video file 150 may include more than one track. Therefore, MOOV box 154 may include TRAK boxes in a number equal to the number of tracks in video file 150. TRAK boxes 158 may describe the characteristics of the corresponding tracks in video file 150. For example, TRAK boxes 158 may describe the temporal and / or spatial information of the corresponding tracks. When encapsulation unit 30 ( Figure 1 When a parameter set track is included in a video file (such as video file 150), a TRAK box similar to the TRAK box 158 of the MOOV box 154 can describe the characteristics of the parameter set track. The encapsulation unit 30 can signal the presence of a sequence level SEI message in the parameter set track within the TRAK box describing the parameter set track.

[0053] The MVEX box 160 can describe the characteristics of the corresponding movie clip 164, for example, by signaling to the video file 150 that, in addition to the video data (if any) included in the MOOV box 154, the movie clip 164 is also included. In the context of streaming video data, decoded video images may be included in the movie clip 164 instead of the MOOV box 154. Therefore, all decoded video samples may be included in the movie clip 164 instead of the MOOV box 154.

[0054] MOOV boxes 154 may include an equal number of MVEX boxes 160 as the number of movie segments 164 in the video file 150. Each MVEX box in the MVEX boxes 160 may describe the characteristics of a corresponding movie segment in the movie segment 164. For example, each MVEX box may include a Movie Extended Header Box (MEHD) box, which describes the time duration of a corresponding movie segment in the movie segment 164.

[0055] As described above, encapsulation unit 30 may store a set of sequence data in a video sample that does not include the actual decoded video data. The video sample may typically correspond to an access unit, which is a representation of a decoded picture at a specific time instance. In the context of AVC, the decoded picture includes one or more VCL NAL units (containing information about all pixels used to construct the access unit) and other associated non-VCL NAL units (such as SEI messages). Therefore, encapsulation unit 30 may include a set of sequence data in a movie clip of movie clip 164, which may include a sequence-level SEI message. Encapsulation unit 30 may also signal the presence of the set of sequence data and / or the sequence-level SEI message as being present in a movie clip of movie clip 164 within an MVEX box of MVEX box 160 corresponding to a movie clip of movie clip 164.

[0056] SIDX box 162 is an optional element of video file 150. That is, video files conforming to 3GPP file formats or other such file formats do not necessarily need to include SIDX box 162. According to the example of the 3GPP file format, SIDX boxes can be used to identify sub-segments of a segment (e.g., a segment contained within video file 150). The 3GPP file format defines a sub-segment as "a self-contained set of one or more consecutive movie clip boxes having a corresponding media data box and a media data box containing data referenced by a movie clip box that must follow that movie clip box and precede the next movie clip box containing information about the same track." The 3GPP file format also instructs that the SIDX box "contains a series of references to sub-segments of the (sub)segment recorded by that box. The referenced sub-segments are consecutive in presentation time. Similarly, bytes referenced by the segment index box are always consecutive within the segment. The size of the reference gives a count of the number of bytes in the referenced material."

[0057] SIDX box 162 typically provides information representing one or more sub-segments of a segment included in video file 150. For example, such information may include the playback time of the start and / or end of the sub-segment, the byte offset of the sub-segment, whether the sub-segment includes (e.g., begins at) a Stream Access Point (SAP), the type of SAP (e.g., whether the SAP is an Instant Decoder Refresh (IDR) picture, a Clean Random Access (CRA) picture, a Broken Link Access (BLA) picture, etc.), the location of the SAP within the sub-segment (in terms of playback time and / or byte offset), etc.

[0058] Movie clip 164 may include one or more decoded video pictures. In some examples, movie clip 164 may include one or more groups of pictures (GOPs), where each GOP may include several decoded video pictures, such as frames or images. Additionally, as described above, in some examples, movie clip 164 may include a sequence of data. Each movie clip in movie clip 164 may include a movie clip header box (MFHD). Figure 2 (Not shown in the image). The MFHD box can describe the characteristics of the corresponding movie clip, such as the movie clip's sequence number. Movie clips 164 can be included in video file 150 in the order of their sequence numbers.

[0059] MFRA box 166 can describe random access points within movie segments 164 of video file 150. This can help perform special effects modes, such as searching for specific time positions (i.e., playback times) within segments encapsulated by video file 150. In some examples, MFRA box 166 is typically optional and does not need to be included in the video file. Similarly, client devices (such as client device 40) do not necessarily need to refer to MFRA box 166 to correctly decode and display the video data of video file 150. MFRA box 166 can include a number equal to the number of tracks in video file 150, or in some examples, a number equal to the number of media tracks (e.g., non-cue tracks) in video file 150 (TFRA) boxes (not shown).

[0060] In some examples, movie clip 164 may include one or more streaming access points (SAPs), such as IDR pictures. Similarly, MFRA box 166 can provide an indication of the location within the SAP video file 150. Therefore, a temporal subsequence of video file 150 can be formed from the SAP of video file 150. The temporal subsequence may also include other pictures, such as SAP-dependent P-frames and / or B-frames. Frames and / or slices of the temporal subsequence can be arranged within segments such that frames / slices of the temporal subsequence that depend on other frames / slices of the subsequence can be correctly decoded. For example, in a hierarchical arrangement of data, data used for prediction of other data may also be included in the temporal subsequence.

[0061] Figure 3 This is a flowchart illustrating an example method for transcoding 3D media data according to the technology of this disclosure. Figure 3 In the example, UE A can correspond to Figure 1 Content preparation device 20, multimedia messaging server can correspond to Figure 1 The server device 60, and UE B can correspond to Figure 1 40 client devices.

[0062] Initially, UE B can register a device profile (100) with the multimedia messaging server, demonstrating its capabilities. UE A can then transmit a multimedia message with 3D content (102). The multimedia messaging server can detect that UE B does not support 3D content and therefore performs transcoding (104). For example, the multimedia messaging server can render 3D media data into one or more images and then encode one or more images. The encoded images can be a sequence of encoded images in HEIF format. The multimedia messaging server can then transmit the transcoded message to UE B (106).

[0063] Figure 4 This is a block diagram illustrating a set of example devices capable of exchanging messages including 3D media data according to the technology of this disclosure. In this example, Figure 4 User equipment (UE) device 120 and UE 122, and multimedia messaging server (MMS) 124 are depicted. In this example, MMS 124 and UE 122 are communicatively coupled via radio access network (RAN) 130. MMS 124 may correspond to an edge computing server.

[0064] Initially (e.g., when UE 122 becomes communicatively coupled to RAN 130), UE 122 may transmit rendering capability data (i.e., data indicating UE 122's ability to render various types of data, such as image and video data, and 3D media data) to MMS 124. In this example, it is assumed that UE 122 cannot render 3D media data. Therefore, UE 122 may transmit data to MMS 124 indicating that UE 122 cannot render 3D media data but is capable of rendering image data.

[0065] At some point, UE 120 and UE 122 can initiate a messaging session, such as a Multimedia Messaging Service (MMS) session. UE 120 can send a message to UE 122 that includes 3D media data (also referred to herein as 3D object data). MMS 124 can receive this message and determine that it is destined for UE 122. MMS 124 can determine that the message includes 3D media data. Based on rendering capability data received from UE 122, MMS 124 can extract the 3D media data and render (i.e., transcode) the 3D media data into image data on behalf of UE 122, and then transmit the 3D media data to UE 122. For example, MMS 124 can update a message received from UE 120 to include rendered image data instead of 3D media data. In this way, UE 122 can participate in an MMS session that includes exchanging 3D media data with UE 120, although it cannot partially render such 3D media data.

[0066] Figure 5 This is a flowchart illustrating an example process of transmitting transcoded image data to a client device according to the technology of this disclosure. Figure 5 The method can be performed by a server device (e.g., a Multimedia Messaging Server (MMS) device), such as Figure 1 Server equipment 60 or Figure 4 MMS 124. For illustrative and illustrative purposes, regarding Figure 4 MMS device 124 to explain Figure 5 The method.

[0067] Initially, MMS 124 receives rendering capability data from the client device (250). The rendering capability data may indicate that the client device cannot render 3D media data. MMS 124 may then receive a message (e.g., an MMBP message or an MMS message) containing 3D media data destined for the client device (252). Based on the client device's rendering capability data, MMS 124 can determine that the client device cannot render 3D media data. Therefore, MMS 124 may transcode (e.g., render) the 3D media data into image data (e.g., one or more images, such as a single image or a sequence of images) (254), and then transmit the image data to the client device (256).

[0068] In this way, Figure 5 The method represents an example of a method for transmitting media data, including: receiving a set of three-dimensional (3D) media data being transmitted to a client device by a server device; transcoding the set of 3D media data into a set of one or more images by the server device; and transmitting the set of one or more images to the client device by the server device.

[0069] The following clauses illustrate various examples of the technology disclosed herein:

[0070] Clause 1. A method for transmitting media data, the method comprising: receiving, by a server device, a set of three-dimensional (3D) media data being transmitted to a client device; transcoding, by the server device, the set of 3D media data into a set of one or more images; and transmitting, by the server device, the set of one or more images to the client device.

[0071] Clause 2. The method according to Clause 1, the method further comprising determining that the client device cannot render the set of 3D media data.

[0072] Clause 3. The method according to Clause 2, wherein determining that the client device cannot render 3D media data includes receiving data representing the capabilities of the client device.

[0073] Clause 4. The method according to any one of Clauses 1 to 3, wherein the client device includes a first client device, and the set of receiving 3D media data includes the set of receiving 3D media data from a second client device.

[0074] Clause 5. The method according to any one of Clauses 1 to 4, wherein receiving the set of 3D media data includes receiving Multimedia Messaging Service (MMS) messages.

[0075] Clause 6. The method described in Clause 5, wherein the MMS message is formatted as a multipart MIME message.

[0076] Clause 7. The method according to any one of Clauses 5 and 6, wherein the MMS message includes a GL Send Format (glTF) or GL Send Format Binary (glB) file that includes the 3D media content.

[0077] Clause 8. The method according to any one of Clauses 1 to 7, the method further comprising receiving data indicating a recommended camera path and rendering the one or more images along the recommended camera path.

[0078] Clause 9. The method according to Clause 8, wherein the recommended camera path comprises an initial pose and a set of additional pose and time sample pairs, wherein each of the additional pose and time sample pairs provides the pose of the virtual camera and the time offset for activating the pose.

[0079] Clause 10. The method according to Clause 8, wherein the recommended camera path includes an initial pose and a description of the recommended camera path, the description including the shape of each segment of the recommended camera path and the speed of camera movement along the segment.

[0080] Clause 11. The method according to any one of Clauses 1 to 10, wherein transcoding the 3D media data comprises forming an image sequence encoded in High Efficiency Image File (HEIF) that includes the set of one or more images.

[0081] Clause 12. The method according to any one of Clauses 1 to 11, wherein transcoding the 3D media data comprises rendering the set of one or more images to include an alpha channel corresponding to a portion of the image surrounding one or more 3D objects in the 3D media data.

[0082] Clause 13. The method according to Clause 1, the method further comprising determining that the client device cannot render the set of 3D media data.

[0083] Clause 14. The method according to Clause 13, wherein determining that the client device cannot render 3D media data includes receiving data representing the capabilities of the client device.

[0084] Clause 15. The method according to Clause 1, wherein the client device includes a first client device, and the set of receiving 3D media data includes the set of receiving 3D media data from a second client device.

[0085] Clause 16. The method according to Clause 1, wherein receiving the set of 3D media data includes receiving Multimedia Messaging Service (MMS) messages.

[0086] Clause 17. The method according to Clause 16, wherein the MMS message is formatted as a multipart MIME message.

[0087] Clause 18. The method according to Clause 16, wherein the MMS message includes a GL Transmission Format (glTF) or GL Transmission Format Binary (glB) file that includes the 3D media content.

[0088] Clause 19. The method according to Clause 1, the method further comprising receiving data indicating a recommended camera path and rendering the one or more images along the recommended camera path.

[0089] Clause 20. The method according to Clause 19, wherein the recommended camera path comprises an initial pose and a set of additional pose and time sample pairs, wherein each of the additional pose and time sample pairs provides the pose of the virtual camera and the time offset for activating the pose.

[0090] Clause 21. The method according to Clause 19, wherein the recommended camera path includes an initial pose and a description of the recommended camera path, the description including the shape of each segment of the recommended camera path and the speed of camera movement along the segment.

[0091] Clause 22. The method according to Clause 1, wherein transcoding the 3D media data comprises forming an image sequence encoded with High Efficiency Image File (HEIF) including the one or more image sets.

[0092] Clause 23. The method according to Clause 1, wherein transcoding the 3D media data includes rendering the set of one or more images to include an alpha channel, the alpha channel corresponding to a portion of the image surrounding one or more 3D objects in the 3D media data.

[0093] Clause 24. An apparatus for retrieving media data, the apparatus comprising one or more components for performing the method according to any one of Clauses 1 to 23.

[0094] Clause 25. The device according to Clause 24, wherein the one or more components include a processing system, the processing system including one or more processors implemented in a circuit.

[0095] Clause 26. The apparatus according to Clause 24, wherein the apparatus includes at least one of: an integrated circuit; a microprocessor; and a wireless communication device.

[0096] Clause 27. A computer-readable storage medium having instructions stored thereon, which, when executed, cause a processor to perform a method according to any one of Clauses 1 to 23.

[0097] Clause 28. An apparatus for transmitting media data, the apparatus comprising: components for receiving a set of three-dimensional (3D) media data being transmitted to a client device; components for transcoding the set of 3D media data into a set of one or more images; and components for transmitting the set of one or more images to the client device.

[0098] Clause 29. A method for transmitting media data, the method comprising: receiving, by a server device, a set of three-dimensional (3D) media data being transmitted to a client device; transcoding, by the server device, the set of 3D media data into a set of one or more images; and transmitting, by the server device, the set of one or more images to the client device.

[0099] Clause 30. The method according to Clause 29, the method further comprising determining that the client device cannot render the set of 3D media data.

[0100] Clause 31. The method according to Clause 30, wherein determining that the client device cannot render 3D media data includes receiving data from the client device representing the rendering capabilities of the client device.

[0101] Clause 32. The method according to Clause 29, wherein the client device includes a first client device, and wherein the set of receiving 3D media data includes receiving the set of 3D media data from a second client device as a message from the second client device to the first client device.

[0102] Clause 33. The method according to Clause 32, wherein the message includes one or more Multimedia Messaging Body Parts (MMBPs), at least one of the MMBPs including the 3D media data.

[0103] Clause 34. The method according to Clause 32, wherein receiving the set of 3D media data as a message includes receiving Multimedia Messaging Service (MMS) messages.

[0104] Clause 35. The method according to Clause 34, wherein the MMS message is formatted as a multipart MIME message.

[0105] Clause 36. The method according to Clause 34, wherein the MMS message includes a GL Transmission Format (glTF) or GL Transmission Format Binary (glB) file that includes the 3D media data.

[0106] Clause 37. The method according to Clause 29, the method further comprising receiving data indicating a recommended camera path and rendering the one or more images along the recommended camera path.

[0107] Clause 38. The method according to Clause 37, wherein the recommended camera path comprises an initial pose and a set of additional pose and time sample pairs, wherein each of the additional pose and time sample pairs provides the pose of the virtual camera and the time offset for activating the pose.

[0108] Clause 39. The method according to Clause 37, wherein the recommended camera path includes an initial pose and a description of the recommended camera path, the description including the shape of each segment of the recommended camera path and the speed of camera movement along the segment.

[0109] Clause 40. The method according to Clause 29, wherein transcoding the 3D media data comprises forming an image sequence encoded in High Efficiency Image File (HEIF) that includes the set of one or more images.

[0110] Clause 41. The method according to Clause 29, wherein transcoding the 3D media data includes rendering the set of one or more images to include an alpha channel, the alpha channel corresponding to a portion of the image surrounding one or more 3D objects in the 3D media data.

[0111] Clause 42. A server device for transmitting media data, the server device comprising: a memory configured to store media data; and a processing system implemented in circuitry and configured to: receive a set of three-dimensional (3D) media data being transmitted to a client device; transcode the set of 3D media data into a set of one or more images; and transmit the set of one or more images to the client device.

[0112] Clause 43. The server device as described in Clause 42, wherein the processing system is further configured to receive from the client device data representing the rendering capabilities of the client device and indicating that the client device cannot render the set of 3D media data.

[0113] Clause 44. The server device according to Clause 42, wherein the client device includes a first client device, and wherein, in order to receive the set of 3D media data, the processing system is configured to receive the set of 3D media data from a second client device as a message from the second client device to the first client device.

[0114] Clause 45. The server device as described in Clause 44, wherein the message includes one or more Multimedia Message Passing Body Parts (MMBPs), at least one of the MMBPs including the 3D media data.

[0115] Clause 46. The server device as described in Clause 44, wherein the message includes a Multimedia Messaging Service (MMS) message, the MMS message including a GL Send Format (glTF) or GL Send Format Binary (glB) file, which includes the 3D media data.

[0116] Clause 47. The server device according to Clause 42, wherein the processing system is further configured to receive data indicating a recommended camera path and render the one or more images along the recommended camera path.

[0117] Clause 48. A server device for transmitting media data, the device comprising: components for receiving a set of three-dimensional (3D) media data being transmitted to a client device; components for transcoding the set of 3D media data into a set of one or more images; and components for transmitting the set of one or more images to the client device.

[0118] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or code on a computer-readable medium or transmitted via a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium (which corresponds to a tangible medium such as a data storage medium) or a communication medium, including, for example, any medium that facilitates the transfer of a computer program from one place to another according to a communication protocol. Thus, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to extract instructions, code, and / or data structures for implementing the techniques described in this disclosure. Computer program products may include computer-readable media.

[0119] By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other medium capable of storing desired program code in the form of instructions or data structures and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies (such as infrared, radio, and microwave), then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies (such as infrared, radio, and microwave) are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but instead refer to non-transient tangible storage media. As used herein, disks and optical discs include compact optical discs (CDs), laser discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, and optical discs optically reproduce data using lasers. The combinations described above should also be included within the scope of computer-readable media.

[0120] Instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, as used herein, the term "processor" can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Furthermore, these techniques can be fully implemented in one or more circuit or logic elements.

[0121] The techniques disclosed herein can be implemented in a wide variety of devices or apparatuses, including wireless mobile phones, integrated circuits (ICs), or a set of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed techniques, but implementation by different hardware units is not necessarily required. Specifically, as described above, various units may be combined in a codec hardware unit, or various units may be provided by a collection of interoperable hardware units (including one or more processors as described above) combined with appropriate software and / or firmware.

[0122] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A method for transmitting media data, the method comprising: A collection of three-dimensional (3D) media data that is being transmitted from a server device to a client device; The server device transcodes the set of 3D media data into a set of one or more images; as well as The server device transmits the set of one or more images to the client device.

2. The method of claim 1, further comprising determining the set of 3D media data that the client device cannot render.

3. The method of claim 2, wherein determining the set of 3D media data that the client device cannot render comprises receiving data from the client device representing the rendering capabilities of the client device.

4. The method of claim 1, wherein the client device includes a first client device, and wherein the set of receiving 3D media data includes receiving the set of 3D media data from a second client device as a message from the second client device to the first client device.

5. The method of claim 4, wherein the message includes one or more Multimedia Message Passing Body Parts (MMBPs), and at least one of the MMBPs includes the 3D media data.

6. The method of claim 4, wherein receiving the set of 3D media data as a message includes receiving a Multimedia Message Service (MMS) message.

7. The method of claim 6, wherein the MMS message is formatted as a multipart MIME message.

8. The method of claim 6, wherein the MMS message comprises a GL Transmission Format (glTF) or GL Transmission Format Binary (glB) file, which includes the 3D media data.

9. The method of claim 1, further comprising receiving data indicating a recommended camera path and rendering the one or more images along the recommended camera path.

10. The method of claim 9, wherein the recommended camera path comprises an initial pose and a set of additional pose and time sample pairs, wherein each of the additional pose and time sample pairs provides the pose of the virtual camera and the time offset for activating the pose.

11. The method of claim 9, wherein the recommended camera path includes an initial pose and a description of the recommended camera path, the description including the shape of each segment of the recommended camera path and the speed of camera movement along the segment.

12. The method of claim 1, wherein transcoding the 3D media data comprises forming an image sequence encoded in High Efficiency Image File (HEIF) that includes the set of one or more images.

13. The method of claim 1, wherein transcoding the 3D media data comprises rendering the set of one or more images to include an alpha channel corresponding to a portion of the image surrounding one or more 3D objects in the 3D media data.

14. A server device for transmitting media data, the server device comprising: A memory configured to store media data; and A processing system, implemented in a circuit and configured to: Receive a collection of three-dimensional (3D) media data that is being transmitted to a client device; The set of 3D media data is transcoded into a set of one or more images; as well as The set of one or more images is transmitted to the client device.

15. The server device of claim 14, wherein the processing system is further configured to receive from the client device data representing the rendering capabilities of the client device and indicating that the client device cannot render the set of 3D media data.

16. The server device of claim 14, wherein the client device includes a first client device, and wherein, in order to receive the set of 3D media data, the processing system is configured to receive the set of 3D media data from a second client device as a message from the second client device to the first client device.

17. The server device of claim 16, wherein the message includes one or more Multimedia Message Passing Body Parts (MMBPs), at least one of the MMBPs including the 3D media data.

18. The server device of claim 16, wherein the message includes a Multimedia Message Service (MMS) message, the MMS message including a GL Send Format (glTF) or GL Send Format Binary (glB) file, which includes the 3D media data.

19. The server device of claim 14, wherein the processing system is further configured to receive data indicating a recommended camera path and render the one or more images along the recommended camera path.

20. A server device for transmitting media data, the server device comprising: A component used to receive a collection of three-dimensional (3D) media data being transmitted to a client device; A component for transcoding the set of 3D media data into a set of one or more images; and A component for transmitting the set of one or more images to the client device.

Images