Method, apparatus, non-transitory computer-readable medium, and computer program for COAP support for an IOT streaming device in a media scene description system

Abstract

Aspects of the present disclosure provide a method and apparatus for accessing a Constrained Application Protocol (CoAP) server in a media scene description system. A CoAP request can be sent to a CoAP server to request a media resource by a media access function (MAF) of a processing circuit implementing the media scene description system using an application programming interface (API). The MAF can be configured as a CoAP client or a Hypertext Transfer Protocol (HTTP)-CoAP proxy. In one example, a CoAP response can be received from the CoAP server by the MAF using the API, the CoAP response including the requested media resource. In one embodiment, the MAF is compatible with both CoAP requests according to the CoAP communication protocol and proxy requests according to the HTTP communication protocol.

Description

Technical Field

[0001] [Cross - Reference to Related Applications] This application claims the benefit of priority to U.S. Patent Application No. 17 / 499,561, filed on October 12, 2021, "METHOD AND APPARATUS OF COAP SUPPORT FOR IOT STREAMING DEVICES IN A MEDIA SCENE DESCRIPTION SYSTEM", which claims the benefit of priority to U.S. Provisional Application No. 63 / 177,783, filed on April 21, 2021, "METHOD AND APPARATUS OF COAP SUPPORT FOR IOT STREAMING DEVICES IN SCENE DESCRIPTION". The entire disclosure of the prior application is hereby incorporated by reference in its entirety.

[0002] This disclosure describes embodiments generally related to a system design that uses 3D modeling syntax to support media objects, implements media syntax for supporting various media codecs, containers, and formats, manages media storage and delivery methods via a predefined programming interface, and provides media buffer control and rendering functionality.

Background Art

[0003] The description of the background art provided herein is for the purpose of generally presenting the context of the disclosure. The work of the inventors, as currently named, is not admitted as prior art to the present disclosure, either expressly or implicitly, to the extent that it is described in this background art section and in aspects of the specification that may not be regarded as prior art at the time of filing.

[0004] The Constrained Application Protocol (CoAP) is an Internet application protocol for constrained devices. CoAP enables constrained devices, called "nodes," to communicate with the wider Internet using a similar protocol. CoAP is designed for use between devices on the same constrained network (e.g., low-power, lossy networks), between a device on the Internet and a general node, and between devices on different constrained networks connected by the Internet.

Summary of the Invention

[0005] Aspects of the present disclosure provide a method and apparatus for accessing a Constrained Application Protocol (CoAP) server in a media scene description system. The apparatus can include a processing circuit implementing the media scene description system. CoAP requests can be sent to the CoAP server by the media access function (MAF) of the processing circuit using an application programming interface (API) to request a media resource. The MAF can be configured as a CoAP client or a Hypertext Transfer Protocol (HTTP)-CoAP proxy. In one embodiment, the MAF is compatible with both CoAP requests according to the CoAP communication protocol and proxy requests according to the HTTP communication protocol.

[0006] CoAP responses can be received by the MAF from the CoAP server using the API. The CoAP response can include the requested media resource.

[0007] In one embodiment, the received media resource can be processed by one of (i) a video decoder, (ii) an audio decoder, or (iii) a data compressor. The processed media resource can be rendered by a presentation engine.

[0008] In one embodiment, the MAF is configured as an HTTP-CoAP proxy and the API is an HTTP-CoAP proxy API. In one example, an HTTP request can be mapped from an HTTP client to a CoAP request by the MAF using the HTTP-CoAP proxy API. In one example, the HTTP request is a proxy request. In one example, a CoAP response is mapped from a CoAP server to an HTTP response by the MAF using the HTTP-CoAP proxy API. The HTTP response can be sent to the HTTP client.

[0009] In one embodiment, the MAF is configured as a CoAP client and the API is a CoAP client API.

[0010] In one embodiment, the MAF is further configured to use HTTP.

[0011] Aspects of the disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform a method of accessing a CoAP server of a media scene description system.

Brief Description of the Drawings

[0012] Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings.

[0013]

Figure 1

[0014]

Figure 2

[0015]

Figure 3A

[0016]

Figure 3B

[0017]

Figure 4

[0018]

Figure 5

[0019]

Figure 6

[0020]

Figure 7

[0021]

Figure 8

[0022]

Figure 9

[0023]

Figure 10

[0024]

Figure 11

DETAILED DESCRIPTION OF THE INVENTION

[0025] FIG. 1 is a schematic diagram of a Graphics Language Transmission Format (glTF) scene description object according to an embodiment of the present disclosure. glTF is a standard file format for three-dimensional (3D) scenes and models. glTF can support 3D model geometry, appearance, scene graph hierarchy, and animation. glTF is a runtime asset 3D modeling distribution format that is neutral to application programming interfaces (APIs). In some embodiments, compared to conventional 3D modeling tools, glTF can provide a more efficient, extensible, and interoperable format for transmitting and loading 3D content. glTF can be a rationalized interoperable format for distributing 3D assets, while minimizing the file size and runtime processing by the application.

[0026] glTF files can use one of two file extensions, namely.gltf (i.e., JSON / ASCII) and.glb (i.e., binary). A.glTF file can be self - contained or can reference external binary and texture resources.

[0027] A glTF scene can be a combination of multiple glTF assets. A glTF asset can be a file in JSON format containing a complete scene description. Referring to FIG. 1, a complete scene description can include a scene object (101), nodes (102), cameras (103), meshes (104), lights (105), animations (106), accessors (107), materials (108), skins (109), buffer views (110), techniques (111), textures (112), buffers (113), programs (114), images (115), samplers (116), and shaders (117). A complete scene description can also include external data it supports.

[0028] glTF can support external data sources that can be referenced by any of the above scene objects. In various examples, binary files can be used for animations (106) or other buffer-based data (113). Image files can be used for object textures (112).

[0029] FIG. 2 shows an example of a glTF JavaScript (TM) object notation (JSON) format representation according to an embodiment of the present disclosure. JSON is an open standard file format, a data exchange format that uses human-readable text to store and transmit data objects containing attribute-value pairs and arrays (or other serializable values). JSON is a common data format with various functions in data exchange including communication with the server of a web application.

[0030] As described above, a glTF scene can be organized in JSON format. A glTF asset can include zero or more scenes (203), for example, a set of visual objects for rendering. A scene can be defined in a scene array. FIG. 2 shows a single scene (206) (e.g., scene 0) with a single node (201) (e.g., node 0 (205)). Various parameters can be associated with each node object. A name (202) can specify the name of the node object. A scene name (204) can specify the name of a single scene (206).

[0031] A glTF scene asset can be used by a presentation engine to render a 3D or immersive scene to a user. In some examples, the glTF syntax supports only 3D objects including static or computer-generated animations. In some examples, the glTF syntax does not support media types such as video or audio and does not render media types of video and / or audio. Certain glTFs cannot describe a scene using a geographic coordinate system, and in some media presentation scenarios, it may be desirable to describe a scene using a geographic coordinate system.

[0032] Therefore, it is necessary to extend glTF to support immersive media content such as 2D flat video, virtual reality (VR), augmented reality (AR), and / or extended reality (XR), and media types including spatial audio. XR can refer to a composite environment of reality and virtuality and the interaction between humans and machines generated by computer technology and wearables, and "X" can represent a variable of any suitable current or future spatial computing technology. In one example, XR includes typical forms such as AR, mixed reality (MR), virtual reality (VR), and the areas interpolated between them. Therefore, in some examples, an extension of glTF that supports the syntax of video and / or audio as well as the system for media delivery and rendering is required.

[0033] In one embodiment, the Moving Picture Experts Group (MPEG) defines specific extensions on top of the glTF specification to support immersive media content. Referring to FIG. 1, the extensions can include MPEG_media(130), MPEG_scene_dynamic(131), MPEG_animation_timing(132), MPEG_texture_video(133), MPEG_audio_spatial(134), MPEG_accessor_timed(135), MPEG_buffer_circular(136), etc.

[0034] In one example, when MPEG_media(130) is specified as the root identifier, MPEG_media(130) can be supported. Referring to FIG. 3A, the syntax supporting MPEG media can be declared as the top-level JSON syntax. In one example, the syntax from 301 to 304 can be accurately displayed as shown if supported. The syntax from 301 to 304 can include extensionsRequired(301), MPEG_media(302), extensionsUsed(303), and MPEG_media(304).

[0035] Scene Updates can be represented using the JSON Patch protocol, and MPEG_scene_dynamic(131) can be used to support the JSON Patch protocol.

[0036] The MPEG texture video extension identified by MPEG_texture_video(133) can provide the possibility of linking glTF texture objects to MPEG media and their respective tracks listed by the MPEG_media object. The MPEG texture video extension can provide a reference to MPEG_accessor_timed(135) that can enable the use of decoded timed textures. The MPEG_audio_spatial(134) extension supports multiple audio types.

[0037] To support timed data access, buffer elements can be extended to provide a circular buffer function. The extension is named MPEG_buffer_circular(136) and can be included as part of the glTF "buffers" object.

[0038] The aforementioned MPEG extensions can enable the creation of immersive experiences using glTF. A glTF asset with MPEG extensions can be loaded into a rendering engine for visualization.

[0039] Figure 3B shows some examples of MPEG gITF extension functions. In one example, a media type such as Multipurpose Internet Mail Extensions (MIME) or MIME type is used. Specifically, the MIME type is indicated by application / dash+xml (305), and the Uniform Resource Identifier (URI) is indicated by manifest-1.mpd (306). Tracks (307) can be specified by (308). Another example is shown by (309) and (314). Figure 3B also shows name (310), autoplay status (311), loop status (312), and alternatives (313).

[0040] Figure 4 is a schematic diagram of a media scene description system reference architecture (also referred to as a reference media scene description architecture) (400) according to an embodiment of the present disclosure. The reference media scene description architecture (400) shows an example of how MPEG extension functions can be used to support various media types such as audio and video. Media content(s) (or media data) can be obtained from an external source such as a cloud or a media cloud (401) using a media search engine (also referred to as a media access function (MAF)) (402). The media content(s) can be processed, for example, by a video decoder (403), an audio decoder (404), or a data compressor (405). The processed media content(s) can be rendered by a presentation engine (409). The media data can be passed from the MAF (402) to the presentation engine (409) in the form of buffers (such as a video buffer (406), an audio buffer (407), other buffers (408), etc.). In some examples, the media content(s) are stored in local storage (410).

[0041] Referring to FIG. 4, the MPEG scene description extension function is designed to separate the presentation engine (409) from the MAF (402). The presentation engine (409) and the MAF (402) can communicate via a predefined programming interface. Thus, the presentation engine (409) can request the media data necessary to render the scene. The MAF (402) can obtain the requested media data and make the media data available in a timely manner and in a format that can be processed immediately, for example, by the presentation engine (409). For example, the requested media asset (or media data) is compressed and is in the network, and the MAF (402) can obtain the requested media asset. The requested media asset can be further decoded. The decoded media data can be passed to the presentation engine (409) for rendering. As described above, the media data can be passed from the MAF (402) to the presentation engine (409) in the form of buffers (such as video buffer (406), audio buffer (407), other buffers (408), etc.). The request for media data can be passed from the presentation engine (409) to the MAF (402) via the media search API. For flexible use of the video decoding resource(s), a video decoding engine or video decoder (403) can be used. When the video decoding engine (403) is used, the presentation engine (409) can provide information about the input format and output format to the video decoder (403) via the application configuration API.

[0042] For example, as described in IETF RFC 7252, the Constrained Application Protocol (CoAP) (also referred to as the CoAP communication protocol) is a lightweight web transfer protocol compared to the Hypertext Transfer Protocol (HTTP). HTTP is also referred to as the HTTP communication protocol. CoAP can be used for Internet of Things (IoT) devices that may have limitations in power, storage, and computing capacity. In various embodiments, CoAP is a specialized web transfer protocol for use in constrained nodes and constrained (e.g., low-power, lossy) networks in the Internet of Things (IoT). A node can have an 8-bit microcontroller with a small amount of ROM and RAM, while a constrained network such as IPv6 over Low-Power Wireless Personal Area Networks (6LoWPANs) can have a relatively high packet error rate and throughput in the tens of kbit / s. CoAP can be designed for machine-to-machine (M2M) applications such as smart energy and building automation.

[0043] CoAP can have similar features to HTTP, such as the use of a request / response mechanism and client / server mode, and support for Representational State Transfer (REST) Application Programming Interfaces (APIs) (also referred to as RESTful APIs). A Representational State Transfer (REST) API (also referred to as a RESTful API) is an API that conforms to the constraints of the REST architectural style and enables interaction with RESTful web services.

[0044] CoAP can use the User Datagram Protocol (UDP) as the underlying transport protocol. CoAP can provide a request / response interaction model between application endpoints, support the built-in detection of services and resources, and can include key web concepts such as URIs and Internet media types. CoAP can be designed to easily interface with HTTP for web integration while meeting special requirements such as multicast support, very low overhead, and simplicity for constrained environments.

[0045] In one embodiment, compared to conventional HTTP-based streaming solutions, CoAP has a lightweight protocol stack. CoAP can be easily introduced into low-power devices such as ultra-low-power video streaming devices. CoAP can also support streaming protocols such as Dynamic Adaptive Streaming over HTTP (also known as MPEG-DASH), and data formats such as text, HTML, XML, and JSON. MPEG-DASH is an adaptive bitrate streaming technology that enables high-quality streaming of media content delivered over the Internet from conventional HTTP web servers.

[0046] One design of glTF takes broad account of IoT sensor data, and glTF can have extensive interactions with IoT sensor data, such as enabling a Web Graphics Library (WebGL) for rendering geospatial datasets of IoT sensors onto maps. WebGL is a JavaScript (registered trademark) API for rendering interactive 2D and 3D graphics within any compatible web browser without using plugins.

[0047] Thus, when performing video streaming using an ultra-low power device, there may be cases where a full-fledged HTTP implementation is not possible on the ultra-low power device. CoAP can be designed as an alternative web transfer protocol for IoT devices.

[0048] CoAP support for MPEG-SD can be enabled in MAFs such as MAF(402). By adding API support, media data such as video data can be retrieved from an IoT device (e.g., a CoAP client) for rendering to the cloud and / or local storage (e.g., a CoAP server).

[0049] For example, glTF designed by the Khronos group can enable the separation of 3D data description text in binary data. "URI" can specify the location of the binary data for media rendering and support customized paths as specified in the glTF2.0 specification. For example, the implementation note indicates that a client can optionally support additional URI components such as the HTTP: / / or file: / / scheme, authority / hostname, absolute path, and query or fragment parameters. Assets containing additional URI components may be less portable. In the case of CoAP, the corresponding URI is coap: / / . CoAP is implemented in many open-source flavors.

[0050] CoAP can be designed to operate like lightweight HTTP using methods such as GET / POST / PUT / DELETE, as shown in Table 1 below (adapted from IETF RFC 7252).

Table 1

[0051] The GET method can obtain the representation of the information currently corresponding to the resource identified by the request URI. The POST method can request that the representation enclosed in the request be processed. The actual function executed by the POST method is determined by the origin server and can depend on the target resource. For example, it can create a new resource or update the target resource. The PUT method can request that the resource identified by the request URI be updated or created with the enclosed representation. The representation format, if provided, can be specified by the media type and content coding given in the Content-Format option. The DELETE method can request that the resource identified by the request URI be deleted.

[0052] Exemplary response codes can be defined, for example, in IETF RFC 7252. Response codes can include the response code class "Success 2.xx (Success 2.xx)", the response code class "Client Error 4.xx (Client Error 4.xx)", and the response code class "Server Error 5.xx (Server Error 5.xx)", where "xx" represents two numerical values. In one example, "xx" is in the range from 00 to 31. The response code class "Success 2.xx" can indicate that the client request has been successfully received, understood, and accepted. The response code class "Client Error 4.xx" can indicate cases where the client is thought to have made an error. The response code class "Server Error 5.xx" can indicate cases where the server has recognized that it has made an error or cannot execute the request. Table 2 shows an example of CoAP response codes according to an embodiment of the present disclosure.

Table 2

[0053] The COAP support for both the CoAP server and the CoAP client can support request / response methods. The semantics of CoAP requests and responses can be transmitted in CoAP messages that include a method code or a response code, respectively. Optional (or default) request and response information, such as the URI and the payload media type, can be transmitted as CoAP options. A token can be used to match the response to a request separately from the underlying message. Requests can be transmitted as confirmable (CON) or non-confirmable (NON) messages. If immediately available, the response to a request transmitted as a confirmable message can be transmitted as a resulting acknowledgement (ACK) message. If the server cannot immediately respond to a request transmitted as a confirmable message, the server can respond with an empty acknowledgement message, as a result of which the client can stop retransmitting the request. When ready to respond, the server can send the response in a new confirmable message. If the request is transmitted as a non-confirmable message, the response is sent using a new non-confirmable message, but the server may send a confirmable message.

[0054] CoAP can use the URI schema: coap: / / host:port / path / to / resource to represent resources that are not security protected, as compared with HTTP.

[0055] CoAP can use the URI schema: coaps: / / host:port / path / to / resource to represent resources that are security protected, as compared with the Hypertext Transfer Protocol Secure (HTTPS) that uses Transport Layer Security (TLS).

[0056] Content formats supported by CoAP, such as audio and / or video, can be specified by the "application / media type". Table 3 shows an exemplary list of supported media type values (adapted from IETF RFC 7252) according to an embodiment of the present disclosure. In the case of video and / or audio content, a specific media type can be defined, for example, in IETF RFC 2046. For example, when CoAP is used for the transfer of video binary data, the content format is specified as "application / video" in RFC 2046.

Table 3

[0057] Figure 5 shows an example of a CoAP deployment scenario (500), such as a general CoAP deployment scenario, according to an embodiment of the present disclosure. CoAP and HTTP can be designed as web transport protocols with different use cases as described above. A cross-protocol network proxy (e.g., an HTTP-CoAP proxy or an HC proxy) (511) can be implemented to perform the conversion from HTTP to CoAP. Thus, an HTTP client (HTTP-C) (512) is enabled to access resources on a CoAP server (CoAP-S) (513) via the cross-protocol network proxy (511). Thus, an HTTP request is mapped to a CoAP request, and a CoAP response is mapped to an HTTP response.

[0058] Referring to FIG. 5, CoAP-S (513) is in the constrained network (501). The constrained network (501) can include multiple CoAP servers such as CoAP-S (513)-(515). In one embodiment, the HC proxy (511) is located at the boundary of the constrained network domain (501). In one example, the HC proxy (511) enables only very special types of traffic, such as permitted inbound HTTP requests (e.g., HTTP request (521)) and related outbound CoAP responses (e.g., HTTP response (524)), to pass through. In one example, other types of traffic are separated within their respective network segments.

[0059] As described above, the HTTP-CoAP proxy (511) can function as middleware between HTTP-C (512) and CoAP-S (513). The HTTP-CoAP proxy (511) can convert the request of HTTP-C (512) and transfer the request to CoAP-S (513), as specified in, for example, IETF RFC 8075. Furthermore, CoAP-S (513) can communicate directly with CoAP-C (514).

[0060] The HC proxy (511) can be accessed by HTTP-C (512) which needs to obtain a resource on CoAP-S (513). The HC proxy (511) can process the HTTP request (521) from HTTP-C (512) by mapping the HTTP request (521) to an equivalent CoAP request (522), and the equivalent CoAP request (522) is then transferred to CoAP-S (513). The CoAP response (523) from CoAP-S (513) is then mapped to an appropriate HTTP response (524), and the appropriate HTTP response (524) is then sent back to the originating HTTP-C (512).

[0061] In various examples, to support an IoT device that streams video content in a CoAP deployment scenario (500), the MAF (402) supports the CoAP protocol and has a function to proxy HTTP requests and / or responses with CoAP-S (513).

[0062] According to an aspect of the present disclosure, in the support of CoAP in scene description, one or more APIs can be used to enable the MAF (e.g., MAF (402)) to support the CoAP protocol and / or function as an HTTP to CoAP proxy (e.g., HC proxy (511)).

[0063] According to an embodiment of the present disclosure, one or more APIs can be used to enable the MAF (e.g., MAF (402)) to support the CoAP protocol. Referring to Table 4, when the MAF (e.g., MAF (402)) operates as a CoAP client (CoAP-C) to obtain media data (e.g., time-limited media, media resources, etc.) from a CoAP server (CoAP-S) (also called a CoAP media server), the API (also called a MAF API) can be applied. The API in Table 4 is also called a CoAP API or a CoAP client API.

Table 4

[0064] Figure 6 shows an exemplary media scene description system reference architecture (or reference media scene description architecture) (600) according to an embodiment of the present disclosure. The reference media scene description architecture or media scene description system (600) can include CoAP-S (601), MAF as CoAP-C (602), a video decoder (603), an audio decoder (604), one or more data compressors (605), a video buffer (606), an audio buffer (607), one or more other buffers (608), a presentation engine (609), a local storage (610), and the like.

[0065] The various components including the video decoder (603), audio decoder (604), one or more data compressors (605), video buffer (606), audio buffer (607), one or more other buffers (608), presentation engine (609), and local storage (610) of the reference media scene description architecture (600) can be the same as or similar to the video decoder (403), audio decoder (404), one or more data compressors (405), video buffer (406), audio buffer (407), one or more other buffers (408), presentation engine (409), and local storage (410) of the reference media scene description architecture (400) shown in FIG. 4, respectively, and thus detailed descriptions are omitted for simplicity.

[0066] Referring to FIG. 6, the MAF (602) or the media search engine can function as a CoAP-C to obtain media data (e.g., time-limited media) from CoAP-S (601). CoAP-S (601) can be a cloud server. The API method (e.g., fetch ()) can be used for the MAF (602) to send a media resource request (or request) (621) to CoAP-S (601). The API method (e.g., receive ()) can be used for the MAF (602) to receive the requested media resource (or resource) (624) from CoAP-S (601). The requested media resource (or resource) (624) can be time-limited media from CoAP-S (601). Thereafter, the resource (624) (e.g., media content (s)) can be processed by, for example, a video decoder (603), an audio decoder (604), or a data compressor (605). The processed resource (624) can be rendered by a presentation engine (609). The processed resource (624) (e.g., media data) can be passed from the MAF (602) to the presentation engine (609) in the form of a buffer (e.g., a video buffer (606), an audio buffer (607), other buffers (608), etc.). The resource (624) can be stored in a local storage (610).

[0067] MAF or CoAP-C (602) can obtain the requested media data and make it available in a format that can be processed in a timely manner, for example, immediately by the presentation engine (609). The request for media data can be passed from the presentation engine (609) to MAF or CoAP-C (602) via the media search API. To use the video decoding resource(s) flexibly, a video decoding engine or a video decoder (603) can be used. When the video decoding engine (603) is used, the presentation engine (609) can provide information on the input format and output format to the video decoder (603) via the application configuration API.

[0068] Figure 6 shows that MAF (602) operating as CoAP-C can send a media resource request (621) to CoAP-S (601) (for example, a cloud server) using the CoAP-C API method (for example, fetch ()). Using the CoAP-C API method (for example, receive ()), MAF (602) operating as CoAP-C can receive the requested media resource from CoAP-S (601).

[0069] According to an embodiment of the present disclosure, one or more APIs can be used to enable MAF (for example, MAF (402)) to operate as an HC proxy. Referring to Table 5, when MAF (for example, MAF (402)) operates as an HC proxy, an API (also referred to as the MAF API or the HTTP-CoAP proxy API) can be applied.

Table 5

[0070] Figure 7 shows an exemplary media scene description system reference architecture (or reference media scene description architecture) (700) according to an embodiment of the present disclosure. The reference media scene description architecture or media scene description system (700) can include CoAP-S (701), MAF (702), a video decoder (703), an audio decoder (704), one or more data compressors (705), a video buffer (706), an audio buffer (707), one or more other buffers (708), a presentation engine (709), a local storage (710), etc.

[0071] The various components including the video decoder (703), audio decoder (704), one or more data compressors (705), video buffer (706), audio buffer (707), one or more other buffers (708), presentation engine (709), and local storage (710) of the reference media scene description architecture (700) can be the same as or similar to the video decoder (403), audio decoder (404), one or more data compressors (405), video buffer (406), audio buffer (407), one or more other buffers (408), presentation engine (409), and local storage (410) of the reference media scene description architecture (400) shown in FIG. 4, respectively, and thus, detailed descriptions are omitted for simplicity.

[0072] Referring to FIG. 7, the MAF (702) or media search engine can function as an HC proxy (e.g., HC-Proxy (731)), and thus map an HTTP request (721) from an HTTP-C (732) to a CoAP request (723) and transfer the CoAP request (723) to a CoAP-S (701) (e.g., cloud server). An API (e.g., hc () in Table 5) can be used by the MAF (702) to map an HTTP request (721) to a CoAP request (723) and transfer the HTTP request (721) to the CoAP-S (701) (as a CoAP request (723)).

[0073] In one example, a CoAP response (724) from the CoAP-S (701) is mapped to an HTTP response (722) by the MAF (702) (e.g., HC proxy (731)), and the HTTP response (722) is then returned to the HTTP-C (732). In one example, the HTTP response (722) (e.g., media data) can be processed by, for example, a video decoder (703), an audio decoder (704), or a data compressor (705) and further rendered by a presentation engine (709).

[0074] FIG. 7 shows that the MAF (702) operating as an HC proxy can map HTTP requests (s) to CoAP using an HC proxy API (e.g., hc ()) and transfer the HTTP requests (s) to the CoAP-S (701). In the example shown in FIG. 7, the MAF (702) can function as an HC proxy (731) and an HTTP-C (732).

[0075] FIG. 8 is a flowchart showing an overview of a process (800) according to an embodiment of the present disclosure. The process (800) can be used when the MAF operates as a CoAP client to obtain time-limited media from a CoAP server. In various embodiments, the process (800) is executed by a processing circuit. In some embodiments, since the process (800) is implemented by software instructions, when the processing circuit executes the software instructions, the processing circuit executes the process (800). The process starts processing from (S801) and proceeds to (S810).

[0076] (S810), a media resource request can be sent to a CoAP server (e.g., (601)) for the MAF (e.g., (602)) to request a media resource using CoAP in a media scene description system using the CoAP client API, as described with reference to FIG. 6. The MAF (e.g., (602)) can be configured as a CoAP client. The MAF can be configured for multiple Internet protocols such as HTTP and CoAP.

[0077] (S820), the requested media resource can be received by the MAF (e.g., (602)) from the CoAP server (e.g., (601)) using the CoAP client API.

[0078] The process (800) can be appropriately adapted. The step(s) of the process (800) can be changed and / or omitted. Additional step(s) can be added. Any suitable implementation order can be used. In one example, the received media resource can be processed by one of (i) a video decoder (e.g., (603)), (ii) an audio decoder (e.g., (604)), (iii) a data compressor (e.g., (605)). The processed media resource can be rendered by a presentation engine (e.g., (609)).

[0079] FIG. 9 is a flowchart showing an overview of a process (900) according to an embodiment of the present disclosure. The process (900) can be used when the MAF operates as a CoAP client to obtain time-limited media from a CoAP server. In various embodiments, the process (900) is executed by a processing circuit. In some embodiments, the process (900) is implemented by software instructions, and thus, when the processing circuit executes the software instructions, the processing circuit executes the process (900). The process starts from (S901) and proceeds to (S910).

[0080] (S910), an HTTP request from an HTTP client can be mapped to a CoAP request using the HTTP-CoAP proxy application programming interface (API) of the MAF of the media scene description system as described with reference to FIG. 7. The MAF can be configured as an HTTP-CoAP proxy. The MAF can be configured for multiple Internet protocols such as HTTP and CoAP.

[0081] (S920), the CoAP request can be sent to the CoAP server.

[0082] (S930), a CoAP response from the CoAP server can be mapped to an HTTP response using the HTTP-CoAP proxy API by the MAF.

[0083] (S940), the HTTP response can be sent to the HTTP client. The process (900) proceeds to (S999) and ends.

[0084] The process (900) can be appropriately adapted. The step(s) of the process (900) can be changed or omitted. Additional step(s) can be added. Any suitable order of implementation can be used. In one example, an HTTP response can be processed by one of (i) a video decoder (e.g., (703)), (ii) an audio decoder (e.g., (704)), or (iii) a data compressor (e.g., (705)). The processed HTTP response can be rendered by a presentation engine (e.g., (709)).

[0085] FIG. 10 is a flowchart showing an overview of a process (1000) according to an embodiment of the present disclosure. The process (1000) can be used to access one or more CoAP servers of a media scene description system (e.g., (600) or (700)). The process (1000) can be used when the MAF operates as a CoAP client to obtain time-limited media from the CoAP server or when the MAF operates as an HTTP-CoAP proxy. In various embodiments, the process (1000) is executed by a processing circuit. In some embodiments, the process (1000) is implemented by software instructions, and thus, when the processing circuit executes the software instructions, the processing circuit executes the process (1000). The process starts at (S1001) and proceeds to (S1010).

[0086] (S1010), a CoAP request can be sent to a CoAP server by the MAF of a processing circuit implementing a media scene description system, for example, using an API, to request a media resource. The MAF can be configured as a CoAP client or an HTTP-CoAP proxy. In one example, the CoAP request is called a media resource request such as request (621) or CoAP request (723).

[0087] In one embodiment, the MAF is configured as an HTTP-CoAP proxy. The API is an HTTP-CoAP proxy API, as shown in FIG. 7. The CoAP server can be CoAP-S(701).

[0088] In one example, the MAF is configured as a CoAP client. The API is a CoAP client API, as shown in FIG. 6. The CoAP server can be CoAP-S(601).

[0089] The MAF can be configured for multiple Internet protocols such as HTTP and CoAP. In one embodiment, the MAF is compatible with both CoAP requests using the CoAP communication protocol and proxy requests using the HTTP communication protocol.

[0090] (S1020), the CoAP response can be received by the MAF using the API from the CoAP server. In one example, the CoAP response includes the requested media resource.

[0091] The process (1000) can be appropriately adapted. The step(s) of the process (1000) can be changed and / or omitted. Additional step(s) can be added. Any suitable implementation order can be used. In one example, the received media resource can be processed by one of (i) a video decoder (e.g., (603) or (703)), (ii) an audio decoder (e.g., (604) or (704)), or (iii) a data compressor (e.g., (605) or (705)). The processed media resource can be rendered by a presentation engine (e.g., (609) or (709)).

[0092] In one embodiment, the MAF is configured as an HTTP-CoAP proxy. Returning to FIG. 7, an HTTP request can be mapped by the MAF from an HTTP client to a CoAP request using the HTTP-CoAP proxy API. In one example, the HTTP request is a proxy request. Further, a CoAP response can be mapped by the MAF from a CoAP server to an HTTP response using the HTTP-CoAP proxy API. The HTTP response can be sent to the HTTP client.

[0093] The disclosed embodiments can be used individually or in any order in combination. Further, each of the methods (or embodiments) can be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored in a non-transitory computer-readable medium.

[0094] The methods described with reference to FIGS. 6-10 can be combined in any order. For example, one or more steps of FIG. 8 can be included in process (900). One or more steps of FIG. 9 can be included in process (800). In one example, the MAF described in FIGS. 6-10 can also be configured to communicate with a server (e.g., cloud server (401)) using HTTP. Thus, the disclosed MAF can be configured to communicate with a device(s) using HTTP. Further, by appropriate API(s) such as CoAP client API, HTTP-CoAP proxy API, etc., when a device (e.g., IoT device(s)) is not configured to use HTTP due to certain limitations such as power, storage, and / or computing capacity limitations, the MAP can be configured to communicate with the device (e.g., IoT device(s)) using CoAP. As shown in FIGS. 6-10, CoAP-C and HTTP-CoAP proxy can be deployed to the MAF, and thus, by appropriate API(s), the MAF can operate as CoAP-C, HTTP-CoAP proxy, etc.

[0095] The above techniques can be implemented as computer software physically stored on one or more computer-readable media using computer-readable instructions. For example, FIG. 11 shows a computer system (1100) suitable for implementing certain embodiments of the disclosed subject matter.

[0096] The computer software can be coded using any suitable machine code or computer language that can be the subject of an assembly, compilation, linking, or similar mechanism to create code that includes instructions that can be executed directly, or through interpretation, by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., such as by execution of microcode.

[0097] The command can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[0098] The components shown in FIG. 11 of the computer system (1100) are exemplary in nature and are not intended to suggest any limitation as to the scope or functionality of the computer software implementing the embodiments of the present disclosure. Also, the component configuration should not be construed as having any dependency or requirement regarding any one or combination of the components shown in the exemplary embodiments of the computer system (1100).

[0099] The computer system (1100) may include a specific human interface input device. Such a human interface input device can respond to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, movements of a data glove), voice input (e.g., voice, clapping), visual input (e.g., gestures), olfactory input (not shown). The human interface device can also be used to capture specific media that is not necessarily directly related to conscious input by humans, such as audio (e.g., speech, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still camera), video (including 2D video, 3D video such as stereoscopic video), and the like.

[0100] The input human interface device may include one or more of a keyboard (1101), a mouse (1102), a trackpad (1103), a touch screen (1110), a data glove (not shown), a joystick (1105), a microphone (1106), a scanner (1107), a camera (1108) (only one of each shown).

[0101] The computer system (1100) may also include certain human interface output devices. Such human interface output devices can stimulate the senses of one or more human users, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include tactile output devices (e.g., tactile feedback by a touch screen (1110), a data glove (not shown), or a joystick (1105)), but tactile feedback devices that do not function as input devices, audio output devices (e.g., speakers (1109), headphones (not shown), etc.), visual output devices (e.g., screens (1110) including CRT screens, LCD screens, plasma screens, OLED screens, each of which may or may not have a touch screen input function, and each of which may or may not have a tactile feedback function - some of these screens may be capable of three-dimensional or higher output by means such as two-dimensional visual output or stereoscopic video output; virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown) may also be possible.

[0102] The computer system (1100) may also include a human-accessible memory device and related media such as an optical medium including a CD / DVD ROM / RW (1120) with a CD / DVD or similar medium (1121), a thumb drive (1122), a removable hard drive or solid state drive (1123), legacy magnetic media such as tapes and floppy (registered trademark) disks (not shown), and special ROM / ASIC / PLD-based devices such as security dongles (not shown).

[0103] One of ordinary skill in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0104] The computer system (1100) can also include an interface (1154) to one or more communication networks (1155). The network can be, for example, wireless, wired, optical. The network can further be local, wide area, metropolitan, vehicle and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet (registered trademark), wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., television wired or wireless wide area digital networks including cable television, satellite television, and terrestrial broadcast television, vehicle and industrial networks including CAN bus, etc. Certain networks generally require an external network interface adapter attached to a specific general-purpose data port or peripheral bus (1149) (e.g., a USB port of the computer system (1100)). Others are generally integrated into the core of the computer system (1100) by attachment to the system bus as follows (e.g., an Ethernet (registered trademark) interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (1100) can communicate with other entities. Such communication can be unidirectional, receive-only (e.g., television broadcast), unidirectional transmit-only (e.g., CAN bus to a specific CAN bus device), or bidirectional, e.g., bidirectional to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks can be used for each of these networks and network interfaces as described above.

[0105] The aforementioned human interface device, human-accessible storage device, and network interface can be attached to the core (1140) of the computer system (1100).

[0106] The core (1140) can include one or more central processing units (CPUs) (1141), a graphics processing unit (GPU) (1142), a special programmable processing device in the form of a field programmable gate array (FPGA) (1143), a hardware accelerator for specific tasks (1144), a graphics adapter (1150), etc. These devices can be connected via a system bus (1148) together with a read-only memory (ROM) (1145), a random access memory (RAM) (1146), an internal mass storage such as an internal hard drive inaccessible to the user, an SSD, etc. (1147). In some computer systems, the system bus (1148) can be accessible in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, etc. Peripheral devices can be attached directly to the system bus (1148) of the core or via a peripheral bus (1149). In one example, a screen (1110) can be connected to the graphics adapter (1150). The architecture of the peripheral bus includes PCI, USB, etc.

[0107] The CPU (1141), GPU (1142), FPGA (1143), and accelerator (1144) can execute specific instructions that can together constitute the aforementioned computer code. The computer code can be stored in the ROM (1145) or the RAM (1146). Transient data can be stored in the RAM (1146), while persistent data can be stored, for example, in the internal mass storage (1147). Fast storage to and retrieval from any of the memory devices can be enabled by the use of cache memory, which can be closely associated with one or more CPUs (1141), GPUs (1142), mass storage (1147), ROM (1145), RAM (1146), etc.

[0108] A computer-readable medium can have computer code thereon for performing various computer-implemented operations. The medium and the computer code may be specially designed and constructed for the purposes of this disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0109] By way of example and not limitation, a computer system (1100) having an architecture, specifically a core (1140), can provide functionality as a result of a processor (s) (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied on one or more tangible computer-readable media. Such computer-readable media can be not only media associated with a mass storage device accessible by a user as introduced above, but also specific storage devices of the core (1140) having a non-transitory nature such as an internal mass storage device (1147) of the core or a ROM (1145). The software implementing various embodiments of the present disclosure can be stored on such devices and executed by the core (1140). The computer-readable media can include one or more memory devices or chips depending on specific needs. The software can cause the core (1140), particularly the processor (including a CPU, GPU, FPGA, etc.) therein, to define data structures stored in a RAM (1146) and modify such data structures according to processes defined by the software, including causing the core to execute a specific process or a specific part of a specific process described herein. Additionally, or alternatively, the computer system can provide functionality as a result of logic hardwired or otherwise embodied in a circuit (e.g., an accelerator (1144)), which can operate instead of or in conjunction with the software to execute a specific process or a specific part of a specific process described herein. References to software can include logic and, where appropriate, vice versa. References to computer-readable media can include circuits (such as integrated circuits (ICs)) that store software for execution, circuits that embody logic for execution, or both where appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0110] Although several exemplary embodiments have been described, there are modifications, permutations, and various alternative equivalents that are within the scope of this disclosure. Accordingly, it will be understood by those skilled in the art that many systems and methods, which are not explicitly shown or described herein but embody the principles of this disclosure and are thus within its spirit and scope, can be devised.

Claims

1. A method for accessing a Constrained Application Protocol (CoAP) server in a media scene description system, comprising: using a HTTP-CoAP proxy application programming interface (API) for mapping an HTTP request to a CoAP request and transferring the CoAP request to the CoAP server, a media access function (MAF) of a processing circuit implementing the media scene description system transmits a CoAP request to the CoAP server to request a media resource, wherein the MAF is configured as an HTTP client and a Hypertext Transfer Protocol (HTTP)-CoAP proxy, and the HTTP-CoAP proxy maps a proxy request, which is an HTTP request generated by the HTTP client according to the HTTP communication protocol based on a request for the media resource including video data by a presentation engine, to the CoAP request according to the CoAP communication protocol using the HTTP-CoAP proxy API and transmits the CoAP request to the CoAP server; and using the HTTP-CoAP proxy API, the HTTP-CoAP proxy receives a CoAP response from the CoAP server, wherein the CoAP response includes the requested media resource; A method.

2. one of (i) a video decoder, (ii) an audio decoder, or (iii) a data compressor processes the received media resource; and the presentation engine renders the processed media resource; The method according to claim 1, further comprising.

3. the MAF maps the CoAP response from the CoAP server to an HTTP response using the HTTP-CoAP proxy API; and transmits the HTTP response to the HTTP client; The method according to claim 1, further comprising.

4. the MAF is further configured as a CoAP client using a CoAP client API to request a media resource; The method according to claim 1.

5. the MAF is further configured to use HTTP; The method according to claim 1. **Claim 6** An apparatus for accessing a Constrained Application Protocol (CoAP) server in a media scene description system, comprising: a processing circuit that executes the method according to any one of claims 1 to 5; the apparatus. **Claim 7** A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to execute the method according to any one of claims 1 to 5. **Claim 8** A computer program that, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 5.

Images