Channel-aware semantic coding

Abstract

A method of channel-aware semantic coding (CASC) by a user equipment (UE) includes determining a quality level of a channel during a time period in which a video frame is transmitted over the channel, determining one or more semantic elements to be included in a semantic tile stream (STS) based on the quality level, encoding the video frame with the one or more elements of the STS, and transmitting the STS to a remote device.

Description

[0001] Priority requirements

[0002] This application claims priority to U.S. Patent Application No. 63 / 248,388, filed September 24, 2021, pursuant to 35 USC §119(e), the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates in whole to wireless communications. Background Technology

[0004] Existing and new services will be categorized into new device classes and will expand from local use to unrestricted cellular use. However, cellular connectivity brings unavoidable challenges, including poor coverage, low throughput, highly dynamic network load variations, and varying scheduling latency. Summary of the Invention

[0005] This application describes a data processing system and process for channel-aware semantic coding (CASC) of a media stream, which includes interactive semantic source coding and channel coding stages. A semantic copy stream provides the story for the media stream and responds to or adapts to channel conditions.

[0006] The abstraction level of semantic copying allows for a significant increase in the compression gain of existing media codecs, enabling high-fidelity target services over compromised links. At the receiver, a single coding instance supports different licensing processes and assets (e.g., multi-layer coding). Semantic coding and channel coding phases together produce the ability to adapt to channel conditions.

[0007] Channel-aware semantic coding (CASC) is configured to extend the high-fidelity service range of cellular devices, enabling high-fidelity experience quality when transmitting high-fidelity content over compromised communication links.

[0008] The disclosed technology is implemented through one or more specific embodiments, including the following as described in the following embodiments section.

[0009] Details of one or more specific embodiments are set forth in the following figures and detailed descriptions. The techniques described herein can be implemented by one or more wireless communication systems, components of wireless communication systems (e.g., sites, access points, user equipment, base stations, etc.) or other systems, devices, methods, or non-transitory computer-readable media. Other features and advantages will become apparent from the detailed descriptions, the figures, and the claims. Attached Figure Description

[0010] Figure 1 Exemplary wireless communication systems according to various specific implementations herein are shown.

[0011] Figure 2 An example of a semantic encoding environment is shown.

[0012] Figure 3A An example of a channel-aware semantic coding environment is shown.

[0013] Figure 3B An example of a channel-aware semantic coding environment is shown.

[0014] Figure 3C An example of a channel-aware semantic coding environment is shown.

[0015] Figure 3D An example of a channel-aware semantic coding environment is shown.

[0016] Figure 3E An example of a channel-aware semantic coding environment is shown.

[0017] Figure 4 An exemplary process for CASC is shown.

[0018] Figure 5 A system for CASC is shown.

[0019] Similar reference symbols in the various figures indicate similar elements. Detailed Implementation

[0020] This application describes a data processing system and process for channel-aware semantic coding (CASC) of a media stream, which includes interactive semantic source coding and channel coding stages. A semantic copy stream provides the story for the media stream and responds to or adapts to channel conditions.

[0021] The abstraction level of semantic copying allows for a significant increase in the compression gain of existing media codecs, enabling high-fidelity target services over compromised links. At the receiver, a single coding instance supports different licensing processes and assets (e.g., multi-layer coding). Semantic coding and channel coding phases together produce the ability to adapt to channel conditions.

[0022] The following detailed description relates to the accompanying drawings. The same reference numerals may be used in different drawings to identify the same or similar elements. In the following description, specific details, such as particular structures, architectures, interfaces, technologies, etc., are set forth for illustrative and non-limiting purposes to provide a thorough understanding of various aspects of the embodiments. However, it will be apparent to those skilled in the art that various aspects of the embodiments may be practiced in other examples departing from these specific details. In some cases, descriptions of well-known devices, circuits, and methods have been omitted so as not to obscure the description of the various embodiments with unnecessary detail. For the purposes of this document, the phrase "A or B" means (A), (B), or (A and B).

[0023] Figure 1 This is a block diagram illustrating components, according to some exemplary embodiments, capable of reading instructions from a machine-readable or computer-readable medium (e.g., a non-transient machine-readable storage medium) and executing one or more methods discussed herein. Specifically, Figure 1 A diagram is shown of hardware resources 100 including one or more processors (or processor cores) 110, one or more memory / storage devices 120, and one or more communication resources 130, wherein each hardware resource can be communicatively coupled via bus 140. For implementations utilizing node virtualization (e.g., NFV), a hypervisor 102 can be executed to provide an execution environment for one or more network slices / subslices to utilize hardware resources 100.

[0024] Processor 110 may include, for example, processor 112 and processor 114. Processor 110 may be, for example, a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a DSP such as a baseband processor, an ASIC, an FPGA, a radio frequency integrated circuit (RFIC), another processor (including those discussed herein), or any suitable combination thereof.

[0025] The memory / storage device 120 may include main memory, disk storage devices, or any suitable combination thereof. The memory / storage device 120 may include, but is not limited to, any type of volatile or non-volatile memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, solid-state storage devices, etc.

[0026] Communication resource 130 may include interconnect or network interface components or other suitable devices for communicating with one or more peripheral devices 104 or one or more databases 106 via network 108. For example, communication resource 130 may include wired communication components (e.g., for coupling via USB), cellular communication components, NFC components, etc. (or Low-power components Components and other communication components.

[0027] Instructions 150 may include software, programs, applications, applets, or other executable code for causing at least any processor in processor 110 to execute one or more sets of methods from the set of methods discussed herein. Instructions 150 may reside wholly or partially within processor 110 (e.g., within the processor's cache memory), at least one of memory / storage device 120, or any suitable combination thereof. Furthermore, any portion of instructions 150 may be transferred from any combination of peripheral device 104 or database 106 to hardware resource 100. Thus, the memory of processor 110, memory / storage device 120, peripheral device 104, and database 106 are examples of computer-readable and machine-readable media.

[0028] As is widely recognized, the use of personally identifiable information should comply with privacy policies and practices that are generally accepted to meet or exceed industry or governmental requirements for protecting user privacy. Specifically, personally identifiable information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use, and the nature of authorized use should be clearly explained to users.

[0029] Figure 2 An example of a semantic coding reference model 200 is shown. Model 200 includes a source 202, metrics 204, 206, 208, and a semantic coding model 210. The source 202 includes material to be semantically encoded / decoded. The source 202 material includes media, such as video or images. Metrics include semantic metric 204, fidelity metric 206, and link metric 208. For reference model 200, the formal information content or media source content to be transmitted via a wireless link is semantically source-coded by eliminating semantic redundancy, and semantically channel-coded by adding redundancy to reduce semantic noise.

[0030] Semantic metric 204 includes a correctness metric. A correctness metric refers to the actual information represented in the source media. The correctness metric uses ontology information 204 to determine how successfully the semantic encoding / decoding was performed (e.g., whether the semantic transmission was successful or failed). For example, if a receiver receives information indicating that a sphere is a square, the correctness metric could indicate a problem with the semantic encoding / decoding (e.g., a point of failure).

[0031] Semantic metric 204 includes consistency metrics. Consistency metrics measure object relationships in the media, details of the depicted scene (e.g., environment or location), spatiotemporal information of the scene, and physical or motion information of the scene. For example, consistency metrics include measurements of the physical relationships between objects within the scene and whether these physical relationships are consistent with what is expected or likely in the scene. For instance, if a ball is falling but lands on the ceiling in the image, a consistency metric could indicate a problem in semantic encoding or decoding. Semantic metric 204 includes rules for rule-based encoding and decoding. These rules include measures of morality, aesthetics, regulations, and other such rules related to the depiction generated from personally identifiable information (PII) of individuals in the synthetic video, etc.

[0032] The coding tasks performed in model 200 include semantic source coding at the transmitter using module 216 and semantic source decoding at the receiver using module 216. Model 200 performs semantic channel coding at module 218 and semantic channel decoding at module 218 at the receiver. The model includes classical source coding (compression / decompression) at module 220 and classical channel coding at module 222.

[0033] Semantic source coding involves extracting semantics from source information (source "material", 202). A semantic channel refers to semantic noise measured relative to semantic metric 204. Semantic channels include measures of correctness and consistency. A classic source channel includes a channel for a media source that introduces source noise measured relative to fidelity metrics 206, such as POLQA or PSNR, on the decompressed media stream at the receiver. A communication link includes a physical channel that can introduce channel noise, measured against link metric 208, into the transmitted signal. The receiver adapts to the noise when decoding the encoded data.

[0034] Semantic metrics 204 include semantic correctness and consistency metrics related to the accurate interpretation and reconstruction of media content. Fidelity metrics 206 include syntactic correctness, sign correctness, and sound correctness. For example, syntactic correctness includes the word-level correctness of objects represented in the source media 202. For example, sign correctness includes character correctness and sign correctness. For example, fidelity includes Perceived Objective Listening Quality Analysis (POLQA), Mean Opinion Score (MOS), and Peak Signal-to-Noise Ratio (PSNR).

[0035] Link metric 208 includes channel metrics associated with data transmission in the medium of source 202. For example, link metric 208 includes BLER or BER.

[0036] Semantic coding module 210 includes submodules for encoding or decoding source information (source "material") 202 relative to metrics 204, 206, and 208. Semantic coding module 210 includes a joint source-channel coding module 212 and a joint semantic source-channel coding module 214. Joint semantic source-channel (JSSC) coding refers to the joint coding and optimization of semantic source coding 216 and semantic channel coding 218. In some specific implementations, JSSC coding refers to the joint coding and optimization of semantic source coding 216 and channel coding 222 tasks, up to the joint coding and optimization of all four elements, including semantic source and channel coding tasks as well as classical source coding 220 and channel coding 222 tasks. Channel-aware semantic coding (CSC) enhances JSSC coding to create a combined coding phase for source 202 and channel 203.

[0037] Semantic copy streams (STS) represent the content that appears in a given media stream. In one example, an STS stream comprises a stream of frames, the content of which is captured with computer-readable annotations. These annotations may include one or more of the following: annotated diagrams, mathematical categories and operators, formal computer languages, or formal natural language represented as text. CASC uses formal natural language represented as text, but is not limited to this format.

[0038] CASC comprises semantic channel-aware / adaptive extraction, STS generation and QoS annotation, and the creation of semantic source and channel-coded SSCC streams. CASC includes lossless source compression and channel coding for each SSCC stream. After channel decoding and source decompression, the SSCC streams are reassembled into an STS. Individual assets, such as realistic models of family members' photos, based on privacy and access authorization, are embedded into a (customized) machine learning inference engine. The machine learning inference engine synthesizes video and audio from the STS at various steps.

[0039] STS includes the following features. STS includes a story flow (configurable by event triggering or frame rate [fps]). STS is a flexible, hierarchical, and structured dataset in a formal, computer-readable format. STS includes channel awareness. The structural depth of STS and the STS data frame rate (e.g., the amount of semantic information present in the STS) are based on predicting the UE's channel capacity and transmission license, and consistently adapting the extracted STS to the UE's actual bandwidth and transmission license. The experience quality of the synthesized video depends on the depth of the channel-aware control information in the STS. This is a semantic copy of the story for quality level 0. STS includes sequences of full frames and incremental frames. Full frames capture features such as the subject (person, animal, object, etc.) and actions detected and extracted from (a set of) classic video frames. (A "full frame" can be understood analogously to an I-frame in a classic video codec). For example, two people are detected, and it is also detected that they are playing with a ball. Incremental frames capture increments to the full frame, such as: new states of semantic story elements, the creator's pose (6 degrees of freedom), timestamps, etc. For example, for a quality level 1 story semantic copy, each action stream of an actor (with a configurable frame rate [fps]) is included. A full frame includes key actors that are interconnected in the story. For example, a subject's body posture, a subject's actions, etc. For instance, an action might indicate that an actor is preparing to kick a ball, the ball's location, and their body posture (e.g., whether they are in motion). A timestamp is provided for each frame. Each actor is associated with a depth variable of actor relationships.

[0040] STS incremental frames indicate changes since the previous frame. This can indicate updates to the main body of the frame. For example, updates to an actor's body pose or action within the frame. For instance, an STS incremental frame could indicate: an actor kicking the ball (now), updates to the body's kicking posture, and the timestamp associated with the frame.

[0041] STS includes details of every actor and action in the stream. This is a quality level 2 story semantic copy. For example, for a full frame, STS provides details for the actor's body, such as leg sub-poses (or updates to leg sub-poses), head sub-poses, etc., for individual parts of the actor.

[0042] STS streams may include actor emotional state streams (e.g., as an optional second-level quality story semantic copy), scene streams (e.g., as an optional first-level quality story semantic copy), and atmosphere streams (e.g., as an optional second-level story semantic copy).

[0043] STS includes annotations and anchors for each subject in the frame. For example, annotations of actor details, such as colors or sounds associated with the actor. STS includes meta-frames that identify the subject's identity, creator, location, etc. On the receiving side, privacy controls are applied to the meta-frames.

[0044] STS enables the transmission of semantic data from a transmitter to a receiver. In some implementations, the transmitter sends no video frames or a limited number of video frames. The receiver is configured to reconstruct the video from the received frames or solely from the semantic data. For example, the receiver may have an object library from which the video is constructed.

[0045] The system's transmitter or receiver is configured to determine the channel quality between them. Depending on the channel quality, the transmitter can send video at different fidelity levels. For example, in a high-throughput or stable channel, the transmitter sends high-fidelity video. If the channel quality is poor, the transmitter only sends STS data, thus reducing the amount of data to be sent by more than 50 times compared to traditional compressed video frames in this example. As the channel quality changes, the channel capacity changes, allowing the amount of data that can be sent to vary.

[0046] For the given examples, the total amount of data (in bytes) used at each semantic coding level is as follows: For a Level 0 story full frame: 20 words, averaging 8 characters (160 bytes); initial privacy and access annotations: 64 bytes (typical) up to 224 bytes. For a Level 0 story incremental frame: 16×2 bytes of 6DoF poses for the creator's view (32 bytes); 30 fps for 32 bytes. For Level 1 and 2 actors and action full frames: 8 key actor body poses (3×2 bytes of spatial anchors, 48 ​​bytes) + 12 key actor sub-poses (3×2 bytes of spatial anchors, 72 bytes); a subset of actor activity with 20 semantic items of 4×8 bytes of contextual information (640 bytes); 15 fps for 952 bytes. For Level 1 and 2 actors and action incremental frames: assumed to be 10% of the full frame, 15 fps for 96 bytes. Assuming a lossless compression rate of 70% at the cell edge in "escape mode" and generally poor radio link conditions across all levels of the full frame (one out of eight frames), robust MCS 1 (code rate = 0.05 – QoS stream 1) and incremental frames MCS 3 (code rate = 0.2 – QoS stream 2) with 8 bytes of L2+RoHC overhead result in a data rate requirement of approximately 57 kbps. However, achieving a sub-data rate of 15 kilobits per second (kbps) is possible.

[0047] For a 4K resolution source video at 30fps with an original bitstream of approximately 3.7Gbps, we assume the best case of compression to approximately 2.5Mbps, and for this, we use a high MCS scheme (bitrate 0.5, with approximately 50x compression gain relative to the latest classic MPEG / VCEG video codecs, while high-fidelity QoE can be established due to semantic synthesis tools on the receiver side).

[0048] In one example, a Semantic Scripting Stream (STS) includes a story stream, actor and action streams, detail streams (e.g., for each actor or action), scene streams, and atmosphere streams. The story stream includes annotations, anchors, and metadata (e.g., privacy and authentication information) for the story. The actor and action streams include annotations, anchors, and metadata for each actor and / or action, including further details that can be associated with a specific actor or action in a nested format. Each actor or action anchor is associated with data such as 6DoF data, timestamp data, etc. Each actor or action metadata may include privacy information or protected asset access data, which can restrict access to a given asset used for video reconstruction. Each scene stream is associated with annotations, anchors, and metadata for that scene. Atmosphere streams can be nested within each scene stream, including their own annotations, anchors, and metadata.

[0049] Figure 3A A framework 300 for channel-aware semantic coding is illustrated. CASC includes selecting the amount of data to be transmitted (e.g., STS stream and video, video only, STS stream only, the amount and frequency of captured optional substreams or actors, etc.) based on the channel quality experienced by the transmitter or receiver (as previously described). STS includes semantic segmentation, object pose and activity recognition, speech and sound recognition, relation reasoning and synthesis, semantic fusion and copy synthesis, and compression of semantic copies, as described later. As previously described, the level of STS provided depends on the QoS measurement of the transmitter or receiver. The receiver is configured to perform decompression of semantic copies, prepare models and individual assets for synthesis, perform copy-to-sound synthesis, perform spatial-temporal, physically consistent rendering, perform texture synthesis, and render lighting or atmosphere, as described later.

[0050] exist Figure 3A In addition to channel-aware / adaptive monitoring and prediction of the effects caused by channel 306, the CASC environment also includes a semantic coding workflow 302 and a semantic decoding workflow 304. For the semantic coding workflow 302, the data processing system performs semantic segmentation 308, generates video annotations 310, performs audio annotations 312, determines the relational reasoning and synthesis of objects in the media 314, performs semantic fusion 316, and performs semantic copy compression 318. Figures 3B to 3E Details of semantic encoding and semantic decoding functions are shown.

[0051] Semantic segmentation 308 involves dividing an image frame (e.g.,) into multiple parts using a data processing system. These parts may include known portions of the scene, such as the ground and sky. In some implementations, segmentation may include the selection of foreground and background. Other segmentation methods are possible.

[0052] like Figure 3B As shown, video frame 342 is extracted from the source media. Frame 340. Frame 342 is divided into segments 344, 346, and 348. Here, segment 344 includes background-level material. Segment 346 includes closer layers, such as layers in front of a distant background but not in the foreground. Segment 348 includes the nearest layers, such as the ground surface.

[0053] Back Figure 3A In environment 300, the data processing system performs object recognition 310. Object recognition includes object extraction. Objects include anything of interest in the media, such as unique objects, foreground objects, or any other subject in the media. Objects can include people, animals, inanimate objects, etc. Typically, for video, objects are identifiable things within the media and are usually the focus of an image frame. Object pose includes the position and orientation of the identified object within the image frame. Pose recognition can include translation and / or rotation of the extracted object. If an object is extracted, additional data is used to generate representations of other poses of the object within the frame. For example, in Figure 3B In the data processing system, the ball object 350, the Anna object 352, the girl object 354, the boy object 356, and the Ben object 358 are identified. Ball 350 is an inanimate object. Boy 356 and girl 354 are objects, but they are general objects. Anna 352 and Ben 358 are unique objects.

[0054] Activity recognition involves the data processing system determining what objects in a frame are doing. Annotations, other objects, scene information, and other details provide context for the actions performed by objects within an image frame. Annotations are provided by the data processing system to describe these objects, their poses, and their activities.

[0055] The data processing system performs audio analysis 312. Audio processing includes speech and sound recognition, and generates annotations to describe these sounds. In some implementations, speech and sound recognition includes identifying sound sources, such as specific people, animal types, vehicle types, etc. Machine learning models can be used to extract and annotate the sounds.

[0056] The data processing system is configured to perform relational reasoning and synthesis 314. Relational reasoning involves generating relationships between objects within an image frame. Objects may be interacting, about to interact, or may have recently interacted. For example, such as Figure 3CAs shown, when describing image frame 342, the data processing system generates an overall annotation 362 describing the relationship between the scene and objects such as Anna 352, Ben 358, boy 356, and girl 354. They are associated with ball 350. Anna 352 is annotated with annotation 364, indicating her pose, motion state (static action), timestamps, etc., for her head, legs, and body. Anna 352 is associated with ball 350 via figure 360. Ball 350 is annotated with annotation 370. Ben 358 is annotated with annotation 366. The scene is annotated with annotation 368, which describes location, area, spatial data, and atmosphere.

[0057] The data processing system performs semantic fusion 316. Semantic fusion 316 includes generating copies 380 for all semantic data to be encoded and transmitted to another device. Semantic fusion 316 includes sorting annotations 362, 364, 366, 368, and 370, such as... Figure 3D As shown. Semantic fusion 316 includes: determining client-specific rules 382 for transmitting data, annotating data, generating synthetic or enhanced data, etc. For example, as shown in rules 382a and 382b, the client is authorized to receive details about the girl Anna and the boy Ben. Rules 382c to 382d indicate that there is no authorization to retrieve semantic details about the objects boy and girl. The semantic details of the scene are retrievable, as shown in rule 382e.

[0058] Back Figure 3A Semantic encoding 302 includes compressing a semantic copy 380 and transmitting the copy via channel 306. Semantic decoding process 304 performs channel monitoring and performs prediction of QoS indicator elements, as described below.

[0059] The semantic decoding process 304 includes decompression of the semantic copy 320, preparation of models and individual assets for copy-to-speech synthesis 324 322, spatial-temporal rendering 326, texture synthesis 328, and lighting rendering 330. The data processing system is configured to decompress the copy and extract annotations for each object, scene, audio data, and relational data. For this video, the data processing system generates synthesized frames. The purpose of this is to generate video when only a portion of the video is received. For example, the receiver can decode video frames and generate one or more additional frames from the received frames by adding new objects, translating and / or rotating existing objects, adding new audio, etc. Therefore, the data processing system can improve video quality compared to traditional compression methods.

[0060] Quality of Service (QoS) metrics are measured by the transmitter and / or receiver. The transmitter can send various amounts of data based on QoS. The decision regarding what data to send is based on the following factors.

[0061] CASC depends on monitoring and prediction components. The transmitting device is configured to: monitor events such as dropped calls; track the frequency and size of uplink (UL) licenses; track handover measurements, reference symbols, etc. The device predicts the future achievable UL data rate (or, in the case of 5G NR, the relevant UL license profile).

[0062] CASC depends on the manipulation quality and depth of STS. The manipulation of semantic extraction quality and depth to generate STS is based on channel quality prediction as described above. This includes, but is not limited to, for example: (1) dynamic incorporation or skipping of Level 1 or Level 2 streams (e.g., skipping Level 2 actors and action detail streams); (2) manipulating the size of the key actor set; (3) controlling the amount of annotations used for photorealism and speech realism enhancement, etc.

[0063] CASC includes: linking the QoS determination process with STS elements of STS. STS elements, such as stream type, stream class, full frame, incremental frame, meta frame, annotation, and anchor, are generated and transmitted based on QoS. STS elements are associated with QoS based on QoS prediction. For example, QoS indications correspond to multiple levels of robustness requirements (e.g., 8 levels, 16 levels, or more), or forward error correction (FEC) requirements, or modulation and coding scheme (MCS) requirements.

[0064] CASC involves assigning STS elements to (e.g., aggregating them into) Semantic Source Coding and Channel Coding (SSCC) streams. Typically, STS elements are assigned to SSCC streams based on the actual channel state. There are n to m mappings from QoS indications to SSCC streams (where n ≥ m). If the actual state cannot be matched to a mapping due to some QoS indications not being correctly mapped, these STS elements are automatically discarded based on the corresponding priority value associated with each STS element.

[0065] like Figure 3E As shown, the data processing system generates a composite video frame of decoded video 390. The data processing system segments frame 342 to generate segmented frame 392. Then, the data processing system generates new composite objects 396 and 398 to be added to the original frame along with existing objects Anna 352 and Ben 358. Objects Boy 356 and Girl 354 are deleted. Target ball 350 is retained. Therefore, frame 394 is a completely new frame that was not received from source 202. Objects 396 and 398 can be composite objects or libraries of existing objects. The added objects can include illustrative or textual or arbitrary avatar-like photorealistic or cartoon substitutes.

[0066] Figure 4 An exemplary process 400 for CASC is shown. In some specific implementations, process 400 is derived from a reference. Figure 1 The data processing system 100 performs the following steps: In step 402, the channel is monitored. The device generates a prediction of the channel quality to be used for the next transmission. In step 404, based on the predicted channel quality, the device determines the depth of the STS to be transmitted and the quality of the video to be transmitted. In step 406, the device performs semantic extraction as described herein. These features form the building blocks of the STS to be generated. In step 408, based on the predicted or determined QoS, the device indicates which STS elements will be transmitted or not transmitted. In step 410, based on the STS elements marked for transmission, the device generates the STS. In step 412, the device generates an SSCC stream. The network scheduler sends data 416 representing the actual channel state. If a mismatch exists between the predicted channel and the actual channel, the device resolves the mismatch by discarding STS elements (e.g., if the channel quality is unexpectedly poor) or adding video frames (e.g., if the channel quality is unexpectedly robust). In step 414, the device generates an SSCC stream for transmission to the receiving device. In step 418, based on the generated SSCC stream, the device transmits the STS stream through the channel.

[0067] In some implementations, the process includes the following routine: extracting semantic copies of the creator's facial, lip, eye, hand, and body expressions. Semantic copies are an enhancement to existing facial camera-to-animation and XR pose extraction because this STS uses words and syntax (instead of commands) available in a (formal) language (rather than a codebook). By extracting semantic items such as people, objects, locations / scenes, events, actions, interactions, and relationships between semantic items, the process extracts semantic copies of the creator's live video.

[0068] In some specific implementations, the process includes: extracting semantic copies of audio such as speech and sound created or accessed by the producer.

[0069] In some specific implementations, STS includes the following routines: synthesizing from semantic copies, or obtaining photorealistic and audio-realistic dynamic people and scenery by decoding multiple quality levels of semantically enhanced video and audio compressed frames.

[0070] In some specific implementations, for privacy protection, the receiving device receives end-to-end encrypted data, regardless of whether the data is a semantic copy, a semantically enhanced compressed frame, or a regular compressed frame.

[0071] In some specific implementations, in order to synthesize photo and audio authenticity of people or private scenes / locations, the receiver receives permission during or before the call to use people or private scenes / location assets queried from the local HW security enclave of JSSCC_v1CASC.

[0072] In some implementations, the synthesis routines create synthesized videos from semantic copy streams, where spatiotemporal and physical consistency is extended to the level of context / annotation information in the semantic copy streams.

[0073] In some implementations, referencing the producer-side audio and the producer's responses to client audio and visual input, the synthesized facial, lip, eye, hand, and body expression stream representing the recreated producer-side expression is optimized based on synchronized marker anchors in the semantic copy. Optimization means ultimately discarding emotional, visual, and audio expressions, or scenery from the producer-side, on the client side to avoid blurring. In the semantic copy, marker annotations of the heard audio are used to potentially discard producer facial, lip, eye, hand, and body expressions associated with the heard audio, such as when the end-to-end latency of the dialogue exceeds 100 milliseconds or if the receiving device's computing power is insufficient.

[0074] In some implementations, the synthesis routines bring the semantic copy stream to a trained generative adversarial network (GAN) or a machine learning-based alternative that creates a synthetic video with potentially initial arbitrary or default (grayscale) photo textures.

[0075] In some specific implementations, during the synthesis routine in step 2, individual and photorealistic human (face) models and private scene / location models from the HW security compartment are used to add individual and photorealistic details and textures that consumer devices can use to enable the use of these models.

[0076] In some implementations, the routine synthesizes producer-side speech based on audio tagging annotations and / or audio embeddings added to the semantic copy. In some implementations, the routine synthesizes producer-side emotional elements in the speech based on audio tagging annotations and / or audio embeddings added to the semantic copy.

[0077] In some implementations, for the semantic encoding of speech (calls), automatic speech recognition systems and / or text-to-speech synthesis are used in the absence of language knowledge. In some implementations, routines are synthesized by adding author-derived voice annotations and / or voice embeddings to the semantic copy. In some implementations, the system generates individual voices or voices with musical elements from text.

[0078] Figure 5An exemplary environment 500 for CASC is shown. Environment 500 includes a transmitting device 502, a receiving device 508, a channel 506, and multiple transmission levels 504. The transmitting device 502 determines the QoS of the channel as previously described. Based on the determined QoS, a transmission level 504 is selected. For example, for a low-quality channel, the transmission level may include a basic STS mode. In this example, no video is transmitted. Based on the elements of the STS, the receiving device fully reconstructs the video. In some specific implementations, at the receiver, stock images, sounds, backgrounds, etc., can be selected from a library to reconstruct the video at the lowest transmission level. Level 504 may include an STS enriched transmission mode that includes more features than the basic STS elements of the lowest mode. The receiving device can construct a richer video. Level 504 may include compressed video and an STS mode. In this mode, multi-part compressed video is reconstructed by the receiving device. For example, the background may be a stock image or a still image reconstructed from a library, but a main object (e.g., a running person and a kicked ball) or multiple parts thereof may be received from the video frame. For example, a subject's face can be received from the video, but the receiver can construct multiple parts of the subject's body or clothing. In high-quality level 504 mode, the transmitter sends fully compressed video.

[0079] As is widely recognized, the use of personally identifiable information should comply with privacy policies and practices that are generally accepted to meet or exceed industry or governmental requirements for protecting user privacy. Specifically, personally identifiable information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use, and the nature of authorized use should be clearly explained to users.

[0080] Specific implementations of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed herein and their equivalents), or in a combination of one or more of these. Software implementations of the subject matter may be implemented as one or more computer programs. Each computer program may include one or more modules of computer program instructions encoded on a tangible, non-transitory, computer-readable computer storage medium for execution by a data processing apparatus or for controlling the operation of the data processing apparatus. Alternatively or additionally, the program instructions may be encoded in / on an artificially generated propagated signal. In one example, the signal may be a machine-generated electrical signal, optical signal, or electromagnetic signal generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of computer storage media.

[0081] The terms “data processing apparatus,” “computer,” and “computing device” (or their equivalents as understood by one of ordinary skill in the art) refer to data processing hardware. For example, a data processing apparatus may encompass various means, devices, and machines for processing data, including programmable processors, computers, or multiple processors or computers. The apparatus may also include special-purpose logic circuitry, including, for example, a central processing unit (CPU), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). In some embodiments, the data processing apparatus or special-purpose logic circuitry (or a combination thereof) may be hardware-based or software-based (or a combination of hardware-based and software-based). The apparatus may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of execution environments. This disclosure contemplates the use of data processing apparatuses with or without conventional operating systems (e.g., LINUX, UNIX, WINDOWS, MAC OS, ANDROID, or IOS).

[0082] A computer program, also referred to or described as a program, software, software application, module, software module, script, or code, can be written in any form of programming language. Programming languages ​​may include, for example, compiled languages, interpreted languages, declarative languages, or procedural languages. A program can be deployed in any form, including as a standalone program, module, component, subroutine, or unit for use in a computing environment. A computer program may, but does not necessarily, correspond to a file in a file system. A program may be stored as part of a file containing other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple harmonized files storing one or more modules, subroutines, or code portions. A computer program may be deployed to execute on a single computer or on multiple computers located at, for example, a single site or distributed across multiple sites interconnected by a communication network. While portions of a program shown in various figures may be depicted as separate modules implementing various features and functions through various objects, methods, or processes, a program may alternatively include multiple submodules, third-party services, components, and libraries. Conversely, the features and functions of various components may be combined into a single component as appropriate. Thresholds determined for computation may be determined statically, dynamically, or simultaneously statically and dynamically.

[0083] While this specification contains numerous specific implementation details, these details should not be construed as limiting the scope of the claims, but rather as descriptions of features that may be specific to particular implementations. Some features described in the context of different implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may be implemented individually or in any suitable sub-combination in multiple implementations. Furthermore, while the features previously described may be described as functioning in certain combinations, and even initially claimed in this manner, one or more features in a claimed combination may be removed from that combination in certain circumstances, and the claimed combination may involve sub-combinations or variations thereof.

[0084] Specific embodiments of the subject matter have been described. Other embodiments, modifications, and arrangements of the described embodiments are within the scope of the following claims, and will be apparent to those skilled in the art. Although the operations are shown in a specific order in the drawings or claims, this should not be construed as requiring such operations to be performed in the specific order or successive order shown, or requiring the performance of all the operations shown (some operations may be considered optional) to achieve the desired result. In some cases, multitasking or parallel processing (or a combination of multitasking and parallel processing) may be advantageous and may be performed as appropriate.

[0085] Furthermore, the division or integration of various system modules and components in the previously described specific implementations should not be construed as requiring such division or integration in all specific implementations, and it should be understood that the program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0086] Therefore, the exemplary embodiments described above do not limit or restrict this disclosure. Other changes, substitutions, and modifications are also possible without departing from the spirit and scope of this disclosure.

[0087] Example

[0088] Further exemplary implementations are provided in the following sections.

[0089] Example 1 includes a method for channel-aware semantic coding (CASC) performed by a user equipment (UE), comprising: determining a quality level of a channel during a period of time in which a video frame is transmitted through the channel; determining one or more semantic elements to be included in a semantic replica stream (STS) based on the quality level; encoding the video frame with the one or more elements of the STS; and transmitting the encoded video frame to a remote device.

[0090] Example 2 includes the method of Example 1 or some other embodiment herein, wherein determining the quality level includes: monitoring channel characteristics, including one or more of upload license size, handover measurement results, or reference symbols; and generating a prediction of the upload data rate used to transmit the video frame.

[0091] Example 3 includes the method of Examples 1 and 2 or some other embodiment of this document, wherein determining one or more semantic elements to be included includes: determining the number of annotations to be extracted from the video frame.

[0092] Example 4 includes the method of Examples 1-3 or some other embodiment herein, wherein determining one or more semantic elements to be included includes: determining the amount of bandwidth available for the semantic elements; assigning a priority to each semantic element; and including the higher-priority semantic elements until the available bandwidth is exhausted.

[0093] Example 5 includes the method of Examples 1-4 or some other example herein, and further includes: assigning a privacy tag to one or more semantic elements, the privacy tag requiring a remote device to have a corresponding permission to access the semantic element.

[0094] Example 6 includes the method of Example 5 or some other embodiment herein, wherein semantic elements include identifiers of people or objects.

[0095] Example 7 includes the method of Example 5 or some other embodiment herein, and further includes: performing end-to-end encryption on one or more semantic elements associated with the privacy tag.

[0096] Example 8 includes the method of Examples 1-7 or some other embodiment herein, wherein the determined channel quality level is a predicted channel quality level, and the method further includes: receiving an actual channel quality level while transmitting an encoded video frame; determining that the quality of the actual channel quality level is reduced relative to the predicted channel quality level; deleting one or more semantic elements from the encoded video frame based on one or more priority rules or by changing one or more configurable frame rates of the semantic elements; and transmitting the encoded video frame without the deleted one or more semantic elements.

[0097] Example 9 includes the method of Examples 1-8 or some other embodiment of this document, wherein transmitting the encoded video frame includes: transmitting only STS data and not transmitting video frame data.

[0098] Example 10 includes the method of Examples 1-9 or some other embodiment herein, wherein transmitting the encoded video frame includes: transmitting the STS data using a fully compressed video frame or transmitting the STS data by attaching one or more portions of the video frame.

[0099] Example 11 includes a user equipment (UE) configured for channel-aware semantic coding (CASC), comprising: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations including: determining a quality level of a channel during a period of time in which video frames are transmitted through the channel; determining one or more semantic elements to be included in a semantic replica stream (STS) based on the quality level; encoding the video frames with the one or more elements of the STS; and transmitting the encoded video frames to a remote device.

[0100] Example 12 includes a UE of Example 11 or some other embodiment herein, wherein determining the quality level includes: monitoring channel characteristics, including one or more of upload license size, handover measurement results, or reference symbols; and generating a prediction of the upload data rate used to transmit the video frame.

[0101] Example 13 includes the UE of Examples 11 and 12 or some other embodiment herein, wherein determining one or more semantic elements to be included includes: determining the number of annotations to be extracted from the video frame.

[0102] Example 14 includes the UE of Examples 11-13 or some other embodiment herein, wherein determining one or more semantic elements to be included includes: determining the amount of bandwidth available for the semantic elements; assigning a priority to each semantic element; and including the higher-priority semantic elements until the available bandwidth is exhausted.

[0103] Example 15 includes the UE of Examples 11-14 or some other example herein, and these operations further include: assigning a privacy tag to one or more semantic elements, the privacy tag requiring a remote device to have corresponding permission to access the semantic element.

[0104] Example 16 includes the UE of Example 15 or some other embodiment herein, wherein semantic elements include identifiers of people or objects.

[0105] Example 17 includes the UE of Example 15 or some other embodiment herein, and further includes: performing end-to-end encryption on one or more semantic elements associated with a privacy tag.

[0106] Example 18 includes a UE of Example 15 or some other embodiment herein, wherein the determined channel quality level is a predicted channel quality level, and the method further includes: receiving an actual channel quality level while transmitting an encoded video frame; determining that the quality of the actual channel quality level is reduced relative to the predicted channel quality level; deleting one or more semantic elements from the encoded video frame based on one or more priority rules or by changing one or more configurable frame rates of the semantic elements; and transmitting an encoded video frame that does not contain the deleted one or more semantic elements.

[0107] Example 19 includes the UE of Examples 11-18 or some other example herein, wherein transmitting the encoded video frame includes transmitting only STS data and not transmitting video frame data.

[0108] Example 20 includes the UE of Examples 11-19 or some other embodiment herein, wherein transmitting the encoded video frame includes: transmitting the STS data using a fully compressed video frame or transmitting the STS data by attaching one or more portions of the video frame.

[0109] Example 21 may include a signal, or a portion thereof, described or associated with any of Examples 1-20.

[0110] Example 22 includes datagrams, information elements, packets, frames, segments, PDUs or messages, or portions or components thereof, as described or otherwise in this disclosure, according to any of Examples 1-21.

[0111] Example 23 may include a signal encoded with data, or a portion or component thereof, as described or associated with any of Examples 1 to 22, or otherwise described in this disclosure.

[0112] Example 24 may include a signal, or a portion or component thereof, encoded as a datagram, IE, packet, frame, segment, PDU, or message, as described or associated with any of Examples 1 to 23, or otherwise described in this disclosure.

[0113] Example 25 may include an electromagnetic signal carrying computer-readable instructions, wherein execution of the computer-readable instructions by one or more processors will cause the one or more processors to perform a method, technique, or process, or a portion thereof, as described or associated with any of Examples 1 to 24.

[0114] Example 26 may include a computer program comprising instructions, wherein execution of the program by a processing element will cause the processing element to perform a method, technique, or process, or a portion thereof, as described or associated with any of Examples 1 to 25.

[0115] Example 27 may include signals in a wireless network as shown and described herein.

[0116] Example 28 may include methods for communicating in a wireless network as shown and described herein.

[0117] Example 29 may include a system for providing wireless communication as shown and described herein.

[0118] Example 30 may include a device for providing wireless communication as shown and described herein.

Claims

1. A method for performing channel-aware semantic coding (CASC) by user equipment (UE), comprising: During the period when video frames are transmitted through the channel, the quality level of the channel is determined; Based on the quality level, determine one or more semantic elements to be included in the Semantic Stream (STS); The video frame is encoded using one or more elements of the STS; as well as Transmit the encoded video frames to a remote device.

2. The method of claim 1, wherein determining the quality grade comprises: Monitor channel characteristics, which include one or more of the following: upload license size, handover measurement results, or reference symbols; as well as Generate a prediction of the upload data rate used to transmit the video frames.

3. The method of claim 1, wherein determining the one or more semantic elements to be included comprises: Determine the number of annotations to be extracted from the video frame.

4. The method of claim 1, wherein determining the one or more semantic elements to be included comprises: Determine the amount of bandwidth available for semantic elements; Assign priority to each semantic element; as well as This includes higher-priority semantic elements until the available bandwidth is exhausted.

5. The method according to claim 1, further comprising: A privacy tag is assigned to the one or more semantic elements, the privacy tag requiring the remote device to have corresponding permission to access the semantic elements.

6. The method of claim 5, wherein the semantic element includes an identifier of a person or object.

7. The method according to claim 5, further comprising: End-to-end encryption is performed on one or more semantic elements associated with the privacy tag.

8. The method of claim 1, wherein the determined channel quality level is a predicted channel quality level, the method further comprising: When transmitting encoded video frames, the actual channel quality level is received; It is determined that the actual channel quality level is lower than the predicted channel quality level. One or more semantic elements are removed from the encoded video frame based on one or more priority rules or by changing one or more configurable frame rates of the semantic elements. as well as Transmit encoded video frames that do not contain one or more of the deleted semantic elements.

9. The method of claim 1, wherein transmitting the encoded video frame comprises: Only the STS data is transmitted, not the video frame data.

10. The method of claim 1, wherein transmitting the encoded video frame comprises: The STS data can be transmitted using fully compressed video frames or by attaching one or more portions of the video frames.

11. A user equipment (UE) configured for channel-aware semantic coding (CASC), comprising: At least one processor; as well as A memory that stores instructions that, when executed by the at least one processor, cause the at least one processor to perform operations including: During the period when video frames are transmitted through the channel, the quality level of the channel is determined; Based on the quality level, determine one or more semantic elements to be included in the Semantic Stream (STS); The video frame is encoded using one or more elements of the STS; as well as Transmit the encoded video frames to a remote device.

12. The UE of claim 11, wherein determining the quality level comprises: Monitor channel characteristics, which include one or more of the following: upload license size, handover measurement results, or reference symbols; as well as Generate a prediction of the upload data rate used to transmit the video frames.

13. The UE of claim 11, wherein determining the one or more semantic elements to be included comprises: Determine the number of annotations to be extracted from the video frame.

14. The UE of claim 11, wherein determining the one or more semantic elements to be included comprises: Determine the amount of bandwidth available for semantic elements; Assign priority to each semantic element; as well as This includes higher-priority semantic elements until the available bandwidth is exhausted.

15. The UE according to claim 11, further comprising: A privacy tag is assigned to the one or more semantic elements, the privacy tag requiring the remote device to have corresponding permission to access the semantic elements.

16. The UE of claim 15, wherein the semantic element includes an identifier of a person or object.

17. The UE according to claim 15, further comprising: End-to-end encryption is performed on one or more semantic elements associated with the privacy tag.

18. The UE of claim 11, wherein the determined channel quality level is a predicted channel quality level, the method further comprising: When transmitting encoded video frames, the actual channel quality level is received; It is determined that the actual channel quality level is lower than the predicted channel quality level. One or more semantic elements are removed from the encoded video frame based on one or more priority rules or by changing one or more configurable frame rates of the semantic elements. as well as Transmit encoded video frames that do not contain one or more of the deleted semantic elements.

19. The UE of claim 11, wherein transmitting the encoded video frame comprises: Only the STS data is transmitted, not the video frame data.

20. The UE of claim 11, wherein transmitting the encoded video frame comprises: The STS data can be transmitted using fully compressed video frames or by attaching one or more portions of the video frames.

Images