Active computation-memory co-orchestration method for holographic point cloud streams on mobile devices

Abstract

This invention provides an active computation-memory collaborative orchestration method for holographic point cloud streams on mobile devices. The method includes: an intelligent resource orchestrator in an MEC server collects wireless transmission network information and state information of heterogeneous UEs, constructs a state space and state vectors, determines an encoding strategy based on the state space and state vectors, and a V-PCC encoder generates V-PCC encoding actions according to the encoding strategy; the intelligent resource orchestrator sets memory cost constraints, calculates the peak memory and composite QoE utility function corresponding to the UE executing the V-PCC encoding action, calculates the optimal encoding strategy through iterative feedback calibration, generates the final V-PCC encoding action based on the optimal encoding strategy, generates a point cloud GOP sequence based on the final V-PCC encoding action, and sends it to the UE. This invention establishes a computation-memory physical coupling model for point cloud streams, characterizing the nonlinear relationship between V-PCC encoding parameters and GOP bitstream size, client processing latency, and DPB peak memory, so that resource orchestration is no longer limited to the network bandwidth dimension.

Description

Technical Field

[0001] This invention relates to the field of mobile communication technology, and in particular to an active computation-memory collaborative orchestration method for holographic point cloud streams in mobile devices. Background Technology

[0002] The sixth-generation mobile communication technology (6G) is driving holographic communication, vehicle-to-everything (V2X) and extended reality services from proof-of-concept to deployment in highly mobile scenarios. Holographic point clouds describe objects and their environment through three-dimensional geometry and attribute information, offering a stronger sense of immersion and interactivity compared to traditional video. This enables applications such as in-vehicle and out-of-vehicle environmental perception, remote collaboration, immersive conferencing, and holographic entertainment.

[0003] Point cloud videos are typically encoded using techniques such as V-PCC (Video-based Point Cloud Compression). V-PCC organizes the point cloud into sub-streams, including geometry, attributes, and occupancy maps, and then compresses and transmits them using video encoding tools. This mechanism can reduce network bandwidth pressure, but changes in encoding parameters can simultaneously affect the GOP (Group of Pictures) bitstream size, client decoding dependency chains, geometry reconstruction complexity, and the number of reference frames in the DPB. Therefore, practical systems exhibit significant computation-memory coupling characteristics.

[0004] On the mobile UE (User Equipment) side, there are significant differences in CPU frequency, effective memory limit, and system load between devices. If the traditional approach of selecting bitrate or quality solely based on network conditions is still used, high-quality or high-complexity encoding configurations may be selected when network bandwidth is sufficient, which may lead to decoding lag, depletion of the receive buffer, or memory overflow on the mobile device. Therefore, resource orchestration for holographic point cloud streams needs to be aware of network conditions, computational load, and DPB (Decoded Picture Buffer) memory status simultaneously.

[0005] One existing method for network adaptive transmission of volumetric video and point cloud streams includes improving transmission efficiency from the perspectives of architecture, bandwidth prediction, viewpoint awareness, or adaptive bitrate control. For example, research on volumetric video streaming summarizes the challenges and opportunities at the architectural level; low-latency adaptive live streaming research reduces stuttering through accurate bandwidth prediction; and frameworks such as NAVA optimize volumetric video transmission through network adaptation and viewpoint awareness.

[0006] These solutions typically prioritize network throughput, buffer status, and transmission bitrate as key decision-making factors, improving playback continuity under network fluctuations. However, they generally don't incorporate UE-side CPU frequency, single-frame processing latency, and peak DPB memory as hard constraints. Therefore, on heterogeneous mobile devices, seemingly feasible high-quality encoding configurations on the network side may still cause client-side computational lag or OutOfMemoryError (OOM).

[0007] The drawbacks of the existing network adaptive transmission methods for volumetric video and point cloud streams include: These methods use network bandwidth, throughput prediction, and buffer status as core inputs, assuming the client can complete decoding and reconstruction at the corresponding quality level. This assumption does not hold true for heterogeneous mobile UEs: when network conditions are good, the system may choose higher point cloud density, longer GOPs, or more complex reference structures, significantly increasing client processing time and memory usage. This method may introduce computational stuttering or DPB overflow, leading to a decrease in QoE or even system crashes. For continuous immersive interactive scenarios in holographic communication, a single OOM (Out-of-Memory) or prolonged processing lag can damage the user experience.

[0008] One existing point cloud compression and V-PCC rate control method includes: V-PCC and G-PCC standards provide basic tools for point cloud compression; subsequent research reduces the bit rate of point cloud broadcasting or transmission from aspects such as non-occupied pixel optimization, geometry-attribute joint rate control, rate-distortion modeling and distortion propagation modeling.

[0009] The drawbacks of the aforementioned point cloud compression and V-PCC rate control method include: this method primarily focuses on improving compression efficiency under given rate or distortion constraints, without fully considering the nonlinear coupling relationship between coding parameters and UE-side computation-memory consumption. For V-PCC, GOP length, geometric quantization parameters, and reference structure not only determine the bitstream size but also the decoding dependency chain, geometric reconstruction complexity, and the number of active reference frames in the DPB.

[0010] Highly compressed or complex reference designs aimed solely at reducing bandwidth can translate network stress into computational and memory stress, especially on devices with low effective memory, making them more prone to triggering OutOfMemoryError (OOM). Compression strategies lacking memory safety constraints are ill-suited to the stability requirements of mobile holographic communication. Summary of the Invention

[0011] This invention provides an active computation-memory collaborative orchestration method for holographic point cloud streams on mobile devices, which effectively improves the stability of point cloud stream services and the collaborative enhancement of QoE for V-PCC encoders.

[0012] To achieve the above objectives, the present invention adopts the following technical solution.

[0013] An active computation-memory co-arrangement method for holographic point cloud streams on mobile devices includes: The intelligent resource orchestrator in the mobile edge computing (MEC) server collects wireless transmission network information and heterogeneous UE state information, and constructs a state space and state vector based on the collected information. The intelligent resource orchestrator, based on deep reinforcement learning (DRL), determines the encoding strategy according to the state space and state vector, and the V-PCC encoder generates V-PCC encoding actions according to the encoding strategy. The intelligent resource orchestrator sets memory cost constraints, calculates the peak memory and composite QoE utility function corresponding to the UE performing the V-PCC encoding action, and calculates the optimal encoding strategy that maximizes the expected cumulative QoE under the premise of satisfying the memory cost constraints through iterative feedback calibration. The V-PCC encoder generates the final V-PCC encoding action according to the optimal encoding strategy, generates a point cloud GOP sequence based on the final V-PCC encoding action, and sends the point cloud GOP sequence to the UE through a wireless transmission network.

[0014] Preferably, the intelligent resource orchestrator in the mobile edge computing (MEC) server collects wireless transmission network information and heterogeneous UE state information, and constructs a state space and state vector based on the collected information, including: The CM-JRO architecture consists of an MEC server, a wireless transmission network, and heterogeneous UEs. An intelligent resource orchestrator and a V-PCC encoder are configured on the MEC server. The CM-JRO architecture processes points cloud flows in discrete time steps (GOPs). The point cloud compression V-PCC point cloud flow orchestration problem is modeled as a CMDP problem. The CMDP problem comprises a state space, an action space, state transition probabilities, a reward function, a constraint cost function, and a discount factor. At each time step k, the intelligent resource orchestrator on the MEC server reads historical bandwidth, the current receive buffer, the actual processing latency of the previous GOP, the actual memory usage of the previous GOP, the UE state, and metadata of the GOP to be encoded. Based on the read data, it constructs a state space, and then constructs a state vector based on the state space:

[0015] in, It's the past Historical bandwidth per time step; This is the current receive buffer level; and These are the actual processing time and memory usage of the previous GOP reported by the client. It is a limitation of the device's physical memory; Includes metadata features of the GOP to be encoded.

[0016] Preferably, the intelligent resource orchestrator determines the encoding strategy based on the state space and state vector according to deep reinforcement learning (DRL), and the V-PCC encoder generates V-PCC encoding actions according to the encoding strategy, including: The intelligent resource orchestrator establishes an aggregated V-PCC bitstream model. In this model, the original dynamic point cloud sequence is stored on the server side and divided into K consecutive Groups of Points (GOPs). For the k-th GOP, the intelligent resource orchestrator uses the currently perceived multidimensional state vector... Input to parameterization In the Actor neural network, after the network performs forward computation, it outputs the probability distribution corresponding to the joint action space, and a strategy is formulated based on the probability distribution. Sampling, i.e. The specific encoding strategy vector is dynamically determined for the k-th GOP. The encoding strategy vector From GOP length Geometric quantization parameters and reference structure composition:

[0017] in, This indicates the number of frames contained in each GOP. Used to adjust point cloud density and reconstruction quality. Used to distinguish between one-way forecasting and two-way forecasting; V-PCC encoder for encoding strategy vector Perform parameter decoupling and physical mapping processing, and encode the strategy vector. The first dimension is resolved to the GOP length. Used to define the number of point cloud frames to be processed in the current batch; encoding strategy vector The second dimension is resolved into geometric quantization parameters. Used to dynamically adjust the bit rate and point cloud spatial precision at the underlying encoding level; encoding strategy vector The third dimension is resolved into a frame reference structure. These parameters are converted into Boolean instructions to indicate whether the encoder is currently using low-memory-overhead unidirectional prediction or high-compression-rate bidirectional prediction. These three decoupled parameters constitute the V-PCC encoding action.

[0018] Preferably, the intelligent resource orchestrator sets memory cost constraints, calculates the peak memory and composite QoE utility function corresponding to the UE performing the V-PCC encoding action, and calculates the optimal encoding strategy that maximizes the expected cumulative QoE while satisfying the memory cost constraints through iterative feedback calibration, including: The calculation formula for the peak memory model corresponding to the UE performing the V-PCC encoding action is shown in equation (8), and the hard security constraint of the bandwidth security margin is:

[0019] in, This is the baseline memory overhead of the decoder; This indicates the number of active reference frames buffered in the DPB. This indicates the memory usage for decoding a single frame; It is the safety margin coefficient; The constraints for setting the GOP length, quantization parameters, reference structure, DPB peak memory, and receive buffer capacity are shown in equations (9)-(13):

[0020] The cumulative constraint cost for determining whether the peak memory of the DPB predicted for the current action exceeds the physical limit of the device is set as shown in equations (14)-(15):

[0021] in, It is a discount factor used to balance current and future rewards and ensure the convergence of the cumulative goal; the safety budget is strictly set to... ; At each time step, the DPB shadow corrects the current memory prediction based on the actual feedback error from the previous GOP. The calibrated DPB memory is:

[0022] in, It is an initial prediction derived from the memory model; It is the feedback compensation coefficient; It is the true feedback residual from time step k to time step (k-1); Set the composite QoE utility function as follows:

[0023] in, These are weighting coefficients; It's about visual quality; It's a network lag issue; It is a calculation lag; It refers to the smoothness of the quality; It's about visual quality; It's a network lag issue; It is a calculation lag; The formulas for calculating the smoothness of the quality are shown in equations (18)-(21).

[0024] in, It is a set of sensitivity coefficients for perceiving visual quality, representing the slope of a curve in an image; It is the set visual perception constant; the square of the ReLU function is used to calculate the penalty term for system timeout processing delay; Client CPU frequency; It is the visual quality at time k-1; Set to simultaneously satisfy the above equation (9) - The optimization objective for maximizing the expected cumulative QoE under various constraints such as GOP length, quantization parameters, reference structure, DPB peak memory, and receive buffer capacity is:

[0025] The DPB shadow maintains the mirror state of the client's decoded image buffer on the server side. The DPB shadow calculates the peak memory corresponding to the V-PCC encoding action performed by the UE, and performs feedback calibration on the encoding strategy based on the peak memory. If the current encoding strategy has a constraint violation risk, CRPO enters safe recovery mode; otherwise, it enters QoE optimization mode. Through iterative feedback calibration, when the actual feedback residual... When the value is below the minimum tolerance threshold, the feedback compensation coefficient When the value approaches 0, it indicates that the server shadow model and the client's real state have reached synchronization, the iterative feedback calibration stops, and the intelligent resource orchestrator determines the optimal coding strategy that maximizes the expected cumulative QoE while satisfying the memory cost constraint, and thus the optimal coding strategy corresponding to the maximum QoE.

[0026] Preferably, the V-PCC encoder generates a final V-PCC encoding action according to the optimal encoding strategy, generates a point cloud GOP sequence according to the final V-PCC encoding action, and transmits the point cloud GOP sequence to the UE through a wireless transmission network, including: The V-PCC encoder performs parameter decoupling and physical mapping processing on the optimal coding strategy, and the first dimension of the optimal coding strategy is parsed as the GOP length. The second dimension of the optimal encoding strategy is parsed into geometric quantization parameters. The third dimension of the optimal coding strategy is parsed into a frame reference structure. The specific parameters after decoupling , and The final V-PCC coding action is formed, and a point cloud GOP sequence is generated based on the final V-PCC coding action. The point cloud GOP sequence is then sent to the UE via the wireless transmission network. The UE reports the actual processing latency and memory usage when executing the point cloud GOP sequence to the MEC server for the next round of model calibration, enabling the agent in the CM-JRO structure to perceive the model offset and adaptively adjust the coding strategy.

[0027] As can be seen from the technical solutions provided by the embodiments of the present invention above, the CM-JRO architecture proposed in the embodiments of the present invention improves QoE while ensuring memory safety by jointly adjusting the GOP length, quantization parameters, and reference structure through intelligent decision-making on the MEC side. The present invention establishes a computation-memory physical coupling model for point cloud flow, and through offline profiling and data-driven analysis, it characterizes the nonlinear relationship between V-PCC encoding parameters and GOP bitstream size, client processing latency, and DPB peak memory, so that resource orchestration is no longer limited to the network bandwidth dimension.

[0028] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of the invention. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1This is a structural diagram of a Joint Compute-Memory Resource Orchestration (CM-JRO) architecture proposed in an embodiment of the present invention; Figure 2 This is a flowchart illustrating an active computation-memory collaborative orchestration method for holographic point cloud flow on mobile devices, provided as an embodiment of the present invention. Detailed Implementation

[0031] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0032] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or couplings. The term “and / or” as used herein includes any and all combinations of one or more of the associated listed items.

[0033] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0034] To facilitate understanding of the embodiments of the present invention, the following will provide further explanation and description with reference to the accompanying drawings and several specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0035] This invention realizes server-side real-time mirroring and hard constraint control of the memory state of the UE: This invention constructs a DPB shadow on the MEC server side, corrects the current prediction by the actual memory error fed back from the previous GOP, reduces open loop model drift, and treats OOM as a hard constraint.

[0036] This invention achieves decoupling optimization of QoE (Quality of Experience) and security: Instead of simply mixing memory security into QoE rewards, this invention treats QoE as a performance objective and memory violations as a constraint cost, dynamically switching the optimization direction through CRPO (Constraint-Rectified Policy Optimization) within the CMDP (Constrained Markov Decision Process) framework.

[0037] An embodiment of the present invention proposes a CM-JRO system architecture as follows: Figure 1 As shown, the CM-JRO system architecture includes an MEC (Mobile Edge Computing) server, a wireless transmission network, and heterogeneous UEs. An intelligent resource orchestrator and a V-PCC encoder are configured on the MEC server. The intelligent resource orchestrator makes decisions based on DRL (Deep Reinforcement Learning) and works in conjunction with the V-PCC encoder on the MEC side. Due to the low-latency link between the MEC server and the UE, the system can obtain client feedback and control encoding parameters in near real-time, thus forming a closed-loop computation-memory resource orchestration.

[0038] Based on the aforementioned CM-JRO system architecture, the processing flow of the active computation-memory collaborative orchestration method for holographic point cloud flow on mobile devices proposed in this embodiment of the invention is as follows: Figure 2 As shown, the processing steps include the following: Step S10: The intelligent resource orchestrator in the MEC server collects wireless transmission network information and heterogeneous UE status information, and constructs a state space and state vector based on the collected information. Step S20: The intelligent resource orchestrator determines the coding strategy based on the state space and state vector according to the DRL. The V-PCC encoder generates V-PCC coding actions according to the coding strategy. The V-PCC coding actions include GOP length, geometric quantization parameters and reference structure.

[0039] Step S30: The intelligent resource orchestrator sets memory cost constraints, calculates the peak memory and composite QoE utility function corresponding to the UE performing V-PCC encoding actions, and calculates the optimal encoding strategy that maximizes the expected cumulative QoE under the premise of satisfying memory cost constraints through iterative feedback calibration. The optimal encoding strategy includes visual quality, network stuttering, computational lag and quality smoothness.

[0040] Step S40: The V-PCC encoder generates the final V-PCC encoding action according to the optimal encoding strategy, generates the point cloud GOP sequence corresponding to the GOP metadata to be encoded according to the final V-PCC encoding action, and sends the point cloud GOP sequence to the UE through the wireless transmission network. Step S50: The UE reports the actual processing latency and memory usage when executing the point cloud GOP sequence to the MEC server for use in the next round of model calibration.

[0041] Step S10 specifically includes: The CM-JRO system architecture processes data in discrete time steps, with each Group of Pictures (GOP). At each time step k, the intelligent resource orchestrator in the MEC server reads historical bandwidth, current receive buffer, actual processing latency of the previous GOP, actual memory usage of the previous GOP, UE status, and metadata of the GOP to be encoded. It then outputs the joint encoding action, including GOP length, geometric quantization parameters, and reference structure. The UE status includes the device's physical memory limit.

[0042] This invention models the V-PCC point cloud flow orchestration problem as a CMDP problem. The CMDP problem consists of a state space, action space, state transition probabilities, a reward function, a constraint cost function, and a discount factor. The goal of the CMDP problem is to find the optimal encoding strategy that maximizes the expected cumulative QoE while satisfying memory cost constraints.

[0043] The intelligent resource orchestrator constructs a state space and a state vector. The state space includes historical bandwidth, current receive buffer, actual processing latency of the previous GOP, actual memory usage of the previous GOP, upper limit of device physical memory, and metadata characteristics of the GOP to be encoded.

[0044] The state vector is:

[0045] in, It's the past Historical bandwidth per time step; This is the current receive buffer level; and These are the actual processing time and memory usage of the previous GOP reported by the client. It is a limitation of the device's physical memory; Includes metadata features of the GOP to be encoded.

[0046] Step S20 specifically includes: the intelligent resource orchestrator establishing an aggregated V-PCC bitstream model, in which the original dynamic point cloud sequence is stored on the server side and divided into K consecutive GOPs. For the k-th GOP, the intelligent resource orchestrator obtains an encoding strategy vector based on the current state space and state vector. The encoding strategy vector is determined by the GOP length. Geometric quantization parameters and reference structure composition

[0047] in, This indicates the number of frames contained in each GOP; Used to adjust point cloud density and reconstruction quality; Used to distinguish between one-way prediction and two-way prediction. One-way prediction usually has a lower memory footprint, while two-way prediction can improve compression efficiency but increase DPB pressure.

[0048] The intelligent resource orchestrator is based on the state space and state vector. Obtain the encoding strategy vector The process is driven by a reinforcement learning Actor network. First, the intelligent resource orchestrator processes the currently perceived multidimensional state vector. Input to parameterization In the Actor neural network, after forward computation, the network outputs the probability distribution corresponding to the joint action space; finally, the agent formulates a policy based on this probability distribution. Sampling, i.e. This allows for the dynamic determination of a specific encoding strategy vector for the k-th GOP. This includes GOP length Geometric quantization parameters and reference structure .

[0049] Encoding strategy vector It is composed of the joint discrete action space A multidimensional decision tuple obtained through sampling. The system acquires the encoded policy vector. Then, the V-PCC encoder will process the encoding strategy vector. Parameter decoupling and physical mapping are performed to generate specific V-PCC encoding actions: First, the encoding strategy vector... The first dimension is directly parsed as the GOP length. First, it is used to define the number of point cloud frames that need to be processed in the current batch; second, the encoding strategy vector. The second dimension is resolved into geometric quantization parameters. This is used to dynamically adjust the bit rate and point cloud spatial precision at the underlying encoding level; finally, the encoding strategy vector... The third dimension is resolved into a frame reference structure. This is converted into Boolean instructions to indicate whether the encoder is currently using low-memory-overhead unidirectional prediction or high-compression-rate bidirectional prediction. These three decoupled parameters will be synchronously configured and applied to the backend V-PCC encoder, thereby achieving precise physical drive for the current point cloud sequence compression process.

[0050] The above step S30 specifically includes: The V-PCC bitstream consists of three sub-bitstreams: geometry, attributes, and occupancy graph. In this embodiment of the invention, the total bit size of the V-PCC bitstream of the k-th GOP is modeled as a nonlinear aggregation of the compression efficiency of each component.

[0051]

[0052] in, It depends on the frame type and reference structure. The basic frame size function; It has quantization parameters The bit rate attenuation coefficient; It is the basic overhead of GOP header information.

[0053] In the transmission and buffering model, the UE's receive buffer is affected not only by network transmission time but also by client processing latency. This invention, referencing the adaptive video stream buffer evolution concept, represents the buffer state at time step (k+1) as a function of the current buffer, GOP transmission time, average single-frame processing latency, and playback duration.

[0054] in, This indicates the remaining buffer time required for the system to fully render the image needed by the UE; This represents the average bandwidth of the system under its current state. This is the average processing latency per frame; It is the playback duration of a single frame. This will be used as the current buffer level of the system at time k+1 and encapsulated into the next state vector. In the process, the feedback is given to the reinforcement learning agent to formulate the encoding strategy for the next stage and to calculate subsequent network stuttering penalties. .

[0055] This invention further establishes a client-side processing parsing model. The total computational load generated by the k-th GOP at the UE consists of decoding dependency chain overhead and geometric reconstruction overhead. Under a given UE CPU frequency, the average single-frame processing latency is...

[0056] in, and These represent the decoding dependency chain overhead and the geometry reconstruction overhead, respectively. Client CPU frequency. Total compute load. This will be used for subsequent calculations of timeout penalty. The average single-frame processing latency Substituted into the aforementioned In the calculation.

[0057] To prevent memory overflow, the system must strictly control DPB usage. This invention models peak memory usage as the sum of the decoder's basic memory overhead and the memory footprint of active reference frames, where the number of active reference frames depends on the GOP length and reference structure, and the single-frame decoding memory footprint decreases power-lawably with increasing quantization parameters. The calculation formula for the peak memory model is shown in equation (8), and the hard safety constraint for bandwidth safety margin is:

[0058] in, This is the baseline memory overhead of the decoder; This indicates the number of active reference frames buffered in the DPB. This indicates the memory usage for decoding a single frame; It is the safety margin coefficient.

[0059] Based on the above model, the present invention sets the objective as maximizing the long-term cumulative QoE under the constraints of dynamic holographic scenarios and heterogeneous UE resources. The constraints of GOP length, quantization parameters, reference structure, DPB peak memory and receive buffer capacity are set as shown in equations (9)-(13):

[0060] To explicitly handle hard memory constraints, this invention defines a binary cost function to indicate whether the peak memory of the DPB predicted for the current action exceeds the physical upper limit of the device with a safety margin. The cumulative constraint cost is shown in equations (14) and (15):

[0061] in, It is a discount factor used to balance current and future rewards and ensure the convergence of the cumulative goal; the safety budget is strictly set to... .

[0062] The security budget of this invention is strictly set to near zero, indicating that the system adopts a near-zero tolerance target for OOM crashes. To avoid long-term drift in server-side open loop predictions, the DPB shadow corrects the current memory prediction at each time step based on the actual feedback error of the previous GOP. The calibrated DPB memory is:

[0063] in, It is an initial prediction derived from the memory model; It is the feedback compensation coefficient; It is the actual feedback residual between the k-th time step and the (k-1)-th time step.

[0064] Regarding the QoE utility function, this invention constructs a composite QoE score by integrating four factors: visual quality, network stuttering, computational lag, and quality smoothness. Unlike traditional methods that use memory constraints as rewards and penalties, this invention strictly decouples memory safety from QoE rewards; QoE only reflects performance goals, while memory safety is handled separately by hard constraints. The composite QoE utility function is as follows:

[0065] in, These are weighting coefficients; It's about visual quality; It's a network lag issue; It is a calculation lag; It refers to the smoothness of the quality.

[0066] The visual quality term describes user perception using the natural logarithm based on the Weber-Fechner law; network lag time is determined by the portion of GOP transmission time exceeding the current buffer; the lag penalty is calculated using the square of the ReLU function to characterize the timeout processing delay; the quality smoothness term penalizes the visual quality difference between adjacent GOPs, as shown in equations (18)-(21).

[0067] in, It is a set of sensitivity coefficients for perceiving visual quality, representing the slope of a curve in an image; It is the set visual perception constant; the square of the ReLU function is used to calculate the penalty term for system timeout processing delay; Client CPU frequency; It is the visual quality at time k-1.

[0068] Set to simultaneously satisfy the above equation (9) - equation The optimization objective for maximizing the expected cumulative QoE under various constraints such as GOP length, quantization parameters, reference structure, DPB peak memory, and receive buffer capacity is:

[0069] The DPB shadow maintains the mirror state of the client's decoded image buffer on the server side and uses the real memory feedback from the previous GOP to correct the current prediction. It can identify dangerous coding actions before the actual OOM occurs and provide a real-time state basis for memory hard constraints.

[0070] The DPB shadow calculates the peak memory corresponding to the UE performing V-PCC encoding actions. Based on the peak memory, it performs feedback calibration on the encoding strategy. If the current encoding strategy has a constraint violation risk, the CRPO enters the safety recovery mode; otherwise, it enters the QoE optimization mode. Through iterative feedback calibration, the intelligent resource orchestrator determines the optimal encoding strategy that maximizes the expected cumulative QoE while satisfying memory cost constraints, and thus the optimal QoE.

[0071] Specifically, in safe recovery mode, the priority of encoding policy updates is forcibly set to minimize safety constraint violations. By updating the policy network parameters along the negative gradient direction of cost advantage, the memory usage caused by point cloud decoding is rapidly reduced, thereby forcibly pulling the system back to a safe region to avoid OOM (Out of Memory) crashes. In QoE optimization mode, maximizing user experience is the primary goal, and the positive gradient direction of reward advantage guides the network to explore higher-resolution and smoother encoding actions. This dynamic alternating update process based on the primal dual domain effectively avoids the overly conservative or frequent crash problems existing in traditional soft penalty mechanisms, mathematically guaranteeing that the policy gradient can strictly converge to the feasible solution region that satisfies heterogeneous physical memory constraints.

[0072] When the actual feedback residual When the value is below the minimum tolerance threshold, the feedback compensation coefficient When the value approaches 0, it indicates that the server shadow model and the client's real state have reached synchronization, and the iterative feedback calibration stops.

[0073] The CRPO strategy correction module prioritizes reducing constraint costs when memory risk is high and prioritizes increasing QoE when memory is safe.

[0074] Based on the aforementioned CMDP modeling and CRPO mechanism, this invention proposes a joint constraint-aware optimization algorithm, CM-JRO. This algorithm employs an Actor-Critic architecture. During the data acquisition phase, it utilizes the DPB shadow on the MEC side to generate real-time memory overhead signals to alleviate client feedback latency. During the parameter update phase, it dynamically switches the gradient update direction according to the CRPO mechanism, prioritizing minimizing constraint costs when memory risk is high and prioritizing maximizing QoE when memory is safe. The specific algorithm process is as follows: ① Initialize the V-PCC dataset, UE hardware memory limit, Actor network, Reward / Critic dual-Critic network, CRPO switching threshold, and DPB shadow calibration factor; ② In each training round, initialize the environment according to the network and UE state, and sample joint encoding actions step by step of GOP; ③ Utilize DPB shadow to predict memory and calibrate based on the error of the previous time step. After executing the action, observe the actual QoE and memory usage, determine whether OOM is triggered, generate cost indicators, and store the transferred samples in the experience buffer; ④ When the buffer is full, calculate the reward advantage, cost advantage, and average constraint violation degree. If the violation degree is higher than the threshold, switch to the safety recovery mode to minimize memory cost. Otherwise, maintain the performance optimization mode to maximize QoE, and simultaneously update the dual-Critic network before clearing the buffer. Finally, output the point cloud flow encoding strategy that satisfies the memory safety constraint.

[0075] In terms of deployment complexity, each GOP time step requires only one forward propagation of the Actor multilayer perceptron to sample and encode actions; subsequent DPB shadow prediction, feedback calibration, and cost determination only involve scalar arithmetic operations. Therefore, the real-time execution overhead of CM-JRO is mainly determined by the lightweight Actor network, with minimal impact on real-time V-PCC stream processing on the MEC side. More complex policy and value function network updates are triggered only when the experience buffer is full and can be executed asynchronously in the background of the MEC server GPU without blocking real-time point cloud stream transmission. This design makes the present invention suitable for deployment in mobile holographic communication scenarios with high requirements for low latency and high stability.

[0076] In summary, this invention systematically solves the problems of stuttering and OOM caused by blindly selecting high-complexity encoding actions when client-side physical resources are limited, as well as the problems of insufficient or overly conservative soft penalty constraints in the security reinforcement learning baseline, through explicit modeling of UE-side computational processing latency and DPB peak memory, DPB shadow real-time memory state estimation, CRPO dynamic policy correction, and an adaptive adjustment mechanism for cross-heterogeneous device configuration. Furthermore, this invention incorporates the impact of compression parameters on decoding dependency chains, geometric reconstruction complexity, and reference frame buffering into a unified decision model, ensuring precise matching of compression strategies with the actual processing capabilities of mobile terminals. It also proposes a real-time resource orchestration algorithm based on DPB shadow prediction and constraint-aware policy optimization, achieving QoE improvement while maintaining a low crash rate, covering multiple heterogeneous device configurations, and providing a reliable resource orchestration solution with high QoE, low stuttering, low OOM risk, and lightweight real-time deployment for 6G high-mobility holographic communication services.

[0077] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0078] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0079] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for apparatus or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The apparatus and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0080] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An active computation-memory co-arrangement method for holographic point cloud streams on mobile devices, characterized in that, include: The intelligent resource orchestrator in the mobile edge computing (MEC) server collects wireless transmission network information and heterogeneous UE state information, and constructs a state space and state vector based on the collected information. The intelligent resource orchestrator, based on deep reinforcement learning (DRL), determines the encoding strategy according to the state space and state vector, and the V-PCC encoder generates V-PCC encoding actions according to the encoding strategy. The intelligent resource orchestrator sets memory cost constraints, calculates the peak memory and composite QoE utility function corresponding to the UE performing the V-PCC encoding action, and calculates the optimal encoding strategy that maximizes the expected cumulative QoE under the premise of satisfying the memory cost constraints through iterative feedback calibration. The V-PCC encoder generates the final V-PCC encoding action according to the optimal encoding strategy, generates a point cloud GOP sequence based on the final V-PCC encoding action, and sends the point cloud GOP sequence to the UE through a wireless transmission network.

2. The method according to claim 1, characterized in that, The intelligent resource orchestrator in the mobile edge computing (MEC) server collects wireless transmission network information and heterogeneous UE state information, and constructs a state space and state vector based on the collected information, including: The CM-JRO architecture consists of an MEC server, a wireless transmission network, and heterogeneous UEs. An intelligent resource orchestrator and a V-PCC encoder are configured on the MEC server. The CM-JRO architecture processes points cloud flows in discrete time steps (GOPs). The point cloud compression V-PCC point cloud flow orchestration problem is modeled as a CMDP problem. The CMDP problem comprises a state space, an action space, state transition probabilities, a reward function, a constraint cost function, and a discount factor. At each time step k, the intelligent resource orchestrator on the MEC server reads historical bandwidth, the current receive buffer, the actual processing latency of the previous GOP, the actual memory usage of the previous GOP, the UE state, and metadata of the GOP to be encoded. Based on the read data, it constructs a state space, and then constructs a state vector based on the state space: in, It's the past Historical bandwidth per time step; This is the current receive buffer level; and These are the actual processing time and memory usage of the previous GOP reported by the client. It is a limitation of the device's physical memory; Includes metadata features of the GOP to be encoded.

3. The method according to claim 2, characterized in that, The intelligent resource orchestrator, based on deep reinforcement learning (DRL), determines the encoding strategy according to the state space and state vector. The V-PCC encoder generates V-PCC encoding actions according to the encoding strategy, including: The intelligent resource orchestrator establishes an aggregated V-PCC bitstream model. In this model, the original dynamic point cloud sequence is stored on the server side and divided into K consecutive Groups of Points (GOPs). For the k-th GOP, the intelligent resource orchestrator uses the currently perceived multidimensional state vector... Input to parameterization In the Actor neural network, after the network performs forward computation, it outputs the probability distribution corresponding to the joint action space, and a strategy is formulated based on the probability distribution. Sampling, i.e. The specific encoding strategy vector is dynamically determined for the k-th GOP. The encoding strategy vector From GOP length Geometric quantization parameters and reference structure composition: in, This indicates the number of frames contained in each GOP. Used to adjust point cloud density and reconstruction quality. Used to distinguish between one-way forecasting and two-way forecasting; V-PCC encoder for encoding strategy vector Perform parameter decoupling and physical mapping processing, and encode the strategy vector. The first dimension is resolved to the GOP length. Used to define the number of point cloud frames to be processed in the current batch; encoding strategy vector The second dimension is resolved into geometric quantization parameters. Used to dynamically adjust the bit rate and point cloud spatial precision at the underlying encoding level; encoding strategy vector The third dimension is resolved into a frame reference structure. These parameters are converted into Boolean instructions to indicate whether the encoder is currently using low-memory-overhead unidirectional prediction or high-compression-rate bidirectional prediction. These three decoupled parameters constitute the V-PCC encoding action.

4. The method according to claim 3, characterized in that, The intelligent resource orchestrator sets memory cost constraints, calculates the peak memory and composite QoE utility function corresponding to the UE performing the V-PCC encoding action, and calculates the optimal encoding strategy that maximizes the expected cumulative QoE while satisfying the memory cost constraints through iterative feedback calibration, including: The calculation formula for the peak memory model corresponding to the UE performing the V-PCC encoding action is shown in equation (8), and the hard security constraint of the bandwidth security margin is: in, This is the baseline memory overhead of the decoder; This indicates the number of active reference frames buffered in the DPB. This indicates the memory usage for decoding a single frame; It is the safety margin coefficient; The constraints for setting the GOP length, quantization parameters, reference structure, DPB peak memory, and receive buffer capacity are shown in equations (9)-(13): The cumulative constraint cost for determining whether the peak memory of the DPB predicted for the current action exceeds the physical limit of the device is set as shown in equations (14)-(15): in, It is a discount factor used to balance current and future rewards and ensure the convergence of the cumulative goal; the safety budget is strictly set to... ; At each time step, the DPB shadow corrects the current memory prediction based on the actual feedback error from the previous GOP. The calibrated DPB memory is: in, It is an initial prediction derived from the memory model; It is the feedback compensation coefficient; It is the true feedback residual from time step k to time step (k-1); Set the composite QoE utility function as follows: in, These are weighting coefficients; It's about visual quality; It's a network lag issue; It is a calculation lag; It refers to the smoothness of the quality; It's about visual quality; It's a network lag issue; It is a calculation lag; The formulas for calculating the smoothness of the quality are shown in equations (18)-(21). in, It is a set of sensitivity coefficients for perceiving visual quality, representing the slope of a curve in an image; It is the set visual perception constant; the square of the ReLU function is used to calculate the penalty term for system timeout processing delay; Client CPU frequency; It is the visual quality at time k-1; Set to simultaneously satisfy the above equation (9) - The optimization objective for maximizing the expected cumulative QoE under various constraints such as GOP length, quantization parameters, reference structure, DPB peak memory, and receive buffer capacity is: The DPB shadow maintains the mirror state of the client's decoded image buffer on the server side. The DPB shadow calculates the peak memory corresponding to the V-PCC encoding action performed by the UE, and performs feedback calibration on the encoding strategy based on the peak memory. If the current encoding strategy has a constraint violation risk, CRPO enters safe recovery mode; otherwise, it enters QoE optimization mode. Through iterative feedback calibration, when the actual feedback residual... When the value is below the minimum tolerance threshold, the feedback compensation coefficient When the value approaches 0, it indicates that the server shadow model and the client's real state have reached synchronization, the iterative feedback calibration stops, and the intelligent resource orchestrator determines the optimal coding strategy that maximizes the expected cumulative QoE while satisfying the memory cost constraint, and thus the optimal coding strategy corresponding to the maximum QoE.

5. The method according to claim 4, characterized in that, The V-PCC encoder generates a final V-PCC encoding action according to the optimal encoding strategy, generates a point cloud GOP sequence based on the final V-PCC encoding action, and transmits the point cloud GOP sequence to the UE via a wireless transmission network, including: The V-PCC encoder performs parameter decoupling and physical mapping processing on the optimal coding strategy, and the first dimension of the optimal coding strategy is parsed as the GOP length. The second dimension of the optimal encoding strategy is parsed into geometric quantization parameters. The third dimension of the optimal coding strategy is parsed into a frame reference structure. The specific parameters after decoupling , and The final V-PCC coding action is formed, and a point cloud GOP sequence is generated based on the final V-PCC coding action. The point cloud GOP sequence is then sent to the UE via the wireless transmission network. The UE reports the actual processing latency and memory usage when executing the point cloud GOP sequence to the MEC server for the next round of model calibration, enabling the agent in the CM-JRO structure to perceive the model offset and adaptively adjust the coding strategy.

Images