A method for MR virtual picture display based on a generative adversarial network

Abstract

The application discloses a kind of MR virtual picture display methods based on generative adversarial network, comprising the following steps: build display residual account book, gather anchor point, visibility, illumination, interframe residual and attitude compensation;In SIREN generator set phase, frequency, visibility, frequency division, wraparound and recharge node;With anchor point phase residual modulation phase item and frequency item, form phase-locked implicit state;With visibility residual generation occlusion deduction gate, correct interlayer transmission;With illumination band residual frequency division deduction, correct brightness and boundary;With interframe phase residual wraparound phase, form continuous phase implicit state;Decoding facet element display record, identify and recharge residual, splice output MR virtual picture.The application relates to the technical field of mixed reality display and artificial intelligence image generation, realizes the stable display of MR virtual picture, improves anchor point accuracy, occlusion fusion and interframe continuity.

Description

Technical Field

[0001] This invention relates to the fields of mixed reality display and artificial intelligence image generation technology, and in particular to a method for displaying MR virtual images based on generative adversarial networks. Background Technology

[0002] MR (Mixed Reality) virtual image display technology is used to overlay virtual images onto real-world scenes. The display process relies on spatial anchor point positioning, real-world scene depth perception, lighting matching, and continuous frame rendering. Existing methods mostly employ 3D rendering, depth occlusion masking, image fusion, or generative adversarial networks to generate virtual images, which are then overlaid and displayed based on the coordinate relationships of the MR device. The SIREN structure has continuous implicit representation capabilities and can be used to represent image and spatial signals, capable of expressing high-frequency textures and continuous phase changes.

[0003] Existing generative model-based MR display methods primarily focus on the visual generation of virtual images, with spatial anchor point offsets, differences in real-world occlusion, lighting frequency band differences, and inter-frame phase jumps mostly corrected during post-rendering processing. The implicit sinusoidal layers within the generator lack phase modulation, frequency modulation, inter-layer visibility subtraction, lighting frequency division subtraction, and phase wrap-around processing for MR display residuals. Discriminators are also primarily used for image realism assessment, making it difficult to feed display errors back to the next display cycle. This easily leads to problems such as virtual image anchor point drift, misaligned occlusion boundaries, unnatural lighting blending, and continuous frame flickering. Therefore, providing a generative adversarial network-based MR virtual image display method is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] One objective of this invention is to propose a method for displaying MR virtual images based on generative adversarial networks. This invention utilizes an improved SIREN generator and a display residual ledger to achieve phase-locked propagation, visibility deduction, illumination correction, and residual refeedback of virtual face elements, and has the advantages of stable anchor points, accurate occlusion, and high image continuity.

[0005] A method for displaying MR virtual images based on generative adversarial networks according to an embodiment of the present invention includes the following steps: Establish a display residual ledger indexed by display cycle, spatial anchor point, and virtual surface element, and register anchor point phase residual, visibility residual, illumination band residual, inter-frame phase residual, and attitude compensation mark. Set up the phase modulation node, frequency modulation node, visible subtraction node, frequency division subtraction node, phase wrap-around node, and residual backfeed node in the SIREN generator; The phase modulation node forms a phase compensation term using the phase residual of the same index anchor point and corrects the sinusoidal implicit layer phase term. The frequency modulation node forms a frequency scaling term using the phase residual of the same anchor point and corrects the sinusoidal implicit layer frequency term, thus obtaining the phase-locked implicit state. It can be seen that the subtraction node maps the visibility residual to the occlusion subtraction gate. The occlusion subtraction gate is multiplied into the inter-layer output and then passed to the next sinusoidal implicit layer to obtain the visibility implicit state. The frequency subtraction node decomposes the illumination frequency band residual into low-frequency brightness residual and high-frequency boundary residual. The low-frequency layer group subtracts the low-frequency brightness residual, and the high-frequency layer group subtracts the high-frequency boundary residual to obtain the implicit illumination correction state. The phase wrap-around node marks the cross-cycle phase with the inter-frame phase residual, and replaces the cross-cycle phase with the phase mark of the previous cycle and the attitude compensation mark to obtain the continuous phase implicit state; The face element decoding layer forms face element display records from continuous phase implicit states. The discriminator calculates the update residuals of the face element display records. The residual backfeed node replaces the next cycle's same index residual item with the update residual. Face element display records with all types of update residuals below the corresponding thresholds are retained and stitched together into an MR virtual screen.

[0006] Optionally, the establishment of the display residual ledger includes: The ledger rows are determined using the same display cycle, the same spatial anchor point, and the same virtual element; In the ledger line, the phase deviation of the virtual surface element relative to the spatial anchor point is recorded as the anchor point phase residual, the occlusion level difference between the virtual surface element and the real scene is recorded as the visibility residual, the brightness difference and boundary difference between the virtual surface element and the real scene are merged and recorded as the illumination frequency band residual, the period span difference between the current period phase mark and the previous period phase mark is recorded as the inter-frame phase residual, and the attitude compensation amount corresponding to the change in observation attitude is recorded as the attitude compensation mark.

[0007] Optionally, the node settings of the generative adversarial network include: The SIREN generator retains the sinusoidal implicit layers arranged by layer number; A phase modulation node is connected in series before the phase term of each sinusoidal implicit layer, and a frequency modulation node is connected in series before the frequency term of each sinusoidal implicit layer. Insert visible deduction nodes between adjacent sinusoidal implicit layers, and use the output of the previous sinusoidal implicit layer as the deduction amount of the visible deduction node; The low-frequency layer group and the high-frequency layer group are divided according to the layer frequency of the sinusoidal implicit layer, and frequency division and subtraction nodes are connected in series in the low-frequency layer group and the high-frequency layer group respectively. A phase wrap-around node is connected in series before the surface decoding layer, and a residual feed-back node is connected in series between the discriminator output and the display residual ledger.

[0008] Optionally, the operations of the phase modulation node and frequency modulation node in processing the anchor point phase residual include: Read the anchor point phase residuals corresponding to the same display period, the same spatial anchor point, and the same virtual surface element in the display residual ledger; The compensation direction is determined based on the sign of the anchor point phase residual. When the anchor point phase residual deviates in the positive direction, reverse compensation is used, and when the anchor point phase residual deviates in the negative direction, positive compensation is used. The compensation level is selected based on the amplitude range of the anchor point phase residual, and the phase compensation term is formed by the compensation direction and the compensation level. Before sinusoidal propagation occurs in the sinusoidal implicit layer, the phase compensation term is superimposed on the phase term to obtain the phase-locked phase term. The frequency participation interval is selected based on the amplitude range of the phase residual at the same anchor point, and the sinusoidal implicit layer participating in frequency modulation is selected from the frequency participation interval. The selected sinusoidal implicit layers use a frequency scaling term corresponding to the compensation level to form a phase-locked frequency term, while the unselected sinusoidal implicit layers retain the original frequency term. The sinusoidal implicit layer propagates layer by layer based on the phase-locked phase term and the phase-locked frequency term or the original frequency term to obtain the phase-locked implicit state.

[0009] Optionally, the operation of the visible deduction node in processing the visibility residual includes: Read the visibility residuals corresponding to the same display period, the same spatial anchor point, and the same virtual surface in the display residual ledger, and obtain the previous sinusoidal implicit layer output as the deducted amount; Based on the differences in occlusion levels in the visibility residual, virtual facets are divided into real occlusion state, virtual display state, and boundary transition state; The actual occlusion state matches the first reduction level, the virtual display state matches the hold level, and the boundary transition state matches the second reduction level. The first deduction gear, the hold gear, and the second deduction gear respectively generate the first obstruction deduction gate, the hold gate, and the second obstruction deduction gate; The first occlusion gate reduces the transmission amplitude of the virtual element body in the subtracted amount, the holding gate retains the transmission amplitude of the virtual element body in the subtracted amount, and the second occlusion gate reduces the transmission amplitude of the virtual element boundary in the subtracted amount. The amount deducted after being processed by the first occlusion deduction gate, the holding gate, and the second occlusion deduction gate is used as the interlayer visible transmission amount and enters the next sinusoidal implicit layer to obtain the visible implicit state.

[0010] Optionally, the operation of the frequency division subtraction node in processing the illumination frequency band residual includes: Read the illumination frequency band residuals corresponding to the same display period, the same spatial anchor point, and the same virtual surface in the display residual ledger, and extract the brightness differences and boundary differences in the illumination frequency band residuals; Using the layer frequency boundary of the sinusoidal implicit layer as the boundary, the low-frequency layer group receives the low-frequency components in the visible implicit state, and the high-frequency layer group receives the high-frequency components in the visible implicit state. Brightness differences are entered into the low-frequency subtraction table to obtain the low-frequency subtraction gate. The low-frequency subtraction gate is connected in series before the amplitude term of the low-frequency layer group to subtract the low-frequency components layer by layer, forming the low-frequency subtraction feature. Boundary differences are entered into the high-frequency subtraction table to obtain the high-frequency subtraction gate. The high-frequency subtraction gate is connected in series before the implicit component corresponding to the virtual surface element boundary in the high-frequency layer group, and the high-frequency component is subtracted layer by layer to form the high-frequency subtraction feature. Low-frequency subtraction features and high-frequency subtraction features are merged under the same virtual surface index to obtain the illumination-corrected implicit state.

[0011] Optionally, the operation of the phase wrap-around node in processing inter-frame phase residuals includes: Using the display cycle, spatial anchor point, and virtual surface element as indexes, the inter-frame phase residual, attitude compensation mark, and previous cycle phase mark are retrieved from the display residual ledger. Extract the current period phase marker from the implicit state of illumination correction; retain the current period phase marker when the inter-frame phase residual falls into an uncrossed interval; When the inter-frame phase residual falls into the positive crossing interval, the result of the superposition of the phase mark of the previous cycle and the attitude compensation mark replaces the phase mark of the current cycle. When the inter-frame phase residual falls into the negative crossing interval, the result of subtracting the attitude compensation mark from the previous cycle phase mark replaces the current cycle phase mark; After the replacement, the current period phase marker is backfilled into the position of the corresponding virtual element in the illumination correction implicit state to obtain the continuous phase implicit state.

[0012] Optionally, the operation of the face element decoding layer in forming face element display records from continuous phase implicit states includes: Positioning continuous phase implicit states using the same display cycle, the same spatial anchor point, and the same virtual surface element; Based on the coverage area of ​​the virtual surface element, the implicit state of the continuous phase is classified into the main state of the surface element and the boundary state of the surface element. The main state of the element is converted into the main display value and the display layer value of the element by the element decoding layer; The boundary state of a virtual surface element is paired with the boundary state of an adjacent virtual surface element, and the phase continuity and hierarchical continuity of the paired elements are compared. Boundary states that satisfy both phase continuity and hierarchical continuity conditions are assigned to the current virtual surface element; boundary states that do not satisfy phase continuity or hierarchical continuity conditions are transferred to the boundary candidate record of the adjacent virtual surface element. The face element decoding layer combines the face element main display value, face element display level value, boundary state belonging to the current virtual face element, and current period phase mark into a face element display record.

[0013] Optionally, the discriminator calculates the update residual of the surface cell display record, including: After the face element decoding layer forms a face element display record, the discriminator locates the ledger line corresponding to the same display period, the same spatial anchor point, and the same virtual face element, and establishes a temporary discrimination bit in the ledger line; When determining the anchor point, the current period phase marker in the surface cell display record is used to replace the anchor point phase verification value in the temporary determination position. When the anchor point phase verification value falls into the compensated threshold range, the anchor point phase residual is rewritten as a zero residual marker. When it does not fall into the compensated threshold range, the original anchor point phase residual is retained. During visibility determination, the visibility level to be verified value in the temporary determination bit is replaced with the surface display level value. When the visibility level to be verified value meets the occlusion level, the visibility residual is rewritten as a zero residual mark. When the occlusion level is not met, the original visibility residual is retained. During illumination discrimination, the illumination frequency band to be verified value in the temporary discrimination position is replaced with the main display value of the surface element and the boundary state. When the illumination frequency band to be verified value meets both the brightness threshold and the boundary threshold, the illumination frequency band residual is rewritten as a zero residual mark. When both the brightness threshold and the boundary threshold are not met, the original illumination frequency band residual is retained. During inter-frame discrimination, the inter-frame phase to be verified value in the temporary discrimination bit is replaced with the current period phase mark. When the inter-frame phase to be verified value falls into a continuous interval, the inter-frame phase residual is rewritten as a zero residual mark. When it does not fall into a continuous interval, the original inter-frame phase residual is retained. The residual backfeed node writes the residual items that have not been rewritten to zero residual markers into the ledger row under the corresponding index in the next display period, and retains the cell display records where all residual items are marked as zero residual markers.

[0014] Optionally, the stitching of the residual recharge node and the MR virtual image includes: After the discriminator rewrites the residual items in the ledger line, the residual feedback node checks the anchor point phase residual, visibility residual, illumination band residual and inter-frame phase residual line by line. If there are ledger lines that have not been rewritten to zero residual markers, the residual feedback node will retain the unrewritten residual items under the corresponding index in the next display cycle, and retain the corresponding attitude compensation markers; For ledger rows where all residual items are marked as zero residuals, the corresponding face value display record is entered into the splicing queue. The splicing queue reads the boundary state and current period phase marker of the adjacent face display record according to the adjacency relationship of the virtual face; When the boundary states of adjacent surface cell display records are consistent and the phase markers of the current period are continuous, the adjacent surface cell display records are spliced ​​together; when the boundary states of adjacent surface cell display records are inconsistent or the phase markers of the current period are discontinuous, the adjacent surface cell display records do not participate in the splicing of this period. The completed spliced ​​element display record is output as an MR virtual image according to the spatial anchor points.

[0015] The beneficial effects of this invention are: (1) By displaying the residual ledger to register the anchor point phase residual, visibility residual, illumination band residual and inter-frame phase residual, and introducing the anchor point phase residual into the phase and frequency terms of the SIREN generator, the present invention can complete the spatial anchor point locking in the virtual image generation stage, reduce the drift of virtual surface elements relative to the spatial anchor point, and improve the spatial stability of MR virtual images.

[0016] (2) The present invention transforms the visibility residual into the occlusion subtraction gate through the visible subtraction node, so that the occlusion relationship participates in the interlayer propagation between adjacent sinusoidal implicit layers. The low-frequency brightness residual and high-frequency boundary residual are processed separately through the frequency subtraction node, which can reduce the problems of occlusion boundary misalignment and unnatural lighting fusion, and improve the fusion consistency between virtual images and real scenes.

[0017] (3) The present invention processes the inter-frame phase residual through the phase wrap-around node, and after the discriminator calculates and updates the residual, the residual feed-back node updates the display residual ledger for the next display cycle. This enables unclosed residuals to continue to participate in the generation and correction in subsequent display cycles, reducing continuous frame flicker and jumps, and improving the continuous display effect of MR virtual images. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a method for displaying MR virtual images based on generative adversarial networks proposed in this invention; Figure 2 This is a structural diagram of the improved SIREN generator for a generative adversarial network-based MR virtual image display method proposed in this invention. Figure 3 This is a residual backfeeding and image stitching diagram for a generative adversarial network-based MR virtual image display method proposed in this invention. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0020] refer to Figures 1-3 A method for displaying MR virtual images based on generative adversarial networks includes the following steps: Establish a display residual ledger indexed by display cycle, spatial anchor point, and virtual surface element, and register anchor point phase residual, visibility residual, illumination band residual, inter-frame phase residual, and attitude compensation mark. The display cycle corresponds to one processing cycle of the MR device continuously refreshing the virtual image, with each display cycle corresponding to one frame of MR virtual image to be output. Spatial anchor points correspond to fixed spatial positions in the real scene used to mount the virtual image. Virtual facets are obtained by segmenting the virtual image to be displayed according to its outline boundary, transparent area, texture continuity area, and boundary transition area. The display residual ledger uses the display cycle number, spatial anchor point number, and virtual facet number as a triple index, with each triple index corresponding to a ledger line. The ledger line contains anchor point phase residual columns, visibility residual columns, illumination band residual columns, inter-frame phase residual columns, and pose compensation marker columns. The anchor point phase residual column records the deviation of the virtual facet phase from the spatial anchor point phase. The visibility residual column records the difference in the virtual facet display level relative to the real scene occlusion level. The illumination band residual column records the differences between the virtual facet and the real scene in the brightness band and boundary band. The inter-frame phase residual column records the cycle crossing status of the current cycle phase marker relative to the previous cycle phase marker. The attitude compensation marker registers the compensation direction and amount of the current observation attitude relative to the observation attitude of the previous display cycle.

[0021] Set up the phase modulation node, frequency modulation node, visible subtraction node, frequency division subtraction node, phase wrap-around node, and residual backfeed node in the SIREN generator; The SIREN generator retains sinusoidal implicit layers arranged by layer number. Each sinusoidal implicit layer contains a phase term, a frequency term, an offset term, and a layer output. Phase modulation nodes are connected in series before the phase term of the sinusoidal implicit layer, and frequency modulation nodes are connected in series before the frequency term. Visible subtraction nodes are connected in series between two adjacent sinusoidal implicit layers; the output of the previous sinusoidal implicit layer enters the visible subtraction node before entering the next sinusoidal implicit layer. Frequency division subtraction nodes are connected in series in the propagation paths of the low-frequency layer group and the high-frequency layer group. Phase wrap-around nodes are connected in series between the frequency division subtraction nodes and the surface decoding layer. Residual backfeed nodes are connected in series between the discriminator output and the display residual ledger for the next display cycle.

[0022] The phase modulation node forms a phase compensation term using the phase residual of the same index anchor point and corrects the sinusoidal implicit layer phase term. The frequency modulation node forms a frequency scaling term using the phase residual of the same anchor point and corrects the sinusoidal implicit layer frequency term, thus obtaining the phase-locked implicit state. The phase modulation node locates and displays the ledger line in the residual ledger according to the display cycle number, spatial anchor point number, and virtual surface element number, and reads the anchor point phase residual. After sign determination, the anchor point phase residual yields a positive deviation, negative deviation, or no deviation result. Positive deviation matches reverse compensation, negative deviation matches positive compensation, and no deviation matches empty compensation. The amplitude of the anchor point phase residual is entered into the amplitude interval table to obtain the compensation level. The compensation direction and compensation level are combined to form a phase compensation item. The phase compensation item is superimposed on the phase item of the sinusoidal implicit layer to form a phase-locked phase item. The frequency modulation node uses the amplitude interval of the same anchor point phase residual to search the frequency participation interval table to obtain the frequency range of the sinusoidal implicit layer participating in phase-locked propagation. Sinusoidal implicit layers whose layer frequencies fall within the frequency range of the sinusoidal implicit layer use a frequency scaling item to form a phase-locked frequency item; sinusoidal implicit layers whose layer frequencies do not fall within the frequency range of the sinusoidal implicit layer retain their original frequency items. The sinusoidal implicit layer propagates layer by layer based on the phase-locked phase term and the phase-locked frequency term or the original frequency term to obtain the phase-locked implicit state.

[0023] It can be seen that the subtraction node maps the visibility residual to the occlusion subtraction gate. The occlusion subtraction gate is multiplied into the inter-layer output and then passed to the next sinusoidal implicit layer to obtain the visibility implicit state. The deduction node locates the ledger line according to the display cycle number, spatial anchor point number, and virtual element number, reads the visibility residual, and obtains the output of the previous sinusoidal implicit layer as the deducted amount. The occlusion level difference in the visibility residual is categorized into the real occlusion state, virtual display state, and boundary transition state. The real occlusion state matches the first deduction level, the virtual display state matches the hold level, and the boundary transition state matches the second deduction level. The first deduction level generates a first occlusion deduction gate, the hold level generates a hold gate, and the second deduction level generates a second occlusion deduction gate. The first occlusion deduction gate reduces the virtual element main body transmission amplitude in the deducted amount, the hold gate retains the virtual element main body transmission amplitude in the deducted amount, and the second occlusion deduction gate reduces the virtual element boundary transmission amplitude in the deducted amount. The processed deducted amount enters the next sinusoidal implicit layer as the inter-layer visibility transmission amount, resulting in the visibility implicit state.

[0024] The frequency subtraction node decomposes the illumination frequency band residual into low-frequency brightness residual and high-frequency boundary residual. The low-frequency layer group subtracts the low-frequency brightness residual, and the high-frequency layer group subtracts the high-frequency boundary residual to obtain the implicit illumination correction state. The frequency subtraction node locates the ledger line according to the display cycle number, spatial anchor point number, and virtual surface element number, and reads the illumination frequency band residual. Brightness differences in the illumination frequency band residual are classified as low-frequency brightness residuals, and boundary differences are classified as high-frequency boundary residuals. The visibility implicit state is divided into low-frequency and high-frequency components according to the layer frequency boundary value of the sinusoidal implicit layer. Low-frequency components enter the low-frequency layer group, and high-frequency components enter the high-frequency layer group. The low-frequency brightness residual retrieves the low-frequency subtraction table to obtain the low-frequency subtraction gate. The low-frequency subtraction gate is concatenated before the amplitude term of the low-frequency layer group, subtracting the low-frequency components layer by layer to obtain the low-frequency subtraction feature. The high-frequency boundary residual retrieves the high-frequency subtraction table to obtain the high-frequency subtraction gate. The high-frequency subtraction gate is concatenated before the implicit component corresponding to the virtual surface element boundary in the high-frequency layer group, subtracting the high-frequency components layer by layer to obtain the high-frequency subtraction feature. The low-frequency subtraction feature and the high-frequency subtraction feature are merged under the same virtual surface element index to obtain the illumination correction implicit state.

[0025] The phase wrap-around node marks the cross-cycle phase with the inter-frame phase residual, and replaces the cross-cycle phase with the phase mark of the previous cycle and the attitude compensation mark to obtain the continuous phase implicit state; The phase wrap-around node locates the ledger line according to the display cycle number, spatial anchor point number, and virtual surface number, and reads the inter-frame phase residual, attitude compensation mark, and previous cycle phase mark. The phase wrap-around node extracts the current cycle phase mark from the illumination correction implicit state. When the inter-frame phase residual falls within an uncrossed interval, the current cycle phase mark is retained. When the inter-frame phase residual falls within a positive crossing interval, the result of superimposing the previous cycle phase mark and the attitude compensation mark replaces the current cycle phase mark. When the inter-frame phase residual falls within a negative crossing interval, the result of subtracting the attitude compensation mark from the previous cycle phase mark replaces the current cycle phase mark. The replaced current cycle phase mark is then filled back into the corresponding virtual surface position in the illumination correction implicit state, resulting in a continuous phase implicit state.

[0026] The face element decoding layer forms face element display records from continuous phase implicit states. The discriminator calculates the update residuals of the face element display records. The residual backfeed node replaces the next cycle's same index residual item with the update residual. Face element display records with all types of update residuals below the corresponding thresholds are retained and stitched together into an MR virtual screen.

[0027] The facet decoding layer reads the continuous phase implicit state according to the display cycle number, spatial anchor point number, and virtual facet number. The continuous phase implicit state is divided into facet main state and facet boundary state according to the virtual facet coverage area. The facet main state is converted into the facet main display value and facet display level value by the facet decoding layer. The facet boundary state is paired with the boundary states of adjacent virtual facets to calculate phase continuity and level continuity. Boundary states that meet the boundary retention conditions are assigned to the current virtual facet; boundary states that do not meet the boundary retention conditions are transferred to the boundary candidate record of the adjacent virtual facet. The facet main display value, facet display level value, the boundary state assigned to the current virtual facet, and the current cycle phase marker are combined to form the facet display record. After the facet display record is formed, the discriminator locates the ledger line corresponding to the same display cycle, the same spatial anchor point, and the same virtual facet, and establishes a temporary discrimination bit in the ledger line. Anchor point discrimination, visibility discrimination, illumination discrimination, and inter-frame discrimination rewrite the anchor point phase residual, visibility residual, illumination band residual, and inter-frame phase residual, respectively. Residual items that meet the corresponding discrimination conditions are rewritten with zero residual markers, while residual items that do not meet the corresponding discrimination conditions retain their original residuals. The residual feedback node registers residual items that have not been rewritten with zero residual markers to the ledger line under the corresponding index in the next display period. Surface display records with all residual items marked with zero residual markers enter the stitching queue. The stitching queue checks the boundary states and current period phase markers of adjacent surface display records according to the virtual surface adjacency relationship. Adjacent surface display records with consistent boundary states and consecutive current period phase markers participate in stitching, and the stitched surface display records are output as MR virtual images according to spatial anchor points.

[0028] In this embodiment, the establishment of the residual ledger includes: The ledger rows are determined using the same display cycle, the same spatial anchor point, and the same virtual element; The ledger line uses the display cycle number, spatial anchor point number, and virtual facet number as search keys. The display cycle number corresponds to the current refresh cycle of the MR virtual image. The spatial anchor point number corresponds to the selected anchor point position in the real scene. The virtual facet number corresponds to a facet display unit in the virtual image to be displayed. Only one ledger line is generated for the same search key. The ledger line is used to store the anchor point phase residual, visibility residual, illumination band residual, inter-frame phase residual, and pose compensation marker for the same virtual facet in the current display cycle.

[0029] In the ledger line, the phase deviation of the virtual surface element relative to the spatial anchor point is recorded as the anchor point phase residual, the occlusion level difference between the virtual surface element and the real scene is recorded as the visibility residual, the brightness difference and boundary difference between the virtual surface element and the real scene are merged and recorded as the illumination frequency band residual, the period span difference between the current period phase mark and the previous period phase mark is recorded as the inter-frame phase residual, and the attitude compensation amount corresponding to the change in observation attitude is recorded as the attitude compensation mark.

[0030] The virtual surface element phase marker is compared with the spatial anchor point phase marker to obtain the anchor point phase residual. The virtual surface element display level is compared with the real scene occlusion level to obtain the visibility residual. The brightness of the virtual surface element subject is compared with the brightness of the real scene to obtain the brightness difference. The boundary change of the virtual surface element is compared with the boundary change of the real scene to obtain the boundary difference. The brightness difference and boundary difference are merged and recorded in the same ledger line as the illumination band residual. The phase marker of the current cycle is compared with the phase marker of the previous cycle to obtain the inter-frame phase residual. The direction and magnitude of the change in the current viewing posture relative to the viewing posture of the previous display cycle are recorded as the posture compensation marker. After the ledger line is generated, the anchor point phase residual is read by the phase modulation node and the frequency modulation node, the visibility residual is read by the visibility subtraction node, the illumination band residual is read by the frequency subtraction node, and the inter-frame phase residual and the posture compensation marker are read by the phase wrap-around node.

[0031] In this embodiment, the node settings of the generative adversarial network include: The SIREN generator retains the sinusoidal implicit layers arranged by layer number; The sinusoidal implicit layers are arranged sequentially from the input side to the front of the surface decoding layer according to their layer numbers. Each sinusoidal implicit layer retains the original phase term, original frequency term, original bias term, and layer output. The layer number is used to determine the propagation order of the sinusoidal implicit layers in the SIREN generator. The layer number is also used to subsequently divide the layer into low-frequency layer groups and high-frequency layer groups according to the layer frequency.

[0032] A phase modulation node is connected in series before the phase term of each sinusoidal implicit layer, and a frequency modulation node is connected in series before the frequency term of each sinusoidal implicit layer. Phase modulation nodes are placed before the original phase term is called by the sinusoidal implicit layer. Frequency modulation nodes are placed before the original frequency term is called by the sinusoidal implicit layer. Phase modulation nodes output phase-locked phase terms. Frequency modulation nodes output phase-locked frequency terms or retain the original frequency term. Phase-locked phase terms and phase-locked frequency terms participate in the phase-locked propagation of the selected sinusoidal implicit layer. Phase-locked phase terms and original frequency terms participate in the propagation of the unselected sinusoidal implicit layer.

[0033] Insert visible deduction nodes between adjacent sinusoidal implicit layers, and use the output of the previous sinusoidal implicit layer as the deduction amount of the visible deduction node; The visible subtraction node is located between the preceding sinusoidal implicit layer and the following sinusoidal implicit layer. The output of the preceding sinusoidal implicit layer enters the visible subtraction node. After the visible subtraction node completes the processing of the first occlusion subtraction gate, the hold gate, or the second occlusion subtraction gate, it outputs the interlayer visible propagation. The following sinusoidal implicit layer receives the interlayer visible propagation and continues to propagate.

[0034] The low-frequency layer group and the high-frequency layer group are divided according to the layer frequency of the sinusoidal implicit layer, and frequency division and subtraction nodes are connected in series in the low-frequency layer group and the high-frequency layer group respectively. Sinusoidal implicit layers with layer frequencies below the layer frequency threshold are classified into the low-frequency layer group. Sinusoidal implicit layers with layer frequencies not below the layer frequency threshold are classified into the high-frequency layer group. The low-frequency layer group subtracts node brightness differences. The high-frequency layer group subtracts node boundary differences. The low-frequency layer group outputs low-frequency subtraction features. The high-frequency layer group outputs high-frequency subtraction features.

[0035] A phase wrap-around node is connected in series before the surface decoding layer, and a residual feed-back node is connected in series between the discriminator output and the display residual ledger.

[0036] The phase wrap-around node is located between the illumination correction implicit state and the surface decoding layer, and completes the cross-cycle phase replacement before surface decoding. The residual backfeed node is located between the discriminator output and the display residual ledger, and registers the residual items that have not been rewritten to the zero residual mark by the discriminator to the ledger line under the corresponding index in the next display cycle.

[0037] In this embodiment, the operations of the phase modulation node and the frequency modulation node in processing the anchor point phase residual include: Read the anchor point phase residuals corresponding to the same display period, the same spatial anchor point, and the same virtual surface element in the display residual ledger; The read operation locates the ledger line by displaying the period number, spatial anchor point number, and virtual element number, and then extracts the anchor point phase residual from the ledger line. The anchor point phase residual retains two pieces of information: deviation direction and deviation magnitude. The deviation direction is used to determine the subsequent compensation direction. The deviation magnitude is used to determine the subsequent compensation level and frequency participation range.

[0038] The compensation direction is determined based on the sign of the anchor point phase residual. When the anchor point phase residual deviates in the positive direction, reverse compensation is used, and when the anchor point phase residual deviates in the negative direction, positive compensation is used. A positive deviation corresponds to the virtual surface phase deviating in the positive direction relative to the spatial anchor phase; reverse compensation is used to cancel the positive deviation. A negative deviation corresponds to the virtual surface phase deviating in the negative direction relative to the spatial anchor phase; positive compensation is used to cancel the negative deviation. When there is no deviation in the anchor phase residual, the phase compensation direction is marked as empty compensation.

[0039] The compensation level is selected based on the amplitude range of the anchor point phase residual, and the phase compensation term is formed by the compensation direction and the compensation level. The deviation of the anchor point phase residual falls within a preset range. This preset range corresponds to a compensation level. The greater the deviation, the higher the matching compensation level. The phase compensation item consists of a compensation direction marker and a compensation level. The compensation direction controls the positive or negative direction of the phase compensation item. The compensation level controls the compensation intensity of the phase compensation item.

[0040] Before sinusoidal propagation occurs in the sinusoidal implicit layer, the phase compensation term is superimposed on the phase term to obtain the phase-locked phase term. Before executing sinusoidal propagation, each sinusoidal implicit layer first reads the original phase term and the phase compensation term. The original phase term and the phase compensation term are merged to form a phase-locked phase term. The phase-locked phase term replaces the original phase term in the current sinusoidal implicit layer propagation.

[0041] The frequency participation interval is selected based on the amplitude range of the phase residual at the same anchor point, and the sinusoidal implicit layer participating in frequency modulation is selected from the frequency participation interval. The deviation magnitude of the phase residual at the same anchor point is mapped to the frequency participation interval. The frequency participation interval is used to define the range of sinusoidal implicit layers participating in frequency modulation. When the layer frequency of a sinusoidal implicit layer is within the frequency participation interval, the sinusoidal implicit layer is marked as a selected sinusoidal implicit layer. When the layer frequency of a sinusoidal implicit layer is not within the frequency participation interval, the sinusoidal implicit layer is marked as an unselected sinusoidal implicit layer.

[0042] The selected sinusoidal implicit layers use a frequency scaling term corresponding to the compensation level to form a phase-locked frequency term, while the unselected sinusoidal implicit layers retain the original frequency term. The selected sinusoidal implicit layers read the frequency scaling term corresponding to the compensation level. The original frequency term is processed by the frequency scaling term to form the phase-locked frequency term. Unselected sinusoidal implicit layers do not read the frequency scaling term, and the original frequency term is directly retained.

[0043] The sinusoidal implicit layer propagates layer by layer based on the phase-locked phase term and the phase-locked frequency term or the original frequency term to obtain the phase-locked implicit state.

[0044] The selected sinusoidal implicit layers are propagated using phase-locked phase and phase-locked frequency terms. Unselected sinusoidal implicit layers are propagated using phase-locked phase and original frequency terms. The propagation result of each layer serves as the input for the next layer. After all sinusoidal implicit layers have completed propagation, the phase-locked implicit states corresponding to the display period, spatial anchor points, and virtual facets are obtained.

[0045] In this embodiment, the visible deduction node's operation for processing visibility residuals includes: Read the visibility residuals corresponding to the same display period, the same spatial anchor point, and the same virtual surface in the display residual ledger, and obtain the previous sinusoidal implicit layer output as the deducted amount; The visible deduction node operates between adjacent sinusoidal implicit layers. The visible deduction node locates ledger rows and reads visibility residuals by displaying the period number, spatial anchor number, and virtual polygon number. The output of the previous sinusoidal implicit layer, after entering the visible deduction node, serves as the deduction amount to be deducted for occlusion deduction, preservation, or boundary deduction.

[0046] Based on the differences in occlusion levels in the visibility residual, virtual facets are divided into real occlusion state, virtual display state, and boundary transition state; The difference in occlusion levels indicates the front-to-back relationship between the real scene and the virtual polygons. When the real scene is in front of the virtual polygon, the virtual polygon enters a real occlusion state. When the virtual polygon is in front of the real scene, the virtual polygon enters a virtual display state. When the real scene and the virtual polygon alternate at the boundary sampling band, the virtual polygon enters a boundary transition state.

[0047] The actual occlusion state matches the first reduction level, the virtual display state matches the hold level, and the boundary transition state matches the second reduction level. The first reduction level is used to handle the area where the real scene occludes the main body of the virtual element. The maintain level is used to handle the area where the virtual element is displayed normally. The second reduction level is used to handle the boundary transition area where the real scene and the virtual element meet.

[0048] The first deduction gear, the hold gear, and the second deduction gear respectively generate the first obstruction deduction gate, the hold gate, and the second obstruction deduction gate; The first occlusion subtraction gate corresponds to the strong subtraction parameters of the main body region. The preservation gate corresponds to the preservation parameters of the main body region. The second occlusion subtraction gate corresponds to the weak subtraction parameters of the boundary region. All three gates are bound to the subtraction amount of the same virtual element.

[0049] The first occlusion gate reduces the transmission amplitude of the virtual element body in the subtracted amount, the holding gate retains the transmission amplitude of the virtual element body in the subtracted amount, and the second occlusion gate reduces the transmission amplitude of the virtual element boundary in the subtracted amount. In a real occlusion state, the first occlusion reduction gate acts on the main region component of the subtracted amount, and the main region component is reduced after passing through the first occlusion reduction gate. In a virtual display state, the maintenance gate acts on the main region component, and the main region component maintains its original transmitted amplitude. In a boundary transition state, the second occlusion reduction gate acts on the boundary region component, and the boundary region component is reduced after passing through the second occlusion reduction gate.

[0050] The amount deducted after being processed by the first occlusion deduction gate, the holding gate, and the second occlusion deduction gate is used as the interlayer visible transmission amount and enters the next sinusoidal implicit layer to obtain the visible implicit state.

[0051] The inter-layer visible propagation is passed to the next sinusoidal implicit layer according to the virtual surface index. The next sinusoidal implicit layer continues sinusoidal propagation after receiving the inter-layer visible propagation. After all visible deduction nodes have been processed, the visibility implicit state of the corresponding virtual surface is obtained.

[0052] In this embodiment, the operation of the frequency division subtraction node for processing the residual illumination frequency band includes: Read the illumination frequency band residuals corresponding to the same display period, the same spatial anchor point, and the same virtual surface in the display residual ledger, and extract the brightness differences and boundary differences in the illumination frequency band residuals; Frequency division deduction nodes locate ledger rows by displaying cycle number, spatial anchor point number, and virtual element number. Brightness differences in the illumination frequency band residual correspond to the brightness deviation between the main area of ​​the virtual element and the corresponding area in the real scene. Boundary differences correspond to the texture transition deviation between the boundary area of ​​the virtual element and the boundary area of ​​the real scene.

[0053] Using the layer frequency boundary of the sinusoidal implicit layer as the boundary, the low-frequency layer group receives the low-frequency components in the visible implicit state, and the high-frequency layer group receives the high-frequency components in the visible implicit state. The layer frequency boundary value is used to divide the sinusoidal implicit layer into a low-frequency layer group and a high-frequency layer group. After the visibility implicit state enters the frequency subtraction node, the low-frequency layer group receives the low-frequency component corresponding to the overall brightness change. The high-frequency layer group receives the high-frequency component corresponding to the boundary texture change.

[0054] Brightness differences are entered into the low-frequency subtraction table to obtain the low-frequency subtraction gate. The low-frequency subtraction gate is connected in series before the amplitude term of the low-frequency layer group to subtract the low-frequency components layer by layer, forming the low-frequency subtraction feature. The low-frequency subtraction table records the correspondence between brightness difference intervals and low-frequency subtraction gates. A low-frequency subtraction gate is obtained when the brightness difference falls within the corresponding brightness difference interval. The low-frequency subtraction gate performs subtraction processing before each sinusoidal implicit layer amplitude term in the low-frequency layer group. After layer-by-layer subtraction is completed, the low-frequency layer group forms a low-frequency subtraction feature.

[0055] Boundary differences are entered into the high-frequency subtraction table to obtain the high-frequency subtraction gate. The high-frequency subtraction gate is connected in series before the implicit component corresponding to the virtual surface element boundary in the high-frequency layer group, and the high-frequency component is subtracted layer by layer to form the high-frequency subtraction feature. The high-frequency deduction table records the correspondence between boundary difference intervals and high-frequency deduction gates. A high-frequency deduction gate is obtained when a boundary difference falls into its corresponding boundary difference interval. The high-frequency deduction gate performs deduction processing before the implicit component corresponding to the virtual surface element boundary in the high-frequency layer group. After layer-by-layer deduction is completed, the high-frequency layer group forms a high-frequency deduction feature.

[0056] Low-frequency subtraction features and high-frequency subtraction features are merged under the same virtual surface index to obtain the illumination-corrected implicit state.

[0057] Low-frequency and high-frequency subtraction features under the same virtual surface index are merged at the end of the frequency subtraction node. The merged implicit state retains both the brightness correction results completed by the low-frequency layer group and the boundary correction results completed by the high-frequency layer group, serving as the illumination correction implicit state.

[0058] In this embodiment, the operation of the phase wrap-around node to process inter-frame phase residuals includes: Using the display cycle, spatial anchor point, and virtual surface element as indexes, the inter-frame phase residual, attitude compensation mark, and previous cycle phase mark are retrieved from the display residual ledger. Phase wrap-around nodes locate ledger rows by displaying the cycle number, spatial anchor point number, and virtual surface element number. Inter-frame phase residuals are used to determine whether the current cycle phase crosses the sinusoidal cycle boundary. Attitude compensation markers describe the direction and amount of phase compensation resulting from changes in observation attitude. The previous cycle phase marker provides a reference for the current cycle phase wrap-around.

[0059] Extract the current period phase marker from the implicit state of illumination correction; retain the current period phase marker when the inter-frame phase residual falls into an uncrossed interval; After the implicit illumination correction state completes frequency division and subtraction, it carries the current period phase marker. If the inter-frame phase residual falls into an uncrossed interval, it means that the current period phase is still within the continuous range of the previous period phase, and the current period phase marker is not replaced.

[0060] When the inter-frame phase residual falls into the positive crossing interval, the result of the superposition of the phase mark of the previous cycle and the attitude compensation mark replaces the phase mark of the current cycle. A positive crossing interval indicates that the current cycle phase crosses the sinusoidal cycle boundary in a positive direction relative to the previous cycle phase. The phase looping node takes the previous cycle phase marker and attitude compensation marker to generate a positive looping phase marker. The positive looping phase marker replaces the current cycle phase marker.

[0061] When the inter-frame phase residual falls into the negative crossing interval, the result of subtracting the attitude compensation mark from the previous cycle phase mark replaces the current cycle phase mark; A negative crossover interval indicates that the current cycle phase crosses the sinusoidal cycle boundary in a negative direction relative to the previous cycle phase. The phase loopback node takes the previous cycle phase marker and attitude compensation marker to generate a negative loopback phase marker. The negative loopback phase marker replaces the current cycle phase marker.

[0062] After the replacement, the current period phase marker is backfilled into the position of the corresponding virtual element in the illumination correction implicit state to obtain the continuous phase implicit state.

[0063] The phase wrap-around node writes the preserved current period phase marker, positive wrap-around phase marker, or negative wrap-around phase marker back to the illumination correction implicit state position of the corresponding virtual surface element. The written-back implicit state is then entered into the surface element decoding layer as a continuous phase implicit state.

[0064] In this embodiment, the operation of the face element decoding layer to form face element display records from continuous phase implicit states includes: Positioning continuous phase implicit states using the same display cycle, the same spatial anchor point, and the same virtual surface element; The face element decoding layer retrieves the implicit states of continuous phases according to the display cycle number, spatial anchor point number, and virtual face element number. The same virtual face element corresponds to a face element display record to be formed.

[0065] Based on the coverage area of ​​the virtual surface element, the implicit state of the continuous phase is classified into the main state of the surface element and the boundary state of the surface element. Implicit states located inside virtual facets within the virtual facet coverage area are classified as the facet's main state. Implicit states located in the facet boundary sampling zone within the virtual facet coverage area are classified as the facet's boundary state. The facet's main state is used to calculate the main display result. The facet's boundary state is used to determine the boundary assignment with adjacent virtual facets.

[0066] The main state of the element is converted into the main display value and the display layer value of the element by the element decoding layer; The face element decoding layer performs value conversion on the face element's main state to obtain the face element's main display value. The face element decoding layer also performs layer conversion on the layer information within the face element's main state to obtain the face element's display layer value. The face element's main display value and display layer value are bound to the same virtual face element number.

[0067] The boundary state of a virtual surface element is paired with the boundary state of an adjacent virtual surface element, and the phase continuity and hierarchical continuity of the paired elements are compared. The virtual facet decoding layer reads the adjacency relationships of virtual facests and determines the adjacent virtual facests of the current virtual facet. The boundary states of the current virtual facet are paired with the boundary states of adjacent virtual facets according to boundary sampling points. After pairing, the phase continuity and hierarchical continuity are calculated.

[0068] Boundary states that satisfy both phase continuity and hierarchical continuity conditions are assigned to the current virtual surface element; boundary states that do not satisfy phase continuity or hierarchical continuity conditions are transferred to the boundary candidate record of the adjacent virtual surface element. Boundary retention conditions include phase continuity thresholds and hierarchical continuity thresholds. When a paired boundary state satisfies both thresholds, the boundary state is retained in the current virtual surface element. When a paired boundary state does not satisfy either threshold, the boundary state is not included in the display record of the current virtual surface element, but is transferred to the boundary candidate record of the adjacent virtual surface element for subsequent discrimination by the adjacent virtual surface element.

[0069] The face element decoding layer combines the face element main display value, face element display level value, boundary state belonging to the current virtual face element, and current period phase mark into a face element display record.

[0070] The element display record uses the display period, spatial anchor point, and virtual element as indexes to record the element's main display value, element display level value, boundary state belonging to the current virtual element, and current period phase marker. The element display record enters the discriminator for residual rewriting.

[0071] In this embodiment, the discriminator calculates the update residual of the surface cell display record, including: After the face element decoding layer forms a face element display record, the discriminator locates the ledger line corresponding to the same display period, the same spatial anchor point, and the same virtual face element, and establishes a temporary discrimination bit in the ledger line; The discriminator locates and displays the ledger line in the residual ledger according to the display period number, spatial anchor point number, and virtual element number in the element display record. Temporary discrimination bits temporarily store the anchor point phase verification value, visibility level verification value, illumination band verification value, and inter-frame phase verification value in the ledger line. Temporary discrimination bits are only used during the discrimination process of the current display period.

[0072] When determining the anchor point, the current period phase marker in the surface cell display record is used to replace the anchor point phase verification value in the temporary determination position. When the anchor point phase verification value falls into the compensated threshold range, the anchor point phase residual is rewritten as a zero residual marker. When it does not fall into the compensated threshold range, the original anchor point phase residual is retained. The anchor point discriminator reads the current period phase marker and registers it in the anchor point phase verification value. The compensated threshold range is the range of phases that can be retained after the anchor point phase residual has been processed by the phase compensation term. When the anchor point phase verification value falls within the compensated threshold range, the discriminator rewrites the anchor point phase residual in the corresponding ledger line as a zero residual marker. When the anchor point phase verification value does not fall within the compensated threshold range, the discriminator retains the original anchor point phase residual in the corresponding ledger line.

[0073] During visibility determination, the visibility level to be verified value in the temporary determination bit is replaced with the surface display level value. When the visibility level to be verified value meets the occlusion level, the visibility residual is rewritten as a zero residual mark. When the occlusion level is not met, the original visibility residual is retained. The visibility discrimination function reads the display layer value of the virtual facet and registers it as a visible layer to be verified value. The occlusion layer represents the target layer relationship between the real scene and the virtual facet, corresponding to the visibility residual. When the visible layer to be verified value satisfies the occlusion layer, the discriminator rewrites the visibility residual as a zero residual. When the visible layer to be verified value does not satisfy the occlusion layer, the discriminator retains the original visibility residual.

[0074] During illumination discrimination, the illumination frequency band to be verified value in the temporary discrimination position is replaced with the main display value of the surface element and the boundary state. When the illumination frequency band to be verified value meets both the brightness threshold and the boundary threshold, the illumination frequency band residual is rewritten as a zero residual mark. When both the brightness threshold and the boundary threshold are not met, the original illumination frequency band residual is retained. The illumination discrimination function reads the main display value of the virtual element and the boundary state of the current virtual element, and registers both in the illumination frequency band to be verified. The brightness threshold is used to determine whether the brightness difference corresponding to the main display value of the virtual element is closed. The boundary threshold is used to determine whether the boundary difference corresponding to the boundary state is closed. When the illumination frequency band to be verified simultaneously meets both the brightness threshold and the boundary threshold, the discriminator rewrites the illumination frequency band residual as a zero residual. When the illumination frequency band to be verified does not simultaneously meet both the brightness threshold and the boundary threshold, the discriminator retains the original illumination frequency band residual.

[0075] During inter-frame discrimination, the inter-frame phase to be verified value in the temporary discrimination bit is replaced with the current period phase mark. When the inter-frame phase to be verified value falls into a continuous interval, the inter-frame phase residual is rewritten as a zero residual mark. When it does not fall into a continuous interval, the original inter-frame phase residual is retained. The inter-frame discriminator reads the current period phase marker and registers it in the inter-frame phase verification value. The continuous interval is the phase range allowed to be retained after the previous period phase marker has been processed by the phase wrap-around node. When the inter-frame phase verification value falls within the continuous interval, the discriminator rewrites the inter-frame phase residual as a zero residual marker. When the inter-frame phase verification value does not fall within the continuous interval, the discriminator retains the original inter-frame phase residual.

[0076] The residual backfeed node writes the residual items that have not been rewritten to zero residual markers into the ledger row under the corresponding index in the next display period, and retains the cell display records where all residual items are marked as zero residual markers.

[0077] After the discriminator completes the discrimination of the four types of residuals, residual items in the ledger line that have not yet been rewritten to the zero residual mark are defined as unclosed residual items. The residual backfeed node registers the unclosed residual items to the ledger line under the corresponding index of the next display period according to the display period, spatial anchor point, and virtual cell index. Cell display records where all four types of residual items have been rewritten to the zero residual mark are retained and entered into the splicing queue.

[0078] In this embodiment, the stitching of the residual recharge node and the MR virtual image includes: After the discriminator rewrites the residual items in the ledger line, the residual feedback node checks the anchor point phase residual, visibility residual, illumination band residual and inter-frame phase residual line by line. The residual feedback node reads the anchor point phase residual, visibility residual, illumination band residual and inter-frame phase residual line by line according to the ledger line index, and checks whether each residual item has been rewritten to the zero residual mark.

[0079] If there are ledger lines that have not been rewritten to zero residual markers, the residual feedback node will retain the unrewritten residual items under the corresponding index in the next display cycle, and retain the corresponding attitude compensation markers; Ledger lines that are not rewritten to zero residual markers are marked as unclosed ledger lines. The residual backfeed node reads the display period, spatial anchor point, and virtual polygon index of the unclosed ledger line, and establishes the ledger line under the corresponding index in the next display period. The unrewritten residual items and corresponding attitude compensation markers in the unclosed ledger line are registered to the ledger line under the corresponding index in the next display period.

[0080] For ledger rows where all residual items are marked as zero residuals, the corresponding face value display record is entered into the splicing queue. Ledger lines with all residuals marked as zero are marked as closed ledger lines. The corresponding face value display records for closed ledger lines are sent to the concatenation queue. Face value display records for unclosed ledger lines are not entered into the concatenation queue for this display period.

[0081] The splicing queue reads the boundary state and current period phase marker of the adjacent face display record according to the adjacency relationship of the virtual face; The concatenation queue reads the adjacency relationships of virtual face elements and determines the display records of adjacent face elements based on these relationships. Each set of adjacent face element display records reads its own boundary state and current period phase marker.

[0082] When the boundary states of adjacent surface cell display records are consistent and the phase markers of the current period are continuous, the adjacent surface cell display records are spliced ​​together; when the boundary states of adjacent surface cell display records are inconsistent or the phase markers of the current period are discontinuous, the adjacent surface cell display records do not participate in the splicing of this period. Consistent boundary states indicate that adjacent surface element display records share the same boundary attribution, boundary level, and boundary transition state on adjacent boundaries. Continuous phase markers in the current cycle indicate that the phase difference between adjacent surface element display records on adjacent boundaries falls within the continuous splicing interval. When boundary states are consistent and current cycle phase markers are continuous, adjacent surface element display records are spliced ​​along adjacent boundaries. When boundary states are inconsistent or current cycle phase markers are discontinuous, adjacent surface element display records are delayed until the next display cycle for processing.

[0083] The completed spliced ​​element display record is output as an MR virtual image according to the spatial anchor points.

[0084] The completed pixel display records are grouped according to their spatial anchor point numbers. Pixel display records under the same spatial anchor point form a local virtual image based on the adjacency relationship of the virtual pixels. Local virtual images under different spatial anchor points are combined according to the positional relationship of the spatial anchor points and output as an MR virtual image.

[0085] Example 1: To verify the feasibility of this invention in practice, it was applied to an industrial equipment MR-assisted display scenario. Users view virtual control panels, indicator arrows, and component labels superimposed on the surface of the real equipment using an MR device. The real equipment surface has edge occlusion, reflective areas, and localized shadows. As the viewing posture changes during continuous display, the virtual image is prone to anchor point drift, misalignment of occlusion boundaries, unnatural lighting blending, and continuous frame flickering. These problems cause unstable adhesion between the virtual control panel and the real equipment surface, broken or accidentally occluded indicator arrows at the edges of the equipment, and jittering of component labels when the head is slightly rotated.

[0086] In this implementation scenario, the virtual image to be displayed is divided into multiple virtual facets. Each virtual facet is bound to a spatial anchor point, display level, boundary state, and current cycle phase marker. The method establishes a display residual ledger using the display cycle, spatial anchor point, and virtual facets as indices. The ledger rows record anchor point phase residuals, visibility residuals, illumination band residuals, inter-frame phase residuals, and pose compensation markers, which are used to drive subsequent propagation within the generator. The display residual ledger is updated once per display cycle; unclosed residual items are retained for the next display cycle, while closed residual items are rewritten with a zero residual marker.

[0087] The generative adversarial network (GAN) uses the SIREN generator as its backbone. In the sinusoidal implicit layer, phase modulation nodes and frequency modulation nodes correct the phase and frequency terms based on the anchor point phase residual, completing the anchor point phase-locked propagation of the virtual surface element. The visibility subtraction node forms an occlusion subtraction gate based on the visibility residual, allowing the occlusion relationship to propagate between adjacent sinusoidal implicit layers. The frequency division subtraction node splits the illumination frequency band residual into low-frequency brightness residual and high-frequency boundary residual, and subtracts them in the low-frequency layer group and high-frequency layer group respectively, obtaining the illumination correction implicit state. After the above processing, the virtual surface element completes the joint correction of anchor point, occlusion, and illumination during the generation stage.

[0088] The phase wrap-around node reads the inter-frame phase residual, attitude compensation marker, and previous cycle phase marker, and replaces the cross-cycle phase to obtain a continuous phase implicit state. The surface decoding layer forms a surface display record based on the continuous phase implicit state. The discriminator distinguishes between the anchor point phase residual, visibility residual, illumination band residual, and inter-frame phase residual. Residual items that meet the discrimination conditions are rewritten as zero residual markers, and unclosed residual items are registered by the residual feedback node to the ledger line under the corresponding index in the next display cycle. The surface display record with all residual items closed enters the stitching queue and is stitched together according to the virtual surface adjacency relationship to output the MR virtual image.

[0089] To verify the display effect, the same batch of real-world scene images, depth data, spatial anchor points, and virtual polygon display data were selected. The traditional MR rendering post-processing method, the ordinary generative adversarial network display method, the ordinary SIREN display method, and the method of this invention were compared. All four methods completed the same number of display cycles, and indicators such as anchor point stability, occlusion consistency, lighting fusion effect, inter-frame continuity, and processing time were statistically analyzed. The comparison results are shown in the table below: Table 1: Comparison of MR Display Effects

[0090] The table shows that the method of this invention significantly improves the stability of spatial anchor points. The average spatial anchor point offset is 1.4 pixels, compared to 5.8 pixels for the traditional MR rendering post-processing method, 4.6 pixels for the ordinary generative adversarial network display method, and 3.7 pixels for the ordinary SIREN display method. The average anchor point phase residual also decreased from 0.172, 0.139, and 0.106 in the comparison methods to 0.038. These data indicate that after the anchor point phase residual enters the phase modulation node and frequency modulation node, the virtual polygons are constrained by the spatial anchor points during the generation stage, and the image overlay position no longer mainly depends on post-processing correction.

[0091] Regarding occlusion and illumination, the occlusion boundary misalignment of the method in this invention is 1.1 pixels, lower than the 4.9 pixels, 4.1 pixels, and 3.2 pixels of the three comparative methods; the average visibility residual is 0.049, also lower than 0.214, 0.181, and 0.142. The average illumination band residual is 0.052, the low-frequency brightness deviation is 0.041, and the high-frequency boundary deviation is 0.057, all superior to the comparative methods. This demonstrates that the visibility subtraction node differentiates between real occlusion, virtual display, and boundary transition, while the frequency subtraction node corrects the overall brightness and boundary texture separately, resulting in a more stable fusion of the virtual image and the real scene.

[0092] In terms of continuous display, the inter-frame phase jump rate of the method of this invention is 1.8%, and the number of consecutive frame flickers is 2.1 times / 100 frames, which is lower than the 5.9% and 7.1 times / 100 frames of the ordinary SIREN display method, respectively. The surface residual closure rate reaches 93.2%, and the surface splicing pass rate reaches 95.1%, indicating that the discriminator residual closure and residual backfeed can reduce the recurrence of unclosed residuals in the next display cycle. The average processing time is 18.6 milliseconds / frame, which is lower than the 22.4 milliseconds / frame of the ordinary generative adversarial network display method and close to the 18.9 milliseconds / frame of the ordinary SIREN display method, indicating that the present invention can still meet the real-time requirements of continuous MR display while improving display stability.

[0093] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for displaying MR virtual images based on generative adversarial networks, characterized in that, Includes the following steps: Establish a display residual ledger indexed by display cycle, spatial anchor point, and virtual surface element, and register anchor point phase residual, visibility residual, illumination band residual, inter-frame phase residual, and attitude compensation mark. Set up the phase modulation node, frequency modulation node, visible subtraction node, frequency division subtraction node, phase wrap-around node, and residual backfeed node in the SIREN generator; The phase modulation node forms a phase compensation term using the phase residual of the same index anchor point and corrects the sinusoidal implicit layer phase term. The frequency modulation node forms a frequency scaling term using the phase residual of the same anchor point and corrects the sinusoidal implicit layer frequency term, thus obtaining the phase-locked implicit state. It can be seen that the subtraction node maps the visibility residual to the occlusion subtraction gate. The occlusion subtraction gate is multiplied into the inter-layer output and then passed to the next sinusoidal implicit layer to obtain the visibility implicit state. The frequency subtraction node decomposes the illumination frequency band residual into low-frequency brightness residual and high-frequency boundary residual. The low-frequency layer group subtracts the low-frequency brightness residual, and the high-frequency layer group subtracts the high-frequency boundary residual to obtain the implicit illumination correction state. The phase wrap-around node marks the cross-cycle phase with the inter-frame phase residual, and replaces the cross-cycle phase with the phase mark of the previous cycle and the attitude compensation mark to obtain the continuous phase implicit state; The face element decoding layer forms face element display records from continuous phase implicit states. The discriminator calculates the update residuals of the face element display records. The residual backfeed node replaces the next cycle's same index residual item with the update residual. Face element display records with all types of update residuals below the corresponding thresholds are retained and stitched together into an MR virtual screen.

2. The method for displaying MR virtual images based on generative adversarial networks according to claim 1, characterized in that, The establishment of the residual ledger includes: The ledger rows are determined using the same display cycle, the same spatial anchor point, and the same virtual element; In the ledger line, the phase deviation of the virtual surface element relative to the spatial anchor point is recorded as the anchor point phase residual, the occlusion level difference between the virtual surface element and the real scene is recorded as the visibility residual, the brightness difference and boundary difference between the virtual surface element and the real scene are merged and recorded as the illumination frequency band residual, the period span difference between the current period phase mark and the previous period phase mark is recorded as the inter-frame phase residual, and the attitude compensation amount corresponding to the change in observation attitude is recorded as the attitude compensation mark.

3. The method for displaying MR virtual images based on generative adversarial networks according to claim 2, characterized in that, The node configuration of the generative adversarial network includes: The SIREN generator retains the sinusoidal implicit layers arranged by layer number; A phase modulation node is connected in series before the phase term of each sinusoidal implicit layer, and a frequency modulation node is connected in series before the frequency term of each sinusoidal implicit layer. Insert visible deduction nodes between adjacent sinusoidal implicit layers, and use the output of the previous sinusoidal implicit layer as the deduction amount of the visible deduction node; The low-frequency layer group and the high-frequency layer group are divided according to the layer frequency of the sinusoidal implicit layer, and frequency division and subtraction nodes are connected in series in the low-frequency layer group and the high-frequency layer group respectively. A phase wrap-around node is connected in series before the surface decoding layer, and a residual feed-back node is connected in series between the discriminator output and the display residual ledger.

4. The method for displaying MR virtual images based on generative adversarial networks according to claim 3, characterized in that, The operations performed by the phase modulation node and frequency modulation node to process the anchor point phase residual include: Read the anchor point phase residuals corresponding to the same display period, the same spatial anchor point, and the same virtual surface element in the display residual ledger; The compensation direction is determined based on the sign of the anchor point phase residual. When the anchor point phase residual deviates in the positive direction, reverse compensation is used, and when the anchor point phase residual deviates in the negative direction, positive compensation is used. The compensation level is selected based on the amplitude range of the anchor point phase residual, and the phase compensation term is formed by the compensation direction and the compensation level. Before sinusoidal propagation occurs in the sinusoidal implicit layer, the phase compensation term is superimposed on the phase term to obtain the phase-locked phase term. The frequency participation interval is selected based on the amplitude range of the phase residual at the same anchor point, and the sinusoidal implicit layer participating in frequency modulation is selected from the frequency participation interval. The selected sinusoidal implicit layer uses a frequency scaling term corresponding to the compensation level to form a phase-locked frequency term, while the unselected sinusoidal implicit layer retains the original frequency term. The sinusoidal implicit layer propagates layer by layer based on the phase-locked phase term and the phase-locked frequency term or the original frequency term to obtain the phase-locked implicit state.

5. The method for displaying MR virtual images based on generative adversarial networks according to claim 4, characterized in that, The operation of the visible deduction node in processing visibility residuals includes: Read the visibility residuals corresponding to the same display period, the same spatial anchor point, and the same virtual surface in the display residual ledger, and obtain the previous sinusoidal implicit layer output as the deducted amount; Based on the differences in occlusion levels in the visibility residual, virtual facets are divided into real occlusion state, virtual display state, and boundary transition state; The actual occlusion state matches the first reduction level, the virtual display state matches the hold level, and the boundary transition state matches the second reduction level. The first deduction gear, the hold gear, and the second deduction gear respectively generate the first obstruction deduction gate, the hold gate, and the second obstruction deduction gate; The first occlusion gate reduces the transmission amplitude of the virtual element body in the subtracted amount, the holding gate retains the transmission amplitude of the virtual element body in the subtracted amount, and the second occlusion gate reduces the transmission amplitude of the virtual element boundary in the subtracted amount. The amount deducted after being processed by the first occlusion deduction gate, the holding gate, and the second occlusion deduction gate is used as the interlayer visible transmission amount and enters the next sinusoidal implicit layer to obtain the visible implicit state.

6. The method for displaying MR virtual images based on generative adversarial networks according to claim 5, characterized in that, The operation of the frequency division subtraction node in processing the residual of the illumination frequency band includes: Read the illumination frequency band residuals corresponding to the same display period, the same spatial anchor point, and the same virtual surface in the display residual ledger, and extract the brightness differences and boundary differences in the illumination frequency band residuals; Using the layer frequency boundary of the sinusoidal implicit layer as the boundary, the low-frequency layer group receives the low-frequency components in the visible implicit state, and the high-frequency layer group receives the high-frequency components in the visible implicit state. Brightness differences are entered into the low-frequency subtraction table to obtain the low-frequency subtraction gate. The low-frequency subtraction gate is connected in series before the amplitude term of the low-frequency layer group to subtract the low-frequency components layer by layer, forming the low-frequency subtraction feature. Boundary differences are entered into the high-frequency subtraction table to obtain the high-frequency subtraction gate. The high-frequency subtraction gate is connected in series before the implicit component corresponding to the virtual surface element boundary in the high-frequency layer group, and the high-frequency component is subtracted layer by layer to form the high-frequency subtraction feature. Low-frequency subtraction features and high-frequency subtraction features are merged under the same virtual surface index to obtain the illumination-corrected implicit state.

7. The method for displaying MR virtual images based on generative adversarial networks according to claim 6, characterized in that, The operation of the phase wrap-around node to process inter-frame phase residuals includes: Using the display cycle, spatial anchor point, and virtual surface element as indexes, the inter-frame phase residual, attitude compensation mark, and previous cycle phase mark are retrieved from the display residual ledger. Extract the current period phase marker from the implicit state of illumination correction; retain the current period phase marker when the inter-frame phase residual falls into an uncrossed interval; When the inter-frame phase residual falls into the positive crossing interval, the result of the superposition of the phase mark of the previous cycle and the attitude compensation mark replaces the phase mark of the current cycle. When the inter-frame phase residual falls into the negative crossing interval, the result of subtracting the attitude compensation mark from the previous cycle phase mark replaces the current cycle phase mark; After the replacement, the current period phase marker is backfilled into the position of the corresponding virtual element in the illumination correction implicit state to obtain the continuous phase implicit state.

8. The method for displaying MR virtual images based on generative adversarial networks according to claim 7, characterized in that, The operation of the face element decoding layer in forming face element display records from continuous phase implicit states includes: Positioning continuous phase implicit states using the same display cycle, the same spatial anchor point, and the same virtual surface element; Based on the coverage area of ​​the virtual surface element, the implicit state of the continuous phase is classified into the main state of the surface element and the boundary state of the surface element. The main state of the element is converted into the main display value and the display layer value of the element by the element decoding layer; The boundary state of a virtual surface element is paired with the boundary state of an adjacent virtual surface element, and the phase continuity and hierarchical continuity of the paired elements are compared. Boundary states that satisfy both phase continuity and hierarchical continuity conditions are assigned to the current virtual surface element; boundary states that do not satisfy phase continuity or hierarchical continuity conditions are transferred to the boundary candidate record of the adjacent virtual surface element. The face element decoding layer combines the face element main display value, face element display level value, boundary state belonging to the current virtual face element, and current period phase mark into a face element display record.

9. A method for displaying MR virtual images based on generative adversarial networks according to claim 8, characterized in that, The discriminator calculates the update residual of the surface element display record, including: After the face element decoding layer forms a face element display record, the discriminator locates the ledger line corresponding to the same display period, the same spatial anchor point, and the same virtual face element, and establishes a temporary discrimination bit in the ledger line; When determining the anchor point, the current period phase marker in the surface cell display record is used to replace the anchor point phase verification value in the temporary determination position. When the anchor point phase verification value falls into the compensated threshold range, the anchor point phase residual is rewritten as a zero residual marker. When it does not fall into the compensated threshold range, the original anchor point phase residual is retained. During visibility determination, the visibility level to be verified value in the temporary determination bit is replaced with the surface display level value. When the visibility level to be verified value meets the occlusion level, the visibility residual is rewritten as a zero residual mark. When the occlusion level is not met, the original visibility residual is retained. During illumination discrimination, the illumination frequency band to be verified value in the temporary discrimination position is replaced with the main display value of the surface element and the boundary state. When the illumination frequency band to be verified value meets both the brightness threshold and the boundary threshold, the illumination frequency band residual is rewritten as a zero residual mark. When both the brightness threshold and the boundary threshold are not met, the original illumination frequency band residual is retained. During inter-frame discrimination, the inter-frame phase to be verified value in the temporary discrimination bit is replaced with the current period phase mark. When the inter-frame phase to be verified value falls into a continuous interval, the inter-frame phase residual is rewritten as a zero residual mark. When it does not fall into a continuous interval, the original inter-frame phase residual is retained. The residual backfeed node writes the residual items that have not been rewritten to zero residual markers into the ledger row under the corresponding index in the next display period, and retains the cell display records where all residual items are marked as zero residual markers.

10. A method for displaying MR virtual images based on generative adversarial networks according to claim 9, characterized in that, The stitching of the residual recharge node and the MR virtual image includes: After the discriminator rewrites the residual items in the ledger line, the residual feedback node checks the anchor point phase residual, visibility residual, illumination band residual and inter-frame phase residual line by line. If there are ledger lines that have not been rewritten to zero residual markers, the residual feedback node will retain the unrewritten residual items under the corresponding index in the next display cycle, and retain the corresponding attitude compensation markers; For ledger rows where all residual items are marked as zero residuals, the corresponding face value display record is entered into the splicing queue. The splicing queue reads the boundary state and current period phase marker of the adjacent face display record according to the adjacency relationship of the virtual face; When the boundary states of adjacent surface cell display records are consistent and the phase markers of the current period are continuous, the adjacent surface cell display records are spliced ​​together; when the boundary states of adjacent surface cell display records are inconsistent or the phase markers of the current period are discontinuous, the adjacent surface cell display records do not participate in the splicing of this period. The completed spliced ​​element display record is output as an MR virtual image according to the spatial anchor points.

Images