Holographic image news scene real-time generation method and system based on deep learning

Abstract

The application provides a holographic image news scene real-time generation method and system based on deep learning, relates to the fields of news dissemination and holographic display technology, and comprises the following steps: performing cross-modal semantic analysis on multi-modal news source data, extracting structured semantic scene features, constructing a three-dimensional space representation by using a deep neural network, calculating an angle spectrum distribution, and then performing reverse propagation to a holographic plane by light ray tracing, and finally obtaining holographic pattern coding data and driving a holographic display device. The method realizes automatic and efficient conversion from multi-modal news sources to holographic images, and improves news performance and immersion.

Description

Technical Field

[0001] This invention relates to the fields of news dissemination and holographic display technology, and in particular to a method and system for real-time generation of holographic news scenes based on deep learning. Background Technology

[0002] With the rapid development of information technology, the way news media is presented is undergoing profound changes. Holographic imaging, as an advanced display technology that can realistically present three-dimensional scenes, provides a completely new visual experience for news reporting. By recording and reconstructing the amplitude and phase information of light waves, holographic technology can present three-dimensional scenes with depth and three-dimensionality, allowing viewers to observe news events from different angles and enhancing the immersion and authenticity of news content.

[0003] Traditional news scene reconstruction mainly relies on manual modeling and rendering, requiring professional 3D designers to manually construct 3D models and perform texture mapping based on 2D images and text descriptions. This method is not only time-consuming and labor-intensive, but also difficult to guarantee real-time performance. With the development of deep learning technology, neural network-based 3D reconstruction and hologram generation methods have gradually attracted attention. Through semantic understanding and geometric reconstruction of multimodal data, they have made it possible to automatically generate holographic news scenes. Summary of the Invention

[0004] This invention provides a method and system for real-time generation of holographic news scenes based on deep learning, which can solve the problems in the prior art.

[0005] A first aspect of this invention provides a method for real-time generation of holographic news scenes based on deep learning, comprising:

[0006] Acquire multimodal news source data containing text content and two-dimensional images;

[0007] Cross-modal semantic parsing is performed on the multimodal news source data to extract multiple objects and label them with object categories, spatiotemporal location relationships and importance markers, generating structured semantic scene features;

[0008] Based on the structured semantic scene features and the two-dimensional image, a three-dimensional geometric mesh is constructed using a deep neural network, and combined with visual textures to form a three-dimensional spatial representation.

[0009] Based on the three-dimensional spatial representation, the light radiation characteristics of different observation directions are calculated to obtain the angular spectral distribution. Then, according to the preset viewing angle range of the holographic display, the spectral components of the corresponding directions are filtered to obtain the viewing angle-limited angular spectral data.

[0010] The angle spectral data with limited viewing angle is back-propagated to the holographic plane through ray tracing. The phase accumulation and amplitude attenuation of each ray are calculated. Rays from different directions from the same spatial point are coherently superimposed at the same position on the holographic plane to obtain the complex amplitude distribution of the holographic plane.

[0011] Using the holographic plane complex amplitude distribution as the target field, a transfer function is established based on the Fresnel number. Diffraction propagation calculations are performed on the randomly initialized complex coding matrix, and the total loss function is constructed and iteratively optimized to obtain holographic pattern coding data.

[0012] The holographic pattern encoding data drives the holographic display device to present holographic news scenes.

[0013] Cross-modal semantic parsing is performed on the multimodal news source data to extract multiple objects and label them with object categories, spatiotemporal relationships, and importance markers, generating structured semantic scene features, including:

[0014] Entity recognition and semantic relationship extraction are performed on the text content in the multimodal news source data to generate a text semantic graph; target detection is performed on the two-dimensional image data, and monocular depth estimation is performed simultaneously to obtain the depth information of visual objects, resulting in a set of visual objects with depth estimation values.

[0015] Semantic matching and binding relationships are established between entities in the text semantic graph and visual objects. Based on the depth estimation value of the visual objects, the bound objects are divided into the depth range of the corresponding holographic display depth layer to obtain a depth-layered object set.

[0016] Based on the semantic relationship types between entities in the text semantic graph, the objects in the deep hierarchical object set are classified into scene subject objects, spatial environment objects, and event-related objects to obtain the category identifier of each object; the distance difference between the depth estimate of each object and the depth position of the holographic display focal plane is calculated, and the distance difference is converted into an importance label according to the absolute value of the distance difference and a preset threshold function;

[0017] Extract the bounding box coordinates of each object in the 2D image, integrate the bounding box coordinates and depth interval labels to form a spatiotemporal positional relationship, and combine the category label, depth interval label and importance label of each object to form a structured semantic scene feature.

[0018] Based on the structured semantic scene features and the two-dimensional image, a three-dimensional geometric mesh is constructed using a deep neural network, and combined with visual textures to form a three-dimensional spatial representation, including:

[0019] Based on the structured semantic scene features, the geometric voxel distribution and surface confidence map are predicted synchronously through a deep neural network. The initial vertices are obtained by extracting the centroids of differentiable voxels. Adjacency relationships are constructed based on surface confidence screening. The initial vertices and topology are iteratively optimized using gradient feedback for authenticity evaluation to generate a three-dimensional geometric mesh including geometric vertices.

[0020] The texture features of the target region are extracted from the two-dimensional image. The texture features are spatially correlated with the geometric vertices to generate vertex color values. Semantic consistency constraints are introduced to optimize the semantic matching degree between the texture and the category, as well as the spatial smoothness of the color values ​​of adjacent vertices. Through iterative optimization, a visual texture that conforms to the semantics of the object and has a continuous transition is obtained.

[0021] A bidirectional mapping relationship is established between the three-dimensional geometric mesh and the visual texture, and the adjacency topology information and surface normal vectors of the geometric vertices are calculated and combined to form a three-dimensional spatial representation.

[0022] Based on the aforementioned three-dimensional spatial representation, the light radiation characteristics of different viewing directions are calculated to obtain the angular spectral distribution. Then, according to the preset viewing angle range of the holographic display, the spectral components of the corresponding directions are filtered to obtain viewing angle-limited angular spectral data, including:

[0023] Geometric vertices, corresponding surface normals, and vertex color values ​​are extracted from the three-dimensional spatial representation. A local coordinate system is constructed for the surface normals of the geometric vertices, and the upper hemisphere space is discretized and sampled with the geometric vertices as the center to generate a set of observation direction samples containing zenith angle parameters and azimuth angle parameters. The cosine value of the angle between each observation direction and the surface normal is calculated.

[0024] The vertex color value is decomposed into spectral intensity components of multiple wavelength channels. Based on the cosine of the included angle and the spectral intensity components, the light radiation intensity of the geometric vertex in each observation direction is calculated to obtain the directional spectral data of the geometric vertex.

[0025] Obtain the center direction and half-angle parameters of the field of view of the holographic display, transform the center direction of the field of view to the local coordinate system, calculate the deviation angle of each observation direction, retain the observation direction and its radiation intensity with a deviation angle not exceeding the half-angle of the field of view, and arrange them according to the zenith angle and azimuth angle parameters to form the filtered spectral data;

[0026] Perform directional spectral data generation and viewpoint filtering on all geometric vertices, establish the mapping relationship between geometric vertices, observation direction, and light radiation intensity, and organize the filtered spectral data into viewpoint-limited angular spectral data.

[0027] The vertex color value is decomposed into spectral intensity components of multiple wavelength channels. Based on the cosine of the included angle and the spectral intensity components, the radiant intensity of the geometric vertex in each viewing direction is calculated to obtain the directional spectral data of the geometric vertex, including:

[0028] The vertex color values ​​are projected onto the linear space of the spectral basis function to obtain a continuous representation of the color values ​​in the spectral domain. By discretizing the continuous representation in the wavelength range, the spectral intensity components of each channel are obtained, and the reconstruction correspondence between the projection coefficients and the spectral intensity components is established.

[0029] Based on the cosine of the angle between the observation direction and the surface normal vector, the degree of spatial deviation is determined. Based on the degree of spatial deviation, weights are assigned to adjacent observation directions. The weights are applied to the corresponding spectral intensity components and interpolation is performed to generate interpolated spectral components for each observation direction.

[0030] The interpolated spectral components are fused and corrected with the spectral intensity components in the reconstructed correspondence to obtain the light radiation intensity of the geometric vertex in each observation direction. A two-dimensional index structure is established according to the zenith angle parameter and the azimuth angle parameter to store the radiation intensity values ​​of multiple wavelength channels under the corresponding observation direction, thereby obtaining the directional spectral data of each geometric vertex.

[0031] The angle spectral data with limited viewing angle is backpropagated to the holographic plane via ray tracing. The phase accumulation and amplitude attenuation of each ray are calculated. Complex coherent superposition of rays from different directions originating from the same spatial point at the same position on the holographic plane is performed to obtain the complex amplitude distribution of the holographic plane, including:

[0032] Based on the angle spectral data with limited viewing angle, the observation direction and corresponding light radiation intensity of each geometric vertex are extracted. Taking the geometric vertex as the light source and the opposite direction of the observation direction as the light propagation direction, a set of reverse tracking rays pointing from the geometric vertex to the holographic plane is constructed.

[0033] Obtain the spatial position parameters and normal vector of the holographic plane, calculate the coordinates of the intersection point of each backward tracing ray with the holographic plane, and map them to the two-dimensional coordinate system of the holographic plane to obtain the position of the ray landing point. Calculate the spatial distance from the geometric vertex to the coordinates of the intersection point as the ray propagation path length.

[0034] Based on the light propagation path length and preset wavelength parameters, the propagation phase change is calculated and added to the initial phase to obtain the phase accumulation value. Based on the light propagation path length and spatial attenuation coefficient, the amplitude attenuation factor is calculated and multiplied by the light radiation intensity to obtain the amplitude value.

[0035] The phase accumulation value and the amplitude value are converted into complex light field components. A discrete pixel array is established on the holographic plane. The light field components are merged into the corresponding pixels based on the position of the light rays. The light field components in the same pixel are coherently superimposed to obtain the complex amplitude superposition value of the pixel position.

[0036] The complex amplitude superposition values ​​of all pixel positions on the holographic plane are organized into a two-dimensional complex matrix according to the spatial arrangement order of the pixel array, forming the complex amplitude distribution of the holographic plane.

[0037] Using the holographic plane complex amplitude distribution as the target field, a transfer function is established based on the Fresnel number. Diffraction propagation calculations are performed on the randomly initialized complex encoding matrix, and the total loss function is constructed and iteratively optimized to obtain holographic pattern encoded data, including:

[0038] Using the holographic plane complex amplitude distribution as the target field, the pixel size of the spatial light modulator is obtained, the complex coding matrix is ​​randomly initialized, the diffraction propagation distance and preset wavelength parameters are obtained, the Fresnel number is calculated, the sampling interval is determined and a frequency grid is established, and the transfer function of diffraction propagation is obtained.

[0039] The complex coding matrix is ​​subjected to Fourier transform and multiplied with the transfer function, and then an inverse transform is performed to obtain the reconstructed complex field. The reconstruction error in the spatial domain and the energy deviation in the frequency domain between the reconstructed complex field and the target field are calculated respectively, and the total loss function is obtained by weighted combination.

[0040] Calculate the gradient of the total loss function with respect to the coding parameters, update the complex coding matrix, perform frequency domain bandpass filtering on the updated complex coding matrix, attenuate the frequency components outside the cutoff frequency according to the frequency domain attenuation ratio, and use the inverse transformation as the input for the next iteration.

[0041] The iteration terminates when the total loss change of adjacent iterations is less than the loss convergence threshold. The phase angle and magnitude of the complex coding matrix of the current iteration are extracted, normalized and mapped to the modulator bit depth range and quantized to obtain the phase modulation parameters and amplitude modulation parameters. These parameters are then organized in pixel order, and a header identifier and checksum are added to obtain the holographic pattern coding data.

[0042] A second aspect of this invention provides a real-time generation system for holographic image news scenes based on deep learning, comprising:

[0043] The first unit is used to acquire multimodal news source data containing text content and two-dimensional images;

[0044] The second unit is used to perform cross-modal semantic parsing on the multimodal news source data, extract multiple objects and label object categories, spatiotemporal location relationships and importance markers, and generate structured semantic scene features;

[0045] The third unit is used to construct a three-dimensional geometric mesh through a deep neural network based on the structured semantic scene features and the two-dimensional image, and combine it with visual texture to form a three-dimensional spatial representation.

[0046] The fourth unit is used to calculate the light radiation characteristics of different viewing directions based on the three-dimensional spatial representation, obtain the angular spectral distribution, and filter the spectral components of the corresponding directions according to the preset viewing angle range of the holographic display to obtain viewing angle-limited angular spectral data.

[0047] The fifth unit is used to back-propagate the angle spectral data with limited viewing angle to the holographic plane through ray tracing, calculate the phase accumulation and amplitude attenuation of each ray, and perform complex coherent superposition of rays from different directions from the same spatial point at the same position on the holographic plane to obtain the complex amplitude distribution of the holographic plane.

[0048] The sixth unit is used to take the complex amplitude distribution of the holographic plane as the target field, establish the transfer function based on the Fresnel number, perform diffraction propagation calculation on the randomly initialized complex coding matrix, construct the total loss function for iterative optimization, and obtain the holographic pattern coding data.

[0049] The seventh unit is used to drive the holographic display device to present holographic news scenes based on the holographic pattern encoding data.

[0050] A third aspect of the embodiments of the present invention,

[0051] An electronic device is provided, comprising:

[0052] processor;

[0053] Memory used to store processor-executable instructions;

[0054] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0055] Fourth aspect of the present invention,

[0056] A computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0057] The beneficial effects of this application are as follows:

[0058] By using deep neural networks to construct a three-dimensional geometric mesh and combining it with visual textures to form a three-dimensional spatial representation, the traditional limitation of holographic generation on physical scene dependence is broken. It can directly construct virtual three-dimensional scenes from news source data, which greatly improves the automation and efficiency of holographic content generation.

[0059] By calculating the angular spectral distribution based on three-dimensional spatial characterization and filtering it within a preset viewing angle range, angular spectral data with limited viewing angles are obtained. This effectively solves the problem of limited viewing angle range in holographic displays, optimizes the utilization of computing resources, and improves the efficiency of holographic reconstruction.

[0060] By back-propagating ray tracing to the holographic plane and performing complex coherent superposition, combined with Fresnel number-based transfer function optimization of holographic pattern encoding, high-quality holographic pattern generation was achieved, significantly improving the visual effect and realism of holographic images. Attached Figure Description

[0061] Figure 1 This is a flowchart illustrating the real-time generation method for holographic image news scenes based on deep learning, according to an embodiment of the present invention.

[0062] Figure 2 This is a schematic flowchart of the angle spectral data generation method with limited viewing angle according to an embodiment of the present invention. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0065] Figure 1 This is a flowchart illustrating the real-time generation method for holographic news scenes based on deep learning, as described in an embodiment of the present invention. Figure 1 As shown, the method includes:

[0066] Acquire multimodal news source data containing text content and two-dimensional images;

[0067] Cross-modal semantic parsing is performed on the multimodal news source data to extract multiple objects and label them with object categories, spatiotemporal location relationships and importance markers, generating structured semantic scene features;

[0068] Based on the structured semantic scene features and the two-dimensional image, a three-dimensional geometric mesh is constructed using a deep neural network, and combined with visual textures to form a three-dimensional spatial representation.

[0069] Based on the three-dimensional spatial representation, the light radiation characteristics of different observation directions are calculated to obtain the angular spectral distribution. Then, according to the preset viewing angle range of the holographic display, the spectral components of the corresponding directions are filtered to obtain the viewing angle-limited angular spectral data.

[0070] The angle spectral data with limited viewing angle is back-propagated to the holographic plane through ray tracing. The phase accumulation and amplitude attenuation of each ray are calculated. Rays from different directions from the same spatial point are coherently superimposed at the same position on the holographic plane to obtain the complex amplitude distribution of the holographic plane.

[0071] Using the holographic plane complex amplitude distribution as the target field, a transfer function is established based on the Fresnel number. Diffraction propagation calculations are performed on the randomly initialized complex coding matrix, and the total loss function is constructed and iteratively optimized to obtain holographic pattern coding data.

[0072] The holographic pattern encoding data drives the holographic display device to present holographic news scenes.

[0073] In one optional implementation, cross-modal semantic parsing is performed on the multimodal news source data to extract multiple objects and label them with object categories, spatiotemporal relationships, and importance markers, generating structured semantic scene features, including:

[0074] Entity recognition and semantic relationship extraction are performed on the text content in the multimodal news source data to generate a text semantic graph; target detection is performed on the two-dimensional image data, and monocular depth estimation is performed simultaneously to obtain the depth information of visual objects, resulting in a set of visual objects with depth estimation values.

[0075] Semantic matching and binding relationships are established between entities in the text semantic graph and visual objects. Based on the depth estimation value of the visual objects, the bound objects are divided into the depth range of the corresponding holographic display depth layer to obtain a depth-layered object set.

[0076] Based on the semantic relationship types between entities in the text semantic graph, the objects in the deep hierarchical object set are classified into scene subject objects, spatial environment objects, and event-related objects to obtain the category identifier of each object; the distance difference between the depth estimate of each object and the depth position of the holographic display focal plane is calculated, and the distance difference is converted into an importance label according to the absolute value of the distance difference and a preset threshold function;

[0077] Extract the bounding box coordinates of each object in the 2D image, integrate the bounding box coordinates and depth interval labels to form a spatiotemporal positional relationship, and combine the category label, depth interval label and importance label of each object to form a structured semantic scene feature.

[0078] In this specific embodiment, during the processing of multimodal news source data, it is necessary to acquire multimodal news source data containing text and two-dimensional images. Multimodal news source data can be news reports, social media content, or other information sources containing text and images.

[0079] This study performs entity recognition and semantic relation extraction on text content in multimodal news source data to generate a text semantic graph. Specifically, named entity recognition technology is used to identify entities such as people, places, organizations, and times in the text and label their categories. Dependency parsing is used to extract semantic relations between entities, such as subject-verb-object relations, spatiotemporal relations, and causal relations. A text semantic graph is constructed based on entities and relations, where nodes represent entities and edges represent semantic relations between entities. For example, in a news report about a traffic accident, entities such as "vehicles," "pedestrians," and "roads" can be identified, along with relations such as "collision" and "located" between them.

[0080] Simultaneously, target detection is performed on the 2D image data to identify visual objects in the image. Deep learning target detection algorithms such as Faster R-CNN or YOLO are used to obtain the location, category, and confidence of objects in the image. At the same time as target detection, monocular depth estimation is performed. By analyzing visual cues such as texture, perspective, and occlusion in the image, the depth value of each pixel or visual object relative to the camera is estimated. This step outputs a set of visual objects with depth estimates. Each object includes its category label, bounding box coordinates, and corresponding depth estimate.

[0081] Semantic matching and binding relationships are established between entities and visual objects in the text semantic graph. The degree of matching is calculated based on the semantic similarity between entity name and visual object category. A word vector model such as Word2Vec or BERT is used to calculate the semantic similarity between text entities and visual object labels. A similarity threshold is set. When the similarity exceeds the threshold, a binding relationship between text entities and visual objects is established. For one-to-many or many-to-one cases, the pair with the highest similarity is selected.

[0082] Based on the depth estimate of the visual object, the bound object is divided into the depth range of the corresponding holographic display depth layer. Multiple depth layers of the holographic display system are predefined, such as the foreground layer (0-2 meters), the midground layer (2-5 meters), and the background layer (above 5 meters). Based on the depth estimate of each visual object, it is assigned to the corresponding depth range to obtain a set of objects with different depth layers. For example, a pedestrian object with a depth estimate of 1.5 meters is assigned to the foreground layer, while a building with a depth of 6 meters is assigned to the background layer.

[0083] Based on the semantic relationship types between entities in the text semantic graph, objects in the deep hierarchical object set are classified. The connectivity and semantic relationship types of each entity in the semantic relationship graph are analyzed, and the objects are divided into three categories: scene subject objects (core objects of news events, such as the protagonist of the event), spatial environment objects (objects that provide scene background, such as buildings and roads), and event-related objects (secondary objects that interact with the subject objects). For each object, it is classified according to its centrality, connectivity, and relationship type in the semantic graph, and corresponding category labels are added.

[0084] The distance difference between the estimated depth of each object and the depth position of the holographic display focal plane is calculated. The holographic display focal plane is usually set at the most comfortable viewing distance for the audience. For each object, the difference between its estimated depth and the depth of the focal plane is calculated. According to a preset threshold function, the distance difference is converted into an importance label. The threshold function can be designed as a non-linear function, so that the closer the object is to the focal plane, the higher its importance. For example, the importance can be divided into three levels: high, medium, and low. Objects with an absolute distance difference of less than 1 meter are labeled as "high importance", those between 1 and 3 meters are "medium importance", and those more than 3 meters are "low importance".

[0085] Extract the bounding box coordinates of each object in the 2D image, and integrate the bounding box coordinates and depth interval identifiers to form a spatiotemporal position relationship. The bounding box coordinates are usually represented as pixel coordinate pairs of the upper left and lower right corners (x1, y1, x2, y2). Combined with the object's depth interval identifier, a 3D spatiotemporal position description is formed. For example, a pedestrian object has bounding box coordinates (100, 200, 150, 300) and foreground layer depth interval identifiers.

[0086] By integrating the category identifiers (scene main object, spatial environment object, or event-related object), bounding box coordinates, depth interval identifiers, and importance markers of each object, a structured semantic scene feature is formed. This feature describes the semantic type, spatial position relationship, and importance of each object in the scene, providing a foundation for subsequent holographic scene reconstruction and rendering.

[0087] Structured semantic scene features can be represented as an array of objects, each containing the following attributes: object ID, category identifier, bounding box coordinates, depth interval identifier, importance marker, and a list of relationships with other objects. This structured representation supports flexible querying and processing, facilitating subsequent scene understanding and holographic rendering optimization.

[0088] In one optional implementation, based on the structured semantic scene features and the two-dimensional image, a three-dimensional geometric mesh is constructed using a deep neural network, and combined with visual textures to form a three-dimensional spatial representation, including:

[0089] Based on the structured semantic scene features, the geometric voxel distribution and surface confidence map are predicted synchronously through a deep neural network. The initial vertices are obtained by extracting the centroids of differentiable voxels. Adjacency relationships are constructed based on surface confidence screening. The initial vertices and topology are iteratively optimized using gradient feedback for authenticity evaluation to generate a three-dimensional geometric mesh including geometric vertices.

[0090] The texture features of the target region are extracted from the two-dimensional image. The texture features are spatially correlated with the geometric vertices to generate vertex color values. Semantic consistency constraints are introduced to optimize the semantic matching degree between the texture and the category, as well as the spatial smoothness of the color values ​​of adjacent vertices. Through iterative optimization, a visual texture that conforms to the semantics of the object and has a continuous transition is obtained.

[0091] A bidirectional mapping relationship is established between the three-dimensional geometric mesh and the visual texture, and the adjacency topology information and surface normal vectors of the geometric vertices are calculated and combined to form a three-dimensional spatial representation.

[0092] In this specific embodiment, when generating a 3D geometric mesh based on structured semantic scene features, a two-stream deep neural network architecture is used to process the scene feature input. This network consists of a shared encoder and two dedicated decoders. The shared encoder uses residual connections and attention mechanisms to extract features, while the two decoders are responsible for predicting the geometric voxel distribution and the surface confidence map, respectively. The geometric voxel distribution is represented by a 3D volume and the space is organized in an octree structure. Each voxel contains an occupancy probability value. The surface confidence map reflects the probability that a voxel is on the surface of an object. The value ranges from 0 to 1. The larger the value, the more likely the location is to be on the surface. In news scene applications, the surface confidence values ​​of anchors and important news event object areas are usually high to ensure the priority of their geometric reconstruction.

[0093] When performing differentiable voxel centroid extraction, an adaptive thresholding process is applied to the geometric voxel distribution. Regions with an occupancy probability greater than the threshold are considered entities. For each connected voxel cluster, the centroid position is calculated as the initial vertex. A differentiable centroid extraction operator is used to ensure that the gradient propagates smoothly during training. For high-importance regions such as the face of a news anchor, the voxel cluster size threshold is reduced, and the vertex sampling density is increased to ensure detail preservation. Experiments show that the vertex density of the main object region in a news scene is three to five times higher than that of the background region, effectively improving the reconstruction quality.

[0094] When constructing adjacency relationships based on surface confidence maps, an adaptive radius spherical neighborhood search is employed to find potential connection points for each initial vertex. For connections between vertex pairs, uniform sampling is performed along the connection line, and surface confidence interpolation values ​​are calculated. If a connection passes through a low-confidence region, the connection is discarded. This surface-guided topology construction ensures that the generated triangular mesh conforms to the geometric properties of the object's surface. In real-time news holographic scenarios, this method can correctly handle complex topological changes caused by anchor gestures, maintaining geometric continuity.

[0095] When using realism assessment for iterative optimization, a rendering-based loss function is constructed. The current geometric mesh is rendered in a differentiable manner to generate a depth map, normal map, and contour map, which are compared with a reference map derived from the input image. The loss function comprehensively considers geometric consistency, surface smoothness, and visual realism. Gradients are calculated through backpropagation, and the initial vertex positions and topology are iteratively optimized. In each iteration, vertex coordinates are fine-tuned, and certain adjacency relationships are re-evaluated. The optimization process employs an adaptive learning rate strategy, gradually reducing the step size as iterations progress to ensure convergence stability.

[0096] When extracting texture features from 2D images, a multi-scale feature pyramid network is used to capture both local details and global semantic information. Targeting the characteristics of news scenes, the network design includes an attention enhancement module that focuses on the texture details of facial expressions and important event areas. The extracted texture features include color, texture gradient, and semantic information, forming a high-dimensional feature map.

[0097] When establishing a spatial correspondence between texture features and geometric vertices, a mapping from two-dimensional image space to three-dimensional vertex space is achieved. For each geometric vertex, its corresponding position in the original image is determined by view projection, and the texture features at that position are extracted and converted into vertex color values. For multi-view input, the texture contributions from different viewpoints are weighted and fused, with the weights based on the view angle and visibility score. This method effectively solves the problem of discontinuous texture mapping caused by the dynamic changes of the anchor in news scenes.

[0098] When optimizing textures by introducing semantic consistency constraints, a dual loss function is constructed, simultaneously considering the semantic matching degree between the texture and its category, and the spatial smoothness of the color values ​​of adjacent vertices. The semantic matching degree is calculated by comparing the vertex color value with the typical color distribution of its semantic category; spatial smoothness is measured by the weighted sum of the color differences between adjacent vertices, with the weights inversely proportional to the semantic differences between vertices. Through an alternating optimization strategy, vertex color values ​​are iteratively updated, ensuring a smooth texture transition while maintaining semantic consistency.

[0099] In news holographic presentation, a bidirectional mapping relationship is established between 3D geometric meshes and visual textures, supporting bidirectional queries from geometry to texture and from texture to geometry. Each geometric vertex records its corresponding texture coordinates, and each texture element is associated with its mapped geometric region. This bidirectional mapping supports real-time interaction, allowing viewers to select the corresponding geometric part by touching the texture area, enhancing the interactivity of the news holographic experience. When calculating the adjacency topology information of geometric vertices, a half-edge data structure is constructed to record the one-ring neighborhood information and facet connection relationships of each vertex. Based on the adjacency information, a weighted average method is used to calculate the surface normal vector of each vertex, with the weight proportional to the area of ​​adjacent faces. For vertices at the boundary, a boundary-preserving normal vector calculation strategy is applied to avoid smoothing artifacts. The vertex coordinates, normal vectors, color values, and topological relationships are combined to form a complete 3D spatial representation.

[0100] Figure 2 This is a flowchart illustrating the angle spectral data generation method with limited viewing angle according to an embodiment of the present invention. In one optional embodiment, based on the three-dimensional spatial representation, the light radiation characteristics of different viewing directions are calculated to obtain the angle spectral distribution, and the spectral components of the corresponding directions are filtered according to the preset viewing angle range of the holographic display to obtain angle spectral data with limited viewing angle, including:

[0101] Geometric vertices, corresponding surface normals, and vertex color values ​​are extracted from the three-dimensional spatial representation. A local coordinate system is constructed for the surface normals of the geometric vertices, and the upper hemisphere space is discretized and sampled with the geometric vertices as the center to generate a set of observation direction samples containing zenith angle parameters and azimuth angle parameters. The cosine value of the angle between each observation direction and the surface normal is calculated.

[0102] The vertex color value is decomposed into spectral intensity components of multiple wavelength channels. Based on the cosine of the included angle and the spectral intensity components, the light radiation intensity of the geometric vertex in each observation direction is calculated to obtain the directional spectral data of the geometric vertex.

[0103] Obtain the center direction and half-angle parameters of the field of view of the holographic display, transform the center direction of the field of view to the local coordinate system, calculate the deviation angle of each observation direction, retain the observation direction and its radiation intensity with a deviation angle not exceeding the half-angle of the field of view, and arrange them according to the zenith angle and azimuth angle parameters to form the filtered spectral data;

[0104] Perform directional spectral data generation and viewpoint filtering on all geometric vertices, establish the mapping relationship between geometric vertices, observation direction, and light radiation intensity, and organize the filtered spectral data into viewpoint-limited angular spectral data.

[0105] In this specific embodiment, when calculating the angular spectral distribution based on the three-dimensional spatial representation, it is necessary to extract geometric vertex information from the three-dimensional spatial representation. The three-dimensional spatial representation stores the geometric and material properties of the scene. During the extraction process, the three-dimensional coordinate position, surface normal vector and color value of each geometric vertex are obtained. The surface normal vector is usually a unit vector, which represents the orientation of the surface where the vertex is located. The vertex color value is in RGB format, which represents the visual attributes of the vertex.

[0106] For each geometric vertex, a local coordinate system is established based on its surface normal vector. The normal vector is defined as the z-axis direction of the local coordinate system. Two mutually orthogonal vectors perpendicular to this normal vector are selected as the x-axis and y-axis. The x-axis direction can be obtained by performing a cross product operation between the normal vector and any non-parallel reference vector (such as the y-axis of the global coordinate system), and the y-axis direction can be obtained by performing a cross product between the x-axis and the normal vector. After the local coordinate system is established, the direction is discretized and sampled in the upper hemisphere space with the geometric vertex as the center.

[0107] The sampling process employs uniform zenith and azimuth step sizes. The zenith angle θ is divided from 0 degrees (normal vector direction) to 90 degrees (surface tangent plane), and the azimuth angle φ is divided from 0 degrees to 359 degrees. For example, the zenith angle can be divided into 6 equal intervals, with one sampling point every 15 degrees; the azimuth angle can be divided into 24 equal intervals, with one sampling point every 15 degrees. This results in a sample set of observation directions containing 144 discrete directions, each uniquely determined by the (θ, φ) parameter pair. For each observation direction, the cosine of its angle with the surface normal vector, cos(θ), is calculated; this value will be used in subsequent calculations of light radiation intensity.

[0108] Vertex color values ​​are usually given in RGB form and need to be decomposed into spectral intensity components of multiple wavelength channels. The visible spectrum range (approximately 380nm to 780nm) can be divided into several wavelength intervals, such as converting RGB values ​​into multiple narrowband spectral components covering the entire visible spectrum. For simplified implementation, the three RGB channels can be directly used as representatives of the three main wavelength intervals of red, green, and blue. Alternatively, a more refined spectral decomposition method can be used, such as converting RGB values ​​into multiple (e.g., 8 or 16) narrowband spectral intensity values ​​through a color space transformation matrix.

[0109] Based on the spectral intensity components of the vertex and the cosine of the angle between the viewing direction and the normal vector, the radiant intensity of the geometric vertex in each viewing direction is calculated. For diffuse materials, Lambert's cosine law can be applied, and the radiant intensity is proportional to cos(θ). For more complex materials, the bidirectional reflectance distribution function (BRDF) can be used to calculate the radiant intensity in different viewing directions. The base intensity of each wavelength channel is multiplied by the direction factor to obtain the radiant intensity value of the vertex in a specific viewing direction and at a specific wavelength, forming a three-dimensional dataset: vertex-viewing direction-wavelength, which stores the complete directional spectral data of the geometric vertex.

[0110] Holographic display systems have inherent field-of-view limitations. Based on these limitations, effective angular spectral data needs to be selected to obtain the holographic display's field-of-view center direction (usually represented by a unit vector) and field-of-view half-angle parameters (expressed in angles, such as horizontal ±30 degrees, vertical ±15 degrees). The field-of-view center direction then needs to be transformed from the global coordinate system to the local coordinate system of each vertex. This transformation operation involves taking the inner product of the field-of-view center direction vector in the global coordinate system and the basis vectors of the local coordinate system to obtain the direction representation in the local coordinate system.

[0111] For each observation direction in the sampling set, calculate its deviation angle from the transformed field of view center direction. The deviation angle can be obtained by taking the inverse cosine function of the dot product of the two direction vectors. Compare the calculated deviation angle with the field of view half-angle parameter, and retain the observation directions with deviation angles not exceeding the field of view half-angle and their corresponding light radiation intensity values. These retained observation directions and their radiation intensity values ​​are arranged in the order of zenith angle and azimuth angle parameters to form the filtered spectral data.

[0112] The above-described process of generating directional spectral data and filtering viewpoints is repeated for all geometric vertices in the scene to establish a mapping relationship from geometric vertices to the viewing direction and the intensity of light radiation. This mapping relationship can be stored using a multidimensional array or dictionary structure, where the vertex index is used as the first-level key, the viewing direction parameter is used as the second-level key, and the corresponding value is an array of spectral radiation intensities in that direction. The spectral data of all filtered vertices are organized together to form the final viewpoint-constrained angular spectral data. This data will be used as input for subsequent holographic calculations to generate a holographic image that conforms to the viewpoint constraints.

[0113] In practical applications, the field of view limitation can be adjusted according to the actual parameters of the holographic display device. For example, for a head-mounted holographic display device with a horizontal viewing angle range of ±30 degrees and a vertical viewing angle range of ±15 degrees, only the observation direction data falling within this elliptical viewing cone is retained. This effectively reduces the amount of computation while ensuring the visual quality of the holographic display. The angle spectral data with limited viewing angle can be further used to calculate the holographic interference pattern, ultimately achieving high-quality 3D scene reconstruction within a limited field of view.

[0114] In one optional implementation, the vertex color value is decomposed into spectral intensity components of multiple wavelength channels. Based on the cosine of the included angle and the spectral intensity components, the radiant intensity of the geometric vertex in each viewing direction is calculated to obtain the directional spectral data of the geometric vertex, including:

[0115] The vertex color values ​​are projected onto the linear space of the spectral basis function to obtain a continuous representation of the color values ​​in the spectral domain. By discretizing the continuous representation in the wavelength range, the spectral intensity components of each channel are obtained, and the reconstruction correspondence between the projection coefficients and the spectral intensity components is established.

[0116] Based on the cosine of the angle between the observation direction and the surface normal vector, the degree of spatial deviation is determined. Based on the degree of spatial deviation, weights are assigned to adjacent observation directions. The weights are applied to the corresponding spectral intensity components and interpolation is performed to generate interpolated spectral components for each observation direction.

[0117] The interpolated spectral components are fused and corrected with the spectral intensity components in the reconstructed correspondence to obtain the light radiation intensity of the geometric vertex in each observation direction. A two-dimensional index structure is established according to the zenith angle parameter and the azimuth angle parameter to store the radiation intensity values ​​of multiple wavelength channels under the corresponding observation direction, thereby obtaining the directional spectral data of each geometric vertex.

[0118] In this specific embodiment, it is necessary to obtain the color value and surface normal vector information of the geometric vertex. The color value is usually represented in RGB three-channel form, and the surface normal vector is represented as a three-dimensional unit vector, indicating the orientation of the surface where the vertex is located.

[0119] By projecting vertex color values ​​onto the linear space of spectral basis functions, a continuous representation of color values ​​to the spectral domain is achieved. Specifically, a set of spectral basis functions, such as Gaussian functions, is selected. Suppose six Gaussian functions are chosen, with center wavelengths at 420nm, 460nm, 500nm, 540nm, 580nm, and 620nm, respectively, covering the main range of the visible spectrum. The standard deviation of each Gaussian function is set to 20nm. This parameter controls the width of the basis functions, ensuring appropriate overlap between them to express a smooth spectral transition. The advantage of choosing Gaussian functions is that their shape is close to the natural spectral distribution curve and they have good mathematical properties, which facilitates subsequent calculations and processing.

[0120] When projecting the vertex color values ​​of a 3D geometric mesh from RGB representation to the linear space of spectral basis functions, the least squares method is used to solve for the projection coefficients. For each vertex's RGB color value, a transformation matrix is ​​constructed. This matrix contains the inner product of the spectral sensitivity functions corresponding to the three primary colors in the RGB color space and the six Gaussian functions. By analytically or iteratively solving the least squares problem, six projection coefficients are obtained that minimize the error between the RGB color values ​​and the integral results of the spectral basis functions linearly combined with the projection coefficients. After obtaining the projection coefficients, the color representation of each vertex is transformed from a 3D RGB vector into a six-dimensional projection coefficient vector. This transformation expands the dimension of color expression. Through the linear combination of the projection coefficients and the spectral basis functions, spectral intensity values ​​can be calculated at any wavelength, forming a continuous spectral representation.

[0121] By discretizing the continuous representation within wavelength ranges, the continuous spectrum is decomposed into multiple discrete wavelength channels. The sampling strategy is based on the characteristics of human visual perception and the spectral response curve of holographic display devices, increasing sampling density in visually sensitive areas. In a typical setting, the visible spectrum range (380nm to 780nm) is divided into thirty-two wavelength channels, each with a width of approximately 10 to 15nm. For key color areas in news scenes, such as landmark buildings or event-related objects, the sampling interval is further refined to ensure accurate color reproduction. When calculating the spectral intensity components of each wavelength channel, the continuous representation is evaluated at each sampling wavelength. Specifically, for each sampling wavelength, the function values ​​of all basis functions at that wavelength are calculated and weighted summed with the projection coefficients to obtain the spectral intensity at that wavelength. These spectral intensity components collectively describe the complete spectral distribution, providing detailed color information for holographic rendering.

[0122] A reconstruction correspondence between projection coefficients and spectral intensity components is established, and a bidirectional conversion model is constructed. This model allows for efficient switching between projection coefficients and discrete spectral intensities to meet different rendering requirements. The conversion model uses sparse matrix representation to optimize computational efficiency and support real-time rendering. To improve reconstruction accuracy, a nonlinear correction term is introduced to compensate for errors in the linear model in specific spectral regions, especially in areas with abrupt spectral changes. For high-contrast regions in news holographic scenes, such as the transition between strongly lit and shadowed areas, nonlinear correction significantly improves the accuracy of color reproduction.

[0123] Multiple observation directions are determined to sample the directional spectral characteristics of the vertex. The observation directions are parameterized using the zenith angle θ and azimuth angle φ in spherical coordinates. For example, five zenith angle values ​​are uniformly sampled (0°, 22.5°, 45°, 67.5°, 90°), and eight azimuth angle values ​​are uniformly sampled (0°, 45°, 90°, 135°, 180°, 225°, 270°, 315°), resulting in a total of 40 observation directions.

[0124] For each observation direction, the cosine of its angle with the surface normal vector is calculated. A cosine value of 1 indicates that the observation direction is completely aligned with the normal vector, while a cosine value of 0 indicates that the observation direction is perpendicular to the normal vector. The degree of spatial deviation is determined based on the cosine value; the smaller the spatial deviation, the closer the observation direction is to the normal vector direction. Weights are assigned to adjacent observation directions based on the degree of spatial deviation, using a distance-based weighting strategy. When the cosine of the angle between the observation direction and the normal vector is cos(θ), the weights of adjacent sampling directions can be calculated as w = max(0, cos(θ) - cos(θ)). threshold )), where θ threshold The preset threshold angle is used. Weight normalization ensures that the sum of the weights is 1.

[0125] The calculated weights are applied to the corresponding spectral intensity components, and weight-based interpolation is performed. The bilinear interpolation method is used to interpolate in two dimensions: zenith angle and azimuth angle. Interpolation is performed separately for each wavelength channel to generate interpolated spectral components for each observation direction.

[0126] To improve the accuracy of spectral characterization, the interpolated spectral components and the spectral intensity components in the reconstructed correspondence are fused and corrected. An adaptive weighting strategy is used for this fusion correction: when the observation direction is close to the normal vector, the spectral intensity components in the reconstructed correspondence have a higher weight; when the observation direction deviates significantly from the normal vector, the interpolated spectral components have a higher weight. The fusion formula is: P final =α·P recon +(1-α)·P interp , where α is the adaptive weighting coefficient, which decreases as the cosine of the included angle decreases.

[0127] After fusion correction, the light radiation intensity of the geometric vertex in each observation direction is obtained. A two-dimensional index structure is established according to the zenith angle parameter and the azimuth angle parameter. Each index position stores the radiation intensity value of multiple wavelength channels in the corresponding observation direction. The index structure is implemented using a two-dimensional array. The row index corresponds to the zenith angle sampling point, and the column index corresponds to the azimuth angle sampling point. Each array element stores a vector containing the radiation intensity value of each wavelength channel.

[0128] The generated directional spectral data supports quick querying of spectral responses under any observation direction. When it is necessary to query the spectral response in a non-sampling direction, an approximate value is obtained by performing interpolation operations in a two-dimensional index structure. The directional spectral data can be used to render realistic material appearances, especially for materials with strong directional optical properties, such as metals, pearlescent materials, and holographic materials.

[0129] In practical applications, the number of wavelength channels and the sampling density in the observation direction can be adjusted according to the material characteristics and accuracy requirements. For slowly changing diffuse reflective materials, the number of sampling points in the observation direction can be reduced; for high-frequency changing high-gloss materials, the sampling density needs to be increased to accurately capture directional characteristics.

[0130] In one optional implementation, the view-limited angular spectral data is backpropagated to the holographic plane via ray tracing. The phase accumulation and amplitude attenuation of each ray are calculated. Complex coherent superposition of rays from different directions originating from the same spatial point at the same position on the holographic plane is performed to obtain the complex amplitude distribution of the holographic plane, including:

[0131] Based on the angle spectral data with limited viewing angle, the observation direction and corresponding light radiation intensity of each geometric vertex are extracted. Taking the geometric vertex as the light source and the opposite direction of the observation direction as the light propagation direction, a set of reverse tracking rays pointing from the geometric vertex to the holographic plane is constructed.

[0132] Obtain the spatial position parameters and normal vector of the holographic plane, calculate the coordinates of the intersection point of each backward tracing ray with the holographic plane, and map them to the two-dimensional coordinate system of the holographic plane to obtain the position of the ray landing point. Calculate the spatial distance from the geometric vertex to the coordinates of the intersection point as the ray propagation path length.

[0133] Based on the light propagation path length and preset wavelength parameters, the propagation phase change is calculated and added to the initial phase to obtain the phase accumulation value. Based on the light propagation path length and spatial attenuation coefficient, the amplitude attenuation factor is calculated and multiplied by the light radiation intensity to obtain the amplitude value.

[0134] The phase accumulation value and the amplitude value are converted into complex light field components. A discrete pixel array is established on the holographic plane. The light field components are merged into the corresponding pixels based on the position of the light rays. The light field components in the same pixel are coherently superimposed to obtain the complex amplitude superposition value of the pixel position.

[0135] The complex amplitude superposition values ​​of all pixel positions on the holographic plane are organized into a two-dimensional complex matrix according to the spatial arrangement order of the pixel array, forming the complex amplitude distribution of the holographic plane.

[0136] In practical applications, based on angle spectral data with limited viewing angles, the observation direction and corresponding ray radiation intensity of each geometric vertex are extracted. Angled spectral data typically contains the light intensity distribution of multiple vertices at different observation angles, which can be represented as a set of four-dimensional functions. This function records the light intensity emitted by each spatial vertex in each direction. For each geometric vertex, a local coordinate system is established, which is used as the starting point of the ray, and the opposite direction of the observation direction is used as the ray propagation direction to construct a set of reverse-tracing rays. For example, for a vertex located at coordinates (x0, y0, z0), if its observed light intensity in the direction (θ, φ) is I(θ, φ), then a reverse ray is constructed with this vertex as the starting point and the direction (-θ, -φ), and the corresponding light intensity value is associated with it.

[0137] Obtain the spatial position parameters and normal vector of the holographic plane. The holographic plane can be defined as a plane in three-dimensional space, passing through the coordinates (x, y) of its center point. h y h , z h ) and normal vector (n x n y n z ) Determine. For each backward tracing ray, calculate the coordinates of its intersection point with the holographic plane. Let the ray's origin be P0(x0, y0, z0), and its direction vector be d(d x d y d z If the ray equation is P(t), then the equation of the ray can be expressed as P(t) = P0 + t·d. Substituting this equation into the plane equation and solving for the parameter t, we can obtain the coordinates of the intersection point P. i Establish a two-dimensional coordinate system for the holographic plane, and set the intersection point P... i Mapping to this coordinate system, we obtain the position of the light ray's point of impact on the holographic plane (u). i v i Simultaneously, calculate the Euclidean distance |P0-P| from the geometric vertices to the intersection points. i |, as the length of the light propagation path L i .

[0138] Based on the calculated light propagation path length and the preset wavelength parameter λ, the phase change during propagation is calculated. The phase change during light propagation is related to the propagation distance and wavelength, and can be expressed as Δφ = 2π·L i / λ. This phase change is added to the initial phase φ0 of the light ray to obtain the phase accumulation value φtotal = φ0 + Δφ. Simultaneously, considering the energy attenuation of the light wave during propagation, the amplitude attenuation factor is calculated based on the propagation path length and the spatial attenuation coefficient α. In free space, light intensity typically attenuates according to the inverse square law; therefore, the amplitude attenuation factor can be expressed as 1 / L. i Or a more complex function form such as exp(-α·L) iMultiplying the attenuation factor by the radiation intensity I of the light, we obtain the final amplitude value A = I·attenuation factor.

[0139] The calculated phase accumulation value φ total The amplitude value A is converted into a complex form of the light field component, i.e., E = A·exp(i·φ). total A discrete pixel array is constructed on the holographic plane, with the pixel size determined based on the resolution requirements of the hologram and the physical characteristics of the recording medium. This is based on the previously calculated positions of the light rays (u...). i v i The light field components are merged into their respective pixels. When multiple rays fall within the same pixel, these light field components are coherently superimposed, i.e., E0. pixel =∑E j , where j represents all the light rays falling within the pixel. This complex superposition takes into account the coherence of light waves and can correctly simulate the interference effect.

[0140] The complex amplitude values ​​at all pixel positions on the holographic plane are superimposed and organized into a two-dimensional complex matrix H(u, v) according to the spatial arrangement of the pixel array, forming the complex amplitude distribution of the holographic plane. This distribution contains both amplitude and phase information, which can be used for subsequent holographic recording or digital holographic display.

[0141] In practical applications, the visual effect of holograms can be optimized by adjusting wavelength parameters, spatial attenuation coefficients, and holographic plane parameters. For example, when synthesizing multicolor holograms, the complex amplitude distributions of the red, green, and blue wavelength channels need to be calculated separately and combined using appropriate encoding methods. Furthermore, the quality and computational efficiency of holograms can be improved by increasing sampling point density and optimizing ray tracing algorithms. The holographic plane complex amplitude distribution generated by the above methods can accurately record the light field information of a three-dimensional scene, achieving high-quality holographic reconstruction effects and providing fundamental data support for applications such as holographic display and holographic projection.

[0142] In one optional implementation, the holographic plane complex amplitude distribution is used as the target field. A transfer function is established based on the Fresnel number. Diffraction propagation calculations are performed on the randomly initialized complex coding matrix. The total loss function is constructed and iteratively optimized to obtain holographic pattern coding data, including:

[0143] Using the holographic plane complex amplitude distribution as the target field, the pixel size of the spatial light modulator is obtained, the complex coding matrix is ​​randomly initialized, the diffraction propagation distance and preset wavelength parameters are obtained, the Fresnel number is calculated, the sampling interval is determined and a frequency grid is established, and the transfer function of diffraction propagation is obtained.

[0144] The complex coding matrix is ​​subjected to Fourier transform and multiplied with the transfer function, and then an inverse transform is performed to obtain the reconstructed complex field. The reconstruction error in the spatial domain and the energy deviation in the frequency domain between the reconstructed complex field and the target field are calculated respectively, and the total loss function is obtained by weighted combination.

[0145] Calculate the gradient of the total loss function with respect to the coding parameters, update the complex coding matrix, perform frequency domain bandpass filtering on the updated complex coding matrix, attenuate the frequency components outside the cutoff frequency according to the frequency domain attenuation ratio, and use the inverse transformation as the input for the next iteration.

[0146] The iteration terminates when the total loss change of adjacent iterations is less than the loss convergence threshold. The phase angle and magnitude of the complex coding matrix of the current iteration are extracted, normalized and mapped to the modulator bit depth range and quantized to obtain the phase modulation parameters and amplitude modulation parameters. These parameters are then organized in pixel order, and a header identifier and checksum are added to obtain the holographic pattern coding data.

[0147] In this specific embodiment, the holographic plane complex amplitude distribution is used as the target field, and the pixel size parameters of the spatial light modulator are obtained. In practical applications, the pixel size of the spatial light modulator is usually 8μm×8μm or 15μm×15μm. For example, in a specific embodiment, a phase spatial light modulator with a resolution of 1920×1080 is used, and the pixel size is 8μm×8μm.

[0148] A complex coding matrix is ​​randomly initialized, the size of which matches the resolution of the spatial light modulator. Random initialization can use random numbers uniformly distributed in the range [0, 1] to initialize the real and imaginary parts of the complex matrix, respectively, to obtain the diffraction propagation distance and preset wavelength parameters, for example, the propagation distance is set to 500 mm and the wavelength parameter is a green light wavelength of 532 nm.

[0149] Based on the above parameters, the Fresnel number is calculated. The Fresnel number is an important parameter in the diffraction propagation process. It is calculated by the formula: the Fresnel number is equal to the square of the pixel size divided by the wavelength multiplied by the propagation distance. When the Fresnel number is less than 1, the Fresnel diffraction model is used; when the Fresnel number is greater than 10, the Fraunhofer diffraction model is used; and in the intermediate region, a hybrid algorithm is used.

[0150] Based on the Fresnel number, the sampling interval is determined and a frequency grid is established. In the frequency domain, a two-dimensional frequency grid is constructed, with the frequency range from -1 / (2×sampling interval) to 1 / (2×sampling interval). Based on the frequency grid and the propagation distance, the transfer function of Fresnel diffraction is calculated, which describes how the complex amplitude at one point in space propagates to another point.

[0151] The complex coding matrix is ​​transformed to the frequency domain using Fast Fourier Transform (FFT), multiplied by the previously calculated transfer function, and then reconstructed using Inverse Fourier Transform (IFFT). This step simulates the propagation process of the light field from the coding plane to the target plane.

[0152] The reconstruction error between the reconstructed complex field and the target field is calculated, using the mean square error (MSE) as the evaluation metric in the spatial domain. Simultaneously, the energy deviation between the two in the frequency domain is calculated using the relative error of the spectral energy distribution. These two errors are weighted and combined using weighting coefficients α and β to obtain the total loss function. In practical applications, α can be set to 0.7 and β to 0.3 to balance the spatial domain reconstruction quality and the frequency domain energy distribution.

[0153] Based on the calculated total loss function, the gradient of the loss function with respect to the coding parameters is calculated using the gradient descent method, and the complex coding matrix is ​​updated. The update step size can adopt an adaptive adjustment strategy, with the initial step size set to 0.01 and gradually decreasing as the number of iterations increases.

[0154] The updated complex coding matrix undergoes frequency domain bandpass filtering to suppress high-frequency noise and low-frequency drift. Specifically, the matrix is ​​transformed to the frequency domain using FFT, and a cutoff frequency is set, for example, 5% of the original frequency for low frequencies and 80% for high frequencies. Frequency components exceeding the cutoff frequency range are attenuated according to a frequency domain attenuation ratio, which can be set as a linear or exponential attenuation function. The filtered frequency domain information is then converted back to the spatial domain using IFFT and used as input for the next iteration.

[0155] During the iteration process, the change in total loss between two consecutive iterations is continuously monitored. The iteration process is terminated when the change value is less than a preset loss convergence threshold. The loss convergence threshold can be set to 0.0001, meaning that the algorithm is considered to have converged when the change in the loss function between two consecutive iterations is less than this value.

[0156] Phase angle and magnitude information are extracted from the complex coding matrix obtained from the final iteration. The phase angle is obtained by calculating the argument of the complex number, and the magnitude is obtained by calculating the modulus of the complex number. The phase angle value is normalized from the range of [-π, π] to the bit depth range of the spatial light modulator. For example, an 8-bit modulator is mapped to [0, 255] and then quantized. Similarly, the magnitude value is normalized from the range of [0, 1] to the amplitude modulation range supported by the modulator and then quantized.

[0157] The quantized phase modulation and amplitude modulation parameters are organized into a data stream according to pixel order. A header identifier and checksum information are added. The header identifier contains information such as data format version, matrix size, and bit depth. The checksum uses the CRC32 algorithm to ensure data integrity. The final data packet is the holographic pattern encoded data, which can be directly used to drive a spatial light modulator to display holographic images.

[0158] In practical applications, this method can be used in fields such as digital holographic display, holographic storage, and optical information processing. It can effectively improve the reconstruction quality of holographic images, reduce noise and artifacts, and enhance the visual experience.

[0159] This invention provides a real-time holographic image news scene generation system based on deep learning, comprising:

[0160] The first unit is used to acquire multimodal news source data containing text content and two-dimensional images;

[0161] The second unit is used to perform cross-modal semantic parsing on the multimodal news source data, extract multiple objects and label object categories, spatiotemporal location relationships and importance markers, and generate structured semantic scene features;

[0162] The third unit is used to construct a three-dimensional geometric mesh through a deep neural network based on the structured semantic scene features and the two-dimensional image, and combine it with visual texture to form a three-dimensional spatial representation.

[0163] The fourth unit is used to calculate the light radiation characteristics of different viewing directions based on the three-dimensional spatial representation, obtain the angular spectral distribution, and filter the spectral components of the corresponding directions according to the preset viewing angle range of the holographic display to obtain viewing angle-limited angular spectral data.

[0164] The fifth unit is used to back-propagate the angle spectral data with limited viewing angle to the holographic plane through ray tracing, calculate the phase accumulation and amplitude attenuation of each ray, and perform complex coherent superposition of rays from different directions from the same spatial point at the same position on the holographic plane to obtain the complex amplitude distribution of the holographic plane.

[0165] The sixth unit is used to take the complex amplitude distribution of the holographic plane as the target field, establish the transfer function based on the Fresnel number, perform diffraction propagation calculation on the randomly initialized complex coding matrix, construct the total loss function for iterative optimization, and obtain the holographic pattern coding data.

[0166] The seventh unit is used to drive the holographic display device to present holographic news scenes based on the holographic pattern encoding data.

[0167] A third aspect of the present invention provides an electronic device, comprising:

[0168] processor;

[0169] Memory used to store processor-executable instructions;

[0170] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0171] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0172] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for real-time generation of holographic news scenes based on deep learning, characterized in that, include: Acquire multimodal news source data containing text content and two-dimensional images; Cross-modal semantic parsing is performed on the multimodal news source data to extract multiple objects and label them with object categories, spatiotemporal location relationships and importance markers, generating structured semantic scene features; Based on the structured semantic scene features and the two-dimensional image, a three-dimensional geometric mesh is constructed using a deep neural network, and combined with visual textures to form a three-dimensional spatial representation. Based on the three-dimensional spatial representation, the light radiation characteristics of different observation directions are calculated to obtain the angular spectral distribution. Then, according to the preset viewing angle range of the holographic display, the spectral components of the corresponding directions are filtered to obtain the viewing angle-limited angular spectral data. The angle spectral data with limited viewing angle is back-propagated to the holographic plane through ray tracing. The phase accumulation and amplitude attenuation of each ray are calculated. Rays from different directions from the same spatial point are coherently superimposed at the same position on the holographic plane to obtain the complex amplitude distribution of the holographic plane. Using the holographic plane complex amplitude distribution as the target field, a transfer function is established based on the Fresnel number. Diffraction propagation calculations are performed on the randomly initialized complex coding matrix, and the total loss function is constructed and iteratively optimized to obtain holographic pattern coding data. The holographic pattern encoding data drives the holographic display device to present holographic news scenes. The process of performing cross-modal semantic parsing on the multimodal news source data, extracting multiple objects and labeling them with object categories, spatiotemporal relationships, and importance markers, and generating structured semantic scene features includes: Entity recognition and semantic relation extraction are performed on the text content in the multimodal news source data to generate a text semantic graph; Target detection is performed on 2D image data, and monocular depth estimation is performed simultaneously to obtain depth information of visual objects, resulting in a set of visual objects with depth estimation values; Semantic matching and binding relationships are established between entities in the text semantic graph and visual objects. Based on the depth estimation value of the visual objects, the bound objects are divided into the depth range of the corresponding holographic display depth layer to obtain a depth-layered object set. Based on the semantic relationship types between entities in the text semantic graph, the objects in the deep hierarchical object set are classified into scene subject objects, spatial environment objects, and event-related objects, and the category identifiers of each object are obtained. Calculate the distance difference between the estimated depth value of each object and the depth position of the focal plane of the holographic display. Based on the absolute value of the distance difference and a preset threshold function, convert the distance difference into an importance label. Extract the bounding box coordinates of each object in the 2D image, integrate the bounding box coordinates and depth interval labels to form a spatiotemporal positional relationship, and combine the category label, depth interval label and importance label of each object to form a structured semantic scene feature.

2. The method according to claim 1, characterized in that, Based on the structured semantic scene features and the two-dimensional image, a three-dimensional geometric mesh is constructed using a deep neural network, and combined with visual textures to form a three-dimensional spatial representation, including: Based on the structured semantic scene features, the geometric voxel distribution and surface confidence map are predicted synchronously through a deep neural network. The initial vertices are obtained by extracting the centroids of differentiable voxels. Adjacency relationships are constructed based on surface confidence screening. The initial vertices and topology are iteratively optimized using gradient feedback for authenticity evaluation to generate a three-dimensional geometric mesh including geometric vertices. The texture features of the target region are extracted from the two-dimensional image. The texture features are spatially correlated with the geometric vertices to generate vertex color values. Semantic consistency constraints are introduced to optimize the semantic matching degree between the texture and the category, as well as the spatial smoothness of the color values ​​of adjacent vertices. Through iterative optimization, a visual texture that conforms to the semantics of the object and has a continuous transition is obtained. A bidirectional mapping relationship is established between the three-dimensional geometric mesh and the visual texture, and the adjacency topology information and surface normal vectors of the geometric vertices are calculated and combined to form a three-dimensional spatial representation.

3. The method according to claim 1, characterized in that, Based on the aforementioned three-dimensional spatial representation, the light radiation characteristics of different viewing directions are calculated to obtain the angular spectral distribution. Then, according to the preset viewing angle range of the holographic display, the spectral components of the corresponding directions are filtered to obtain viewing angle-limited angular spectral data, including: Geometric vertices, corresponding surface normals, and vertex color values ​​are extracted from the three-dimensional spatial representation. A local coordinate system is constructed for the surface normals of the geometric vertices, and the upper hemisphere space is discretized and sampled with the geometric vertices as the center to generate a set of observation direction samples containing zenith angle parameters and azimuth angle parameters. The cosine value of the angle between each observation direction and the surface normal is calculated. The vertex color value is decomposed into spectral intensity components of multiple wavelength channels. Based on the cosine of the included angle and the spectral intensity components, the light radiation intensity of the geometric vertex in each observation direction is calculated to obtain the directional spectral data of the geometric vertex. Obtain the center direction and half-angle parameters of the field of view of the holographic display, transform the center direction of the field of view to the local coordinate system, calculate the deviation angle of each observation direction, retain the observation direction and its radiation intensity with a deviation angle not exceeding the half-angle of the field of view, and arrange them according to the zenith angle and azimuth angle parameters to form the filtered spectral data; Perform directional spectral data generation and viewpoint filtering on all geometric vertices, establish the mapping relationship between geometric vertices, observation direction, and light radiation intensity, and organize the filtered spectral data into viewpoint-limited angular spectral data.

4. The method according to claim 3, characterized in that, The vertex color value is decomposed into spectral intensity components of multiple wavelength channels. Based on the cosine of the included angle and the spectral intensity components, the radiant intensity of the geometric vertex in each viewing direction is calculated to obtain the directional spectral data of the geometric vertex, including: The vertex color values ​​are projected onto the linear space of the spectral basis function to obtain a continuous representation of the color values ​​in the spectral domain. By discretizing the continuous representation in the wavelength range, the spectral intensity components of each channel are obtained, and the reconstruction correspondence between the projection coefficients and the spectral intensity components is established. Based on the cosine of the angle between the observation direction and the surface normal vector, the degree of spatial deviation is determined. Based on the degree of spatial deviation, weights are assigned to adjacent observation directions. The weights are applied to the corresponding spectral intensity components and interpolation is performed to generate interpolated spectral components for each observation direction. The interpolated spectral components are fused and corrected with the spectral intensity components in the reconstructed correspondence to obtain the light radiation intensity of the geometric vertex in each observation direction. A two-dimensional index structure is established according to the zenith angle parameter and the azimuth angle parameter to store the radiation intensity values ​​of multiple wavelength channels under the corresponding observation direction, thereby obtaining the directional spectral data of each geometric vertex.

5. The method according to claim 1, characterized in that, The angle spectral data with limited viewing angle is backpropagated to the holographic plane via ray tracing. The phase accumulation and amplitude attenuation of each ray are calculated. Complex coherent superposition of rays from different directions originating from the same spatial point at the same position on the holographic plane is performed to obtain the complex amplitude distribution of the holographic plane, including: Based on the angle spectral data with limited viewing angle, the observation direction and corresponding light radiation intensity of each geometric vertex are extracted. Taking the geometric vertex as the light source and the opposite direction of the observation direction as the light propagation direction, a set of reverse tracking rays pointing from the geometric vertex to the holographic plane is constructed. Obtain the spatial position parameters and normal vector of the holographic plane, calculate the coordinates of the intersection point of each backward tracing ray with the holographic plane, and map them to the two-dimensional coordinate system of the holographic plane to obtain the position of the ray landing point. Calculate the spatial distance from the geometric vertex to the coordinates of the intersection point as the ray propagation path length. Based on the light propagation path length and preset wavelength parameters, the propagation phase change is calculated and added to the initial phase to obtain the phase accumulation value. Based on the light propagation path length and spatial attenuation coefficient, the amplitude attenuation factor is calculated and multiplied by the light radiation intensity to obtain the amplitude value. The phase accumulation value and the amplitude value are converted into complex light field components. A discrete pixel array is established on the holographic plane. The light field components are merged into the corresponding pixels based on the position of the light rays. The light field components in the same pixel are coherently superimposed to obtain the complex amplitude superposition value of the pixel position. The complex amplitude superposition values ​​of all pixel positions on the holographic plane are organized into a two-dimensional complex matrix according to the spatial arrangement order of the pixel array, forming the complex amplitude distribution of the holographic plane.

6. The method according to claim 1, characterized in that, Using the holographic plane complex amplitude distribution as the target field, a transfer function is established based on the Fresnel number. Diffraction propagation calculations are performed on the randomly initialized complex encoding matrix, and the total loss function is constructed and iteratively optimized to obtain holographic pattern encoded data, including: Using the holographic plane complex amplitude distribution as the target field, the pixel size of the spatial light modulator is obtained, the complex coding matrix is ​​randomly initialized, the diffraction propagation distance and preset wavelength parameters are obtained, the Fresnel number is calculated, the sampling interval is determined and a frequency grid is established, and the transfer function of diffraction propagation is obtained. The complex coding matrix is ​​subjected to Fourier transform and multiplied with the transfer function, and then an inverse transform is performed to obtain the reconstructed complex field. The reconstruction error in the spatial domain and the energy deviation in the frequency domain between the reconstructed complex field and the target field are calculated respectively, and the total loss function is obtained by weighted combination. Calculate the gradient of the total loss function with respect to the coding parameters, update the complex coding matrix, perform frequency domain bandpass filtering on the updated complex coding matrix, attenuate the frequency components outside the cutoff frequency according to the frequency domain attenuation ratio, and use the inverse transformation as the input for the next iteration. The iteration terminates when the total loss change of adjacent iterations is less than the loss convergence threshold. The phase angle and magnitude of the complex coding matrix of the current iteration are extracted, normalized and mapped to the modulator bit depth range and quantized to obtain the phase modulation parameters and amplitude modulation parameters. These parameters are then organized in pixel order, and a header identifier and checksum are added to obtain the holographic pattern coding data.

7. A real-time generation system for holographic news scenes based on deep learning, used to implement the method as described in any one of claims 1-6, characterized in that, include: The first unit is used to acquire multimodal news source data containing text content and two-dimensional images; The second unit is used to perform cross-modal semantic parsing on the multimodal news source data, extract multiple objects and label object categories, spatiotemporal location relationships and importance markers, and generate structured semantic scene features; The third unit is used to construct a three-dimensional geometric mesh through a deep neural network based on the structured semantic scene features and the two-dimensional image, and combine it with visual texture to form a three-dimensional spatial representation. The fourth unit is used to calculate the light radiation characteristics of different viewing directions based on the three-dimensional spatial representation, obtain the angular spectral distribution, and filter the spectral components of the corresponding directions according to the preset viewing angle range of the holographic display to obtain viewing angle-limited angular spectral data. The fifth unit is used to back-propagate the angle spectral data with limited viewing angle to the holographic plane through ray tracing, calculate the phase accumulation and amplitude attenuation of each ray, and perform complex coherent superposition of rays from different directions from the same spatial point at the same position on the holographic plane to obtain the complex amplitude distribution of the holographic plane. The sixth unit is used to take the complex amplitude distribution of the holographic plane as the target field, establish the transfer function based on the Fresnel number, perform diffraction propagation calculation on the randomly initialized complex coding matrix, construct the total loss function for iterative optimization, and obtain the holographic pattern coding data. The seventh unit is used to drive the holographic display device to present holographic news scenes based on the holographic pattern encoding data.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

Images