Display picture processing method and device, computer device, readable storage medium and program product

Abstract

The application relates to the technical field of augmented reality, in particular to a display picture processing method and device, computer equipment, a readable storage medium and a program product. The display picture processing method obtains multiple images and attitude data, performs scene sensing and correction, determines three-dimensional key object and dynamic object coordinates, renders a real scene depth map, and generates a shielding mask based on depth comparison to process virtual object rendering, can effectively process virtual-real shielding relations, can optimize the picture display effect between a real scene and a virtual image in augmented reality, and improves picture reality and immersion.

Description

Technical Field

[0001] This application relates to the field of augmented reality technology, and in particular to a display screen processing method, apparatus, computer device, readable storage medium, and program product. Background Technology

[0002] Augmented Reality (AR) technology aims to seamlessly integrate virtual images into the real-world view to enhance the user experience. However, current technologies cannot achieve natural and realistic occlusion of virtual content by real objects when realizing virtual-real fusion. This results in the judgment of virtual-real occlusion always lagging behind changes in real spatial relationships. When an interactive object needs to interact with a virtual image, the occlusion relationship between the real world where the interactive object is located and the virtual image cannot be updated in time, resulting in occlusion delays and the problem of virtual objects being incorrectly placed in front. It cannot provide adaptive dynamic scenes and real-time natural virtual-real occlusion, resulting in poor image realism and interactive intuition. Summary of the Invention

[0003] Therefore, it is necessary to provide a display screen processing method, apparatus, computer device, readable storage medium, and program product to address the aforementioned technical problems, which can optimize and enhance the display effect between real-world scenes and virtual images.

[0004] In a first aspect, this application provides a display screen processing method, the method comprising:

[0005] Acquire multiple frames of image data collected for the target scene, as well as the pose data corresponding to the image data;

[0006] Scene perception is performed on multi-frame image data based on a perceptual coding network to obtain perceptual information in the target scene;

[0007] Based on the posture data, the perceived information is corrected to determine the scene visual features in the target scene, and based on the scene visual features, the three-dimensional key objects in the target scene are identified.

[0008] Identify dynamic objects in multi-frame image data, and determine the three-dimensional coordinates of the dynamic objects in the three-dimensional world coordinate system based on their position information in the image data.

[0009] Render a realistic scene depth map of the target scene based on the 3D coordinate information of key 3D objects and dynamic objects;

[0010] The virtual object is projected onto the depth map of the real scene, and the depth information of the virtual object is compared with the depth information of the real scene in the depth map to determine the occlusion mask corresponding to each frame of image data; wherein, the occlusion mask is used to represent the positional relationship between the virtual object and the real scene in any frame of image data for display.

[0011] The image data is rendered using an occlusion mask to output a target display screen, which is a 3D display screen that includes both virtual objects and real-world scenes.

[0012] In one embodiment, the scene visual features include a density field and a color field, and the perceptual coding network includes a multi-resolution hash encoder and a decoding multi-layer perceptual layer. The step of correcting the perceptual information based on pose data to determine the scene visual features in the target scene includes:

[0013] Based on the pose data, determine the camera extrinsic parameters corresponding to the image data of the current frame;

[0014] Based on the camera's extrinsic parameters, the pixel coordinates of the image data are projected onto the world coordinate system along the line of sight to generate a three-dimensional sampling point sequence; the three-dimensional sampling point sequence includes multiple three-dimensional sampling points;

[0015] Multiple 3D sampling points are input into a multi-resolution hash encoder. The hash table entries corresponding to the 3D sampling points at each resolution level are queried to obtain the hash feature vector corresponding to the image data.

[0016] The hash feature vector is input into the decoding multilayer perceptron to determine the density and color values ​​of the three-dimensional sampling points in the image data.

[0017] Based on the density and color values ​​in the image data, the scene feature parameters are corrected to determine the density field and color field corresponding to the target scene.

[0018] In one embodiment, the step of correcting scene feature parameters based on density and color values ​​in image data to determine the density field and color field corresponding to the target scene includes:

[0019] Based on the density value, target density feature points are extracted from the image data, and the reprojection error loss of the target density feature points in the world coordinate system is determined.

[0020] Based on the color values, the predicted colors of the three-dimensional sampling points are calculated, and the color error loss between the predicted colors and the color values ​​of the corresponding pixels in the image data is determined.

[0021] The total loss function is determined based on the reprojection error loss and the color error loss, and the hash table entries in the scene visual features and the perception parameters of the decoding multi-layer perception layer are updated based on the total loss function.

[0022] Based on the updated hash table entries and the sensory parameters of the decoded multilayer sensory layer, the density field and color field are determined.

[0023] In one embodiment, the step of determining key 3D objects in a target scene based on scene visual features includes:

[0024] Based on the density field, obtain the depth information map corresponding to the image data;

[0025] Density isosurfaces are extracted from the depth information map to determine multiple density feature regions corresponding to the image data.

[0026] Multiple density feature regions are input into the semantic segmentation network to label each density feature region according to the color field and determine the semantic category label corresponding to each density feature region.

[0027] Based on semantic category labels, key semantic regions in the density feature region are identified, and key semantic regions in the target scene are used as 3D key objects.

[0028] In one embodiment, the step of projecting a virtual object onto a real scene depth map and comparing the depth information of the virtual object with the depth information of the real scene in the real scene depth map to determine the occlusion mask corresponding to each frame of image data includes:

[0029] Obtain the 3D geometric model of the virtual object to be rendered and the preset pose of the virtual object in the world coordinate system;

[0030] Switch the 3D geometric model of the virtual object to the current camera view to generate a depth map of the virtual object and the corresponding pixel coordinate set;

[0031] Read the real depth value at the same location as the pixel coordinate set from the real scene depth map;

[0032] Calculate the depth difference between the virtual depth value of each pixel in the pixel coordinate set and the real depth value of the real scene depth map, pixel by pixel;

[0033] When the actual depth value is less than the virtual depth value and the depth difference exceeds the preset tolerance threshold, the corresponding pixel is marked as the first value; otherwise, it is marked as the second value, and a binarized occlusion mask is generated.

[0034] In one embodiment, the method further includes:

[0035] When the absolute value of the depth difference is less than or equal to the preset tolerance threshold, the transparency weight of the corresponding pixel is calculated according to the transparency function to generate a transparent blending mask;

[0036] Based on the transparency blending mask, the colors of the virtual object are blended and rendered with the colors in the real scene to obtain the target display image.

[0037] Secondly, this application also provides a display screen processing apparatus, which includes:

[0038] The acquisition module is used to acquire multiple frames of image data for the target scene, as well as the pose data corresponding to the image data;

[0039] The perception module is used to perform scene perception on multi-frame image data based on the perceptual coding network to obtain perceptual information in the target scene.

[0040] The first processing module is used to correct the perception information based on the posture data to determine the scene visual features in the target scene, and to determine the three-dimensional key objects in the target scene based on the scene visual features.

[0041] The second processing module is used to identify dynamic objects in multi-frame image data and determine the three-dimensional coordinate information of the dynamic objects in the three-dimensional world coordinate system based on the position information of the dynamic objects in the image data.

[0042] The depth map rendering module is used to render a realistic scene depth map of the target scene based on the 3D coordinate information of key 3D objects and dynamic objects.

[0043] The comparison module is used to project virtual objects onto the depth map of the real scene and compare the depth information of the virtual objects with the depth information of the real scene in the depth map of the real scene to determine the occlusion mask corresponding to each frame of image data; wherein, the occlusion mask is used to represent the positional relationship between the virtual objects and the real scene in any frame of image data for display.

[0044] The image rendering module is used to render the virtual objects in the image data according to the occlusion mask and output the target display screen; wherein, the target display screen is a three-dimensional display screen including virtual objects and real scene.

[0045] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the display screen processing method of the first aspect.

[0046] Fourthly, this application also provides a readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the display screen processing method of the first aspect.

[0047] Fifthly, this application also provides a program product, including a computer program that, when executed by a processor, implements the steps of the display screen processing method of the first aspect.

[0048] The aforementioned display screen processing method, apparatus, computer equipment, readable storage medium, and program product, wherein the display screen processing method includes: acquiring multiple frames of image data collected for a target scene, and pose data corresponding to the image data; performing scene perception on the multiple frames of image data based on a perceptual coding network to obtain perceptual information in the target scene; correcting the perceptual information based on the pose data to determine the scene visual features in the target scene, and determining the three-dimensional key objects in the target scene based on the scene visual features; identifying dynamic objects in the multiple frames of image data, and determining the position of the dynamic objects in the three-dimensional world based on the position information of the dynamic objects in the image data. The system acquires 3D coordinate information in a bounded coordinate system; renders a realistic scene depth map of the target scene based on the 3D coordinate information of key 3D objects and dynamic objects; projects virtual objects onto the realistic scene depth map and compares the depth information of the virtual objects with that of the real scene in the realistic scene depth map to determine the occlusion mask corresponding to each frame of image data; the occlusion mask is used to represent the positional relationship between the virtual objects and the real scene in any frame of image data for display; renders the virtual objects in the image data according to the occlusion mask, and outputs the target display screen; the target display screen is a 3D display screen including virtual objects and the real scene. In this way, by acquiring multiple frames of images and pose data, performing scene perception and correction, determining the coordinates of key 3D objects and dynamic objects, rendering a realistic scene depth map, and generating an occlusion mask based on depth comparison to process virtual object rendering, it can effectively handle the occlusion relationship between virtual and real objects, optimize and enhance the display effect between the real scene and the virtual image, and improve the realism and immersion of the image. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram of the system architecture of the display screen processing method in this embodiment;

[0051] Figure 2 This is a schematic diagram of the first process of the display screen processing method in this embodiment;

[0052] Figure 3This is a schematic diagram of the second process of the display screen processing method in this embodiment;

[0053] Figure 4 This is a schematic diagram of the third process of the display screen processing method in this embodiment;

[0054] Figure 5 This is a schematic diagram of the display screen processing device in one embodiment;

[0055] Figure 6 This is a schematic diagram of the internal structure of a computer device in one embodiment. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0057] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0058] The display screen processing method provided in this application embodiment can be applied to, for example, Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server. Terminal 102 can be a target display device, i.e., an augmented reality virtual device. It can acquire multiple frames of image data collected for a target scene, as well as the corresponding pose data of the image data; perform scene perception on the multiple frames of image data based on a perceptual coding network to obtain perceptual information in the target scene; correct the perceptual information according to the pose data to determine the scene visual features in the target scene, and determine the three-dimensional key objects in the target scene according to the scene visual features; identify dynamic objects in the multiple frames of image data, and determine the three-dimensional coordinate information of the dynamic objects in the three-dimensional world coordinate system according to the position information of the dynamic objects in the image data; render a real scene depth map of the target scene according to the three-dimensional key objects and the three-dimensional coordinate information of the dynamic objects; project virtual objects onto the real scene depth map, and compare the depth information of the virtual objects with the depth information of the real scene in the real scene depth map to determine the occlusion mask corresponding to each frame of image data; wherein, the occlusion mask is used to represent the positional relationship between the virtual objects and the real scene in any frame of image data for screen display; render the screen of the virtual objects in the image data according to the occlusion mask, and output the target display screen; wherein, the target display screen is a three-dimensional display screen including virtual objects and the real scene.

[0059] Specifically, the target display device can be an augmented reality virtual device that includes a acquisition device and a display device, which can output a display screen with virtual objects added when acquiring the target scene.

[0060] Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, and projection equipment. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0061] In augmented reality systems, existing technologies struggle to achieve natural occlusion of virtual content by real objects. Specifically, traditional methods rely on external supervision or depth maps for virtual-real occlusion processing, which can easily lead to geometric distortion in complex scenes. This results in virtual objects not being correctly occluded by real objects based on physical occlusion relationships. Furthermore, inaccuracies in depth relationship modeling prevent the system from dynamically determining the spatial positions of real objects and virtual content, thereby reducing immersive experience and interaction reliability, and impacting the system's applicability in dynamic environments.

[0062] Based on this, this embodiment proposes a display processing method aimed at achieving natural integration and occlusion handling between virtual objects and real scenes, such as... Figure 2 As shown, the method is applied to a target display device and includes steps 21 to 27, wherein:

[0063] Step 21: Acquire multiple frames of image data collected for the target scene, as well as the pose data corresponding to the image data.

[0064] The target scene can refer to the real physical environment in which the user or the target display device is currently located. This target scene is the real basis for overlaying virtual image content in augmented reality.

[0065] Multi-frame image data refers to a sequence of images acquired continuously at different points in time by an image acquisition device. The image data can contain visual information of the target scene. In particular, each frame of multi-frame image data can be a sequence of binocular images, which can include RGB images of the left and right eyes.

[0066] Attitude data refers to the spatial position and orientation information of an image acquisition device when acquiring image data. Attitude data can include the translation and rotation parameters of the acquisition device and can be used to describe the position and orientation of the image acquisition device in three-dimensional space. Attitude data can have the same timestamp as multiple frames of image data.

[0067] It should be noted that image data can be continuously captured by one or more cameras at different points in time. For example, a monocular camera or stereo camera system can be used to acquire image information including the target scene, and an inertial measurement unit (IMU) can be used to collect the current pose data of the target display device. For example, in a scenario where a user is wearing smart glasses, the camera on the smart glasses continuously collects environmental images, while the built-in IMU provides real-time motion posture data of the glasses.

[0068] Step 22: Perform scene perception on multi-frame image data based on the perceptual coding network to obtain perceptual information in the target scene.

[0069] The perceptual coding network is a neural network model used to process image data and extract scene features. This network can learn and encode information useful for scene understanding from raw images, such as geometric structure, texture, or semantic attributes.

[0070] Perceptual information refers to the preliminary understanding or representation of a target scene obtained by a perceptual coding network after processing multiple frames of image data. This information is an intermediate product of determining the visual features of the scene.

[0071] This perceptual coding network can be a deep learning model, such as a convolutional neural network, trained to extract features from images that are related to the geometry, texture, or semantics of a scene. By processing multiple frames of image data, the network can learn and encode a preliminary representation of the scene; for example, it can extract feature vectors of different regions in the scene, which contain an abstract understanding of the scene's content.

[0072] Step 23: Based on the posture data, the perception information is corrected to determine the scene visual features in the target scene, and based on the scene visual features, the three-dimensional key objects in the target scene are determined.

[0073] Scene visual features can be an abstract mathematical expression that fully describes the geometric structure and appearance attributes of the current scene, obtained by correcting and remodeling the perceptual information obtained through real-time perception of the real scene using a lightweight Instant-NeRF reconstruction module. Scene visual features can be a continuous implicit representation based on neural radiation field technology, including density field and color field.

[0074] The density field can be a radiation density function, denoted as σ(x). This function takes the coordinates x of any point in the 3D space of the real scene as input and outputs the radiation density value corresponding to that point. The radiation density in the density field can characterize the degree to which an object occupies that point. Specifically, a higher density value on or inside an object's surface means that light is blocked there; in open areas, the density value approaches zero, indicating that light can pass through freely. Therefore, the radiation density function essentially characterizes the geometric structure of the scene and determines the location information of objects within the target scene.

[0075] Furthermore, a color field can be a color function, denoted as c(x, d). This color function takes the coordinates x of any point in the 3D space of the real scene and the viewing direction d as input, and outputs the color value presented when viewing that point from that direction. Color functions can not only include the texture and material information of an object's surface, but also accurately describe visual effects such as light reflection and highlights that change with the viewing angle by introducing the viewpoint direction parameter. In other words, color functions can be viewpoint-dependent and can be used to represent the appearance information of objects in a target scene.

[0076] One implementation approach is to reconstruct the perception module using lightweight Instant-NeRF, thereby generating updated perception information in real time. Specifically, the image data of the most recent N frames and their corresponding pose data are selected as input using a sliding window approach. Utilizing a compressed multilayer perceptron and hash grid coding technology, through differentiable volume rendering and error backpropagation, the joint optimization and parameter update of the scene within the current field of view of the target scene are completed in a short time. The output is a radiation density function and a color function that only cover the current view frustum and are valid only for the most recent N frames, thus obtaining the scene's visual features.

[0077] In subsequent processing, the radiation density function in the scene visual features is used to extract the 3D mesh, construct the depth buffer, and compare the depth with the virtual object to achieve accurate judgment of virtual and real occlusion; the color function in the scene visual features plays a role in the rendering stage to ensure that the virtual object and the real scene blend naturally in appearance.

[0078] Furthermore, three-dimensional key objects representing key objects in the target scene can be extracted based on the radiation density function (density field) in the scene's visual features.

[0079] In one implementation, the step of determining key 3D objects in a target scene based on scene visual features includes:

[0080] (1) Obtain the depth information map corresponding to the image data based on the density field.

[0081] Among them, the density field is an implicit representation used to describe the volume density information of any point in three-dimensional space, and can be used to reflect the geometry and spatial distribution of objects in a scene.

[0082] Specifically, it can be represented by the volume density function in the neural radiation field, which maps three-dimensional coordinates to a scalar density value. The higher the density value, the greater the probability that there is an entity at that point.

[0083] A depth information map is an image that represents the distance of each pixel in an image to the camera.

[0084] One implementation involves emitting a ray along the viewing direction for each pixel and calculating the depth of the first opaque surface the ray encounters as it passes through the scene, based on the density field. Another implementation sets a threshold in the density field, treating points with densities higher than this threshold as solid surfaces, and calculating the distances from these surface points to the camera.

[0085] (2) Extract density isosurfaces from the depth information map to determine multiple density feature regions corresponding to the image data.

[0086] Density isosurfaces are surfaces in three-dimensional space formed by all points with the same density value. In scene reconstruction, density isosurfaces are often used to approximate the geometric boundaries of objects. Specifically, density isosurfaces can be extracted using the traveling cube algorithm or the traveling tetrahedron algorithm.

[0087] Density feature regions refer to areas in a depth information map that consist of continuous pixels with similar density isosurface features. These regions can be used to represent individual objects or different parts of an object in a target scene. Specifically, connected component analysis or region growing algorithms can be used to identify connected regions with similar depth or density features in the depth information map, thereby determining multiple density feature regions within the depth information map.

[0088] (3) Input multiple density feature regions into the semantic segmentation network to label each density feature region according to the color field and determine the semantic category label corresponding to each density feature region.

[0089] A color field is an implicit representation similar to a density field, used to describe the color information of any point in three-dimensional space. It can be used to reflect the surface texture and color of objects in a scene.

[0090] Specifically, it can be represented by a color function in the neural radiation field, which maps the three-dimensional coordinates and the direction of the gaze to an RGB color value.

[0091] (4) Based on the semantic category label, determine the key semantic regions in the density feature region and take the key semantic regions in the target scene as three-dimensional key objects.

[0092] Semantic category labels are category identifiers assigned to each density feature region by the semantic segmentation network. They indicate the semantic type of the object represented by that region. Semantic category labels can be predefined and used as supervisory labels during the training of the semantic segmentation network. Key semantic regions can refer to regions containing key objects in the target scene among multiple semantic regions.

[0093] Here, the three-dimensional key objects can be those identified and screened in the target scene as having special semantic information and three-dimensional spatial attributes, used for the screen fusion, interaction and screen rendering of virtual objects.

[0094] Step 24: Identify dynamic objects in multi-frame image data, and determine the three-dimensional coordinate information of the dynamic objects in the three-dimensional world coordinate system based on the position information of the dynamic objects in the image data.

[0095] Dynamic objects refer to objects whose position or shape changes over time in the target scene, such as a user's hand, a moving tool, or other active objects.

[0096] As one implementation method, a target detection model can be used to identify moving objects in an image, such as a user's hand or a tool. After identifying a dynamic object in the target scene, the three-dimensional coordinates of the dynamic object in the three-dimensional world coordinate system can be calculated based on the two-dimensional position information of the dynamic object in the image, such as a bounding box or pixel region, combined with the device's pose data and scene depth information.

[0097] Step 25: Render a realistic scene depth map of the target scene based on the 3D coordinate information of the key 3D objects and dynamic objects.

[0098] Among them, the real scene depth map is an image that can be used to represent the distance from each pixel in the target scene to the camera, and is used to provide three-dimensional geometric information of the target scene.

[0099] By projecting the geometric information of identified 3D key objects and dynamic objects in the 3D world coordinate system from the current camera viewpoint onto the 2D image plane, a depth map can be generated to represent the distance from each visible point in the scene to the camera. The real scene depth map can be used to reflect the 3D geometry of the real scene.

[0100] Step 26: Project the virtual object onto the real scene depth map, and compare the depth information of the virtual object with the depth information of the real scene in the real scene depth map to determine the occlusion mask corresponding to each frame of image data.

[0101] Among them, the occlusion mask is used to represent the positional relationship between virtual objects and real scenes in any frame of image data.

[0102] Virtual objects refer to digital content generated by computers and superimposed on a real scene, such as 3D models, animations, or information interfaces. Virtual objects need to be integrated with the real scene and displayed in the correct positional relationship.

[0103] Here, depth information refers to the distance of an object in three-dimensional space relative to the observer or reference plane. In virtual-real fusion, comparing the depth information of virtual objects with that of the real scene can be used to determine the occlusion relationship between key three-dimensional objects in the real scene and virtual objects.

[0104] An occlusion mask can be a binary or multi-valued image used to indicate which areas of a virtual object are occluded by the real scene and which areas are visible in each frame. Occlusion masks can be used to guide the rendering of virtual objects to resemble the real scene.

[0105] In practical applications, the 3D model of the virtual object is first rendered to the current camera's viewpoint, generating its own depth map. This virtual object's depth map is then compared pixel-by-pixel with the depth map of the real scene. If the depth value of the real scene is less than the depth value of the virtual object, it indicates that the real scene object is in front of the virtual object, and that pixel should be marked as occluded. This generates a binary occlusion mask that clearly indicates which parts of the virtual object are occluded by the real scene.

[0106] Step 27: Render the virtual objects in the image data according to the occlusion mask and output the target display screen.

[0107] The target display screen is a three-dimensional display screen that includes virtual objects and real scenes.

[0108] The target display screen refers to the augmented reality screen that is presented to the user after virtual-real fusion rendering, which includes virtual objects and real scenes.

[0109] When rendering virtual objects, occlusion masks guide the rendering process, ensuring that only the unoccluded portions of the virtual object are fully rendered and overlaid onto the real-world scene image. Occluded portions are either not rendered or subjected to special processing. The processed virtual object is then composited with the original real-world scene image to generate a 3D display that blends virtual and real content, which is then output to the user.

[0110] In the above-mentioned display processing method, this embodiment acquires multiple frames of images and pose data, performs scene perception and correction, determines the coordinates of three-dimensional key objects and dynamic objects, renders a real scene depth map, and generates an occlusion mask based on depth comparison to process virtual object rendering. This can effectively handle the relationship between virtual and real occlusion, optimize and enhance the display effect between real scenes and virtual images, and improve the realism and immersion of the image.

[0111] In one exemplary embodiment, such as Figure 3 As shown, the scene visual features include a density field and a color field. The perceptual coding network includes a multi-resolution hash encoder and a decoding multi-layer perceptual layer. Step 23, which involves correcting the perceptual information based on the pose data to determine the scene visual features in the target scene, may include steps 231 to 235, wherein:

[0112] In this context, the density field can be understood as a function or data structure describing the opacity or probability of existence of points in three-dimensional space. In ray propagation models, the density field determines the degree to which light is absorbed or scattered at a point in space, thus affecting that point's contribution to the final pixel color. The density field can be represented as a three-dimensional grid, with each grid point storing a density value; alternatively, it can be a continuous function that maps spatial coordinates through a neural network to output density values.

[0113] Furthermore, a color field describes the color information of each point in three-dimensional space from different viewpoints, defining the color that light may appear at a specific point when it passes through space. A color field can be a three-dimensional grid, with each grid point storing an RGB color value; or it can be a continuous function that outputs color values ​​by mapping spatial coordinates and viewpoint direction through a neural network.

[0114] Multi-resolution hash encoders are efficient spatial encoding techniques that capture scene features at different scales by mapping 3D spatial points to multiple hash tables with different resolutions. Multi-resolution hash encoders can balance memory consumption and feature representation capabilities, accelerating the training and inference processes. For example, they can employ multi-level (e.g., L=16 levels) hash tables, each with different resolutions and feature dimensions; alternatively, they can combine linear interpolation or trilinear interpolation to extract feature vectors from the hash tables, ensuring feature continuity.

[0115] A decoding multilayer perceptron is one or more fully connected neural network layers used to decode feature vectors obtained from a hash encoder into specific scene attributes, such as density and color values. It can transform abstract feature representations into physical quantities that can be used for rendering. For example, a decoding multilayer perceptron can consist of multiple linear layers and activation functions stacked together, with the output layer directly outputting density and color; or it can include an additional view-dependent branch to handle the sensitivity of the color field to changes in viewpoint.

[0116] Step 231: Determine the camera extrinsic parameters corresponding to the image data of the current frame based on the pose data.

[0117] Camera extrinsic parameters describe the camera's position and orientation in the world coordinate system; that is, the amount of rotation and translation in the world coordinate system. They can be used to transform points in the world coordinate system to the camera coordinate system. Specifically, camera extrinsic parameters can be determined in real time through visual inertial odometry or simultaneous localization and mapping (SMR); or they can be calculated through pre-calibration or using image matching algorithms.

[0118] Step 232: Based on the camera extrinsic parameters, project the pixel coordinates of the image data onto the world coordinate system along the line of sight to generate a three-dimensional sampling point sequence.

[0119] The three-dimensional sampling point sequence includes multiple three-dimensional sampling points.

[0120] A 3D sampling point sequence is a series of 3D points obtained by sampling image pixel coordinates along the line of sight from the camera center through the pixel center in the world coordinate system. These points represent discrete positions along the path of light rays passing through the scene. The 3D sampling point sequence can be generated by uniformly sampling N points along each line of sight; alternatively, layered sampling or importance sampling can be used to sample more densely at locations where objects may exist in the scene.

[0121] Step 233: Input multiple 3D sampling points into the multi-resolution hash encoder, query the hash table entries corresponding to the 3D sampling points at each resolution level, and obtain the hash feature vector corresponding to the image data.

[0122] Among them, the hash feature vector extracts the corresponding multi-scale feature representation for each three-dimensional sampling point.

[0123] For each 3D sampling point, world coordinates are input into a multi-resolution hash encoder. The multi-resolution hash encoder can look up the corresponding feature vector in hash tables of different resolutions based on the point's coordinates. Hash table entries store local feature information of the scene. Querying a hash table entry can be done by mapping the 3D coordinates to an index in the hash table using a hash function and reading the feature vector from that index; alternatively, for non-integer coordinates, the feature vector can be calculated from adjacent hash table entries through interpolation.

[0124] Step 234: Input the hash feature vector into the decoding multilayer perceptron to determine the density and color values ​​of the three-dimensional sampling points in the image data.

[0125] Decoding a multilayer perceptron can convert hash feature vectors into density values ​​(representing opacity) and color values ​​(representing the color of that point) for each 3D sampling point through learned mapping relationships. The output layer of the decoding multilayer perceptron can directly predict the density values ​​and RGB color values.

[0126] Step 235: Based on the density and color values ​​in the image data, the scene feature parameters are corrected to determine the density field and color field corresponding to the target scene.

[0127] Specifically, through iterative optimization, the perceptual coding network can more accurately learn and represent the density and color fields of a scene. Specifically, by comparing the predicted density and color values ​​with the actually observed image data, a loss function is calculated, and the parameters of the perceptual coding network are updated using the backpropagation algorithm. The parameters of the perceptual coding network can include hash table entries and perceptual parameters of multiple perceptual layers.

[0128] In one implementation, scene feature parameters can be corrected by updating network parameters by minimizing the pixel-level difference between the rendered image and the real image; alternatively, regularization terms can be combined to smooth the density field and color field, avoiding overfitting.

[0129] In one embodiment, step 235, which involves correcting the scene feature parameters based on the density and color values ​​in the image data to determine the density field and color field corresponding to the target scene, includes:

[0130] (1) Based on the density value, extract the target density feature points in the image data and determine the reprojection error loss of the target density feature points in the world coordinate system.

[0131] Among them, target density feature points refer to regions in a scene that have significant geometric structure or density changes, such as object edges and surface textures. They are important indicators of the scene's geometric structure.

[0132] Target density feature points can be identified by setting a preset density threshold to identify areas with large density gradient changes; or by using edge detection algorithms commonly used in image processing in combination with density information.

[0133] The reprojection error loss measures the difference between the projected positions of target density feature points in 3D space and the actual observed positions of the corresponding feature points in the original image after being projected back to the 2D image plane using camera parameters. This loss function is used to optimize the positions of 3D points in the world coordinate system, making them more consistent with the 2D image observations, thereby ensuring the accuracy of the scene geometry.

[0134] (2) Calculate the predicted color of the three-dimensional sampling point based on the color value, and determine the color error loss between the color value of the corresponding pixel in the image data.

[0135] Among them, predicted color is the color information of a point output by the decoding multilayer perceptual layer in the perceptual coding network based on the position information of the three-dimensional sampling point.

[0136] Color error loss is used to represent the difference between the predicted color and the color of the corresponding pixel in the real image. The color loss function is used to optimize the perceptual coding network's ability to model the scene color field, making its predicted color closer to the actual color of the real scene, thereby ensuring the realism of the scene appearance.

[0137] (3) Determine the total loss function based on the reprojection error loss and color error loss, and update the hash table entries in the scene visual features and the perception parameters of the decoding multi-layer perception layer based on the total loss function.

[0138] The total loss function is usually a weighted sum of reprojection error loss and color error loss. By combining the two losses, density information can be used to optimize the geometry of the scene, and color information can be used to optimize the appearance of the scene.

[0139] (4) Based on the updated hash table entries and the perceptual parameters of the decoded multilayer perceptual layer, determine the density field and color field.

[0140] One implementation approach is to use gradient descent to calculate the gradient of the total loss function with respect to hash table entries and perceptual parameters, and then adjust the hash table entries and perceptual parameters in the opposite direction of the gradient to minimize the total loss function. Here, the hash table entries store feature encodings at different resolutions, while the perceptual parameters are the weights and biases in the decoding multi-layer perceptual layers, jointly determining the scene representation capability of the perceptual coding network. Based on the updated hash table entries and the perceptual parameters of the decoding multi-layer perceptual layers, a more accurate density field and color field can be determined. Therefore, the optimized perceptual coding network can retrieve more accurate density and color values ​​from any 3D sampling point, thereby constructing high-precision scene visual features.

[0141] In this embodiment, by concretizing abstract perceptual information into density and color fields, and constructing a perceptual coding network using a multi-resolution hash encoder and a decoding multi-layer perceptual layer, the 3D geometry and appearance information of the scene can be represented and learned in a more refined and efficient manner. The precise determination of camera extrinsic parameters and the generation of 3D sampling points provide accurate spatial positioning for subsequent feature extraction. The multi-resolution hash encoder can effectively capture scene details at different scales, balancing feature representation capability and computational efficiency. The decoding multi-layer perceptual layer transforms these abstract features into density and color values ​​that can be directly used for rendering. Through iterative correction based on density and color values, a high-precision density and color field can be constructed, improving the accuracy of 3D key object recognition, dynamic object tracking, and the final rendering of a realistic scene depth map, thereby enhancing the realism and immersion of the displayed image.

[0142] In one exemplary embodiment, such as Figure 4 As shown, the biometric parameters also include the refractive power of the object. Step 26, projecting the virtual object onto the real scene depth map and comparing the depth information of the virtual object with the depth information of the real scene in the real scene depth map to determine the occlusion mask corresponding to each frame of image data, may include steps 261 to 265, wherein:

[0143] Step 261: Obtain the 3D geometric model of the virtual object to be rendered and the preset pose of the virtual object in the world coordinate system.

[0144] The three-dimensional geometric model can be a mathematical representation used to describe the shape and size of a virtual object, and can be represented by point clouds, polygon meshes, solid models, or parametric surface models. The three-dimensional geometric model can include vertex coordinates, normal vectors, and texture coordinate information.

[0145] Here, the preset pose can be the position and orientation information of a virtual object in a three-dimensional world coordinate system, which can be represented by a transformation matrix containing rotation and translation components or independent rotation and translation vectors.

[0146] Step 262: Switch the 3D geometric model of the virtual object to the current camera viewpoint and generate the depth map of the virtual object and the corresponding pixel coordinate set.

[0147] The pixel coordinate set can be the set of all pixels occupied by the virtual object when projected onto the image plane from the current camera viewpoint, and each pixel contains its two-dimensional coordinates in the image.

[0148] One implementation involves transforming the 3D geometric model of the virtual object from the world coordinate system to the coordinate system of the current rendering camera, and further projecting it onto the camera's 2D image plane to determine the set of coordinate points in the virtual object. Specifically, this can be achieved through row coordinate transformation and perspective projection based on an intrinsic parameter matrix including camera focal length, principal point, and distortion parameters, and an extrinsic parameter matrix representing the camera's position and orientation in the world coordinate system.

[0149] Step 263: Read the real depth value at the same position as the pixel coordinate set from the real scene depth map.

[0150] Additionally, it can read the real depth value at the same position as the pixel coordinate set from the real scene depth map, and use it to compare with the depth information of each pixel in the virtual object.

[0151] Step 264: Calculate the depth difference between the virtual depth value of each pixel in the pixel coordinate set and the real depth value of the real scene depth map.

[0152] The depth difference can be the difference between the depth value in the depth map of a virtual object and the depth value in the depth map of the real scene at the same image location. The depth difference can be used to reflect the relative front-back relationship between the virtual object and the real scene at that pixel location.

[0153] Step 265: When the actual depth value is less than the virtual depth value and the depth difference exceeds the preset tolerance threshold, the corresponding pixel is marked as the first value; otherwise, it is marked as the second value, and a binarized occlusion mask is generated.

[0154] The preset tolerance threshold can be a pre-set depth difference value, used to handle depth measurement errors, floating-point precision issues, or to allow for slight overlap between virtual objects and the real scene without being misjudged as occlusion.

[0155] At the current pixel position, the real-world object is closer to the camera than the virtual object; that is, the real object is occluding the virtual object.

[0156] It should be noted that the first value and the second value are two different numerical values ​​used to represent the occlusion state. For example, the first value can be 1 (indicating occlusion), and the second value can be 0 (indicating no occlusion). The binarized occlusion mask can be a two-dimensional matrix with the same size as the image, where each pixel has either the first value or the second value. The binarized occlusion mask can be used to clearly indicate whether a virtual object is partially occluded by the real scene in the image, and whether it is partially visible.

[0157] In this embodiment, a refined depth comparison mechanism ensures a high degree of realism in the fusion of virtual objects and the real scene. During depth comparison with the real scene, alignment with the real scene depth map is achieved within the same image space. For each pixel occupied by the virtual object, the corresponding real depth value is read from the previously rendered real scene depth map. By comparing the depth values ​​of the virtual object and the real scene pixel by pixel and calculating the depth difference, the relative front-back relationship between the virtual object and the real scene at each pixel can be accurately determined. Occlusion masks guide the rendering of the virtual object, ensuring that it can be correctly occluded by objects in the real scene, thereby outputting a visually highly integrated and realistic target display.

[0158] Through the above technical solution, this application overcomes the shortcomings of traditional depth comparison methods in handling depth errors in real-world scenes and close contact between virtual objects and real-world scenes. By introducing pixel-by-pixel depth difference calculation and preset tolerance threshold judgment, this solution can more accurately and robustly determine the occlusion relationship between virtual objects and real-world scenes. This ensures that the generated occlusion mask accurately reflects the visibility of virtual objects in the real-world scene, thereby ensuring that virtual objects are correctly occluded by objects in the real-world scene during subsequent rendering. This greatly enhances the realism and visual consistency of the fusion between virtual objects and real-world scenes, providing users with a more immersive 3D display experience.

[0159] In one implementation, the method further includes:

[0160] (1) When the absolute value of the depth difference is less than or equal to the preset tolerance threshold, the transparency weight of the corresponding pixel is calculated according to the transparency function to generate a transparent blending mask.

[0161] The preset tolerance threshold can be set according to the needs of the actual application scenario. For example, it can be a very small distance value or a certain range based on the pixel depth value.

[0162] When comparing the depth information of virtual objects with the depth information of real scenes, if the distance between the two is within a preset very small range, they are considered to be in a close contact or slightly overlapping position.

[0163] When the depth difference falls within this threshold, it indicates that the virtual object and the corresponding part of the real scene are very close in depth. At this time, it is not advisable to simply occlude or fully display the virtual object. Instead, hybrid rendering should be considered to improve the display effect of the image.

[0164] Here, the transparency function is a mathematical function used to map depth differences to transparency weights. It can be used to dynamically calculate the transparency of a virtual object at the current pixel position based on the depth proximity between the virtual object and the real scene.

[0165] It's important to note that the alpha value can be a value between 0 and 1, representing the degree to which the color of a virtual object contributes to the displayed image during the blending rendering process. When the alpha value is 1, the virtual object is completely opaque, and its color will completely cover the color of the real scene; when the alpha value is 0, the virtual object is completely transparent, and the color of the real scene will be fully displayed; when the alpha value is between 0 and 1, the virtual object is semi-transparent, and its color will blend with the color of the real scene.

[0166] To further clarify, the transparency blending mask can be an image or data structure that stores the transparency weight of each pixel. Each pixel value in the transparency blending mask can be a continuous transparency weight, rather than a simple 0 or 1. This mask provides fine-grained transparency information for subsequent blending rendering, allowing virtual objects to blend with the real scene in a semi-transparent or gradient manner.

[0167] For example, the transparency function can be (α = (-k·|D_v - D_r|)).

[0168] Where k is the transparency weight, D_v is the depth information of the virtual object, and D_r is the depth information of the object displayed in the real scene.

[0169] (2) Based on the transparency blending mask, the color of the virtual object is blended with the color in the real scene to obtain the target display screen.

[0170] Based on the transparency weight of each pixel in the transparency blending mask, the color of the virtual object is blended with the color of the corresponding pixel in the real scene to generate the final target display image. Common blending rendering methods include alpha blending, that is, the final color of the target display image = transparency weight * virtual object color + (1 - transparency weight) * real scene color.

[0171] Therefore, through hybrid rendering, a smooth visual transition between virtual objects and real scenes can be achieved, avoiding the abruptness caused by binary occlusion. This is especially suitable for processing semi-transparent virtual objects or scenes where virtual objects are closely attached to real scene objects.

[0172] By introducing a depth difference threshold, calculating transparency weights using a transparency function, and generating a transparency blending mask, the above technical solution enables fine-grained blending rendering when the depth of virtual objects and the real scene is similar. Specifically, during the determination of the occlusion mask, when the absolute value of the depth difference between the virtual object and the real scene is less than or equal to a preset tolerance threshold, the system no longer simply marks the pixel as completely occluded or not occluded. Instead, it calculates a transparency weight between 0 and 1 using a transparency function. This weight is recorded in the transparency blending mask. Subsequently, during the rendering stage, this transparency blending mask is used to weight and blend the colors of the virtual object and the real scene according to their respective transparency weights, thereby generating the final target display image. This processing method allows virtual objects to blend with the real scene in a semi-transparent or gradient manner, greatly enhancing the realism and visual effect of the mixed reality scene.

[0173] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0174] Based on the same inventive concept, this application also provides a display screen processing apparatus for implementing the display screen processing method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more display screen processing apparatus embodiments provided below can be found in the limitations of the display screen processing method described above, and will not be repeated here.

[0175] In one exemplary embodiment, such as Figure 5 As shown, a display image processing device is provided, including: an acquisition module, a sensing module, a first processing module, a second processing module, a depth map rendering module, a comparison module, and an image rendering module, wherein:

[0176] The acquisition module is used to acquire multiple frames of image data for the target scene, as well as the pose data corresponding to the image data;

[0177] The perception module is used to perform scene perception on multi-frame image data based on the perceptual coding network to obtain perceptual information in the target scene.

[0178] The first processing module is used to correct the perception information based on the posture data to determine the scene visual features in the target scene, and to determine the three-dimensional key objects in the target scene based on the scene visual features.

[0179] The second processing module is used to identify dynamic objects in multi-frame image data and determine the three-dimensional coordinate information of the dynamic objects in the three-dimensional world coordinate system based on the position information of the dynamic objects in the image data.

[0180] The depth map rendering module is used to render a realistic scene depth map of the target scene based on the 3D coordinate information of key 3D objects and dynamic objects.

[0181] The comparison module is used to project virtual objects onto the depth map of the real scene and compare the depth information of the virtual objects with the depth information of the real scene in the depth map of the real scene to determine the occlusion mask corresponding to each frame of image data; wherein, the occlusion mask is used to represent the positional relationship between the virtual objects and the real scene in any frame of image data for display.

[0182] The image rendering module is used to render the virtual objects in the image data according to the occlusion mask and output the target display screen; wherein, the target display screen is a three-dimensional display screen including virtual objects and real scene.

[0183] In one embodiment, the first processing module includes an extrinsic parameter determination unit, a projection unit, a query unit, a perception unit, and a correction unit, wherein:

[0184] The extrinsic parameter determination unit is used to determine the camera extrinsic parameters corresponding to the image data of the current frame based on the pose data;

[0185] The projection unit is used to project the pixel coordinates of the image data onto the world coordinate system along the line of sight based on the camera's extrinsic parameters, thereby generating a three-dimensional sampling point sequence; wherein the three-dimensional sampling point sequence includes multiple three-dimensional sampling points;

[0186] The query unit is used to input multiple 3D sampling points into the multi-resolution hash encoder, query the hash table entries corresponding to the 3D sampling points at each resolution level, and obtain the hash feature vector corresponding to the image data.

[0187] The perception unit is used to input the hash feature vector into the decoding multi-layer perception layer to determine the density and color values ​​of the three-dimensional sampling points in the image data.

[0188] The correction unit is used to correct the scene feature parameters based on the density and color values ​​in the image data, so as to determine the density field and color field corresponding to the target scene.

[0189] In one embodiment, the correction unit includes a first error calculation subunit, a second error calculation subunit, a total loss calculation subunit, and an eigenvalue determination subunit, wherein:

[0190] The first error calculation subunit is used to extract target density feature points in image data based on density values ​​and determine the reprojection error loss of target density feature points in the world coordinate system.

[0191] The second error calculation subunit is used to calculate the predicted color of the three-dimensional sampling point based on the color value, and determine the color error loss between the color value and the color value of the corresponding pixel in the image data.

[0192] The total loss calculation subunit is used to determine the total loss function based on the reprojection error loss and color error loss, and update the hash table entries in the scene visual features and the perception parameters of the decoded multi-layer perception layer based on the total loss function;

[0193] The eigenvalue determination subunit is used to determine the density field and color field based on the updated hash table entries and the perceptual parameters of the decoded multilayer perceptual layer.

[0194] In one embodiment, the first processing module further includes a first determining unit, a second determining unit, a marking unit, and a third determining unit, wherein:

[0195] The first determining unit is used to obtain the depth information map corresponding to the image data based on the density field;

[0196] The second determining unit is used to extract density isosurfaces from the depth information map to determine multiple density feature regions corresponding to the image data.

[0197] The labeling unit is used to input multiple density feature regions into the semantic segmentation network to label each density feature region according to the color field and determine the semantic category label corresponding to each density feature region.

[0198] The third determining unit is used to determine the key semantic regions in the density feature region based on the semantic category label, and to take the key semantic regions in the target scene as three-dimensional key objects.

[0199] In one embodiment, the comparison module includes a virtual pose acquisition unit, a depth information determination unit, a depth value reading unit, a pixel-by-pixel comparison unit, and a first mask generation unit, wherein:

[0200] The virtual pose acquisition unit is used to acquire the 3D geometric model of the virtual object to be rendered and the preset pose of the virtual object in the world coordinate system.

[0201] The depth information determination unit is used to switch the three-dimensional geometric model of the virtual object to the current camera viewpoint and generate a depth map of the virtual object and the corresponding pixel coordinate set.

[0202] The depth value reading unit is used to read the real depth value at the same position as the pixel coordinate set from the real scene depth map;

[0203] The pixel-by-pixel comparison unit is used to calculate the depth difference between the virtual depth value of each pixel in the pixel coordinate set and the real depth value of the real scene depth map.

[0204] The first mask generation unit is used to mark the corresponding pixel as the first value when the real depth value is less than the virtual depth value and the depth difference exceeds a preset tolerance threshold, and otherwise mark it as the second value, thereby generating a binarized occlusion mask.

[0205] In one embodiment, the display screen processing apparatus further includes a screen processing module, wherein the screen processing module includes: a second mask generation module and a hybrid rendering module, wherein:

[0206] The second mask generation module is used to calculate the transparency weight of the corresponding pixel according to the transparency function when the absolute value of the depth difference is less than or equal to the preset tolerance threshold, and generate a transparent and transparent hybrid mask.

[0207] The hybrid rendering module is used to blend the colors of virtual objects with the colors in the real scene based on the transparency blending mask to obtain the target display image.

[0208] Each module in the aforementioned display processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0209] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores matrix data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a display screen processing method.

[0210] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0211] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the steps of the above-described display screen processing method.

[0212] In one embodiment, a readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, performs the steps of the above-described display screen processing method.

[0213] In one embodiment, a program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0214] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0215] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0216] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for processing display images, characterized in that, The method includes: Acquire multiple frames of image data collected for the target scene, as well as the pose data corresponding to the image data; Based on a perceptual coding network, scene perception is performed on multiple frames of the image data to obtain perceptual information in the target scene; Based on the posture data, the perceived information is corrected to determine the scene visual features in the target scene, and based on the scene visual features, the three-dimensional key objects in the target scene are determined; wherein, the scene visual features are a continuous implicit representation based on neural radiation field technology, including a density field and a color field, the radiation density in the density field is used to characterize the degree to which any spatial point in the three-dimensional space where the real scene is located is occupied by an object, and the color field has viewpoint correlation and is used to represent the appearance information of objects in the target scene; Identify dynamic objects in multiple frames of image data, and determine the three-dimensional coordinate information of the dynamic objects in the three-dimensional world coordinate system based on the position information of the dynamic objects in the image data; Based on the three-dimensional coordinate information of the key three-dimensional objects and the dynamic objects, render a realistic scene depth map of the target scene; The virtual object is projected onto the depth map of the real scene, and the depth information of the virtual object is compared with the depth information of the real scene in the depth map of the real scene to determine the occlusion mask corresponding to each frame of the image data; wherein, the occlusion mask is used to represent the positional relationship between the virtual object and the real scene in any frame of image data for display. The image data is rendered based on the occlusion mask to output a target display screen; wherein the target display screen is a three-dimensional display screen including the virtual object and the real scene.

2. The method according to claim 1, characterized in that, The perceptual coding network includes a multi-resolution hash encoder and a decoding multi-layer perceptual layer. The step of correcting the perceptual information based on the pose data to determine the scene visual features in the target scene includes: Based on the pose data, determine the camera extrinsic parameters corresponding to the image data of the current frame; Based on the camera extrinsic parameters, the pixel coordinates of the image data are projected onto the world coordinate system along the line of sight to generate a three-dimensional sampling point sequence; wherein, the three-dimensional sampling point sequence includes multiple three-dimensional sampling points; The multiple three-dimensional sampling points are input into the multi-resolution hash encoder, and the hash table entries corresponding to the three-dimensional sampling points at each resolution level are queried to obtain the hash feature vector corresponding to the image data. The hash feature vector is input into the decoding multilayer perceptron to determine the density and color values ​​of the three-dimensional sampling points in the image data; Based on the density and color values ​​in the image data, the scene feature parameters are corrected to determine the density field and color field corresponding to the target scene.

3. The method according to claim 2, characterized in that, The step of correcting the scene feature parameters based on the density value and color value in the image data to determine the density field and color field corresponding to the target scene includes: Based on the density value, target density feature points are extracted from the image data, and the reprojection error loss of the target density feature points in the world coordinate system is determined. Based on the color value, calculate the predicted color of the three-dimensional sampling point and determine the color error loss between it and the color value of the corresponding pixel in the image data; The total loss function is determined based on the reprojection error loss and the color error loss, and the hash table entries in the scene visual features and the perception parameters of the decoding multilayer perception layer are updated based on the total loss function. Based on the updated hash table entries and the perception parameters of the decoded multilayer perception layer, the density field and the color field are determined.

4. The method according to claim 1, characterized in that, The step of determining the three-dimensional key objects in the target scene based on the scene visual features includes: Based on the density field, obtain the depth information map corresponding to the image data; Density isosurfaces are extracted from the depth information map to determine multiple density feature regions corresponding to the image data; Multiple density feature regions are input into a semantic segmentation network to label each density feature region according to the color field and determine the semantic category label corresponding to each density feature region. Based on the semantic category label, the key semantic regions in the density feature region are determined, and the key semantic regions in the target scene are used as three-dimensional key objects.

5. The method according to claim 1, characterized in that, The step of projecting a virtual object onto the real scene depth map and comparing the depth information of the virtual object with the depth information of the real scene in the real scene depth map to determine the occlusion mask corresponding to each frame of image data includes: Obtain the three-dimensional geometric model of the virtual object to be rendered and the preset pose of the virtual object in the world coordinate system; Switch the three-dimensional geometric model of the virtual object to the current camera viewpoint to generate a depth map of the virtual object and the corresponding pixel coordinate set; Read the real depth value at the same position as the pixel coordinate set from the real scene depth map; Calculate the depth difference between the virtual depth value of each pixel in the pixel coordinate set and the real depth value of the real scene depth map, pixel by pixel; When the actual depth value is less than the virtual depth value and the depth difference exceeds a preset tolerance threshold, the corresponding pixel is marked as the first value; otherwise, it is marked as the second value, thus generating a binarized occlusion mask.

6. The method according to claim 5, characterized in that, The method further includes: When the absolute value of the depth difference is less than or equal to a preset tolerance threshold, the transparency weight of the corresponding pixel is calculated according to the transparency function to generate a transparent blending mask; Based on the transparency blending mask, the color of the virtual object is blended and rendered with the color in the real scene to obtain the target display image.

7. A display screen processing device, characterized in that, The device includes: The acquisition module is used to acquire multiple frames of image data for the target scene, as well as the pose data corresponding to the image data; The perception module is used to perform scene perception on multiple frames of image data based on a perceptual coding network to obtain perceptual information in the target scene; The first processing module is used to correct the perceived information based on the posture data to determine the scene visual features in the target scene, and to determine the three-dimensional key objects in the target scene based on the scene visual features; wherein, the scene visual features are a continuous implicit representation based on neural radiation field technology, including a density field and a color field, the radiation density in the density field is used to characterize the degree to which any spatial point in the three-dimensional space where the real scene is located is occupied by an object, and the color field has viewpoint correlation and is used to represent the appearance information of the objects in the target scene; The second processing module is used to identify dynamic objects in multiple frames of image data, and determine the three-dimensional coordinate information of the dynamic objects in the three-dimensional world coordinate system based on the position information of the dynamic objects in the image data. The depth map rendering module is used to render a real scene depth map of the target scene based on the three-dimensional coordinate information of the three-dimensional key objects and the dynamic objects. A comparison module is used to project a virtual object onto the real scene depth map and compare the depth information of the virtual object with the depth information of the real scene in the real scene depth map to determine the occlusion mask corresponding to each frame of the image data; wherein, the occlusion mask is used to represent the positional relationship between the virtual object and the real scene in any frame of image data for display. The image rendering module is used to render the virtual object in the image data according to the occlusion mask and output the target display image; wherein, the target display image is a three-dimensional display image including the virtual object and the real scene.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images