Three-dimensional model intelligent display method and system

Abstract

The invention discloses a three-dimensional model intelligent display method and system, and relates to the technical field of model display, and the method comprises the steps: obtaining geometric data, image data and environment illumination parameters of a displayed product, and the position, sight focus and electroencephalogram signal of a user, and generating a multi-modal three-dimensional display data set; the method comprises the following steps: constructing NeRF, training the NeRF through a multi-view stereoscopic vision algorithm, generating an initial three-dimensional model, and generating a real-time three-dimensional model by positioning an update area, performing incremental training on the NeRF and combining PBR rendering; and the hierarchical display priority of the real-time three-dimensional model is adjusted by calculating the sight line retention time and the electroencephalogram attention. According to the method, the updating area is positioned, incremental training is carried out, the real-time three-dimensional model is generated in combination with PBR rendering, and efficient updating and realistic presentation of the model in a dynamic environment are ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of model display, and in particular to a three-dimensional model intelligent display method and system. BACKGROUND

[0002] In recent years, three-dimensional modeling and interactive display technology has made significant progress in virtual reality, augmented reality and product display fields. The three-dimensional reconstruction method based on neural radiation field (NeRF) generates high-precision three-dimensional models through multi-view images and depth data, and is widely used in industrial design, e-commerce and cultural heritage digitization fields. The existing technology usually uses RGB-D cameras to collect multi-view images, combines depth information to generate point clouds, and constructs three-dimensional models through volume density field optimization. In addition, interactive technologies based on eye tracking and brain-computer interfaces are gradually emerging, which realize human-computer interaction by capturing user gaze focus and electroencephalogram signals, and enhance user experience. These technologies provide high realism and interactivity for product display, laying a solid foundation.

[0003] However, the existing three-dimensional modeling and interactive technology still has significant deficiencies in practical application. First, the traditional NeRF method often lacks feature extraction when dealing with complex geometric structures, resulting in missing model details, especially in areas with large curvature changes, making it difficult to accurately express local geometric features. Second, the existing interactive technology relies on single modal data (such as gaze or electroencephalogram), lacks multi-modal data fusion mechanism, and cannot dynamically adjust the display content according to user attention, resulting in insufficient interactive accuracy and immersion. These deficiencies limit the display effect and user interaction experience of three-dimensional models in dynamic environments, especially in scenarios that require real-time updating and personalized display, an integrated solution of multi-modal fusion and adaptive adjustment is urgently needed. SUMMARY

[0004] In view of the above existing problems, the present application is proposed.

[0005] Therefore, the present application provides a three-dimensional model intelligent display method to solve the problem of insufficient geometric detail expression.

[0006] To solve the above technical problems, the present application provides the following technical solutions: In a first aspect, the present application provides a three-dimensional model intelligent display method, which comprises: acquiring geometric data, image data and environmental lighting parameters of a product to be displayed, and the position, visual focus and electroencephalogram signal of a user, generating a multi-modal three-dimensional display data set; extracting multi-scale features of the multi-modal three-dimensional display data set, and adaptively adjusting a fusion weight combined with the visual focus of the user and the electroencephalogram signal of the user, to generate a fusion feature vector; constructing a NeRF, training the NeRF through a multi-view stereo vision algorithm, generating an initial three-dimensional model, locating an update area by searching for a point cloud area closest to the visual focus of the user in the initial three-dimensional model, incrementally training the NeRF, and generating a real-time three-dimensional model combined with PBR rendering; determining an interaction trigger condition combined with a visual focus displacement threshold and an electroencephalogram attention threshold through calculation of visual dwell time and electroencephalogram attention, and adjusting a hierarchical display priority of the real-time three-dimensional model through division of hierarchical display areas.

[0007] As a preferred scheme of the three-dimensional model intelligent display method, the multi-scale features of the multi-modal three-dimensional display data set are extracted through the following specific steps: detecting key points in the image data to generate low-scale features; performing illumination intensity normalization processing on a multi-view image sequence in the image data, extracting middle-scale features through depth convolution and global pooling; selecting a user with the smallest distance from the geometric center of the product to be displayed as a main user, and taking the position of the main user as a main user viewpoint; generating an initial point cloud based on the main user viewpoint, and extracting high-scale features through construction of a dynamic graph and multi-layer convolution.

[0008] As a preferred scheme of the three-dimensional model intelligent display method, the fusion weight is adaptively adjusted through the following specific steps: calculating a relative distance from the visual focus coordinate of the main user to the product to be displayed, and performing normalization to obtain a normalized distance score of each scale feature; performing fast Fourier transform on the electroencephalogram signal of the main user, and calculating an attention index according to the spectral power distribution of the electroencephalogram signal of the main user; and calculating a dynamic fusion weight of each scale feature according to the normalized distance score and the attention index of each scale feature.

[0009] As a preferred scheme of the three-dimensional model intelligent display method, the relative distance, for low-scale features, represents the Euclidean distance from the visual focus coordinate of the main user to the geometric edge of the product to be displayed; for middle-scale features, the relative distance represents the distance from the visual focus coordinate of the main user to the surface center area of the bounding box of the product to be displayed; and for high-scale features, the relative distance represents the Euclidean distance from the visual focus coordinate of the main user to the geometric center of the product to be displayed.

[0010] As a preferred scheme of the intelligent three-dimensional model display method, the NeRF is trained by a multi-view stereo algorithm, and the specific steps are as follows, For each multi-view image in the image data, a plurality of rays are randomly sampled, and a plurality of points are uniformly sampled along the rays; the coordinates of the sampling points and the fusion feature vectors are spliced into vectors, and then input into the NeRF to output the predicted values of the volume density and color of each sampling point; a rendering loss function is defined by comparing the mean square error of the pixel color predicted by the NeRF and the pixel color of the real multi-view image, and adding a regularization term of the predicted volume density; the Adam optimizer is used to update the weight matrix of the NeRF by iterating the image data of the displayed image.

[0011] As a preferred scheme of the intelligent three-dimensional model display method, the NeRF is trained by a multi-view stereo algorithm, and the specific steps are as follows, The point cloud region closest to the user's line of sight focus in the initial three-dimensional model is found as the update region; it is detected whether there is new image data of the update region in the image data; the NeRF is incrementally trained to update the NeRF weight.

[0012] As a preferred scheme of the intelligent three-dimensional model display method, the PBR rendering is performed, and the specific steps are as follows, According to the ambient light parameters, the PBR ambient light coefficient is set; the pixel color is extracted from the image data of the displayed product, and is mapped to the vertices of the three-dimensional model based on the camera projection matrix; the diffuse reflection coefficient and the specular reflection coefficient are initialized according to the geometric data, the texture map of the displayed product is generated, and is bound to the three-dimensional model to obtain the real-time three-dimensional model.

[0013] In a second aspect, the present application provides an intelligent three-dimensional model display system, comprising a multi-modal data module, a feature fusion module, a three-dimensional model module and a hierarchical display module, The multi-modal data module is used to obtain the geometric data, image data and ambient light parameters of the displayed product, and the position, line of sight focus and electroencephalogram signal of the user, and generate a multi-modal three-dimensional display data set; The feature fusion module is used to extract multi-scale features of the multi-modal three-dimensional display data set, and adaptively adjust the fusion weight in combination with the user's line of sight focus and the user's electroencephalogram signal to generate a fusion feature vector; The three-dimensional model module is used to construct the NeRF, train the NeRF by a multi-view stereo algorithm, generate an initial three-dimensional model, locate the update region by finding the point cloud region closest to the user's line of sight focus in the initial three-dimensional model, incrementally train the NeRF, and generate a real-time three-dimensional model by PBR rendering; The layered display module is used to determine the interaction triggering conditions by calculating the gaze dwell time and EEG attention, and combining the gaze focus displacement threshold and EEG attention threshold. By dividing the layered display area, the layered display priority of the real-time 3D model is adjusted.

[0014] Thirdly, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements any step of the intelligent display method for three-dimensional models as described in the first aspect of the present invention.

[0015] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the intelligent display method for three-dimensional models as described in the first aspect of the present invention.

[0016] The beneficial effects of this invention are as follows: This invention extracts multi-scale features based on the geometric data, image data and ambient lighting parameters of the product being displayed, generates a fused feature vector, enhances NeRF's ability to express complex geometric structures, and generates a real-time 3D model by locating and updating the region and performing incremental training, combined with PBR rendering, ensuring efficient updating and realistic presentation of the model in dynamic environments. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Fig. 1 A flowchart for an intelligent display method for 3D models.

[0019] Fig. 2 A module diagram for an intelligent 3D model display system.

[0020] Fig. 3 A flowchart for incremental updates and PBR rendering.

[0021] Fig. 4 Adjust the flowchart to prioritize the hierarchical display. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without the specific details set forth in this description. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the present application.

[0024] Second, the "one embodiment" or "an embodiment" referred to herein means a specific feature, structure, or characteristic under discussion. Thus, "one embodiment" does not mean a single embodiment nor is it to be taken to be an alternative to, or mutually exclusive of, the other embodiments described herein.

[0025] Reference will now be made to Figs. 1-4 For one embodiment of the present application, the embodiment provides a three-dimensional model intelligent display method, comprising the following steps: S1, acquiring the geometric data, image data and environmental lighting parameters of the product to be displayed, and the position, visual focus and electroencephalogram signal of the user, and generating a multi-modal three-dimensional display data set.

[0026] According to the shape and size of the product to be displayed, the geometric data of the product to be displayed is extracted, including geometric parameters (size, curvature) and material properties (reflectivity), the image data of the product to be displayed is collected by an RGB-D camera such as Azure Kinect DK, including 360° multi-view images and depth maps (one frame every 8°), the environmental lighting parameters are obtained by using a light sensor to collect the lighting intensity in real time, the number of users and the distance from each user to the geometric center of the product to be displayed are detected by an infrared camera to reflect the user position, an eye tracking instrument is used to capture the visual focus of the user, and a brain-computer interface device is integrated to record the electroencephalogram signal of the user; The various types of data are time-stamped synchronized and Gaussian filtered to remove noise, and a multi-modal three-dimensional display data set is generated. Specifically, a time-stamp alignment algorithm is used to synchronize all data, Gaussian filtering (kernel size 5x5, standard deviation 1.0) is applied to the image data of the product to be displayed to remove noise, Gaussian filtering (kernel size 3x3, standard deviation 0.5) is applied to the user's electroencephalogram signal to smooth the waveform of the user's electroencephalogram signal, one-dimensional Gaussian filtering is applied to the geometric data to smooth the fluctuations of the geometric data such as curvature changes, one-dimensional Gaussian filtering is applied to the environmental lighting parameters to remove lighting mutations, three-dimensional Gaussian filtering is applied to smooth the jitter of the user's position, two-dimensional Gaussian filtering is applied to smooth the offset of the user's visual focus, and all data that has been time-stamped synchronized and denoised is integrated into a unified multi-modal three-dimensional display data set.

[0027] S2, multi-scale features of the multi-modal three-dimensional display data set are extracted, and the fusion weight is adaptively adjusted in combination with the user's visual focus and the user's electroencephalogram signal to generate a fusion feature vector.

[0028] According to the mean value and standard deviation of the curvature value in the geometric parameters of the displayed product, a curvature threshold is set, and geometric parameters and corresponding three-dimensional curvature coordinates with a curvature value greater than the curvature threshold are extracted; the curvature coordinates are projected by using the internal and external parameters of the RGB-D camera, the key area is cropped, and a key image set is obtained; Specifically, according to the external parameters (rotation matrix, translation vector) of the RGB-D camera, the rigid body transformation formula is used to transform the three-dimensional curvature coordinates with a curvature value greater than the curvature threshold from the world coordinate system to the camera coordinate system, the internal parameters (focal length, principal point) of the RGB-D camera are used, and the pinhole camera projection method is used to project the three-dimensional curvature coordinates in the camera coordinate system to the image plane to generate two-dimensional pixel coordinates with a curvature value greater than the curvature threshold, and a rectangular area with a side length of, such as 50 pixels, is defined as a key area on the basis of the two-dimensional pixel coordinates in each frame of image data of the displayed product, and the key area is cropped to obtain a key image set; The BRISK algorithm is initialized, including the number of octaves and the scale factor; the key image set is converted into a key grayscale image set by using the OpenCV grayscale conversion method such as cv2.cvtColor; and the key points in the key grayscale image set are detected by using the BRISK algorithm to generate a key point coordinate set; Specifically, the BRISK algorithm is used to downsample each frame of key grayscale image from the original resolution to generate four octaves, and two intermediate scales are generated in each octave by Gaussian blur to construct a scale space pyramid and generate a multi-scale grayscale image set. For 12 scale images (4 octaves, 3 scales in each octave) in each frame of image in the multi-scale grayscale image set, the second-order partial derivative of each pixel point is calculated by using the Sobel operator to obtain the Hessian response value of the pixel point. According to the mean value and standard deviation of the Hessian response value, a Hessian threshold is set, and the pixel points with a Hessian response value greater than the Hessian threshold are used as candidate key points. In a 3x3 neighborhood of each octave, the non-maximum suppression method is used to compare the Hessian response values of the candidate key points, and only the candidate key point with the maximum value in the 3x3 neighborhood is reserved as the final key point. The number of final key points in each frame is limited to not more than 4000, the final key points with the highest Hessian response value are preferentially retained, the two-dimensional pixel coordinates of the final key points are summarized, and a key point coordinate set is generated; For each frame of image in the key grayscale image set, a circular sampling region is defined with each final key point in the key point coordinate set as the center and a default radius of 15 pixels. In the sampling region, 60 sampling points predefined by the BRISK algorithm are selected and distributed in 4 concentric circles with radii of 2, 5, 10 and 15 pixels respectively to form a pair of sampling points such as point A and point B. The grayscale values of the two points are compared to obtain a binary descriptor set (0 or 1). If the grayscale value of point A is greater than that of point B, then 1 is obtained, otherwise 0 is obtained. The key point coordinate set and the descriptor set are aligned by linear interpolation to generate low-scale features; For multi-view images, the time-stamped ambient light parameters are matched to associate the corresponding light intensity with each frame of multi-view image. If a frame of multi-view image has no corresponding light value, the light intensity is estimated based on the light intensity of adjacent time-stamped images by linear interpolation to generate a multi-view image sequence, each frame of which contains multi-view images and light intensity; The pixel value scaling method based on light intensity is used to normalize the multi-view image sequence, and the calculation formula is as follows: ; wherein, is the normalized pixel value, is the pixel value of the multi-view image, is the light intensity; The bilinear interpolation method is applied to calculate the weighted average value of adjacent pixels in the normalized multi-view image sequence to resample the multi-view image pixels of the displayed product, maintain the texture smoothness, and adjust the resolution to adapt to the ConvNeXt-T network; The multi-view image sequence is input into the pre-trained ConvNeXt-T network. For each input image, the forward propagation of the pre-trained ConvNeXt-T network is performed to extract the global average pooling layer output and generate a single-frame feature vector set. The single-frame feature vector set is averaged by pooling, i.e. the mean value is calculated dimension by dimension, and the medium-scale features are obtained; The user with the minimum distance to the geometric center of the displayed product is selected as the main user, and the position of the main user is extracted as the main user's viewpoint; The median filter is used to denoise the depth map of the displayed product. According to the intrinsic and extrinsic parameters of the RGB-D camera, the camera projection formula is applied to convert the pixel coordinates in the depth map of the displayed product into three-dimensional coordinates. Meanwhile, the main user's viewpoint is defined as the origin, and the main user's viewing angle range is defined, such as a spherical region with a radius of 2m. The three-dimensional coordinates beyond the main user's viewing angle range are removed to generate an initial point cloud; The initial point cloud is normalized and randomly sampled based on the uniform distribution of the point cloud to obtain a set of sampled point clouds which are converted into tensor format to generate a set of point cloud tensors; The point cloud tensor set is input into the PointNeXt network for feature extraction, generating a single-frame point cloud feature vector. Specifically, the topological relationships (adjacency matrix, 16 neighbors) between point cloud tensors are generated through the dynamic graph construction (k nearest neighbors, k=16, search radius 0.4) of the PointNeXt network. Three layers of dynamic graph convolution are used, with ReLU activation function and BatchNorm normalization in each layer. Global pooling is used to aggregate the features of each frame in the point cloud tensor set into a single frame point cloud feature vector. High-scale features are generated by merging single-frame point cloud feature vectors through max pooling. Specifically, max pooling is performed on all single-frame point cloud feature vectors in the point cloud tensor set to extract the maximum value in each dimension and merge them into high-scale features. Global average pooling is applied to low-scale features to reduce their dimensionality, while linear transformations are applied to mid-scale and high-scale features to align their dimensions. Calculate dynamic fusion weights based on the user's gaze focus and EEG signals; Specifically, the relative distance from the main user's gaze focus coordinates to the displayed product is calculated. For low-scale features, the relative distance represents the Euclidean distance from the main user's gaze focus coordinates to the geometric edge of the displayed product. For mid-scale features, the relative distance represents the Euclidean distance from the main user's gaze focus coordinates to the boundary of the surface center region of the displayed product's bounding box (e.g., a spherical region with a radius of 5 cm centered at the geometric center). For high-scale features, the relative distance represents the Euclidean distance from the main user's gaze focus coordinates to the geometric center of the displayed product. The relative distance is then normalized to obtain a normalized distance score. If the relative distance is greater than the maximum distance reference value, the normalized distance score is 0. Bandpass filtering was applied to each channel of the main user's EEG signal to remove low-frequency drift and high-frequency noise. Fast Fourier Transform was performed on the EEG signal to convert the time-domain EEG signal into a frequency-domain EEG signal, generating the spectral power distribution of the main user's EEG signal. The average alpha wave power and average beta power were extracted, the ratio of average alpha wave power to average beta power was calculated, and the Sigmoid function was used for normalization to obtain the main user's attention index. Based on the normalized distance score and attention metric, the dynamic fusion weights of features at each scale are calculated as follows: ; in, For the first The fusion weights of features at each scale Indicates the first Normalized distance scores for each scale feature attention indicator; According to the dynamic fusion weight of each scale feature, the low, medium and high scale features are weighted and fused to generate a fusion feature vector.

[0029] S3, constructing NeRF, training NeRF through multi-view stereo algorithm, generating an initial three-dimensional model, locating the update area by searching the point cloud area closest to the user's line of sight focal point in the initial three-dimensional model, incrementally training NeRF, and generating a real-time three-dimensional model combined with PBR rendering.

[0030] Configure the neural network structure of NeRF (Neural Radiance Field); Specifically, an 8-layer multilayer perceptron is used, each layer has 128-dimensional hidden neurons, the input is processed using a sine activation function, the weight matrix of each layer of multilayer perceptron is initialized using the Xavier method, and the bias term is initially set to 0, and the construction of NeRF is completed; According to the intrinsic and extrinsic parameters of the RGB-D camera, the pixel depth value in the depth map of the product to be displayed is converted into a three-dimensional coordinate by applying the camera projection formula, and uniform sampling is performed, for each sampling point, the three-dimensional coordinate and the fusion feature vector are vector spliced, and input into NeRF, which enhances the expression ability of NeRF to scene geometry and semantics, outputs the volume density of each sampling point, and performs average pooling to generate a unified volume density distribution, and obtains the initial volume density field; Train NeRF through a multi-view stereo algorithm to generate an initial three-dimensional model; For each multi-view image, 10,000 rays are randomly sampled (from the camera center through the pixel point), and 64 points are uniformly sampled along each ray, and after the sampling point coordinates and the fusion feature vector are vector spliced, they are input into NeRF to predict the volume density and color of each sampling point. By comparing the mean square error of the pixel color predicted by NeRF and the true multi-view image pixel color, and adding a volume density regularization term, a rendering loss function is defined, and the calculation formula is as follows: ; ; Wherein, is the rendering loss value, represents the true RGB pixel color of the light, represents the color predicted by NeRF, is the regularization coefficient, is the predicted volume density, which represents the opacity of a three-dimensional space point, is the cumulative transmittance, which represents the transmission probability of the light from the proximal end to the point , is the volume density of the point , For the point of color, is a point on the light ray, indicating the distance along the light ray, is the near end distance of the light ray, indicating the minimum distance of the light ray into the scene, is the far end distance of the light ray, indicating the maximum distance of the light ray out of the scene, is a small distance interval of light ray integration, indicating the step size of sampling along the light ray for numerical integration; Using the Adam optimizer, iteratively update the weight matrix of NeRF using the image data of the displayed image; According to the stereoscopic density distribution of the initial stereoscopic density field, set the stereoscopic density threshold, filter the predicted stereoscopic density of each sampling point, only keep the sampling points with stereoscopic density greater than the stereoscopic density threshold, and collect the three-dimensional coordinates of the retained sampling points to arrange them into a point cloud set, and remove the isolated points in the point cloud set through voxel grid filtering; For each point cloud in the point cloud set, use the nearest neighbor search method to find the nearest neighbor points, construct a covariance matrix of the neighbor points, use principal component analysis to calculate the normal vector of each point cloud, and merge the three-dimensional coordinates and normal vectors of each point cloud one by one to generate point cloud vertex data, and generate an initial three-dimensional model of the displayed product by writing the point cloud vertex data into a PLY file; Using KNN algorithm, find the point cloud area closest to the user's line of sight focal point in the initial three-dimensional model as the update area (such as radius 5CM), record the index of the point cloud in the update area, generate an update area dataset, detect whether there is new image data (multi-view image or depth map) of the update area in the image data of the displayed product, perform incremental training on NeRF, update the NeRF weight, only adjust the update area, generate a locally optimized stereoscopic density field, replace the update area point cloud of the initial three-dimensional model, and obtain a locally optimized three-dimensional model; Using PBR rendering pipeline, setting PBR environment light coefficient according to environment light parameters, extracting pixel color from image data of the displayed product, mapping to the vertex of the locally optimized three-dimensional model based on the camera projection matrix, initializing diffuse reflection coefficient and specular reflection coefficient according to geometric data (curvature and material properties, etc.), generating texture map of the displayed product, and binding to the locally optimized three-dimensional model to generate real-time three-dimensional model of the displayed product.

[0031] S4, determine the interaction trigger condition by calculating the line of sight dwell time and the brain electrical attention, combining the line of sight focal point displacement threshold and the brain electrical attention threshold, and adjusting the hierarchical display priority of the real-time three-dimensional model by dividing the hierarchical display area.

[0032] Calculate the line of sight dwell time and the brain electrical attention to generate an interaction trigger dataset; According to the spatial displacement of the user's visual focus and the size of the real-time three-dimensional model, a visual focus displacement threshold is set, the Euclidean distance of the visual focus corresponding to adjacent time stamps is calculated, and if it is less than the visual focus displacement threshold, it is determined that the two visual focuses are in the same stay area, the sequence of visual focuses that meet the same stay area condition is recorded continuously, and is marked as a stay area. The visual stay time of each stay area, i.e. the start and end time difference of the stay area, is calculated; For each stay area, the average EEG attention in the visual stay time is calculated according to the number of EEG signal records and the attention index of the user; By counting the user's visual stay time, the median of the user's visual stay time distribution is set as the visual stay time threshold, and based on the statistical distribution of the attention index of each user, the attention index threshold is set. For example, according to the mean and standard deviation of the user's attention index, the three-sigma principle is used to set the attention index threshold; When the visual stay time is greater than the visual stay time threshold and the attention index is greater than the attention index threshold, it is determined that the interaction trigger condition is met, and the trigger state is set to true, otherwise the trigger state is set to false; When the trigger state is true, the hierarchical display area is divided by K-means clustering, and the hierarchical display of the real-time three-dimensional model is adjusted; Specifically, the K-means clustering algorithm is used to cluster the point cloud vertex coordinates of the real-time three-dimensional model. According to the complexity of the real-time three-dimensional model, the number of clusters is set, and a number of point clouds are randomly selected from the real-time three-dimensional model as initial cluster centers. The Euclidean distance of each point cloud in the real-time three-dimensional model from the 10 cluster centers is calculated, and each point cloud is assigned to the cluster center with the smallest Euclidean distance. The cluster centers are updated by recalculating the coordinate mean of all point clouds in each cluster. Through multiple iterations, evenly distributed clusters are generated. The point cloud coordinates closest to the user's visual focus are found and filtered from the clusters. The cluster with the highest frequency of occurrence is selected as the active area, and the point clouds in the active area are assigned the highest display priority. The display priority of the remaining clusters is assigned according to the frequency of the point cloud coordinates closest to the user's visual focus.

[0033] The embodiment also provides a three-dimensional model intelligent display system, comprising a multi-modal data module, a feature fusion module, a three-dimensional model module and a hierarchical display module, the multi-modal data module is used for acquiring geometric data, image data and environmental illumination parameters of a product to be displayed, and position, visual focus and electroencephalogram signals of a user, and generating a multi-modal three-dimensional display data set; the feature fusion module is used for extracting multi-scale features of the multi-modal three-dimensional display data set, and adaptively adjusting a fusion weight in combination with the visual focus of the user and the electroencephalogram signals of the user, and generating a fusion feature vector; the three-dimensional model module is used for constructing a NeRF, training the NeRF through a multi-view stereo vision algorithm, generating an initial three-dimensional model, locating an update area by searching for a point cloud area closest to the visual focus of the user in the initial three-dimensional model, incrementally training the NeRF, and generating a real-time three-dimensional model in combination with PBR rendering; and the hierarchical display module is used for determining an interactive trigger condition by calculating visual dwell time and electroencephalogram attention in combination with a visual focus displacement threshold and an electroencephalogram attention threshold, and adjusting a hierarchical display priority of the real-time three-dimensional model by dividing hierarchical display areas.

[0034] The embodiment also provides a computer device suitable for the three-dimensional model intelligent display method, comprising a memory and a processor; the memory is used for storing computer executable instructions, and the processor is used for executing the computer executable instructions to implement the three-dimensional model intelligent display method proposed in the above embodiment.

[0035] The computer device can be a terminal, and the computer device comprises a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. The processor of the computer device is used for providing computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used for wired or wireless communication with external terminals. The wireless communication can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0036] The embodiment also provides a storage medium having a computer program stored thereon, the program being executed by a processor to implement the method for intelligently displaying a three-dimensional model proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, or an optical disk.

[0037] To sum up, the method for intelligently displaying a three-dimensional model is implemented by extracting geometric data, image data, and environmental illumination parameters of a displayed product, combining a BRISK algorithm and a ConvNeXt-T network to extract multi-scale features, generating a fusion feature vector, enhancing the expression capability of NeRF for a complex geometric structure, positioning an update area by using a KNN algorithm and performing incremental training, and generating a real-time three-dimensional model by combining PBR rendering, so that efficient updating and realistic presentation of the model in a dynamic environment are ensured.

[0038] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application but not limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical solutions of the present application, and all of them should be covered in the scope of the claims of the present application.

Claims

1. A method for intelligent display of 3D models, characterized in that: This includes acquiring the geometric data, image data, and ambient lighting parameters of the product being displayed, as well as the user's position, gaze focus, and EEG signals, to generate a multimodal 3D display dataset; Extract multi-scale features from the multimodal 3D display dataset, and adaptively adjust the fusion weights by combining the user's gaze focus and the user's EEG signals to generate a fusion feature vector; A NeRF (Neural Image Rendering Framework) is constructed and trained using a multi-view stereo vision algorithm to generate an initial 3D model. The update region is located by finding the point cloud region in the initial 3D model that is closest to the user's gaze focus. The NeRF is then incrementally trained and combined with PBR (Programmable Rendering) to generate a real-time 3D model. By calculating gaze dwell time and EEG attention, and combining gaze focus displacement threshold and EEG attention threshold, interaction trigger conditions are determined. The layered display priority of the real-time 3D model is adjusted by dividing the display area into layers.

2. The intelligent display method for three-dimensional models as described in claim 1, characterized in that: The specific steps for extracting multi-scale features from the multimodal 3D display dataset are as follows: detect key points in the image data to generate low-scale features; perform illumination intensity normalization on the multi-view image sequence in the image data, and extract mid-scale features through depthwise convolution and global pooling. Select the user whose distance from the geometric center of the product being displayed is the main user, and use the main user's position as the main user's viewpoint. Based on the main user's viewpoint, an initial point cloud is generated, and high-scale features are extracted by constructing a dynamic graph and performing multi-layer convolution.

3. The intelligent display method for three-dimensional models as described in claim 1, characterized in that: The adaptive adjustment of fusion weights involves the following steps: calculating the relative distance from the main user's gaze focus coordinates to the displayed product and normalizing it to obtain the normalized distance score for each scale feature; performing a fast Fourier transform on the main user's EEG signal and calculating the attention index based on the spectral power distribution of the main user's EEG signal; and calculating the dynamic fusion weight for each scale feature based on the normalized distance score and attention index for each scale feature.

4. The intelligent display method for three-dimensional models as described in claim 3, characterized in that: The relative distance, for low-scale features, represents the Euclidean distance from the main user's gaze focus coordinates to the geometric edge of the displayed product; for medium-scale features, it represents the distance from the main user's gaze focus coordinates to the surface center region of the displayed product's bounding box; and for high-scale features, it represents the Euclidean distance from the main user's gaze focus coordinates to the geometric center of the displayed product.

5. The intelligent display method for three-dimensional models as described in claim 4, characterized in that: The specific steps for training NeRF using a multi-view stereo vision algorithm are as follows: For each frame of multi-view image data, several light rays are randomly sampled, and several points are uniformly sampled according to the light rays; the coordinates of the sampled points are concatenated with the fused feature vector and input into NeRF, outputting the predicted values ​​of the stereo density and color of each sampled point; by comparing the mean square error between the pixel color predicted by NeRF and the pixel color of the real multi-view image, and adding a regularization term for the predicted stereo density, a rendering loss function is defined; using the Adam optimizer, the image data of the displayed image is used to iterate and update the weight matrix of NeRF.

6. The intelligent display method for three-dimensional models as described in claim 5, characterized in that: The specific steps for incremental training of NeRF are as follows: find the point cloud region in the initial 3D model that is closest to the user's line of sight as the update region; detect whether there is any newly added image data in the image data for the update region; perform incremental training on NeRF and update the NeRF weights.

7. The intelligent display method for three-dimensional models as described in claim 6, characterized in that: The PBR rendering process involves the following steps: setting the PBR ambient light coefficient based on ambient lighting parameters; extracting pixel colors from the image data of the product being displayed and mapping them to the vertices of the 3D model based on the camera projection matrix; initializing the diffuse reflection coefficient and specular reflection coefficient based on geometric data, generating a texture map of the product being displayed, and binding it to the 3D model to obtain a real-time 3D model.

8. A three-dimensional model intelligent display system, based on the three-dimensional model intelligent display method according to any one of claims 1 to 7, characterized in that: It includes a multimodal data module, a feature fusion module, a 3D model module, and a hierarchical display module. The multimodal data module is used to acquire the geometric data, image data, and ambient lighting parameters of the product being displayed, as well as the user's position, gaze focus, and EEG signals, to generate a multimodal 3D display dataset. The feature fusion module is used to extract multi-scale features from the multimodal 3D display dataset and adaptively adjust the fusion weights by combining the user's gaze focus and the user's EEG signal to generate a fused feature vector. The 3D model module is used to construct NeRF, train NeRF through a multi-view stereo vision algorithm to generate an initial 3D model, locate the update region by finding the point cloud region closest to the user's line of sight in the initial 3D model, incrementally train NeRF, and generate a real-time 3D model by combining PBR rendering. The layered display module is used to determine the interaction triggering conditions by calculating the gaze dwell time and EEG attention, and combining the gaze focus displacement threshold and EEG attention threshold. By dividing the layered display area, the layered display priority of the real-time 3D model is adjusted.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the intelligent display method for three-dimensional models according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the intelligent display method for three-dimensional models as described in any one of claims 1 to 7.

Images