Immersive 3D live-action experience interactive system

Abstract

The application discloses an immersive 3D real scene experience interaction system, and particularly relates to the technical field of 3D experience interaction optimization; the application collects user visual perception data in multiple dimensions, calculates the real-time perception importance of each rendering object in combination with a scientific algorithm, simultaneously quantitatively analyzes rendering overhead, divides three types of object types and executes differentiated strategies, preferentially allocates resources to target guarantee objects that are core concerns, improves rendering quality, accurately degrades low-concern high-overhead objects to be optimized, avoids resource waste, guarantees realistic experience in the core visual area of the user, significantly reduces problems such as system frame rate reduction and delay, reduces visual lag and motion sickness risk, and improves immersion coherence.

Description

Technical Field

[0001] This invention relates to the field of 3D experience interaction optimization technology, and more specifically, to an immersive 3D real-world experience interaction system. Background Technology

[0002] With the rapid development of technologies such as VR (virtual reality), AR (augmented reality), and digital twins, immersive 3D real-scene interactive systems have been widely used in various fields such as culture and tourism, industry, education, and healthcare.

[0003] The rendering strategies of existing immersive 3D reality interaction systems still have the following technical limitations in the actual user experience:

[0004] On the one hand, traditional technologies often adopt a uniform rendering strategy, allocating similar rendering resources to all rendered objects in the scene without considering the differences in user attention to different objects. For core objects that users focus on and frequently interact with, insufficient resource allocation may lead to insufficient rendering accuracy and loss of details. On the other hand, for secondary objects that users pay little attention to and that are only used as background, excessive high-quality rendering wastes rendering resources and increases the hardware load on the device.

[0005] On the other hand, existing systems lack the ability to dynamically analyze the correlation between rendering performance and user-perceived data. During the rendering process, performance indicators such as the geometric complexity and texture resolution of objects are considered alone, while user-perceived data such as visual attention distribution, interaction intent, and physiological feedback are ignored, resulting in a lack of targeted adjustment of rendering strategies.

[0006] Therefore, there is an urgent need for an immersive 3D real-world experience interactive system that can dynamically integrate rendering performance and user perception data and achieve differentiated rendering resource allocation to solve the above-mentioned technical pain points. Summary of the Invention

[0007] In order to overcome the above-mentioned defects of the prior art, embodiments of the present invention provide an immersive 3D real-scene experience interactive system.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] An immersive 3D reality experience interactive system includes the following modules:

[0010] The data acquisition module is used to collect rendering performance data and user visual perception data during the user's experience of 3D real-world scenes;

[0011] The perception and analysis module includes an importance calculation unit, a rendering calculation unit, and a correlation analysis unit;

[0012] The importance calculation unit is used to receive the user's visual perception data in the current time zone and calculate the real-time perceived importance score of each rendered object in the scene in the current time zone.

[0013] The rendering calculation unit is used to analyze the rendering performance data of each rendered object, thereby determining the rendering overhead score of each rendered object and storing it in a pre-built object database.

[0014] The performance correlation analysis unit is used to perform correlation analysis between the real-time perceived importance score of each rendered object in the current time zone and the object database, and construct a correlation analysis matrix. The matrix is ​​a two-dimensional data matrix with rendered objects as rows and perceived importance score and rendering cost score as columns. Based on the correlation analysis matrix, a combination of cluster analysis and threshold judgment is used to classify the object types. The object types include target protection objects, objects to be optimized, and regular maintenance objects.

[0015] The rendering control module receives the object type classification results from the correlation analysis unit and executes pre-edited differential rendering strategies for different types of rendered objects.

[0016] Specifically, rendering performance data and user visual perception data include;

[0017] Rendering performance data includes the number of polygons, texture resolution, and shader complexity for each rendered object;

[0018] Visual perception data includes eye-tracking data, interaction data, and physiological feedback data;

[0019] Eye-tracking data includes gaze location and fixation duration;

[0020] Interactive data includes clicked, dragged, and hovered object objects;

[0021] Physiological feedback data includes skin conductance response.

[0022] Specifically, the calculation logic for the real-time perceived importance score of each rendered object;

[0023] Each rendered object in the 3D real scene is represented by Obi, where i is the number of each rendered object in the real scene; the visual attention factor, interaction intention factor and physiological feedback factor of Obi are calculated based on visual perception data.

[0024] For each Obi within the 3D real-world scene, the visual attention factor is compared;

[0025] If the visual attention factor > 0, the visual attention factor, interaction intention factor and physiological feedback factor of Obi are weighted and fused to output the perception importance coefficient, and then the perception importance coefficient is converted into the real-time perception importance score using the pre-constructed perception importance coefficient-real-time perception importance score mapping rule.

[0026] Specifically, the calculation logic of the visual attention factor;

[0027] Map the Obi in the 3D real scene and the user's eye-tracking gaze point to the device screen's two-dimensional pixel coordinate system; calculate the axis-aligned bounding box of Obi on the screen, obtain the screen space outline of Obi, and output the screen pixel area.

[0028] For all gaze points within the current time zone, those gaze points whose duration exceeds a preset duration are marked as valid gaze points. The screen coordinates of each valid gaze point are extracted to form a set of valid gaze points.

[0029] A gaze point heatmap of the screen space is generated based on the effective gaze point set. With each gaze point as the core, a Gaussian kernel function is set to calculate the gaze density value of each pixel on the screen. The area that overlaps with the Obi screen bounding box in the heatmap is extracted, and the total number of effective pixels with a density value higher than the reference density value is counted as the overlapping area. The spatial coverage ratio of Obi is calculated by using the overlapping area of ​​Obi as the numerator and the screen pixel area as the denominator.

[0030] Iterate through each valid gaze point, determine whether each valid gaze point is within the Obi's screen bounding box, and if so, calculate the duration and sum them up to obtain the total valid gaze duration of the user on the Obi in the current time zone.

[0031] The ratio of Obi's effective gaze duration to the current time zone duration is used to calculate the gaze duration percentage of Obi. The spatial coverage percentage and gaze duration percentage of Obi are weighted and fused to output the visual attention factor of users to Obi in the current time zone.

[0032] Specifically, the calculation logic of interaction intention factor and physiological feedback factor;

[0033] Iterate through all interactive operation data within the current time zone and match the screen pixel coordinates of the operation landing point with the screen bounding box of Obi.

[0034] If interactive operation data exists within the Obi screen bounding box, including click and drag operations, then count the number of times the user clicks and drags on the Obi within the current time zone.

[0035] Set a basic operation score for each different interactive operation; multiply the number of clicks and drags of Obi in the current time zone by the corresponding basic operation scores and then sum them up as the user's interaction intent factor with Obi in the current time zone;

[0036] Extract skin conductance response data when the user gazes at and interacts with Obi, and construct a skin conductance response data sequence;

[0037] The mean of the skin conductance response data sequence is calculated and divided by a pre-set baseline value for skin conductance response, which serves as the physiological feedback factor for the user to Obi in the current time zone.

[0038] Specifically, the calculation logic for rendering overhead score;

[0039] The actual number of triangles rendered by Obi is used as the geometric complexity; the total texture resolution equivalent of Obi is used as the texture memory overhead.

[0040] Identify the number of instructions and the preset complexity level of the shaders bound to Obi; perform comprehensive processing based on the number of instructions and the preset complexity level to obtain the shading complexity;

[0041] The geometric complexity, texture memory overhead, and shading complexity of Obi are weighted and fused to obtain the rendering overhead coefficient of Obi, and then transformed into a rendering overhead score using a pre-built rendering overhead coefficient-rendering overhead score mapping rule.

[0042] Specifically, the calculation logic for coloring complexity;

[0043] Different complexity levels are assigned a set of level values;

[0044] The complexity levels of Obi are converted into level assignments, and the instruction count of the Obi shader is extracted and normalized.

[0045] Using formula The coloring complexity is calculated; where Assign a value to the level. This represents the number of instructions after normalization.

[0046] Specifically, the logic for classifying object types;

[0047] Identify the two-dimensional data matrix, extract the perceived importance score of each rendered object, and mark the objects with perceived importance scores higher than the preset importance scores as target protection objects;

[0048] Objects with a visual attention factor < 0 and an overall sensory evaluation coefficient lower than the preset overall evaluation threshold are marked as objects to be optimized.

[0049] The remaining items are classified as regular maintenance items.

[0050] Specifically, the calculation logic of the overall evaluation coefficient;

[0051] This means that after screening the target objects, the perception importance coefficient and rendering cost coefficient of the remaining objects are weighted and fused together.

[0052] The technical effects and advantages of this invention are as follows:

[0053] (1) By collecting user visual perception data from multiple dimensions, combining scientific algorithms to calculate the real-time perception importance of each rendered object, and at the same time quantifying and analyzing rendering overhead, classifying three types of objects and implementing differentiated strategies, prioritizing the allocation of resources to the core target objects and improving rendering quality, and accurately downgrading the objects with low attention and high overhead to avoid resource waste, it not only ensures the realistic experience of the user's core visual area, but also significantly reduces the problems of system frame rate drop and latency, reduces the risk of visual stuttering and motion sickness, and improves the continuity of immersion.

[0054] (2) By updating the object importance analysis results according to a preset cycle, adapting to user behavior and scene changes, and dynamically adjusting the degradation gradient of the object to be optimized according to the real-time load of the device, dynamically adapting to the scene and device status, the intelligent and flexible rendering optimization is realized. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the immersive 3D real-scene interactive system of the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] like Figure 1 As shown, the modules of the immersive 3D real-scene interactive system are as follows:

[0058] The data acquisition module is used to collect rendering performance data and user visual perception data during the user's experience of 3D real-world scenes;

[0059] The rendering performance data includes the number of polygons, texture resolution, and shader complexity for each rendered object.

[0060] Rendering performance data is collected in real time at a preset sampling frequency of 10-30Hz by interfacing with the graphics card driver and rendering engine of the 3D experience device to ensure the timeliness and accuracy of the data.

[0061] Visual perception data reflects the user's visual focus and perception state during the 3D real-world experience, including eye-tracking data, interaction operation data, and physiological feedback data.

[0062] Eye-tracking data includes gaze location and fixation duration;

[0063] Interactive data includes clicked, dragged, and hovered object objects;

[0064] Physiological feedback data includes skin conductance response; used to help determine the user's level of attention to the scene;

[0065] Eye-tracking data is collected through eye-tracking sensors integrated into VR devices and interactive terminals. Interactive operation data is captured in real time through scene interaction modules. Physiological perception feedback data is collected through matching wearable devices. After synchronous processing, all types of data are used to generate a standardized visual perception dataset.

[0066] The perception and analysis module includes an importance calculation unit, a rendering calculation unit, and a correlation analysis unit;

[0067] The importance calculation unit is used to receive the user's visual perception data in the current time zone and calculate the real-time perceived importance score of each rendered object in the scene in the current time zone.

[0068] The importance calculation unit updates the analysis results once every preset period to adapt to changes in scenarios and user behavior. The preset period can be set by the user before the experience or executed according to the initial system settings.

[0069] Before calculating the importance score, the user's visual perception data is preprocessed to remove abnormal data (such as false trigger data in eye tracking and false operation data in interactive operations).

[0070] Specifically:

[0071] Each rendered object in the 3D real scene is represented by Obi, where i is the number of each rendered object in the real scene;

[0072] Map the Obi in the 3D real scene and the user's eye-tracking gaze point to the device screen's two-dimensional pixel coordinate system.

[0073] The screen pixel area is output after calculating the screen-aligned bounding box of Obi on the screen using the projection matrix of the 3D rendering engine to obtain the screen space outline of Obi. The screen axis-aligned bounding box is a rectangle, and its pixel area is width × height.

[0074] For all gaze points within the current time zone, those gaze points whose duration exceeds a preset duration are marked as valid gaze points;

[0075] Accidental trigger points, blink points, and rapid eye saccades in eye-tracking data are removed (industry-standard criteria: fixation duration ≥ 80ms and pixel offset between adjacent fixations ≤ 50px), and only valid data is retained for calculation.

[0076] Extract the two-dimensional screen coordinates of each valid gaze point to form a set of valid gaze points;

[0077] Based on the effective set of gaze points, a gaze point heatmap in the screen space is generated using kernel density estimation.

[0078] A Gaussian kernel function is set with each gaze point as the core; the kernel radius is an empirical value of 30px to adapt to the screen resolution; the gaze density value of each pixel on the screen is calculated; the higher the density value, the stronger the user's gaze is on that pixel.

[0079] Extract the area in the heatmap that overlaps with the Obi screen bounding box, and count the total number of valid pixels with a density value higher than the reference density value as the overlapping area; the reference density value can be set to 0.

[0080] The spatial coverage ratio of Obi is calculated by using the overlapping area of ​​Obi as the numerator and the screen pixel area as the denominator.

[0081] Iterate through each valid gaze point, determine whether each valid gaze point is within the Obi's screen bounding box, and if so, calculate the duration and sum them up to obtain the total valid gaze duration of the user on the Obi in the current time zone.

[0082] The proportion of Obi's gaze duration is calculated by using the effective gaze duration of Obi as the numerator and the current time zone duration as the denominator.

[0083] The spatial coverage ratio and fixation duration ratio of Obi are weighted and fused to output the visual attention factor of users to Obi in the current time zone;

[0084] That is, using formulas Calculations were performed to obtain the visual attention factor. ;in and These are preset weighting coefficients, with values ​​of 0.4 and 0.6 respectively; and These represent the percentage of spatial coverage and the percentage of fixation time, respectively.

[0085] Iterate through all interactive operation data within the current time zone and filter out valid interactive operations;

[0086] Match the screen pixel coordinates of the operation landing point with the screen bounding box of Obi;

[0087] If interactive operation data exists within the Obi screen bounding box, including click and drag operations, then count the number of times the user clicks and drags on the Obi within the current time zone.

[0088] Valid interaction operations: Remove accidental triggering operations from the interaction operation data (such as accidental clicks caused by device shaking, invalid drags / long presses shorter than 50ms), and only retain user-initiated clicks, selections, drags, and long presses. The operation landing point must be aligned with and overlap with the bounding box of the object Obi's screen axis to be judged as a valid interaction for Obi.

[0089] Set a basic score for each different interactive action; clicking can be set to 2, and dragging can be set to 3.

[0090] The number of clicks and drags by Obi in the current time zone are multiplied by the corresponding basic operation scores and then summed to form the user's interaction intent factor with Obi in the current time zone.

[0091] Before the user enters the 3D real scene, the resting state of the skin conductance response is collected for 10-30 seconds as the baseline value of the skin conductance response;

[0092] Extract skin conductance response data when the user gazes at and interacts with Obi, and construct a skin conductance response data sequence;

[0093] When a user experiences positive feelings such as interest, curiosity, or surprise when encountering an object in a 3D scene, or when emotions are aroused due to interaction or visual impact, the sympathetic nervous system is quickly activated, leading to increased secretion of sweat glands in the skin (even if it is just a little sweat). This significantly improves the skin's conductivity, which is directly reflected in an increase in the GSR value (conductivity value).

[0094] The mean of the skin conductance response data sequence is calculated and divided by a pre-set baseline value for skin conductance response, which serves as the physiological feedback factor for the user to Obi in the current time zone.

[0095] For each Obi in the 3D real scene, the visual attention factor is compared. If the visual attention factor is >0, the visual attention factor, interaction intention factor and physiological feedback factor of the Obi are weighted and fused to output the perception importance coefficient. The perception importance coefficient-real-time perception importance score is converted into a real-time perception importance score using the pre-constructed perception importance coefficient-real-time perception importance score mapping rule.

[0096] Multi-factor data linkage: The calculation of the three factors has a clear preceding linkage relationship:

[0097] The calculation of the physiological feedback factor is based on the premise that the visual attention factor is >0; there is no physiological feedback factor if there is no fixation.

[0098] That is, after normalizing the visual attention factor, interaction intention factor, and physiological feedback factor, the results are substituted into the formula. The perceived importance coefficient was calculated. ;in , as well as These are preset weighting coefficients, with values ​​of 0.4, 0.4, and 0.2 respectively. and These represent the interaction intention factor and the physiological feedback factor, respectively.

[0099] Additional explanation of mapping rules: The normalized perceived importance coefficients are converted into real-time perceived importance scores within a preset range (e.g., 0-10) using a linear mapping method. The mapping rule is that the coefficients and scores are positively correlated, with coefficient 0 corresponding to score 0 and coefficient 1 corresponding to score 10.

[0100] The rendering calculation unit is used to analyze the rendering performance data of each rendered object, thereby determining the rendering overhead score of each rendered object and storing it in a pre-built object database.

[0101] Specifically:

[0102] From the model mesh data of the 3D rendering engine, the actual number of rendered triangles in Obi is counted as the geometric complexity; the effective number of triangles after view frustum clipping and backface culling is removed, and the number of non-original triangles in the model is excluded to ensure that the data matches the actual rendering overhead.

[0103] The number of original polygons (triangles) of an object Obi is a core quantitative indicator of the geometric complexity of a 3D model. The more triangles there are, the greater the overhead of vertex calculation and rasterization rendering.

[0104] Obtain the total texture resolution equivalent of Obi as the texture memory overhead;

[0105] Identify the number of instructions and preset complexity level of the shader bound to Obi, and set a set of level assignments corresponding to different complexity levels; the complexity level range is set to 1-10, and the higher the level, the higher the corresponding level assignment, and the level assignment range is limited to 5-15;

[0106] The complexity level of Obi is converted into a level assignment, and the instruction count of the Obi shader is extracted and normalized to 0-1.

[0107] Using formula The coloring complexity is calculated; where Assign a value to the level. This is the number of instructions after normalization.

[0108] The geometric complexity, texture memory overhead, and shading complexity of Obi are weighted and fused to obtain the rendering overhead coefficient of Obi, and then transformed into a rendering overhead score using a pre-built rendering overhead coefficient-rendering overhead score mapping rule.

[0109] That is, after normalizing the geometric complexity, texture memory overhead, and shading complexity, the results are substituted into the formula. Calculations are performed to obtain the rendering overhead coefficient. ;in , as well as These are preset weighting coefficients, with values ​​of 0.3, 0.2, and 0.4 respectively. , and These represent geometric complexity, texture memory overhead, and shading complexity, respectively.

[0110] Additional explanation of mapping rules: The normalized rendering overhead coefficients are converted into rendering overhead scores within a preset range (e.g., 0-10) using a linear mapping method. The mapping rule is that the coefficients and scores are positively correlated, with coefficient 0 corresponding to score 0 and coefficient 1 corresponding to score 10.

[0111] The performance correlation analysis unit is used to perform correlation analysis between the real-time perceived importance score of each rendered object in the current time zone and the object database, and construct a correlation analysis matrix. The matrix is ​​a two-dimensional data matrix with rendered objects as rows and perceived importance score and rendering cost score as columns. Based on the correlation analysis matrix, a combination of cluster analysis and threshold judgment is used to classify the object types. The object types include target protection objects, objects to be optimized, and regular maintenance objects.

[0112] Specifically:

[0113] Identify the two-dimensional data matrix, extract the perceived importance score of each rendered object, and mark the objects with perceived importance scores higher than the preset importance scores as target protection objects;

[0114] Target objects to be protected: Objects with a perception importance score of ≥7 and no limit on rendering cost score. These objects are the core of the user's visual perception and their rendering quality should be prioritized.

[0115] Objects with a visual attention factor < 0 and an overall sensory evaluation coefficient lower than the preset overall evaluation threshold are marked as objects to be optimized.

[0116] Objects to be optimized: These objects have low user attention or consume a lot of rendering resources, making them the key targets for rendering optimization.

[0117] The perceptual rendering comprehensive score coefficient is obtained by weighting and fusing the perceptual importance coefficient and rendering cost coefficient of objects (excluding target protection objects);

[0118] That is, substitute into the formula Calculations were performed to obtain the overall emotional impact score coefficient. ;in , These are preset weighting coefficients, with values ​​of 0.4 and 0.6 respectively.

[0119] Remaining items are classified as regular maintenance items;

[0120] Regularly maintained objects: Objects other than the target objects and objects to be optimized can maintain their existing rendering quality without any additional optimization or improvement.

[0121] The rendering control module receives the object type classification results from the correlation analysis unit, executes differentiated rendering strategies for different types of rendering objects, and achieves precise allocation of rendering resources.

[0122] Pre-edit the rendering strategies corresponding to different object types and then execute them.

[0123] Examples of rendering strategies for different object types:

[0124] For the objects to be optimized, implement downgraded rendering optimization;

[0125] Specific optimization methods include, but are not limited to: reducing texture resolution (e.g., from 4K to 2K or 1080P), reducing the number of polygons (eliminating redundant vertices and faces through model simplification algorithms), replacing with low-complexity shaders (removing complex lighting calculations and reflection / refraction effects), reducing rendering priority (placing it later in the rendering queue), and using LOD (Level of Detail) technology to dynamically adjust rendering accuracy based on the distance between the object and the user's viewpoint.

[0126] The degree of rendering degradation is dynamically adjusted based on the rendering overhead score and system performance load to ensure that rendering resources are saved without affecting the user's perception of the overall scene.

[0127] To maintain or improve rendering quality for the target protected object;

[0128] Specific measures include, but are not limited to: maintaining the highest texture resolution and polygon precision, enabling advanced shader effects (such as global illumination, real-time shadows, and enhanced material texture details), increasing drawing priority (placing it at the front of the rendering queue), and allocating dedicated rendering resources to avoid the rendering quality of core objects being degraded due to other objects consuming resources.

[0129] For regular objects, keep the existing rendering parameters unchanged, while monitoring changes in rendering overhead and user-perceived importance in real time. If the parameters change significantly, adjust the object type and rendering strategy accordingly.

[0130] During the downgrade rendering optimization of the object to be optimized, the degree of rendering gradient downgrade is dynamically adjusted based on the device's real-time rendering performance data to avoid over-downgrading or under-downgrading.

[0131] Low device load (CPU < 60%, GPU < 65%): Only perform the first-level low-degree degradation;

[0132] Device load (CPU 60%-75%, GPU 65%-80%): Execute first + second gradient degradation;

[0133] High device load (CPU > 75%, GPU > 80%): Execute full degradation of the first, second, and third gradients to quickly release rendering resources and ensure overall smoothness;

[0134] First tier (low-level degradation, prioritized execution): Adjust the priority of drawing calls, place the object to be optimized at the end of the rendering queue, and allocate remaining resources only after the core object has been rendered; enable LOD technology, dynamically switch the low-precision model level according to the distance between the object and the user's viewpoint, and automatically remove redundant polygons at long distances.

[0135] Second tier (moderate degradation): Reduce object texture resolution (e.g., 4K→2K, 2K→1080P, adjust flexibly according to device performance), use texture compression technology to reduce memory usage; simplify shader effects, turn off unnecessary lighting calculations (e.g., real-time shadows, reflections and refractions), and replace them with static baked lighting effects.

[0136] The third gradient (high-level degradation, executed only under high device load): removes 30%-50% of redundant polygons from objects using a model simplification algorithm (preserving object outline features and avoiding visual distortion); reduces the object rendering sampling rate; uses batch rendering calls for distant objects to be optimized to reduce the number of rendering instructions per instance; if the object is a background decoration, its image transparency can be appropriately reduced or it can be blended into the environment for rendering, further reducing rendering overhead.

[0137] The above formulas are all dimensionless calculations. Dimensionless calculations can be performed using various methods such as standardization, which will not be elaborated here. The formulas are derived from software simulations based on a large amount of collected data, and the preset parameters in the formulas can be set by those skilled in the art according to the actual situation.

[0138] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, ATA hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium can be a solid-state ATA hard disk.

[0139] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0140] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0141] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0142] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0143] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0144] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable ATA hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0145] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An immersive 3D real-scene interactive system, characterized in that, Includes the following modules: The data acquisition module is used to collect rendering performance data and user visual perception data during the user's experience of 3D real-world scenes; The perception and analysis module includes an importance calculation unit, a rendering calculation unit, and a correlation analysis unit; The importance calculation unit is used to receive the user's visual perception data in the current time zone and calculate the real-time perceived importance score of each rendered object in the scene in the current time zone. The rendering calculation unit is used to analyze the rendering performance data of each rendered object, thereby determining the rendering overhead score of each rendered object and storing it in a pre-built object database. Rendering performance data includes the number of polygons, texture resolution, and shader complexity for each rendered object; The calculation logic for rendering overhead score; Each rendered object in the 3D real scene is represented by Obi, where i is the number of each rendered object in the real scene; The actual number of triangles rendered by Obi is used as the geometric complexity. Obtain the total texture resolution equivalent of Obi as the texture memory overhead; Identify the instruction count and preset complexity level of the shaders bound to Obi; The coloring complexity is obtained by comprehensively processing based on the number of instructions and the preset complexity level; The geometric complexity, texture memory overhead, and shading complexity of Obi are weighted and fused to obtain the rendering overhead coefficient of Obi, and then transformed into a rendering overhead score using a pre-built rendering overhead coefficient-rendering overhead score mapping rule. The performance correlation analysis unit is used to perform correlation analysis between the real-time perceived importance score of each rendered object in the current time zone and the object database, and construct a correlation analysis matrix. The rendered objects are the rows, and the perceived importance score and rendering cost score are the columns, forming a two-dimensional data matrix. Based on the correlation analysis matrix, a combination of cluster analysis and threshold judgment is used to classify the object types. The object types include target protection objects, objects to be optimized, and regular maintenance objects; The calculation logic for the real-time perceived importance score of each rendered object; Obi's visual attention factor, interaction intent factor, and physiological feedback factor are calculated based on visual perception data. For each Obi within the 3D real-world scene, the visual attention factor is compared; Visual attention factor calculation logic: Map the Obi in the 3D real scene and the user's eye-tracking gaze point to the device screen's two-dimensional pixel coordinate system; calculate the axis-aligned bounding box of Obi on the screen, obtain the screen space outline of Obi, and output the screen pixel area. For all gaze points within the current time zone, those gaze points whose duration exceeds a preset duration are marked as valid gaze points. The screen coordinates of each valid gaze point are extracted to form a set of valid gaze points. A gaze point heatmap of the screen space is generated based on the effective gaze point set. With each gaze point as the core, a Gaussian kernel function is set to calculate the gaze density value of each pixel on the screen. The area that overlaps with the Obi screen bounding box in the heatmap is extracted, and the total number of effective pixels with a density value higher than the reference density value is counted as the overlapping area. The spatial coverage ratio of Obi is calculated by using the overlapping area of ​​Obi as the numerator and the screen pixel area as the denominator. Iterate through each valid gaze point, determine whether each valid gaze point is within the Obi's screen bounding box, and if so, calculate the duration and sum them up to obtain the total valid gaze duration of the user on the Obi in the current time zone. The ratio of Obi's effective gaze duration to the current time zone duration is used to calculate the gaze duration percentage of Obi. The spatial coverage percentage and gaze duration percentage of Obi are weighted and fused to output the visual attention factor of users to Obi in the current time zone. If the visual attention factor > 0, the visual attention factor, interaction intention factor and physiological feedback factor of Obi are weighted and fused to output the perception importance coefficient, and the perception importance coefficient-real-time perception importance score is converted into a real-time perception importance score using the pre-constructed perception importance coefficient-real-time perception importance score mapping rule. The rendering control module receives the object type classification results from the correlation analysis unit and executes pre-edited differential rendering strategies for different types of rendered objects.

2. The immersive 3D real-scene interactive system according to claim 1, characterized in that: User visual perception data includes; Visual perception data includes eye-tracking data, interaction data, and physiological feedback data; Eye-tracking data includes gaze location and fixation duration; Interactive data includes clicked, dragged, and hovered object objects; Physiological feedback data includes skin conductance response.

3. The immersive 3D real-scene interactive system according to claim 1, characterized in that: The calculation logic of interaction intention factor and physiological feedback factor; Iterate through all interactive operation data within the current time zone and match the screen pixel coordinates of the operation landing point with the screen bounding box of Obi. If interactive operation data exists within the Obi screen bounding box, including click and drag operations, then count the number of times the user clicks and drags on the Obi within the current time zone. Set a basic operation score for each different interactive operation; multiply the number of clicks and drags of Obi in the current time zone by the corresponding basic operation scores and then sum them up as the user's interaction intent factor with Obi in the current time zone; Extract skin conductance response data when the user gazes at and interacts with Obi, and construct a skin conductance response data sequence; The mean of the skin conductance response data sequence is calculated and divided by a pre-set baseline value for skin conductance response, which serves as the physiological feedback factor for the user to Obi in the current time zone.

4. The immersive 3D real-scene interactive system according to claim 1, characterized in that: The calculation logic for coloring complexity; Different complexity levels are assigned a set of level values; The complexity levels of Obi are converted into level assignments, and the instruction count of the Obi shader is extracted and normalized. Using formula The coloring complexity is calculated; where Assign a value to the level. This represents the number of instructions after normalization.

5. The immersive 3D real-scene interactive system according to claim 1, characterized in that: Logic for classifying object types; Identify the two-dimensional data matrix, extract the perceived importance score of each rendered object, and mark the objects with perceived importance scores higher than the preset importance scores as target protection objects; Objects with a visual attention factor < 0 and an overall sensory evaluation coefficient lower than the preset overall evaluation threshold are marked as objects to be optimized. Remaining items are classified as regular maintenance items.

6. The immersive 3D real-scene interactive system according to claim 5, characterized in that: The calculation logic of the overall evaluation coefficient for emotional appeal; This means that after screening the target objects, the perception importance coefficient and rendering cost coefficient of the remaining objects are weighted and fused together.

Images