Holographic display method and system based on TOF projection equipment

Abstract

The embodiment of the invention provides a holographic display method and system based on TOF projection equipment, and relates to the technical field of holographic display, and the method comprises the steps: obtaining the observation information of a plurality of users, and the observation information comprises an observation position and a line-of-sight direction; according to the observation information, identifying to obtain a focus area, and according to the number of users in the focus area, the observation stability and the importance of holographic content, obtaining a target weight; calculating based on the focus area and the target weight to obtain a collaborative focusing plane; and generating a holographic light field for the collaborative focusing plane, and performing local phase compensation on the holographic light field according to the relative relationship between the observation position of each user and the collaborative focusing plane. According to the invention, the fluency and immersion of interaction can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of holographic display, and in particular to a holographic display method and system based on a TOF projection device. BACKGROUND

[0002] In the related art, the system will preset a projection surface with stable geometric shape and uniform material, and calculate a hologram based on the projection surface to ensure that the image is in focus on the predetermined surface. However, in actual applications, especially in augmented reality environments that require high interaction, these ideal conditions are often difficult to maintain. The projection target surface may be composed of multiple physical objects that are continuously moving, irregular in shape, and different in material, and the high-frequency interaction actions of multiple users may also frequently block the depth sensor, resulting in incomplete or inaccurate depth data. SUMMARY

[0003] The present application aims to at least solve one of the technical problems existing in the prior art. To this end, the present application proposes a holographic display method and system based on a TOF projection device, aiming to improve the fluency and immersion of interaction.

[0004] In a first aspect, an embodiment of the present application provides a holographic display method based on a TOF projection device, comprising: Obtaining observation information of multiple users, the observation information including observation positions and line-of-sight directions; According to the observation information, a focal point area is identified, and a target weight is obtained according to the number of users in the focal point area, the observation stability, and the importance of the holographic content; Based on the focal point area and the target weight, a collaborative focusing plane is calculated; Generating a holographic light field for the collaborative focusing plane, and performing local phase compensation on the holographic light field according to the relative relationship between the observation position of each user and the collaborative focusing plane.

[0005] According to some embodiments of the present application, the focal point area is identified according to the observation information, and the target weight is obtained according to the number of users in the focal point area, the observation stability, and the importance of the holographic content, comprising: Obtaining observation point information of the user, identifying a virtual component currently gazed by the user, and querying a semantic association tag of the virtual component; When high-frequency, short-term line-of-sight switching between the virtual components with semantic association is identified, a relational focal point area containing the virtual components and the spatial area of the virtual components is dynamically constructed, wherein the relational focal point area is taken as the focal point area; Based on the number of users within the focal area, observation stability, and the importance of the holographic content, a higher weight value than that of a preset regular focal area is assigned to the relational focal area to obtain the target weight.

[0006] According to some embodiments of this application, assigning a weight value higher than that of a preset conventional focus region to the relational focus region includes: Query the task priority of the virtual component contained in the relational focus area; Based on the task priority, a weight value higher than that of a preset regular focus area is assigned to the relational focus area, wherein the task priority is positively correlated with the weight value.

[0007] According to some embodiments of this application, after identifying the focal region based on the observation information, and considering the number of users within the focal region, observation stability, and the importance of the holographic content, the process further includes: The virtual components in the holographic content within the focal area are preset with task stage attributes and user role permission levels, and configurable weight coefficients are provided for the importance assessment of the holographic content under different task stages and role permissions. Real-time monitoring of the current stage of collaborative tasks and acquisition of each user's role permissions; Based on the task stage and the user's role permissions, adjust the importance assessment parameters of the holographic content and the weight contribution coefficient of the number of users to obtain the target weight.

[0008] According to some embodiments of this application, the step of generating a holographic light field for the co-focusing plane and performing local phase compensation on the holographic light field based on the relative relationship between each user's observation position and the co-focusing plane includes: Obtain the preset depth level attribute and importance level of the virtual component in the holographic content. The preset depth level attribute describes the relative depth position of the virtual component in three-dimensional space, and the importance level describes the visual or interactive priority of the virtual component in the holographic content. The system acquires the user's observation position and line of sight in real time, and obtains the user's desired compensation depth range based on the relative relationship between the collaborative focusing plane and the user. Identify the virtual components included within the compensation depth range, and query the depth level attribute and importance level of the virtual components; Based on the depth hierarchy attribute of the virtual component and the importance level, the intensity and range of the local phase compensation are adjusted to obtain the adjusted compensation intensity and range; A holographic light field is generated for the co-focusing plane, and local phase compensation is performed on the holographic light field based on the relative relationship between each user's observation position and the co-focusing plane, as well as the adjusted compensation intensity and range.

[0009] According to some embodiments of this application, obtaining the target weight based on the number of users within the focal area, observation stability, and the importance of the holographic content further includes: Real-time acquisition of each user's role and permission level or professional field tags; Based on the role permission level or the professional field tag, adjust the contribution coefficient of the number of users in the focus area to the weight allocation to obtain the adjusted first contribution coefficient; Based on the adjusted first contribution coefficient, and combined with the number of users in the focal area, observation stability, and the importance of the holographic content, the target weight is obtained.

[0010] According to some embodiments of this application, identifying the focal region based on the observed information includes: Real-time monitoring of events that may trigger non-explicit attention in the holographic content within the focal area, including the appearance of abnormal data, alarm triggering, or updates to critical information; When the event occurs, the system continuously acquires the observation point information and eye movement trajectory of all users within a preset time period, and identifies the collective gaze behavior of users towards the relevant area of ​​the event. The focal area is identified based on the intensity and duration of the collective gaze behavior.

[0011] According to some embodiments of this application, obtaining the target weight based on the number of users within the focal area, observation stability, and the importance of the holographic content further includes: Real-time acquisition of observational behavior data for each user; Based on the observed behavioral data, dynamically assess each user's current level of trust or influence; Based on the level of trust or influence, the contribution coefficient of users in the focus area to the weight allocation is adjusted to obtain the adjusted second contribution coefficient. Based on the adjusted second contribution coefficient, and combined with the number of observers in the focal region, observation stability, and the importance of the holographic content, the target weight is obtained.

[0012] According to some embodiments of this application, the step of dynamically assessing each user's current trust level or influence based on the observed behavioral data includes: The observed behavior data is examined and noisy data points, missing data points, and abnormal data points are identified. When the noise data point is identified, it is smoothed or filtered. When the missing data point is identified, the missing data point is interpolated and filled in according to the observed behavior data at adjacent time points or the user's historical behavior pattern. When the abnormal data point is identified, it is corrected or removed based on a preset abnormal threshold or by comparing it with the behavior data of other users. Based on the processed observation data, the current trust level or influence of each observer is dynamically assessed.

[0013] Secondly, embodiments of this application provide a holographic display system based on a TOF projection device, comprising: The acquisition module is used to acquire observation information from multiple users, including observation position and line of sight direction; The identification module is used to identify the focal region based on the observation information, and to obtain the target weight based on the number of users in the focal region, the observation stability, and the importance of the holographic content. The calculation module is used to calculate, based on the focal region and the target weight, a cooperative focusing plane; The adjustment module is used to generate a holographic light field for the co-focusing plane and to perform local phase compensation on the holographic light field according to the relative relationship between each user's observation position and the co-focusing plane.

[0014] The technical solution according to the embodiments of this application has at least the following beneficial effects: This application discloses a holographic display method based on a TOF projection device. By acquiring observation information from multiple users, including observation positions and viewing directions, it can grasp the focus of user attention in real time. Based on this, this application identifies the focal region based on the observation information and dynamically calculates the target weight by comprehensively considering the number of users within the focal region, observation stability, and the importance of the holographic content. This mechanism enables the system to intelligently determine the area and content that currently needs the most attention and assign it a higher priority. Subsequently, based on the focal region and target weight, the system can accurately calculate the collaborative focusing plane, which represents the optimal focusing position for multiple users. Finally, a holographic light field is generated for the collaborative focusing plane, and local phase compensation is performed on the holographic light field according to the relative relationship between each user's observation position and the collaborative focusing plane. This application effectively solves the problems of image defocusing, display delay, and decreased user experience caused by dynamic changes in the projection surface, incomplete depth data, and excessive computational load in existing holographic display systems in complex, multi-user interactive environments. Specifically, this application dynamically identifies the user's focus and calculates the co-focusing plane, enabling the holographic image to always remain clearly focused on the area of ​​user attention, overcoming the limitation of traditional systems that can only provide clear images for preset areas. Simultaneously, a local phase compensation mechanism further ensures that even if the viewing position deviates from the co-focusing plane, the user can still obtain a good visual experience, significantly reducing visual fatigue. Furthermore, through intelligent weight allocation and co-focusing calculation, the system can utilize computing resources more efficiently, avoiding latency caused by high-precision calculations across the entire scene, thereby improving the smoothness and immersion of the interaction.

[0015] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0016] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0017] Figure 1 A flowchart illustrating a holographic display method based on a TOF projection device provided in one embodiment of this application; Figure 2 This is a schematic diagram of a holographic display system based on a TOF projection device provided in one embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical methods, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] It should be noted that the meaning of "multiple" (or "more than") in the description of the embodiments of this application refers to two or more, and "greater than," "less than," "exceeding," etc. are understood to exclude the number itself, while "above," "below," "within," etc. are understood to include the number itself. If "first," "second," etc. are used in the description, they are only for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of the technical features indicated.

[0020] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: the existence of a alone, the existence of b alone, the existence of c alone, the simultaneous existence of a and b, the simultaneous existence of a and c, the simultaneous existence of b and c, or the simultaneous existence of a, b, and c, where a, b, and c can be single or multiple.

[0021] In the description of this application, unless otherwise expressly defined, terms such as "setup," "installation," and "connection" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this application in conjunction with the specific content of the technical solution.

[0022] Based on the above, this application proposes a holographic display method and system based on a TOF projection device, aiming to improve the smoothness and immersion of the interaction.

[0023] The holographic display method based on TOF projection devices provided in this application can be applied to terminals, servers, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms; the software can be an application implementing the holographic display method based on TOF projection devices, but is not limited to the above forms.

[0024] This application can be applied to numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via communication networks. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices. It should be noted that in various specific embodiments of this invention, when processing is required based on data related to the characteristics of an object (e.g., user attributes or sets of attribute information), permission or consent from the corresponding object is obtained first, and the collection, use, and processing of this data comply with relevant laws and standards. Furthermore, when the embodiments of the present invention need to obtain the attribute information of an object, they will obtain the separate permission or separate consent of the corresponding object through pop-up windows or redirection to a confirmation page. After obtaining the separate permission or separate consent of the corresponding object, they will then obtain the relevant data of the object necessary for the embodiments of the present invention to operate normally.

[0025] See Figure 1 , Figure 1This is a flowchart illustrating a holographic display method based on a Time-of-Flight (TOF) projection device according to an embodiment of this application. The holographic display method based on a TOF projection device provided in this embodiment includes, but is not limited to, steps S110 to S140, which are described in detail below.

[0026] Step S110: Obtain observation information from multiple users, including observation position and line of sight direction; Step S120: Based on the observation information, identify the focal area, and obtain the target weight based on the number of users in the focal area, observation stability, and the importance of the holographic content; Step S130: Calculate the co-focusing plane based on the focal region and target weight; Step S140: Generate a holographic light field for the co-focusing plane, and perform local phase compensation on the holographic light field according to the relative relationship between each user's observation position and the co-focusing plane.

[0027] It's important to note that "observation information" refers to data describing the user's observation state, with core elements including the user's "observation position" and "gaze direction." The observation position typically refers to the coordinates of the user's head or eyes in three-dimensional space, which can be obtained through head-tracking or eye-tracking devices. The gaze direction indicates the direction the user's eyes are looking, usually calculated from eye-tracking data. This information is fundamental for the system to understand user intent and optimize display effects. The "focal area" refers to a specific three-dimensional spatial region in a holographic display scene that the system identifies based on the user's observation information and that is of high interest to the current user or user group. This area may contain one or more virtual components and is the focus of the system's holographic light field optimization.

[0028] "Target weight" is a numerical value assigned to the focal area, quantifying its importance in the co-focusing calculation. It comprehensively considers factors such as the number of users within the focal area, observation stability, and the importance of the holographic content, guiding the system to prioritize providing better display effects for more focused areas. "Co-focusing plane" refers to a virtual plane calculated by the system based on the focal area and target weight. This plane represents the optimal focus position for multiple users, and the holographic light field is primarily generated and optimized around this plane to ensure optimal visual clarity in its vicinity. "Holographic light field" refers to a three-dimensional light field generated through holographic technology, capable of reconstructing three-dimensional images with depth and realism. In TOF projection devices, the holographic light field is typically achieved by modulating the phase and amplitude of a laser or LED array. "Local phase compensation" refers to local phase adjustment of the generated holographic light field based on the relative relationship between each user's observation position and the co-focusing plane. This compensation aims to correct defocusing problems caused by users' observation positions deviating from the co-focusing plane, ensuring a clear visual experience for users in different positions.

[0029] In one embodiment, the first step is to acquire observation information from multiple users. Acquiring this observation information is the starting point of the entire system, providing foundational data for subsequent focus identification and collaborative focusing. For example, eye-tracking sensors integrated into a head-mounted display (HMD) or the environment can monitor users' eye movements in real time to obtain their gaze direction. Simultaneously, infrared cameras or Time-of-Flight (TOF) sensors are used to track the user's head position to determine their observation location. These sensors can continuously acquire data at high frame rates, ensuring the real-time nature and accuracy of the observation information. As a preferred implementation, observation information can also be acquired through built-in sensors on smart glasses or smart helmets worn by the user. These sensors can directly measure the user's head posture and eye movements and wirelessly transmit the data to the processing unit. After acquiring the observation information, the system identifies the focus area based on this information and determines the target weight based on the number of users within the focus area, observation stability, and the importance of the holographic content. Focus area identification is a crucial step in understanding user intent. For example, the system can analyze the intersection points or dense areas of multiple users' gaze directions in three-dimensional space, identifying these areas as potential focus areas. When multiple users' gazes are focused on a virtual object or spatial area for an extended period, that area is determined to be a focus area. Observational stability can be assessed by calculating the time a user's gaze lingers within the focal area and the degree of jitter; a longer dwell time and less jitter indicate higher stability. The importance of holographic content can be pre-marked by content creators or system administrators; for example, mission-critical information or core interactive components are assigned higher importance. Target weight calculation can be a weighted average process; for example, areas with a large number of users, areas with high observational stability, and areas containing important content will have correspondingly higher weights. Based on the focal areas and target weights, a co-focusing plane is obtained. Determining the co-focusing plane is crucial for achieving a clear visual experience for multiple users. For example, the system can calculate a weighted average 3D point as the center of the co-focusing plane based on the geometric centers of one or more identified focal areas and the target weight of each focal area. Then, based on the distribution trends of all focal areas and the average viewing distance of users, the normal direction and optimal depth of this plane are determined. As a preferred implementation, the co-focusing plane can also be determined using an iterative optimization algorithm that aims to minimize the average distance from all focal areas to the plane while considering the influence of target weights. Finally, a holographic light field is generated for the co-focusing plane, and local phase compensation is performed on the holographic light field based on the relative relationship between each user's observation position and the co-focusing plane. The generation of the holographic light field typically involves complex diffraction calculations to encode the three-dimensional holographic content into a two-dimensional phase map, which is then projected through a Time-of-Flight (TOF) projection device.For example, Fresnel diffraction or angular spectral diffraction algorithms can be used to propagate light waves from a 3D virtual scene onto the co-focusing plane and calculate the required phase modulation pattern. Local phase compensation is crucial to ensuring that users at different viewing positions can obtain clear images. Specifically, for each user, the system measures the distance and angular deviation between their viewing position and the co-focusing plane. Then, based on these deviations, the amount of local phase adjustment required for the holographic light field is calculated.

[0030] It's important to clarify that acquiring the user's observation point information can be understood as using eye-tracking devices to monitor the user's eye movement trajectory in real time, thereby determining the user's specific gaze point within the holographic display space. Identifying the virtual component the user is currently gazing at refers to determining, based on the observation point information, which virtual object within the holographic content the user's gaze is focused on. Querying the semantic association tags of virtual components aims to obtain the pre-defined logical or functional relationships between that virtual component and other virtual components. For example, two virtual buttons may both be associated with the same operation process, or a virtual chart and a virtual data table may have a data display relationship. Specifically, when the system detects high-frequency, short-duration gaze switching between semantically related virtual components, it means that by analyzing the user's eye movement trajectory, the system detects that the user frequently shifts their gaze from one virtual component to another semantically related virtual component within a short period. For example, when viewing multiple related indicators on a virtual dashboard, the user's gaze will quickly move between these indicators. In this context, dynamically constructing relational focus areas encompassing virtual components and their spatial regions aims to treat these semantically related virtual components, frequently observed by users, and their surrounding local spaces as a unified, higher-attention focus area. This construction of relational focus areas more accurately reflects the user's deeper concerns in complex interaction scenarios, rather than merely a single gaze point. Assigning higher weight values ​​to relational focus areas than to preset regular focus areas highlights their importance. Preset regular focus areas are typically based on a single gaze point or a simple aggregation of user attention areas, while relational focus areas consider the semantic depth of user behavior and interaction intent. Therefore, assigning them higher weight values ​​ensures that these areas receive higher priority in subsequent calculations of the collaborative focus plane, enabling the holographic display system to more accurately respond to the user's collective focus of attention.

[0031] In one embodiment, assuming a multi-user collaborative design review scenario, multiple users are jointly viewing a holographic display of a complex mechanical device model. This model includes multiple virtual components, such as virtual engine parts, performance parameter charts, and assembly instructions. When the system detects that user A and user B frequently switch their gaze between a key component of the virtual engine (e.g., a virtual piston) and its corresponding virtual performance data chart, the system identifies a semantic relationship between the virtual piston and the virtual performance data chart (e.g., they are both related to engine performance analysis). At this point, the system dynamically constructs a relational focus area containing the virtual piston, the virtual performance data chart, and the spatial regions between them. Because this gaze-switching behavior indicates that users have a deep interest in and intention to interact with these related components, the system assigns this relational focus area a higher weight value than regular focus areas. For example, if the weight coefficient of a regular focus area is 1.0, the weight coefficient of this relational focus area may be increased to 1.5 or higher. This higher weighting value was then used to calculate the collaborative focal plane, ensuring that the key component and its related data could be presented to all observers with greater clarity and detail during holographic display, thereby significantly improving the efficiency and accuracy of collaborative review.

[0032] It's important to note that task priority can be understood as the level of importance, urgency, or criticality of the information or function carried by a virtual component within the current collaborative task or application scenario. For example, in holographic guidance for complex equipment maintenance, the task priority of virtual components related to core equipment fault diagnosis will be higher than that of virtual components related to equipment appearance inspection. Task priorities can be predefined in the system configuration or dynamically adjusted based on real-time task flow or user input. Querying the task priority of virtual components contained within a relational focus area refers to the system accessing a preset priority database, task management system, or semantic association rule base to obtain task priority information associated with the virtual components identified within the relational focus area. For example, each virtual component can be assigned one or more task priority tags when created, or its corresponding task module may have a specific priority setting. Based on task priority, a higher weight value is assigned to the relational focus area than to the preset regular focus area, where task priority and weight value are positively correlated. This means that virtual components with higher task priorities will have higher weight values ​​assigned to their relational focus areas. This positive correlation ensures that the system prioritizes and optimizes the display of holographic content most critical to the current task. For example, a weight allocation function can be defined that takes task priority as input and outputs a corresponding weight coefficient. This coefficient is used to adjust or multiply the base weight value to obtain the final target weight.

[0033] In one embodiment, suppose that in a telemedicine collaboration scenario, multiple doctors are observing a patient's 3D holographic model through a Time-of-Flight (TOF) projection device. This holographic model contains multiple virtual components, such as the heart, lungs, tumor region, and auxiliary diagnostic data charts. If the doctors are discussing tumor resection plans, the virtual component representing the "tumor region" will have the highest task priority. When the system detects frequent, short-duration gaze shifts towards the "tumor region" and identifies it as a relational focal area, the system queries its high task priority. Based on this high task priority, the system assigns a significantly higher weight value to this relational focal area than to other areas (such as the lungs or auxiliary data chart areas). Therefore, when generating a collaborative focal plane, the "tumor region" will achieve higher focusing accuracy and better display quality, ensuring that doctors can clearly and accurately observe the details of the tumor, thereby assisting them in making more precise diagnostic and treatment decisions.

[0034] It's important to note that the task phase attribute refers to the various stages in which a collaborative task is divided, such as the design phase, review phase, revision phase, or final confirmation phase. Different task phases may have different focuses and priorities for different virtual components within the holographic content. User role and permission levels refer to the role each user plays in the collaborative task and their corresponding permissions or influence, such as project manager, senior engineer, regular engineer, or observer. These role and permission levels can reflect the user's weight in the decision-making process or their criticality to task progress. Providing configurable weight coefficients for assessing the importance of holographic content under different task phases and role permissions means that the system pre-sets or allows administrators to configure a series of weight values, which are dynamically applied to the importance assessment of the holographic content based on the current task phase and the user's role and permissions. For example, in the design phase, there might be more emphasis on virtual components with design details; while in the review phase, there might be more emphasis on virtual components with overall structure or key performance indicators. Furthermore, the observation behavior of a senior engineer may have a greater impact on the target weight than that of a regular engineer. Real-time monitoring of the current task stage of the collaborative task refers to the system continuously tracking the task's progress, such as updating the specific stage of the task through the task management system interface or manual user input. Obtaining each user's role and permissions means the system identifies the identity of the user currently participating in the collaboration and queries their preset role and permission levels. Based on the task stage and user role and permissions, the importance assessment parameters of the holographic content and the weight contribution coefficient of the number of users are adjusted. The aim is to make the calculation of the target weight more intelligent and contextualized. The importance assessment parameters of the holographic content can be quantitative indicators of the attention, criticality, or urgency of specific virtual components within the holographic content. The weight contribution coefficient of the number of users is used to adjust the degree of influence of the number of users in the focal area on the final target weight. For example, in a critical task stage, the attention of a specific high-privilege user may be more important than the attention of a large number of low-privilege users. In this case, the weight contribution coefficient of the number of users can be reduced, while the individual contribution of high-privilege users can be increased.

[0035] It's important to note that the depth hierarchy attribute can be understood as the Z-axis coordinate or relative depth value of a virtual component in a 3D holographic scene, used to distinguish the spatial relationship between different components. The importance level can be pre-set, for example, through manual marking by content creators, analysis of user behavior data, or automatic evaluation using artificial intelligence algorithms, to reflect the informational value, interaction frequency, or visual salience of the virtual component within the current holographic content. For example, in medical imaging, virtual components in lesion areas can be assigned a higher importance level. The user's desired compensation depth range can be obtained based on the user's observation position and line of sight, combined with the co-focal plane, through geometric calculations or optical models. This range represents the depth area currently of visual focus for the user, and the system will prioritize ensuring the best visual effect for holographic content within this area. In practical applications, after identifying virtual components within the compensation depth range, the system will query the preset depth hierarchy attributes and importance levels of these components. For example, for virtual components that are within the user's desired depth range and have a high level of importance, the system will correspondingly increase the intensity of local phase compensation and may expand the compensation range to ensure that these key components appear clearer and more realistic to the user. Conversely, for components of lower importance or located in non-core depth regions, the compensation intensity and range can be appropriately reduced to save computational resources and avoid unnecessary visual interference.

[0036] In one embodiment, assuming a remote collaborative engineering design scenario, multiple users are jointly reviewing a 3D mechanical model using a Time-of-Flight (TOF) projection device. This mechanical model includes multiple virtual components, such as a core transmission mechanism, auxiliary support structures, and some sensor data visualization interfaces. The core transmission mechanism is preset to have a high importance level and a specific depth hierarchy attribute, while the auxiliary support structures have a lower importance level. When an engineer's observation position and line of sight indicate that they are closely observing the core transmission mechanism, the system calculates their desired compensation depth range based on the engineer's relative relationship to the collaborative focusing plane. Subsequently, the system identifies virtual components within this depth range that contain the core transmission mechanism and queries their high importance level and depth hierarchy attribute. Based on this, the system dynamically adjusts the local phase compensation intensity and range for the area containing the core transmission mechanism, providing stronger compensation and thus presenting higher clarity and detail from the engineer's perspective. Simultaneously, for auxiliary support structures outside the engineer's desired depth range or with lower importance, the system applies relatively weaker phase compensation. In this way, engineers can clearly see the key details of the transmission mechanism without being distracted by the blurring or overcompensation of other non-core components, greatly improving the efficiency and accuracy of collaborative review.

[0037] It's important to note that real-time acquisition of each user's role and permission level or professional domain tag means the system can dynamically identify and acquire the identity attributes of each user currently participating in the collaboration. Role and permission level can be understood as the user's preset permission level in the current collaborative task or system, such as administrator, expert, operator, ordinary user, or visitor. Professional domain tag can be understood as the user's area of ​​expertise or their professional knowledge background, such as mechanical design, electrical engineering, software development, project management, etc. This information can be obtained through user authentication during login, user profiles, preset user group information, or analysis of user behavior patterns during the collaboration process. The purpose is to provide personalized user attribute basis for subsequent weight adjustments. Based on the role and permission level or professional domain tag, the contribution coefficient of the number of users in the focus area to the weight allocation is adjusted to obtain the adjusted first contribution coefficient. Specifically, different role and permission levels or professional domain tags can be associated with different contribution coefficients according to preset mapping rules or algorithms. For example, when a user with a higher role and permission level (such as an expert or administrator) focuses on a certain focus area, the contribution coefficient of their observation behavior to the importance of that area can be set higher than that of an ordinary user. Similarly, when the holographic content within the focal area is highly relevant to a user's professional domain tag, that user's observation behavior can be assigned a higher contribution coefficient. For example, when viewing mechanical design drawings, a mechanical engineer's observation behavior may have a higher weighted contribution than that of a non-professional. Thus, through this adjustment, the contribution of the number of users is no longer a simple count, but a weighted effective attention, resulting in an adjusted first contribution coefficient. Based on this adjusted first contribution coefficient, combined with the number of users within the focal area, observation stability, and the importance of the holographic content, the target weight is obtained. This means that when calculating the final target weight, the original number of users is no longer used directly, but rather the user number contribution coefficient adjusted for role permission level or professional domain tag. This adjusted first contribution coefficient, along with other factors such as observation stability and the importance of the holographic content, participates in the calculation of the target weight, forming a more comprehensive and accurate weight assessment.

[0038] It's important to note that real-time monitoring refers to the system continuously analyzing the data stream and state changes of the holographic content to detect specific events that may attract users' non-explicit attention. These events can be understood as information that suddenly occurs or updates in the holographic display environment, possessing potential importance but which may not have been actively searched for or explicitly noticed by users. For example, abnormal data occurrence could refer to a metric in data visualization suddenly exceeding the normal range; alarm triggering could refer to the system issuing a warning signal; and critical information updates could be the immediate change of an important parameter or state in a collaborative task. The goal is to capture sudden or critical information that may quickly attract the collective attention of users. When the above events occur, the system continuously collects all users' observation point information and eye movement trajectories within a preset time period, such as several seconds. Observation point information refers to the position of the user's current gaze point in the holographic space, while eye movement trajectories record the path and speed of the user's gaze movement. Through comprehensive analysis of this data, collective gaze behavior of users towards the event-related area can be identified. Collective gaze behavior refers to the phenomenon where multiple users focus their gaze on the area where an event occurred within a short period of time after the event takes place. This indicates that the area has become the common focus of the current user group. Once collective gaze behavior is detected, the system identifies the focal area based on the intensity and duration of the behavior. The intensity of collective gaze behavior can be assessed based on the number of users focusing their gaze on the event-related area within a preset time, the density of these users' gaze points, and the convergence of their eye movement trajectories. Duration refers to the length of time that the collective gaze behavior lasts from beginning to end. By comprehensively evaluating the intensity and duration, for example, when preset intensity and duration thresholds are reached, the event-related area can be confirmed as a new focal area. The purpose is to ensure that the identified focal area truly represents the collective attention of the user group, rather than accidental individual behavior.

[0039] This application's solution proactively captures key moments that may quickly attract users' collective attention by introducing a real-time monitoring mechanism for events in holographic content that may trigger non-explicit attention. Because traditional methods may struggle to effectively identify momentary, collective attention caused by sudden events, this application continuously acquires all users' observation point information and eye-tracking trajectories for a preset time during the event, and identifies users' collective gaze behavior towards event-related areas. This allows for timely and accurate capture of such non-explicit but important collective attention. By further identifying focal areas based on the intensity and duration of collective gaze behavior, this application ensures that the identified focal areas are not only triggered by the event but are also points of sufficient importance confirmed by the user group. This mechanism enables the system to respond more sensitively to dynamic changes, avoiding the problems of delayed or inaccurate focal area identification caused by relying solely on explicit, continuous observation.

[0040] In one embodiment, assuming a multi-user collaborative architectural design review scenario, multiple designers are jointly viewing a holographic architectural model via a Time-of-Flight (TOF) projection device. The model displays structural stress analysis data in real time. When the stress value of a critical load-bearing beam in the model suddenly exceeds a safety threshold and is displayed as a flashing red light in the holographic display (i.e., triggering an "abnormal data occurrence" event), the system immediately initiates continuous monitoring of all designers' observation behavior. Over the next 3-second preset timeframe, the system continuously collects each designer's observation point information and eye movement trajectory. If the system detects that within this short period, more than half of the designers' gazes rapidly and intently move to the flashing red load-bearing beam area, and their gazes linger in that area for at least one second (i.e., identifying a "collective gaze behavior" with a certain intensity and duration), then the flashing red load-bearing beam area will be immediately identified as the new focal area. Subsequently, the collaborative focusing plane will be adjusted based on this new focal area to ensure that the critical area presents optimal visual clarity and detail from all designers' perspectives, thereby promptly alerting designers to this potential structural problem and facilitating their collective discussion and resolution of the issue.

[0041] It should be noted that observational behavior data can include users' eye movement trajectories, gaze points, frequency of gaze switching, interaction operation records, task completion efficiency, and historical decision accuracy. This data reflects the user's engagement, focus, and understanding of the holographic content within the holographic display environment. Trust or influence can be understood as a user's authority, reliability, or ability to guide other users in a specific collaborative task or professional field. For example, in scenarios such as medical surgery simulations or complex equipment repair guidance, the observational behavior of senior experts or surgeons should typically have higher trust or influence. Dynamically assessing each user's current trust or influence can be achieved in various ways. For instance, it can be calculated based on pre-set user profile information (such as professional title, years of experience, professional certifications), historical task performance data (such as error rate, efficiency, and the correctness of key decisions), and real-time observational behavior patterns (such as the duration of attention to key information and the speed of reaction to abnormal data). Trust or influence can be quantified into different levels or values, such as multiple levels from low to high, or a continuous percentage value. Based on the level of trust or influence, the contribution coefficient of users within the focus area to the weight allocation can be adjusted. For example, the contribution coefficient of a user with a higher level of trust or influence to the target weight can be set above average, and vice versa. The adjusted second contribution coefficient will be used to correct the original weight allocation based on the number of users, ensuring that more authoritative or reliable users play a greater role in determining the collaborative focus plane.

[0042] In one embodiment, assuming a telemedicine consultation scenario, multiple doctors observe a patient's 3D holographic image through a Time-of-Flight (TOF) projection device. These include an attending physician, an intern, and a radiologist. Traditional methods might treat all doctors' observation behaviors equally. However, according to the solution of this application, the system acquires each doctor's observation behavior data in real time, such as eye movement trajectory and gaze duration. Based on this data, the system dynamically evaluates each doctor's trust level or influence. For example, the attending physician, due to their rich experience and decision-making authority, has the highest level of trust or influence; the radiologist, with expertise in image analysis, has the next highest level; and the intern, due to insufficient experience, has a relatively low level. When the attending physician exhibits prolonged and stable gazing at a specific lesion area in the holographic image, even if the intern and radiologist may simultaneously focus on other areas, the attending physician's higher trust level or influence significantly increases the contribution coefficient of their observation behavior to the target weight, thus giving that lesion area a higher priority in the co-focusing plane calculation. Therefore, the system will be more inclined to precisely focus the co-focusing plane on the lesion area and perform more refined local phase compensation on the holographic light field of that area, ensuring that the attending physician can observe key details with the best visual effect. This approach avoids interference from non-critical observation behaviors that interns may have on the overall focusing effect, ensuring that holographic display resources can prioritize serving the most critical decision-makers and the most core diagnostic information, thereby improving the efficiency and accuracy of consultations.

[0043] It should be noted that the observed behavior data may include the user's eye movement trajectory, fixation points, head posture, interaction actions, etc. Checking the observed behavior data aims to discover inconsistencies, incompleteness, or inaccuracies in the data. Noise data points refer to data fluctuations or random deviations caused by sensor errors, environmental interference, etc.; missing data points refer to data that fails to be successfully collected within a specific time period; abnormal data points refer to data that significantly deviates from the normal behavior pattern, which may be caused by the user's unintentional behavior or system failures. When noise data points are identified, various smoothing or filtering methods can be adopted. For example, a moving average filter, a Gaussian filter, or a Kalman filter can be used to process the data to eliminate random noise and make the data trend clearer. When missing data points are identified, linear interpolation, polynomial interpolation, or spline interpolation can be performed based on the observed behavior data at the time points before and after them to estimate the missing values. In addition, the historical observed behavior pattern of this user can also be combined, and a machine learning model can be used for predictive completion to improve the accuracy of completion. When abnormal data points are identified, statistical thresholds can be set to determine whether the data is abnormal. Or, the behavior data of the current user can be compared with the behavior data of other users in real time. If the behavior data of a certain user significantly deviates from the group behavior, it may be regarded as abnormal. For abnormal data points, they can be corrected to more reasonable values or directly removed to avoid their negative impact on the evaluation results.

[0044] In one embodiment, it is assumed that in a holographic design review scenario of multi-user collaboration, the system needs to evaluate the user's attention and professionalism towards the design scheme based on the observed behavior. At a certain moment, due to a short-term failure of the eye-tracking device, a segment of the eye movement trajectory data of a certain user is missing. At the same time, due to the user's slight head movement, there is slight random noise in some of the fixation point data. In addition, the system also detects that the user suddenly switches the line of sight from the key design area to the non-critical information at the edge of the screen in a short period of time, which may be an abnormal data point. First, these observed behavior data are checked. For the missing eye movement trajectory data, the system can adopt linear interpolation or predictive completion based on the user's eye movement pattern before and after the failure to restore the continuity of the data. For the fixation point data with random noise, the system can apply a moving average filter for smoothing to eliminate the noise interference and make the true position of the fixation point more accurate. For the abnormal line-of-sight switching behavior, the system can compare it with the normal behavior pattern of the user in other time periods or with the behavior of other users in the same collaboration group. If it is confirmed as abnormal, it can be corrected to a more reasonable fixation behavior or the impact of this abnormal data point on the trustworthiness evaluation can be temporarily removed.

[0045] See Figure 2 ,Figure 2 This is a schematic diagram of a holographic display system based on a Time-of-Flight (TOF) projection device, provided as an embodiment of this application. The TOF-based holographic display system 200 includes: The acquisition module 210 is used to acquire observation information from multiple users, including observation position and line of sight direction; The identification module 220 is used to identify the focal region based on the observation information, and to obtain the target weight based on the number of users in the focal region, the observation stability, and the importance of the holographic content. Calculation module 230 is used to calculate based on the focal region and the target weight to obtain a cooperative focusing plane; The adjustment module 240 is used to generate a holographic light field for the cooperative focusing plane and to perform local phase compensation on the holographic light field according to the relative relationship between each user's observation position and the cooperative focusing plane.

[0046] It should be noted that the information interaction and execution process between the above modules are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0047] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0048] The foregoing has provided a detailed description of the preferred embodiments of this application. However, this application is not limited to the above-described embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined in this application.

Claims

1. A holographic display method based on a TOF projection device, characterized in that, include: Acquire observation information from multiple users, including observation position and line of sight direction; Based on the observation information, a focal region is identified, and a target weight is obtained based on the number of users within the focal region, observation stability, and the importance of the holographic content. A collaborative focusing plane is obtained based on the focal region and the target weight; A holographic light field is generated for the co-focusing plane, and local phase compensation is performed on the holographic light field according to the relative relationship between each user's observation position and the co-focusing plane.

2. The method according to claim 1, characterized in that, The step of identifying a focal region based on the observation information, and obtaining a target weight based on the number of users within the focal region, observation stability, and the importance of the holographic content, includes: Obtain the user's observation point information, identify the virtual component that the user is currently gazing at, and query the semantic association tags of the virtual component; When it is detected that the user is switching between the virtual components with semantic association at a high frequency and for a short period of time, a relational focus area containing the virtual components and the spatial regions of the virtual components is dynamically constructed, wherein the relational focus area is used as the focus area. Based on the number of users within the focal area, observation stability, and the importance of the holographic content, a higher weight value than that of a preset regular focal area is assigned to the relational focal area to obtain the target weight.

3. The method according to claim 2, characterized in that, Assigning a weight value higher than that of a preset regular focus region to the relational focus region includes: Query the task priority of the virtual component contained in the relational focus area; Based on the task priority, a weight value higher than that of a preset regular focus area is assigned to the relational focus area, wherein the task priority is positively correlated with the weight value.

4. The method according to claim 1, characterized in that, The process of identifying the focal region based on the observation information, and then, based on the number of users within the focal region, observation stability, and the importance of the holographic content, further includes: The virtual components in the holographic content within the focal area are preset with task stage attributes and user role permission levels, and configurable weight coefficients are provided for the importance assessment of the holographic content under different task stages and role permissions. Real-time monitoring of the current stage of collaborative tasks and acquisition of each user's role permissions; Based on the task stage and the user's role permissions, adjust the importance assessment parameters of the holographic content and the weight contribution coefficient of the number of users to obtain the target weight.

5. The method according to claim 1, characterized in that, The process of generating a holographic light field for the co-focusing plane and performing local phase compensation on the holographic light field based on the relative relationship between each user's observation position and the co-focusing plane includes: Obtain the preset depth level attribute and importance level of the virtual component in the holographic content. The preset depth level attribute describes the relative depth position of the virtual component in three-dimensional space, and the importance level describes the visual or interactive priority of the virtual component in the holographic content. The system acquires the user's observation position and line of sight in real time, and obtains the user's desired compensation depth range based on the relative relationship between the collaborative focusing plane and the user. Identify the virtual components included within the compensation depth range, and query the depth level attribute and importance level of the virtual components; Based on the depth hierarchy attribute of the virtual component and the importance level, the intensity and range of the local phase compensation are adjusted to obtain the adjusted compensation intensity and range; A holographic light field is generated for the co-focusing plane, and local phase compensation is performed on the holographic light field based on the relative relationship between each user's observation position and the co-focusing plane, as well as the adjusted compensation intensity and range.

6. The method according to claim 1, characterized in that, The step of obtaining the target weight based on the number of users within the focal area, observation stability, and the importance of the holographic content also includes: Real-time acquisition of each user's role and permission level or professional field tags; Based on the role permission level or the professional field tag, adjust the contribution coefficient of the number of users in the focus area to the weight allocation to obtain the adjusted first contribution coefficient; Based on the adjusted first contribution coefficient, and combined with the number of users in the focal area, observation stability, and the importance of the holographic content, the target weight is obtained.

7. The method according to claim 1, characterized in that, The step of identifying the focal region based on the observation information includes: Real-time monitoring of events that may trigger non-explicit attention in the holographic content within the focal area, including the appearance of abnormal data, alarm triggering, or updates to critical information; When the event occurs, the system continuously acquires the observation point information and eye movement trajectory of all users within a preset time period, and identifies the collective gaze behavior of users towards the relevant area of ​​the event. The focal area is identified based on the intensity and duration of the collective gaze behavior.

8. The method according to claim 7, characterized in that, The step of obtaining the target weight based on the number of users within the focal area, observation stability, and the importance of the holographic content also includes: Real-time acquisition of observational behavior data for each user; Based on the observed behavioral data, dynamically assess each user's current level of trust or influence; Based on the level of trust or influence, the contribution coefficient of users in the focus area to the weight allocation is adjusted to obtain the adjusted second contribution coefficient. Based on the adjusted second contribution coefficient, and combined with the number of observers in the focal region, observation stability, and the importance of the holographic content, the target weight is obtained.

9. The method according to claim 8, characterized in that, The step of dynamically assessing each user's current trust level or influence based on the observed behavioral data includes: The observed behavior data is examined and noisy data points, missing data points, and abnormal data points are identified. When the noise data point is identified, it is smoothed or filtered. When the missing data point is identified, the missing data point is interpolated and filled in according to the observed behavior data at adjacent time points or the user's historical behavior pattern. When the abnormal data point is identified, it is corrected or removed based on a preset abnormal threshold or by comparing it with the behavior data of other users. Based on the processed observation data, the current trust level or influence of each observer is dynamically assessed.

10. A holographic display system based on a TOF projection device, characterized in that, include: The acquisition module is used to acquire observation information from multiple users, including observation position and line of sight direction; The identification module is used to identify the focal region based on the observation information, and to obtain the target weight based on the number of users in the focal region, the observation stability, and the importance of the holographic content. The calculation module is used to calculate, based on the focal region and the target weight, a cooperative focusing plane; The adjustment module is used to generate a holographic light field for the co-focusing plane and to perform local phase compensation on the holographic light field according to the relative relationship between each user's observation position and the co-focusing plane.

Images