Imaging Detection Method and System for a Head-Mounted Display Device

Abstract

The present invention relates to the field of display devices, and discloses an imaging detection method and system for a head-mounted display device, which is applicable to a head-mounted display device for mixed reality. With the cooperation of an image sensor, an attitude sensor, etc., real-time virtual synchronization of environmental data and its own movement is achieved, so as to achieve the purpose of enhancing and displaying the visual interaction experience through mixed reality. Moreover, through the mapping correction of the imaging by the perspective shaking angle and the correction of the visual center, a more immersive virtual overlay experience is achieved compared with the prior art, and the picture content can be output more accurately to achieve the positioning coverage of the real scene, and the best output state effect is achieved in cooperation with the perspective positioning.

Description

Technical Field

[0001] The present invention relates to the field of display devices, and specifically to an imaging detection method and system for a head-mounted display device. Background Art

[0002] Mixed Reality (MR) is a hybrid technology solution based on Virtual Reality (VR) and Augmented Reality (AR). In mixed reality, users can not only see the real world but also virtual content generated by a computer. By combining the two, a more immersive real-world interaction effect can be achieved.

[0003] In the prior art, when a mixed reality wearable device is in use, it will collect environmental data through sensing devices and cameras, identify and judge environmental objects and related content, and then generate the content to be output and output it through a display projection device. In this process, an attitude sensor can perform motion calculations through relevant readings such as inertial attitude, so as to achieve synchronization with the motion of the real user. However, in the prior art, most of the superposition of virtual and real uses the method of camera ranging, and due to the different wearing habits and visual habits of different users, there are large errors in the fixed-scale superposition method. Summary of the Invention

[0004] The purpose of the present invention is to provide an imaging detection method and system for a head-mounted display device to solve the problems raised in the above background art.

[0005] To achieve the above purpose, the present invention provides the following technical solutions:

[0006] An imaging detection method for a head-mounted display device, comprising:

[0007] Real-time obtaining environmental space data within a scanning field of view through a sensing device group, and performing high-precision space modeling based on the environmental space data to obtain an environmental twin model, where the environmental space data includes environmental image data and laser ranging data;

[0008] Obtaining the current motion attitude data of the display device through an attitude sensing device group and performing real-time synchronous update, and performing motion integral calculation through the motion attitude data to obtain the spatial motion travel within a time period, where the motion attitude data includes attitude orientation data and motion inertia data;

[0009] Obtaining the viewing angle swing angle of the user, and mapping the display screen based on the viewing angle swing angle to synchronize the user's viewing angle with the display angle of the display device, where the viewing angle swing angle is used to represent the deflection angle of the visual center of the user between two time nodes;

[0010] The user's pupil is tracked, and the user's visual center is obtained through an initial request. The spatial perspective difference between the visual center and the current posture orientation data is calculated, and the display output field of view is calibrated based on the spatial perspective difference. The spatial perspective difference is used to characterize the spatial orientation angle between the user's visual center and the display output center of the display device. The visual center is used to characterize the user's head-on viewing angle.

[0011] As a further solution of the present invention, the step of obtaining the user's viewing angle and mapping the display image based on the viewing angle to synchronize the user's viewing angle with the display viewing angle of the display device specifically includes:

[0012] When it is detected that the wearing state of the display device changes, a viewing angle range synchronization request is generated, and after determining to obtain the visual center of the user, a projection display distance between the user's pupil and the display device is detected;

[0013] In response to the range synchronization request, a pair of synchronization reference nodes are randomly selected based on the environment twin model, and the pair of synchronization reference nodes are displayed and marked in the environment twin model;

[0014] When the visual center of the display device coincides with the synchronization reference node, the current posture orientation data is recorded, and the posture inclination difference of the posture orientation data between a pair of synchronization reference nodes is calculated, that is, the viewing angle shaking angle of the corresponding display device;

[0015] The visual width under the corresponding viewing angle shaking angle is calculated based on the projection display distance, and the display scaling ratio is calculated through the actual display distance of a pair of synchronous reference nodes, and the output picture of the display device is synchronously scaled based on the display scaling ratio.

[0016] As a further solution of the present invention: it also includes a user characterization correction step, specifically including:

[0017] When tracking the user's pupil, generate a view initialization request to guide the user's view center to coincide with the display center of the display device. If the user's pupil angle does not coincide with the display center of the display device, guide the user to perform feature correction.

[0018] Generate multiple sets of correction guide points distributed in multiple plane quadrants of the display range, and record the pupil angles of the user at the multiple correction guide points when the user feedback visual centers coincide;

[0019] Based on the relative position relationship between the multiple correction guide points and the corresponding position relationship between the multiple pupil angles, a fitting calculation is performed to obtain the user's pupil movement mapping model, which is used to characterize the actual visual center change corresponding to when the user's pupil rotates a certain angle.

[0020] As a further aspect of the present invention: It further includes an imaging resource optimization step:

[0021] The visual center of the user is obtained in real time through pupil tracking, and the visual center is set as the first imaging center;

[0022] Based on the first imaging center as the center, a plurality of annular display areas are established. The plurality of annular display areas respectively correspond to a plurality of display output schemes, and the display output resolution and content refresh frequency of the plurality of annular display areas gradually decrease along the radial direction.

[0023] As a further aspect of the present invention: The imaging resource optimization step further includes:

[0024] The environmental space data within the scanning field of view is obtained in real time, and the differential data is updated based on the environmental twin model;

[0025] Based on the object area where the differential data is updated, a second imaging center is established. The second imaging center is different from the first imaging center, and the second imaging center does not include a plurality of accompanying annular display areas;

[0026] The same display output scheme as that of the first imaging center is adopted for the corresponding display data content of the second imaging center.

[0027] An imaging detection system for a head-mounted display device according to an embodiment of the present invention includes:

[0028] An environment synchronization module, configured to obtain the environmental space data within the scanning field of view in real time through a sensor device group, and perform high-precision space modeling based on the environmental space data to obtain an environmental twin model. The environmental space data includes environmental image data and laser ranging data;

[0029] A motion synchronization module, configured to obtain the current motion attitude data of the display device based on an attitude sensor device group and perform real-time synchronization update, and perform motion integration calculation through the motion attitude data to obtain the spatial motion travel within a time period. The motion attitude data includes attitude orientation data and motion inertia data;

[0030] A viewing angle synchronization module, configured to obtain the viewing angle shaking angle of the user, and map the display screen based on the viewing angle shaking angle to synchronize the user's viewing angle with the display viewing angle of the display device. The viewing angle shaking angle is used to represent the deflection angle of the visual center of the user between two time nodes;

[0031] A field of view calibration module is used to perform pupil tracking on the user, obtain the visual center of the user through an initial request, calculate the spatial perspective difference between the visual center and the current attitude orientation data, and calibrate the display output field of view based on the spatial perspective difference. The spatial perspective difference is used to represent the spatial orientation angle between the user's perspective center and the display output center of the display device, and the visual center is used to represent the user's forward-looking perspective.

[0032] As a further solution of the present invention: The perspective synchronization module specifically includes:

[0033] A viewing distance detection unit is used to generate a perspective range synchronization request when it detects that the wearing state of the display device has changed, and detect and obtain the projection display distance between the user's pupil and the display device after determining to obtain the user's visual center;

[0034] A reference marking unit is used to respond to the range synchronization request, randomly select a pair of synchronization reference nodes based on the environmental twin model, and perform display marking on the pair of synchronization reference nodes in the environmental twin model;

[0035] An angle-of-rotation acquisition unit is used to record the current attitude orientation data when the visual center of the display device coincides with the synchronization reference node, and calculate the attitude inclination difference of the attitude orientation data between a pair of synchronization reference nodes, that is, the viewing angle swing angle corresponding to the display device;

[0036] A field-of-view synchronization unit is used to calculate the viewing width at the corresponding viewing angle swing angle based on the projection display distance, calculate the display scaling ratio through the actual display distance between a pair of synchronization reference nodes, and perform perspective synchronization scaling on the output picture of the display device based on the display scaling ratio.

[0037] As a still further solution of the present invention: It further includes a characterization correction module, which specifically includes:

[0038] A correction initial unit is used to generate a perspective initialization request when performing pupil tracking on the user, so as to guide the user's perspective center to coincide with the display center of the display device. If the user's pupil angle does not coincide with the display center of the display device at this time, then guide to perform characterization correction;

[0039] A correction marking unit is used to generate multiple groups of correction guiding points distributed in multiple plane quadrants of the display range, and record the pupil angles of the user at multiple correction guiding points when the user feedbacks that the visual centers coincide;

[0040] A correction mapping unit, configured to perform fitting calculations based on the relative positional relationships of multiple correction guiding points and the positional relationships between corresponding multiple pupil angles, so as to obtain a pupil movement mapping model of the user, where the pupil movement mapping model is used to represent the actual visual center change amount corresponding to the user's pupil when it rotates by a certain angle.

[0041] As a further aspect of the present invention: It further includes a resource optimization module, specifically including:

[0042] A first center determination unit, configured to obtain the visual center of the user in real time through pupil tracking and set the visual center as the first imaging center;

[0043] A display optimization management unit, configured to establish multiple annular display areas with the first imaging center as the center, where the multiple annular display areas respectively correspond to multiple display output schemes, and the display output resolution and content refresh frequency of the multiple annular display areas gradually decrease along the radial direction.

[0044] As a further aspect of the present invention: The resource optimization module further includes:

[0045] An update synchronization unit, configured to obtain the environmental space data within the scanning field of view in real time and perform difference data update based on the environmental twin model;

[0046] A second center determination unit, configured to establish a second imaging center based on the object area where the difference data is updated, where the second imaging center is different from the first imaging center, and the second imaging center does not include multiple accompanying annular display areas;

[0047] A display resource management unit, configured to adopt a display output scheme consistent with that of the first imaging center for the corresponding display data content of the second imaging center.

[0048] Compared with the prior art, the beneficial effects of the present invention are: It is applicable to a head-mounted display device for mixed reality. With the cooperation of an image sensor, an attitude sensor, etc., it realizes real-time virtual synchronization of environmental data and its own movement, thereby achieving the purpose of enhancing the visual interaction experience through mixed reality and display. Moreover, through the mapping correction of imaging by the perspective shaking angle and the correction of the visual center, it has a more immersive virtual overlap experience compared with the prior art, can output the picture content more accurately to achieve the positioning coverage of the real scene, and cooperate with the perspective positioning to achieve the best output state effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a flowchart of an imaging detection method for a head-mounted display device.

[0050] Figure 2It is a flowchart of the step of synchronizing the user's perspective with the display perspective of the display device in an imaging detection method for a head-mounted display device.

[0051] Figure 3 It is a block diagram of the composition of an imaging detection system for a head-mounted display device. Detailed implementation manners

[0052] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0053] The following describes in detail the specific implementation manners of the present invention with reference to specific embodiments.

[0054] As Figure 1 described, an imaging detection method for a head-mounted display device provided by an embodiment of the present invention includes the following steps:

[0055] S10. Real-time obtain environmental space data within the scanning field of view through a sensor device group, and perform high-precision space modeling based on the environmental space data to obtain an environmental twin model, where the environmental space data includes environmental image data and laser ranging data;

[0056] S20. Obtain the current motion attitude data of the display device through an attitude sensor device group and perform real-time synchronous update, and perform motion integral calculation through the motion attitude data to obtain the spatial motion travel within a time period, where the motion attitude data includes attitude orientation data and motion inertia data;

[0057] S30. Obtain the perspective swing angle of the user, and map the display screen based on the perspective swing angle to synchronize the user's perspective with the display perspective of the display device, where the perspective swing angle is used to characterize the deflection angle of the visual center of the user between two time nodes;

[0058] S40. Perform pupil tracking on the user, obtain the visual center of the user through an initial request, calculate the spatial perspective difference between the visual center and the current attitude orientation data, and calibrate the display output field of view based on the spatial perspective difference, where the spatial perspective difference is used to characterize the spatial orientation angle between the user's perspective center and the display output center of the display device, and the visual center is used to characterize the user's level viewing perspective.

[0059] In this embodiment, an imaging detection method for a head-mounted display device is provided, which is applicable to a head-mounted display device for mixed reality. With the cooperation of an image sensor, an attitude sensor, etc., real-time virtual synchronization of environmental data and its own movement is achieved, so as to achieve the purpose of enhancing and displaying the visual interaction experience through mixed reality. Moreover, through the mapping correction of the imaging by the perspective shaking angle and the correction of the visual center, a more immersive virtual overlap experience is provided compared with the prior art, and the picture content can be output more accurately to achieve the positioning coverage of the real scene, and the best output state effect is achieved in cooperation with the perspective positioning; Mixed Reality (MR) is a hybrid technology solution based on virtual reality and augmented reality. In mixed reality, users can not only see the real world but also see virtual content generated by a computer. By combining the two, a more immersive real-world interaction effect is achieved; At present, mixed reality devices mainly include two types. One is a fully virtual mixed reality display device with full-field coverage. This kind of mixed reality display device completely collects image data through a camera module, performs visual occlusion, and then completely provides visual interaction by the display device; The other is semi-hybrid. The user can still directly see the real world. Compared with the former, the latter can provide a more realistic feedback in terms of visual perception because it can still directly see the original external environment, while the former can obtain a better sense of immersion because of the occlusion effect; Generally speaking, in the prior art, when a semi-hybrid wearable device is used, environmental data is collected through a sensing device, a camera, etc., environmental objects and related content are identified and judged, and then the content to be output is generated and output through a display projection device. In this process, an attitude sensor, etc., can perform motion calculation through relevant readings such as inertial attitude, so as to achieve synchronization with the movement of the real user. However, in the prior art, most of the superposition of virtual and real uses the method of camera ranging, and due to the different wearing habits and visual habits of different users, there are large errors in the fixed-mode scaling superposition method; In this embodiment, a motion correction method is adopted. By cooperating with the user, the fixed-point motion angle information of the user is mapped to the display content, so that the perspective obtained by the user in the display device can highly overlap with the real perspective. Here, the initial request is used to represent sending an initialization request to the user. At this time, the user sets his own perspective to the normal perspective in the normal state, so that the standard visual center of the user can be detected, which can be used for the field-of-view correction management of the output image.

[0060] As Figure 2 shown, as another preferred embodiment of the present invention, the steps of obtaining the perspective shaking angle of the user, mapping the display screen based on the perspective shaking angle to synchronize the user's perspective with the display perspective of the display device specifically include:

[0061] S41, when it is detected that the wearing state of the display device changes, a viewing angle range synchronization request is generated, and after determining to obtain the visual center of the user, a distance between the pupil of the user and the projection display of the display device is detected;

[0062] S42, in response to the range synchronization request, randomly selecting a pair of synchronization reference nodes based on the environment twin model, and displaying and marking the pair of synchronization reference nodes in the environment twin model;

[0063] S43, when the visual center of the display device coincides with the synchronization reference node, the current posture orientation data is recorded, and the posture inclination difference of the posture orientation data between a pair of synchronization reference nodes is calculated, that is, the viewing angle shaking angle of the corresponding display device;

[0064] S44, calculating the visual width at the corresponding viewing angle shaking angle based on the projection display distance, calculating the display scaling ratio through the actual display distance of a pair of synchronous reference nodes, and performing viewing angle synchronous scaling on the output picture of the display device based on the display scaling ratio.

[0065] In this embodiment, the steps of mapping based on the viewing angle shaking are further explained. When a change in the wearing state of the device is detected, the correction step is started. First, the pupil distance between the device and the user's pupil after wearing is required as an important parameter for subsequent mapping; two synchronization reference points with a certain distance between them are randomly generated, and the user is informed through content interaction. When the user moves his head to make the synchronization reference point overlap with the visual center, the current posture orientation data is marked. By calculating the plane angle of the user's movement between the two synchronization reference points, the angle information of the convergence in the user's observation can be obtained (with the user's eyeball as the starting point of the angle). If you want to achieve an effect of complete overlap with reality, the two synchronization reference points in the display device should also have the same angle relative to the pupil, so as to achieve an overlapping effect in the user's realization. Therefore, the visual width is calculated by the viewing angle shaking angle and the pupil distance.

[0066] As another preferred embodiment of the present invention, it also includes a user characterization correction step, specifically including:

[0067] When tracking the user's pupil, generate a view initialization request to guide the user's view center to coincide with the display center of the display device. If the user's pupil angle does not coincide with the display center of the display device, guide the user to perform feature correction.

[0068] Generate multiple sets of correction guide points distributed in multiple plane quadrants of the display range, and record the pupil angles of the user at the multiple correction guide points when the user feedback visual centers coincide;

[0069] Performing fitting calculations based on the relative positional relationships of the multiple correction guiding points and the positional relationships between the corresponding multiple pupil angles to obtain a pupil movement mapping model of the user, where the pupil movement mapping model is used to represent the actual visual center change amount corresponding to the user's pupil when it rotates by a certain angle.

[0070] In this embodiment, specifically for individual users, there may still be some users with strabismus problems due to physical reasons. Therefore, there are still serious adaptation problems in the eye scanning and tracking of related display devices in the prior art. For a small number of strabismic users, the system cannot accurately identify the accurate visual center position of the user, resulting in incorrect output of image content. The solution adopted here is to randomly generate correction guiding points distributed in multiple quadrants within the output viewing angle range of the display device, which are aligned and fed back by the user one by one. Finally, the system calculates the strabismus deviation amount when the user's pupil moves in different directions based on the actual positions of several correction guiding points and the relative position of the user's pupil, and uses it as the pupil movement mapping model of the user's characteristics for automatic correction during subsequent pupil tracking.

[0071] As another preferred embodiment of the present invention, it further includes an imaging resource optimization step:

[0072] Real-time obtaining the visual center of the user through pupil tracking and setting the visual center as the first imaging center;

[0073] Based on the first imaging center as the center, multiple annular display areas are established. The multiple annular display areas respectively correspond to multiple display output schemes, and the display output resolution and content refresh frequency of the multiple annular display areas gradually decrease along the radial direction.

[0074] Furthermore, the imaging resource optimization step further includes:

[0075] Real-time obtaining the environmental space data within the scanning field of view and performing differential data update based on the environmental twin model;

[0076] Based on the object area where the differential data is updated, a second imaging center is established. The second imaging center is different from the first imaging center, and the second imaging center does not include multiple accompanying annular display areas;

[0077] Adopting a display output scheme consistent with that of the first imaging center for the corresponding display data content of the second imaging center.

[0078] In this embodiment, it is used to manage and optimize the limited computing power resources and energy consumption of real devices. By weakening the display quality outside the line-of-sight aggregation area, the occupation of computing power resources and energy consumption is reduced. At the same time, when new object content appears in the field of vision, it often easily causes the user's attention to shift, enabling the user to quickly shift the visual viewing object. Therefore, for the corresponding content of differential data update, the speed of this process is much higher than the line-of-sight switching speed when normally browsing objects. To ensure the user's fast perspective switching, the corresponding second imaging center should also adopt complete resource scheduling to ensure the output resolution, refresh frame rate, etc.

[0079] As Figure 3 shown, the present invention also provides an imaging detection system for a head-mounted display device, which includes:

[0080] An environment synchronization module 100, configured to obtain environmental space data within the scanned field of view in real time through a sensor device group, and perform high-precision spatial modeling based on the environmental space data to obtain an environmental twin model. The environmental space data includes environmental image data and laser ranging data;

[0081] A motion synchronization module 200, configured to obtain the current motion posture data of the display device based on a posture sensor device group and perform real-time synchronization and update, and perform motion integration calculation through the motion posture data to obtain the spatial motion travel within a time period. The motion posture data includes posture orientation data and motion inertia data;

[0082] A perspective synchronization module 300, configured to obtain the user's perspective swing angle, map the display screen based on the perspective swing angle to synchronize the user's perspective with the display perspective of the display device. The perspective swing angle is used to represent the deflection angle of the visual center between two time nodes of the user;

[0083] A field-of-view calibration module 400, configured to perform pupil tracking on the user, obtain the user's visual center through an initial request, calculate the spatial perspective difference between the visual center and the current posture orientation data, and calibrate the display output field of view based on the spatial perspective difference. The spatial perspective difference is used to represent the spatial orientation angle between the user's perspective center and the display output center of the display device, and the visual center is used to represent the user's frontal perspective.

[0084] As another preferred embodiment of the present invention, the perspective synchronization module 300 specifically includes:

[0085] A viewing distance detection unit 310, configured to generate a perspective range synchronization request when detecting that the wearing state of the display device has changed, and detect and obtain the projection display distance between the user's pupil and the display device after determining to obtain the user's visual center;

[0086] A reference marker unit 320, configured to randomly select a pair of synchronous reference nodes based on an environmental twin model in response to the range synchronization request, and display and mark the pair of synchronous reference nodes in the environmental twin model;

[0087] A corner acquisition unit 330, configured to record the current attitude and orientation data when the visual center of the display device coincides with the synchronous reference node, and calculate the attitude inclination difference of the attitude and orientation data between the pair of synchronous reference nodes, that is, the viewing angle swing angle corresponding to the display device;

[0088] A field of view synchronization unit 340, configured to calculate the viewing width corresponding to the viewing angle swing angle based on the projection display distance, calculate the display scaling ratio through the actual display distance between the pair of synchronous reference nodes, and perform perspective synchronization scaling on the output image of the display device based on the display scaling ratio.

[0089] As another preferred embodiment of the present invention, it further includes a characterization correction module, specifically including:

[0090] An initial correction unit, configured to generate a viewing angle initialization request when performing pupil tracking on a user to guide the user's viewing angle center to coincide with the display center of the display device. If the user's pupil angle does not coincide with the display center of the display device at this time, it guides the user to perform characterization correction;

[0091] A correction marking unit, configured to generate multiple groups of correction guiding points distributed in multiple plane quadrants of the display range, and record the pupil angles of the user at the multiple correction guiding points when the user feedbacks that the visual centers coincide;

[0092] A correction mapping unit, configured to perform fitting calculation based on the relative position relationship of the multiple correction guiding points and the position relationship between the corresponding multiple pupil angles to obtain the pupil movement mapping model of the user, and the pupil movement mapping model is used to characterize the actual visual center change amount corresponding to the user's pupil when rotating a certain angle.

[0093] As another preferred embodiment of the present invention, it further includes a resource optimization module, specifically including:

[0094] A first center determination unit, configured to obtain the visual center of the user in real time through pupil tracking and set the visual center as the first imaging center;

[0095] A display optimization management unit, configured to establish multiple annular display areas with the first imaging center as the center. The multiple annular display areas respectively correspond to multiple display output schemes, and the display output resolution and content refresh frequency of the multiple annular display areas gradually decrease along the radial direction.

[0096] As another preferred embodiment of the present invention, the resource optimization module further includes:

[0097] An update synchronization unit, configured to obtain environmental space data within the scanning field of view in real time, and update the difference data based on the environmental twin model;

[0098] A second center determination unit, configured to establish a second imaging center based on the object area where the difference data is updated, the second imaging center being different from the first imaging center, and the second imaging center not including a plurality of accompanying annular display areas;

[0099] A display resource management unit, configured to adopt a display output scheme consistent with that of the first imaging center for the corresponding display data content of the second imaging center.

[0100] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0101] After considering the specification and the disclosure of the embodiments, those skilled in the art will readily conceive of other embodiments of the present disclosure. The present application aims to cover any variations, uses, or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common general knowledge or conventional technical means in the technical field not disclosed in the present disclosure. The specification and the embodiments are only regarded as exemplary, and the true scope and spirit of the present disclosure are pointed out by the claims.

[0102] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An imaging detection method for a head-mounted display device, characterized in that, Including: Obtain the environmental space data within the scanning field of view in real time through a group of sensing devices, and perform high-precision spatial modeling based on the environmental space data to obtain an environmental twin model. The environmental space data includes environmental image data and laser ranging data; Obtain the current motion attitude data of the display device based on a group of attitude sensing devices and perform real-time synchronous update, and perform motion integral calculation through the motion attitude data to obtain the spatial motion travel within a time period. The motion attitude data includes attitude orientation data and motion inertia data; Obtain the user's viewing angle of rotation, and map the display screen based on the viewing angle of rotation to synchronize the user's viewing angle with the display angle of the display device. The viewing angle of rotation is used to represent the deflection angle of the user's visual center between two time nodes; Perform pupil tracking on the user, obtain the user's visual center through an initial request, calculate the spatial viewing angle difference between the visual center and the current attitude orientation data, and calibrate the display output field of view based on the spatial viewing angle difference. The spatial viewing angle difference is used to represent the spatial orientation angle between the user's viewing center and the display output center of the display device, and the visual center is used to represent the user's frontal viewing angle; The step of obtaining the user's viewing angle of rotation and mapping the display screen based on the viewing angle of rotation to synchronize the user's viewing angle with the display angle of the display device specifically includes: When it is detected that the wearing state of the display device has changed, generate a viewing range synchronization request, and after determining to obtain the user's visual center, detect and obtain the projection display distance between the user's pupil and the display device; Respond to the range synchronization request, randomly select a pair of synchronization reference nodes based on the environmental twin model, and perform display marking on the pair of synchronization reference nodes in the environmental twin model; When the visual center of the display device coincides with the synchronization reference node, record the current attitude orientation data, and calculate the attitude inclination difference of the attitude orientation data between a pair of synchronization reference nodes, that is, the viewing angle of rotation corresponding to the display device; Calculate the viewing width corresponding to the viewing angle of rotation based on the projection display distance, calculate the display scaling ratio through the actual display distance between a pair of synchronization reference nodes, and perform viewing angle synchronous scaling on the output screen of the display device based on the display scaling ratio; It further includes a user characterization correction step, specifically including: When performing pupil tracking on the user, generate a viewing initialization request to guide the user's viewing center to coincide with the display center of the display device. If the user's pupil angle does not coincide with the display center at this time, guide for characterization correction; Generate multiple groups of correction guidance points distributed in multiple plane quadrants within the display range, and record the pupil angles of the user at the multiple correction guidance points when the user feedbacks that the visual centers coincide; Perform fitting calculation based on the relative position relationship of the multiple correction guidance points and the position relationship between the corresponding multiple pupil angles to obtain the pupil motion mapping model of the user. The pupil motion mapping model is used to represent the actual visual center change amount corresponding to the user's pupil when it rotates a certain angle.

2. The imaging detection method of a head-mounted display device according to claim 1, characterized in that, It further includes an imaging resource optimization step: Obtain the user's visual center in real time through pupil tracking, and set the visual center as the first imaging center; Based on the first imaging center as the center, establish multiple annular display areas. The multiple annular display areas respectively correspond to multiple display output schemes, and the display output resolution and content refresh frequency of the multiple annular display areas gradually decrease along the radial direction.

3. The imaging detection method of a head-mounted display device according to claim 2, characterized in that, The imaging resource optimization step further includes: Obtain the environmental space data within the scanning field of view in real time, and update the difference data based on the environmental twin model; Establish a second imaging center based on the object area where the difference data is updated. The second imaging center is different from the first imaging center, and the second imaging center does not include multiple accompanying annular display areas; Adopt a display output scheme consistent with that of the first imaging center for the corresponding display data content of the second imaging center.

4. An imaging detection system for a head-mounted display device, characterized in that, Include: An environment synchronization module for obtaining the environmental space data within the scanning field of view in real time through a sensor device group, and performing high-precision spatial modeling based on the environmental space data to obtain an environmental twin model. The environmental space data includes environmental image data and laser ranging data; A motion synchronization module for obtaining the current motion posture data of the display device based on an attitude sensor device group and performing real-time synchronization update, and calculating the spatial motion travel within a time period through the motion posture data. The motion posture data includes attitude orientation data and motion inertia data; A perspective synchronization module for obtaining the user's perspective swing angle, and mapping the display screen based on the perspective swing angle to synchronize the user's perspective with the display perspective of the display device. The perspective swing angle is used to represent the deflection angle of the visual center between two time nodes of the user; A field of view calibration module for performing pupil tracking on the user, obtaining the visual center of the user through an initial request, calculating the spatial perspective difference between the visual center and the current attitude orientation data, and calibrating the display output field of view based on the spatial perspective difference. The spatial perspective difference is used to represent the spatial orientation angle between the user's perspective center and the display output center of the display device, and the visual center is used to represent the user's frontal perspective; The perspective synchronization module specifically includes: A viewing distance detection unit for generating a perspective range synchronization request when detecting a change in the wearing state of the display device, and detecting and obtaining the projection display distance between the user's pupil and the display device after determining the user's visual center; A reference marking unit for responding to the range synchronization request and randomly selecting a pair of synchronization reference nodes based on the environmental twin model, and performing display marking on the pair of synchronization reference nodes in the environmental twin model; A rotation angle acquisition unit for recording the current attitude orientation data when the visual center of the display device coincides with the synchronization reference node, and calculating the attitude inclination difference of the attitude orientation data between a pair of synchronization reference nodes, that is, the perspective swing angle corresponding to the display device; A field-of-view synchronization unit, which is used to calculate the visual width at a corresponding viewing angle swing angle based on the projection display distance, calculate the display scaling ratio through the actual display distances of a pair of synchronous reference nodes, and perform perspective synchronization scaling on the output image of the display device based on the display scaling ratio; It further includes a characterization correction module, specifically including: An initial correction unit, which is used to generate a viewing angle initialization request when performing pupil tracking on a user, so as to guide the user's viewing angle center to coincide with the display center of the display device. If the user's pupil angle does not coincide with the display center of the display device at this time, it guides the user to perform characterization correction; A correction marking unit, which is used to generate multiple groups of correction guiding points distributed in multiple plane quadrants within the display range, and record the pupil angles of the user at multiple correction guiding points when the user feedbacks that the visual centers coincide; A correction mapping unit, which is used to perform fitting calculations based on the relative position relationships of the multiple correction guiding points and the position relationships between the corresponding multiple pupil angles, and obtain the pupil movement mapping model of the user. The pupil movement mapping model is used to characterize the actual visual center change amount corresponding to the user's pupil when it rotates a certain angle.

5. The imaging detection system of a head-mounted display device according to claim 4, characterized in that It further includes a resource optimization module, specifically including: A first center determination unit, which is used to obtain the user's visual center in real time through pupil tracking and set the visual center as the first imaging center; A display optimization management unit, which is used to establish multiple annular display areas centered on the first imaging center. The multiple annular display areas respectively correspond to multiple display output schemes, and the display output resolution and content refresh frequency in the radial direction of the multiple annular display areas gradually decrease.

6. The imaging detection system of a head-mounted display device according to claim 5, wherein The resource optimization module further includes: An update synchronization unit, which is used to obtain the environmental space data within the scanned field of view in real time and perform difference data update based on the environmental twin model; A second center determination unit, which is used to establish a second imaging center based on the object area where the difference data is updated. The second imaging center is different from the first imaging center, and the second imaging center does not include multiple accompanying annular display areas; A display resource management unit, which is used to adopt a display output scheme consistent with that of the first imaging center for the corresponding display data content of the second imaging center.

Images