Building environment interaction analysis method and device based on virtual-real space synchronous mapping and multi-modal perception

Abstract

The invention provides a building environment interaction analysis method and device based on virtual-real space synchronous mapping and multi-modal perception, and the method comprises the steps: carrying out the preprocessing of a target region panoramic image set, and obtaining a static target region panoramic image; mapping the panoramic image of the static target area to a double-sphere coordinate system to obtain virtual scene space mapping; obtaining a thermodynamic distribution diagram according to the user eye movement data and the virtual scene space; inputting the panoramic image of the static target area into the trained feature recognition model to obtain all elements contained in the thermodynamic distribution diagram; and generating a visual behavior analysis report according to the staring duration time, the staring frequency and the spatial weight matrix of each element in the thermal distribution diagram in combination with the staring duration time, the staring frequency and the spatial weight matrix of each element.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically to a method and apparatus for interactive analysis of building environment based on synchronous mapping of virtual and real spaces and multimodal perception. Background Technology

[0002] Eye-tracking solutions in the field of Virtual Reality (VR) technology can be divided into two main categories: laboratory-grade dedicated equipment and commercial VR integrated systems. Laboratory-grade eye-tracking systems, typified by Tobii Pro Glasses 3, utilize corneal reflection principles to construct a gaze point localization mechanism, achieving sub-centimeter-level high-precision localization. However, these systems suffer from significant application limitations: high equipment costs (over $30,000 per unit) and the need for dedicated calibration equipment, leading to exponentially increasing deployment costs for large-scale group experiments; stringent environmental requirements, heavily reliant on static scene data collection in controlled laboratory environments, resulting in low public participation and limited sample sizes; furthermore, deep coupling with specific eye-tracking analysis software and a lack of open software interfaces cause significant discrepancies between the generated gaze heatmaps and real-world behavioral patterns, severely impacting the validity and applicability of the data.

[0003] In contrast, consumer-grade VR devices, such as the PICO 4 Pro, achieve eye-tracking functionality through a built-in infrared camera, reaching a gaze direction tracking accuracy of 2.5cm at a 72Hz sampling rate. However, these devices also have functional limitations: they can only acquire eye movement direction data, facing technical obstacles in establishing spatial correspondences with traditional street view images and virtual reality modeling scenes, making it difficult to accurately determine the specific coordinates of eye movements within the scene; furthermore, they lack the function of semantic gaze point data secondary classification for multi-scale urban elements such as building facade textures and street signs, failing to meet the needs of refined data analysis in complex scenes and limiting their in-depth application in professional fields such as urban planning and architectural design. Summary of the Invention

[0004] Based on this, the present invention provides a method and apparatus for interactive analysis of building environment based on synchronous mapping of virtual and real space and multimodal perception. It utilizes panoramic images acquired by a panoramic camera to project onto the inner sphere surface of a dual-sphere coordinate system, and then combines this with collaborative mapping on the outer sphere surface. The eye-tracking focus where the eye-tracking ray intersects with the street scene texture is converted into a precise correspondence of the street scene texture coordinates. The heat analysis of the street scene data is combined with the user's gaze time to provide data support for building space optimization decisions and meet the research needs of large-scale building environment.

[0005] In a first aspect, the present invention provides a method for interactive analysis of the built environment based on synchronous mapping of virtual and real spaces and multimodal perception, comprising:

[0006] Obtain a set of panoramic images of the target area;

[0007] The set of panoramic images of the target area is preprocessed to obtain a static panoramic image of the target area;

[0008] The panoramic image of the static target area is mapped to a dual-sphere coordinate system to obtain a virtual scene space mapping;

[0009] Acquire real-time eye-tracking data from users;

[0010] A heat map is obtained based on the user's eye-tracking data and the virtual scene space;

[0011] The panoramic image of the static target area is input into the trained feature recognition model to obtain all the elements contained in the heat map.

[0012] The duration of gaze, number of gazes, and spatial weight matrix of each element are calculated by combining the heat map, and a visual behavior analysis report is generated based on the duration of gaze, number of gazes, and spatial weight matrix of each element.

[0013] Furthermore, the preprocessing of the panoramic image set of the target area to obtain a static panoramic image of the target area specifically involves:

[0014] The set of panoramic images of the target area is input into the Ghost Town model. After removing moving objects from the panoramic images of the target area, a static panoramic image of the target area is obtained.

[0015] Furthermore, the step of mapping the panoramic image of the static target area to a dual-sphere coordinate system to obtain a virtual scene space mapping specifically involves:

[0016] By using texture-related interfaces in Unity, the two-dimensional texture coordinates of each pixel in the panoramic image of the static target area are mapped to three-dimensional spatial coordinates to obtain the virtual scene spatial mapping.

[0017] Furthermore, the user's real-time eye movement data includes the user's real-time left / right eye spatial coordinates.

[0018] Furthermore, obtaining the heat map based on the user's real-time eye-tracking data and the virtual scene space includes:

[0019] The user's real-time eye movement data is preprocessed to obtain effective eye movement data;

[0020] The effective eye-tracking data is mapped to obtain the user's gaze coordinates;

[0021] A thermal distribution map is obtained based on the user's gaze coordinates using the kernel density estimation method.

[0022] Furthermore, the user's real-time eye-tracking data is preprocessed to obtain valid eye-tracking data, specifically as follows:

[0023] The user's real-time eye movement data is input into a multi-dimensional data filtering model to obtain eye movement data after initial filtering.

[0024] The eye movement data after initial rejection was completed using time window interpolation to obtain valid eye movement data;

[0025] The multidimensional data filtering model filters real-time eye movement data of users in the time dimension, spatial dimension, and dynamic dimension, respectively.

[0026] Furthermore, the elements of the heat map include text label categories, building facade feature categories, functional interface categories, and environmental element categories.

[0027] Secondly, the present invention also provides a building environment interactive analysis device based on virtual-real space synchronous mapping and multimodal perception, comprising:

[0028] The region image acquisition module is used to acquire a set of panoramic images of the target region;

[0029] The regional image processing module is used to preprocess the set of panoramic images of the target region to obtain a static panoramic image of the target region.

[0030] The image mapping module is used to map the panoramic image of the static target area to a dual-sphere coordinate system to obtain a virtual scene space mapping;

[0031] The eye-tracking data acquisition module is used to acquire real-time eye-tracking data from users;

[0032] The gaze mapping module is used to obtain a heat map based on the user's eye movement data and the virtual scene space;

[0033] The feature recognition module is used to input the panoramic image of the static target area into the trained feature recognition model to obtain all the features contained in the heat map.

[0034] The hotspot analysis module is used to calculate the gaze duration, number of gazes, and spatial weight matrix of each element by combining the heat map, and to generate a visual behavior analysis report based on the gaze duration, number of gazes, and spatial weight matrix of each element.

[0035] Thirdly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any of the building environment interaction analysis methods based on virtual-real space synchronous mapping and multimodal perception in the first aspect.

[0036] Fourthly, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform any one of the building environment interaction analysis methods based on virtual-real space synchronous mapping and multimodal perception in the first aspect.

[0037] The beneficial effects of adopting the above technical solution are as follows: This embodiment breaks through the top / bottom image distortion limitation of Unity Sphere, and pioneers the collaborative mapping of inner surface spheres (SphereR) and outer surface spheres (SphereN) to achieve spatial transformation of eye-tracking coordinates; panoramic street view mapping is achieved in the concentric sphere SkyDome (built into Unity) with zero distortion. Simultaneously, through ray casting and texture coordinate normalization algorithms, eye-tracking ray intersection with street view texture coordinates is achieved on the PICO device to obtain eye-tracking focus, which is then converted into a precise correspondence of street view texture coordinates (UV coordinates). Furthermore, the cost of related eye-tracking analysis is greatly optimized by using commercial-grade equipment combinations (such as Insta360 camera + PICO 4 Pro) to replace professional eye trackers, achieving a precipitous cost reduction. Moreover, through VR's low-threshold interaction (requiring only controller clicks), the participation of the elderly, disabled, and other groups is increased, filling the gap in data on vulnerable groups in traditional surveys. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0039] Figure 1 This is a schematic diagram of a building environment interaction analysis method based on virtual-real space synchronous mapping and multimodal perception in one embodiment of this application;

[0040] Figure 2 This is a schematic diagram of a building environment interaction analysis method based on virtual-real space synchronous mapping and multimodal perception in one embodiment of this application;

[0041] Figure 3 This is a schematic diagram illustrating the presentation of the acquisition process in a PICO virtual reality device according to one embodiment of this application;

[0042] Figure 4 This is a schematic diagram of the thermal distribution in one embodiment of this application;

[0043] Figure 5 This is a schematic diagram illustrating the distribution of eye-tracking data and different regions of interest in one embodiment of this application;

[0044] Figure 6 This is a schematic diagram of a building environment interaction analysis device based on virtual-real space synchronous mapping and multimodal perception in one embodiment of this application. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention. To describe the present invention in more detail, the method and apparatus for interactive analysis of building environment based on synchronous mapping of virtual and real space and multimodal perception provided by the present invention will be specifically described below with reference to the accompanying drawings.

[0046] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms "an," "a," or "the" do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms "comprising" or "including" mean that the preceding element or object encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. The terms "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up," "down," "left," "right," etc., are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0047] Eye-tracking solutions in the field of Virtual Reality (VR) technology can be divided into two main categories: laboratory-grade dedicated equipment and commercial VR integrated systems. Laboratory-grade eye-tracking systems, typified by Tobii Pro Glasses 3, utilize corneal reflection principles to construct a gaze point localization mechanism, achieving sub-centimeter-level high-precision localization. However, these systems suffer from significant application limitations: high equipment costs (over $30,000 per unit) and the need for dedicated calibration equipment, leading to exponentially increasing deployment costs for large-scale group experiments; stringent environmental requirements, heavily reliant on static scene data collection in controlled laboratory environments, resulting in low public participation and limited sample sizes; furthermore, deep coupling with specific eye-tracking analysis software and a lack of open software interfaces cause significant discrepancies between the generated gaze heatmaps and real-world behavioral patterns, severely impacting the validity and applicability of the data.

[0048] In contrast, consumer-grade VR devices, such as the PICO 4 Pro, achieve eye-tracking functionality through a built-in infrared camera, reaching a gaze direction tracking accuracy of 2.5cm at a 72Hz sampling rate. However, these devices also have functional limitations: they can only acquire eye movement direction data, facing technical obstacles in establishing spatial correspondences with traditional street view images and virtual reality modeling scenes, making it difficult to accurately determine the specific coordinates of eye movements within the scene; furthermore, they lack the function of semantic gaze point data secondary classification for multi-scale urban elements such as building facade textures and street signs, failing to meet the needs of refined data analysis in complex scenes and limiting their in-depth application in professional fields such as urban planning and architectural design.

[0049] This application provides an application scenario for a building environment interaction analysis method and device based on synchronous mapping of virtual and real spaces and multimodal perception. This application scenario includes the terminal device provided in the embodiment. The terminal device includes, but is not limited to, smartphones and computer devices. The computer device can be at least one of desktop computers, portable computers, laptop computers, mainframe computers, tablet computers, etc. The terminal device obtains a hotspot analysis report of the user's gaze by receiving a set of panoramic images of the target area and user eye-tracking data, combined with attached... Figure 1 The diagram and appendix show a building environment interaction analysis method based on virtual-real space synchronous mapping and multimodal perception. Figure 2 The diagram shows a flowchart of the building environment interaction analysis method based on virtual-real space synchronous mapping and multimodal perception. For details, please refer to the implementation examples of the building environment interaction analysis method based on virtual-real space synchronous mapping and multimodal perception.

[0050] Step S100: Obtain a set of panoramic images of the target area.

[0051] Specifically, in this embodiment, a panoramic camera (such as an Insta360 panoramic camera) can be used to acquire a set of panoramic images of the target area at a preset frequency. For example, the data acquisition personnel can mount an Insta360 ONE X2 camera (with a resolution set to 5.7K and a frame rate set to 30fps) on a handheld gimbal, select 9 consecutive viewpoints within the target area, and take 60 street view images covering the entire field of view at a fixed height (such as 1.6 meters) for each viewpoint. This set of images is recorded as the set of panoramic images of the target area.

[0052] Step S200: Preprocess the set of panoramic images of the target area to obtain a static panoramic image of the target area.

[0053] Specifically, in this embodiment, the Ghost Town algorithm uses multi-frame panoramic image fusion technology (60 frames / point) to eliminate moving objects in the panoramic image of the target area, obtaining a static panoramic image of the target area that retains only the architectural environment. This static panoramic image can be an equidistant cylindrical projection panoramic image with a resolution of 4096×2048, and its highest resolution can reach 7680*7680.

[0054] This study utilizes the motion-compensated temporal median filtering feature of the Ghost Town algorithm to eliminate dynamic interference. It employs pixel-level comparison (e.g., 0.5 seconds per point sampling time) of several temporally adjacent frames (e.g., 60 adjacent frames) at the same location, combined with a multi-frame alignment algorithm using the HSV color space to detect moving objects. After eliminating moving objects (e.g., pedestrians, pets) from the panoramic image of the target area, static elements such as building facades and ground paving are retained. Simultaneously, background image interpolation is used to fill in missing areas, resulting in a static panoramic image of the target area. This static panoramic image is saved in equirectangular projection format.

[0055] Step S300: Map the panoramic image of the static target area to a dual-sphere coordinate system to obtain a virtual scene space mapping.

[0056] Specifically, in conjunction with the appendix Figure 2In the street view projection section, to address the core technical bottleneck of existing VR devices' inability to directly obtain street view eye-tracking coordinates, this embodiment utilizes a self-developed dual-sphere mapping algorithm (SphereR+SphereN). Through the collaborative mapping of the inner and outer sphere surfaces, a panoramic image of the static target area projected onto the inner sphere surface (SphereR) is created, enabling interaction between the eye-tracking ray and the virtual street view image. The outer sphere surface (SphereN) is used to map spatial coordinates. Through binocular viewpoint ray projection technology, the texture coordinates of each pixel in the panoramic image of the static target area are mapped to three-dimensional spatial coordinates, resulting in a virtual scene spatial mapping, as shown in the attached figure. Figure 3 As shown.

[0057] Specifically, by utilizing texture-related interfaces in Unity, the two-dimensional texture coordinates of each pixel in the panoramic image of the static target area are obtained. Mapping to three-dimensional space coordinates This yields a virtual scene space mapping. The specific code is as follows:

[0058] if (Physics.Raycast(ray, out RaycastHit hit)) {

[0059] Vector2 uv = hit.textureCoord;

[0060] }

[0061] In this embodiment, the inner sphere surface (SphereR) serves as the interaction carrier, with its UV coordinates corresponding to the original street scene texture. The outer sphere surface (SphereN) serves as the spatial calculation carrier, establishing three-dimensional spatial coordinates through normal vector projection. with 2D texture coordinates The mapping relationship is as follows. Based on this, when the user's line of sight intersects the inner sphere surface (SphereR) at point HitA, it is projected twice along the normal direction to the outer sphere surface (SphereN) to obtain point HitB. The UV coordinates of HitB correspond to the spatial position of the actual street view image.

[0062] Step S400: Obtain the user's real-time eye movement data.

[0063] The user's real-time eye movement data includes the user's real-time left / right eye spatial coordinates. In this embodiment, the user's real-time eye movement data can be achieved by integrating it into the PICO 4 Pro eye-tracking module (72Hz sampling rate).

[0064] Step S500: Obtain a heat map based on the user's eye movement data and the virtual scene space.

[0065] Step S500, which obtains the heat map based on the user's eye-tracking data and the virtual scene space, includes the following specific steps:

[0066] Step S501: Preprocess the user's real-time eye movement data to obtain valid eye movement data.

[0067] Specifically, a multi-dimensional data filtering model is established. Real-time eye movement data of users is input into the multi-dimensional data filtering model to remove invalid eye movement data (such as blinking, gaze deviation, etc.) to obtain preliminary eye movement data after removal. Then, time window interpolation (such as Window=0.2 seconds) is used to complete the preliminary eye movement data after removal to obtain valid eye movement data.

[0068] This multi-dimensional data filtering model filters real-time eye-tracking data based on the linkage of time, space, and angle. In the time dimension, it removes real-time eye-tracking data where the duration of a single gaze is less than the blink time threshold (e.g., 100ms) and eye-tracking data such as physiological blinking reflexes. In the spatial dimension, it removes real-time eye-tracking data where the gaze deviates from the target area scene boundary by more than the angle threshold (e.g., 10°). In the dynamic dimension, it filters real-time eye-tracking data where the coordinate drift exceeds the static drift angle for several consecutive frames, such as abnormal tracking data where the coordinate drift is greater than 1.5° within 5 consecutive frames.

[0069] Step S502: Map the valid eye movement data to obtain the user's gaze coordinates.

[0070] Specifically, using Unity's 3D coordinate system transformation algorithm, for effective eye-tracking data... The coordinates HitB, the intersection point of the viewpoint direction vector and the scene, are calculated. After noise reduction by Kalman filtering, the input coordinate system transformation module is used. This module uses quaternion interpolation to achieve a smooth transition from the world coordinate system (device space) to the spherical coordinate system (scene space). Finally, the gaze point coordinate sequence in the texture coordinate system (accuracy ±1.5°) is output to obtain the user's gaze coordinates.

[0071] Step S503: Using the kernel density estimation method, obtain the thermal distribution map based on the user's gaze coordinates.

[0072] Specifically, a kernel density estimation algorithm is used to perform spatiotemporal clustering of user gaze coordinates, and hotspot regions are obtained by combining this with the virtual scene space. Among these, user gaze coordinates... Using the input data, the spatiotemporal coupling density value is calculated through the Gaussian kernel function, resulting in a thermodynamic distribution map with time decay characteristics, as shown in the attached figure. Figure 4 As shown.

[0073] Step S600: Input the panoramic image of the static target area into the trained feature recognition model to obtain all the elements contained in the heat map.

[0074] Specifically, the panoramic image of the static target area is input into the trained feature recognition model, and the panoramic image of the static target area is segmented by SegmentAnything to obtain all the elements contained in the heat map. In this embodiment, the elements of the heat map are divided into four categories, including text label category, building facade feature category, functional interface category, and environmental element category. The text label category includes text information such as signs and road signs; the building facade feature category includes window opening form, material contrast, and facade segmentation; the functional interface category includes shop windows and entrance spaces; and the environmental element category includes landscape nodes and public artworks.

[0075] Step S700: Calculate the gaze duration, number of gazes, and spatial weight matrix of each element based on the heat map, and generate a visual behavior analysis report based on the gaze duration, number of gazes, and spatial weight matrix of each element.

[0076] Specifically, for the duration of gaze on each element in the heat map, 300ms can be used as the effective gaze threshold, and the duration of a single gaze on each element can be counted. The number of times an element is looked at can be obtained by counting the number of times the same user repeatedly looks at a specific element. Gazes with an interval of more than 500ms are considered independent access events. For the spatial weight matrix, specific weights can be assigned according to the element category corresponding to each element in the heat map. For example, building facade feature category and environmental element category are primary weights, and text label category and functional interface category are secondary weights.

[0077] A visual behavior analysis report is generated by combining the gaze duration, number of gazes, and spatial weight matrix of each element in the heat map. This report can be output in a standard, data-driven JSON format. The specific data structure of the visual behavior analysis report includes:

[0078] {

[0079] "user_id": "P4_2023",

[0080] "fixation_summary": {

[0081] "text_signage": {"duration": 12.7, "count": 6},

[0082] "facade_windows": {"duration": 8.3, "count": 3}

[0083] },

[0084] "heatmap_coord": [[x1,y1,t1],...[xn,yn,tn]]

[0085] }

[0086] Based on the architectural environment interaction analysis method based on virtual-real space synchronous mapping and multimodal perception provided in this application embodiment, a millimeter-level accurate mapping between three-dimensional world coordinates and two-dimensional texture coordinates is achieved by constructing an inner and outer dual-sphere coordinate transformation system (SphereR for interaction and SphereN for spatial calculation). A normal vector quadratic projection algorithm is adopted. When the user's gaze ray intersects the inner sphere at Hit A, it is quadratically projected along the normal direction to the surface of the outer sphere to obtain the UV coordinates of Hit B, solving the spatial distortion problem of traditional VR scenes. Simultaneously, image coordinates can be directly obtained, providing a geometrically faithful foundation for architectural environment interaction analysis. Furthermore, based on the time, space, and dynamic three-domain linkage thresholds (single gaze duration < 100ms, gaze deviation > 10°, coordinate drift > 1.5° for 5 consecutive frames), invalid eye-tracking data (blinking, gaze deviation, etc.) is automatically removed, ensuring the validity and reliability of gaze point data. Moreover, attention area weights are defined according to feature type, and a structured JSON report is generated by combining Fixation Duration (single gaze duration) and Visit Count (visit count), directly supporting architectural space optimization decisions. It achieves full automation from street view acquisition, virtual scene loading, eye-tracking data acquisition to analysis report generation, supports continuous acquisition and processing of multiple scenes, and has a very short data processing time for a single scene, meeting the needs of large-scale built environment research.

[0087] It should be understood that, although attached Figure 1 The steps in the flowchart are shown sequentially according to the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order requirement for the execution of these steps, and they can be executed in other orders. Furthermore, [the following is a list of steps]. Figure 1 At least some of the steps in the process may include multiple sub-steps or sub-stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0088] The embodiments disclosed above describe in detail a method for interactive analysis of building environment based on synchronous mapping of virtual and real spaces and multimodal perception. Since the above-disclosed method can be implemented using various types of devices, this invention also discloses a device for interactive analysis of building environment based on synchronous mapping of virtual and real spaces and multimodal perception, in conjunction with the appendix. Figure 6 The following are specific embodiments for detailed explanation.

[0089] The region image acquisition module 801 is used to acquire a set of panoramic images of the target region.

[0090] The region image processing module 802 is used to preprocess the set of panoramic images of the target region to obtain a static panoramic image of the target region.

[0091] Image mapping module 803 is used to map the panoramic image of the static target area to a dual-sphere coordinate system to obtain a virtual scene space mapping;

[0092] The eye-tracking data acquisition module 804 is used to acquire real-time eye-tracking data from the user.

[0093] The gaze mapping module 805 is used to obtain a heat map based on the user's eye movement data and the virtual scene space;

[0094] The feature recognition module 806 is used to input the panoramic image of the static target area into the trained feature recognition model to obtain all the features contained in the heat map.

[0095] The hotspot analysis module 807 is used to calculate the gaze duration, number of gazes, and spatial weight matrix of each element by combining the heat map, and to generate a visual behavior analysis report based on the gaze duration, number of gazes, and spatial weight matrix of each element.

[0096] For details regarding the interactive analysis device for building environments based on synchronous mapping of virtual and real spaces and multimodal perception, please refer to the method specifications above; they will not be repeated here. Each module in the aforementioned device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the terminal device in hardware form or independently of it, or stored in the memory of the terminal device in software form, so that the processor can call and execute the corresponding operations of each module.

[0097] In one embodiment, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described building environment interaction analysis method based on virtual-real space synchronous mapping and multimodal perception.

[0098] The computer-readable storage medium may be an electronic storage device such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), hard disk, or ROM. Optionally, the computer-readable storage medium includes a non-transitory computer-readable storage medium. The computer-readable storage medium has storage space for program code that performs any of the method steps described above. This program code can be read from or written to one or more computer program products, and the program code may be compressed in an appropriate form.

[0099] In one embodiment, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the above-described building environment interaction analysis method based on virtual-real space synchronous mapping and multimodal perception.

[0100] The computer device includes a memory, a processor, and one or more computer programs, wherein the one or more computer programs can be stored in the memory and configured to be executed by one or more processors, and the one or more application programs are configured to perform the above-described building environment interaction analysis method based on virtual and real space synchronous mapping and multimodal perception.

[0101] A processor may include one or more processing cores. The processor connects to various parts of the computer device using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory, and by calling data stored in memory. Optionally, the processor may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also be implemented separately as a communication chip, without being integrated into the processor.

[0102] The memory may include random access memory (RAM) or read-only memory (ROM). The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described above. The data storage area may also store data created by the terminal device during use.

[0103] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for interactive analysis of the built environment based on synchronous mapping of virtual and real spaces and multimodal perception, characterized in that, include: Obtain a set of panoramic images of the target area; The set of panoramic images of the target area is preprocessed to obtain a static panoramic image of the target area; The panoramic image of the static target area is mapped to a dual-sphere coordinate system to obtain a virtual scene space mapping; Acquire real-time eye-tracking data from users; A heat map is obtained based on the user's eye-tracking data and the virtual scene space; The panoramic image of the static target area is input into the trained feature recognition model to obtain all the elements contained in the heat map. The duration of gaze, number of gazes, and spatial weight matrix of each element are calculated by combining the heat map, and a visual behavior analysis report is generated based on the duration of gaze, number of gazes, and spatial weight matrix of each element.

2. The building environment interaction analysis method based on virtual-real space synchronous mapping and multimodal perception as described in claim 1, characterized in that, The preprocessing of the panoramic image set of the target area to obtain a static panoramic image of the target area specifically involves: The set of panoramic images of the target area is input into the Ghost Town model. After removing moving objects from the panoramic images of the target area, a static panoramic image of the target area is obtained.

3. The building environment interaction analysis method based on virtual-real space synchronous mapping and multimodal perception as described in claim 2, characterized in that, The step of mapping the panoramic image of the static target area to a dual-sphere coordinate system to obtain a virtual scene space mapping is specifically as follows: By using texture-related interfaces in Unity, the two-dimensional texture coordinates of each pixel in the panoramic image of the static target area are mapped to three-dimensional spatial coordinates to obtain the virtual scene spatial mapping.

4. The building environment interaction analysis method based on virtual-real space synchronous mapping and multimodal perception as described in claim 3, characterized in that, The user's real-time eye movement data includes the user's real-time left / right eye spatial coordinates.

5. The building environment interaction analysis method based on virtual-real space synchronous mapping and multimodal perception as described in claim 4, characterized in that, The step of obtaining the heat map based on the user's real-time eye-tracking data and the virtual scene space includes: The user's real-time eye movement data is preprocessed to obtain effective eye movement data; The effective eye-tracking data is mapped to obtain the user's gaze coordinates; A thermal distribution map is obtained based on the user's gaze coordinates using the kernel density estimation method.

6. The building environment interaction analysis method based on synchronous mapping of virtual and real spaces and multimodal perception as described in claim 5, characterized in that, The user's real-time eye movement data is preprocessed to obtain valid eye movement data as follows: The user's real-time eye movement data is input into a multi-dimensional data filtering model to obtain eye movement data after initial filtering. The eye movement data after initial rejection was completed using time window interpolation to obtain valid eye movement data; The multidimensional data filtering model filters real-time eye movement data of users in the time dimension, spatial dimension, and dynamic dimension, respectively.

7. The building environment interaction analysis method based on virtual-real space synchronous mapping and multimodal perception as described in claim 6, characterized in that, The elements of a heat map include text label categories, building facade feature categories, functional interface categories, and environmental element categories.

8. A building environment interactive analysis device based on synchronous mapping of virtual and real spaces and multimodal perception, characterized in that, include: The region image acquisition module is used to acquire a set of panoramic images of the target region; The regional image processing module is used to preprocess the set of panoramic images of the target region to obtain a static panoramic image of the target region. The image mapping module is used to map the panoramic image of the static target area to a dual-sphere coordinate system to obtain a virtual scene space mapping; The eye-tracking data acquisition module is used to acquire real-time eye-tracking data from users; The gaze mapping module is used to obtain a heat map based on the user's eye movement data and the virtual scene space; The feature recognition module is used to input the panoramic image of the static target area into the trained feature recognition model to obtain all the features contained in the heat map. The hotspot analysis module is used to calculate the gaze duration, number of gazes, and spatial weight matrix of each element by combining the heat map, and to generate a visual behavior analysis report based on the gaze duration, number of gazes, and spatial weight matrix of each element.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of any one of the building environment interaction analysis methods based on virtual-real space synchronous mapping and multimodal perception as described in claims 1-7.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it performs any one of the building environment interaction analysis methods based on virtual-real space synchronous mapping and multimodal perception as described in claims 1-7.

Images