Extended reality man-machine interaction method for road engineering operation and maintenance stage

Abstract

The invention provides an expanded reality man-machine interaction method for a road engineering operation and maintenance stage, and relates to the technical field of intelligent construction and operation and maintenance of road engineering. The method comprises the following steps: in a road construction stage, arranging a sensor system based on road types, structures and functions, and marking spatial positions of various sensors; constructing a road digital asset system based on the road structure physical information and the real-time response data collected by the sensor system; based on the constructed road digital asset system, road digital asset information visualization is realized by using an extended reality technology, and a distributed man-machine interaction system is constructed based on industry regulations and processes of road engineering operation and maintenance work; in a road operation and maintenance stage, road digital asset information and a man-machine interaction system are anchored in a road physical environment based on space calculation. By adopting the method, the problems of data integration visualization, time-space integration mapping of data and interaction of engineering field people, environment and computers in the operation and maintenance process of existing road engineering can be solved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent construction and operation and maintenance of road engineering, and in particular to an extended reality human-computer interaction method used in the operation and maintenance stage of road engineering. Background Art

[0002] In the traditional road engineering field, the data monitored by sensor equipment in the road structure are all displayed through the background computer. However, it is difficult to unify the sensor data collectors and data display platforms of different types and manufacturers, resulting in a lot of time wasted in retrieving road engineering monitoring data. In addition, the background computer is fixed and has a certain degree of privacy, which makes it difficult to combine with the road form on site. As a result, on-site operation and maintenance personnel are limited by the time and space of data information, and cannot effectively perceive, understand and make decisions on the current road status. In addition, there is no basic digital tool to carry these operation and maintenance information to be interacted with. Summary of the invention

[0003] In order to solve the problems of data integration visualization, time-space integration mapping of data and interaction between people, environment and computers on the construction site in the operation and maintenance process of road projects in the prior art, an embodiment of the present invention provides an augmented reality human-computer interaction method for the operation and maintenance stage of road projects.

[0004] The extended reality human-computer interaction method provided in this embodiment for the road engineering operation and maintenance stage includes:

[0005] During the road construction phase, the sensor system is arranged based on the road type, structure and function, and the spatial locations of various sensors are marked;

[0006] Build a road digital asset system based on the physical information of the road structure and the real-time response data collected by the sensor system;

[0007] Based on the constructed road digital asset system, the road digital asset information is visualized using extended reality technology, and a distributed human-computer interaction system is built based on the industry regulations and processes of road engineering operation and maintenance;

[0008] During the road operation and maintenance stage, road digital asset information and human-computer interaction systems are anchored in the road physical environment based on spatial computing.

[0009] Furthermore, the road types include expressways and municipal roads;

[0010] The sensor system is arranged in layers as the roadbed and pavement engineering are constructed in layers;

[0011] The physical quantities monitored by the sensor system include one or more of stress, strain, load, displacement, oxygen content, temperature, humidity and ultraviolet radiation.

[0012] Furthermore, the marking of the spatial positions of various sensors includes:

[0013] Taking the road elevation as a reference, marking the depth of the road, the thickness of each layer of the roadbed and road surface, the length and width of the road surface, and taking the actual road engineering spatial information as a constraint, selecting any point on the road surface that is convenient for spatial marking as the origin O to construct a fixed global three-dimensional vector coordinate system, the global three-dimensional vector coordinate system takes the road length direction as the X direction, the road width direction as the Y direction, and the road depth direction as the Z direction;

[0014] In the global three-dimensional vector coordinate system, the layered road structure L along the road depth is taken as the reference. i All sensors in the group are marked with the digital three-dimensional coordinates of each sensor, with the direction parallel to the road surface as the 0° direction, and the angle θ between the two reference lines perpendicular to the road surface and the horizontal plane of the sensor as the arrangement direction of the sensor. The information of the spatial position of the sensor in the digital space is marked as S i,j (x digital ,y digital ,z digital ,θ), where S i,j Represents the i-th layer road structure L i The spatial position of the jth sensor in (x digital ,y digital ,z digital ,θ) represent the coordinate value and angle value of the sensor in the X direction, Y direction, and Z direction respectively.

[0015] Furthermore, the construction of the road digital asset system based on the physical information of the road structure and the real-time response data collected by the sensor system includes:

[0016] The physical information of the road structure is collected and a database of inherent road attributes is constructed. The inherent road attribute data includes: the spatial geometric dimensions of the layered road structure, the material information of each layer of the road structure, and the engineering function;

[0017] Integrate the type, model, manufacturer, spatial location information of sensors in the road structure and the real-time response data collected by the sensor system to build a road structure response database;

[0018] The road digital asset system is constructed using BIM technology, including: a three-dimensional digital model of the road built at a scale of 1:100 containing road geometry information and materials, a road inherent attribute database, and a road structure response database; among them, BIM stands for Building Information Model.

[0019] Furthermore, the road inherent attribute database and the road structure response database are both relational databases built based on MySQL to store data information in relational road digital assets.

[0020] Furthermore, the road digital asset system constructed based on the above method utilizes the augmented reality technology to realize the visualization of road digital asset information, including:

[0021] Export the road 3D digital model as an .ifc file and import it into the Unity game engine;

[0022] The multi-type data collected by various sensors in the road structure, including text, image and numerical data, are expressed and stored in the form of a matrix;

[0023] The road 3D digital model and sensor data visualization interface are deployed to the head-mounted mixed reality device through the Unity game engine.

[0024] Furthermore, the sensor data visualization interface is a data visualization interface rewritten based on the TCP / IP communication interface of the data acquisition interface provided by various sensor manufacturers and the C# language. The rewritten sensor data visualization interface can reflect the real-time dynamic expression of multiple types of data.

[0025] Furthermore, the distributed human-computer interaction system built based on the industry regulations and processes of road engineering operation and maintenance work includes:

[0026] Import the .ifc file of the geometric information of the road digital assets in BIM into the Unity game engine as a digital physical object for human-computer interaction, and access the interface of the road inherent attribute database and the road structure response database;

[0027] Based on the industry regulations and processes of road engineering operation and maintenance, a human-computer interaction interface was built in the Unity game engine based on the front-end interface development platform Figma, and a human-computer interaction system was developed based on the mixed reality toolkit, including hand recognition and eye motion capture recognition, so as to realize the function of feedback of interactive information through physical hand operation of virtual digital interface in extended reality applications;

[0028] Based on the universal Windows platform, Visual Studio is used to deploy the road digital asset information and human-computer interaction system in the Unity game engine to the distributed application platform according to the application environment.

[0029] Furthermore, the distributed application platform includes a head-mounted mixed reality device and a local background computer, wherein the head-mounted mixed reality device is used for round-the-clock operation and maintenance by on-site staff, and the background computer is used for round-the-clock auxiliary operation and maintenance and decision-making by background managers.

[0030] Furthermore, during the road operation and maintenance phase, anchoring the road digital asset information and the human-computer interaction system in the road physical environment based on spatial computing includes:

[0031] The digital coordinates in the Mixed Reality Toolkit are set to the 3D coordinates of the digital world (X digital ,Y digital ,Z digital );

[0032] Convert the real-world data scanned by the camera system of the head-mounted mixed reality device into digital information that can be understood by the computer device, set spatial anchor points in the physical environment of the road, and map the digital physical objects at the anchor point positions to ensure that the digital information matches the real world;

[0033] Based on the 3D coordinates of the digital world, the 3D coordinates of the digital world are converted into the 3D coordinates of the real world through the camera system (X world ,Y world ,Z world ):

[0034] X world , Y world , Z world =K[R(X digital , Y digital , Z digital )+T]

[0035] Among them, R is the rotation matrix of the anchor coordinate system of the camera system, T is the translation matrix of the anchor coordinate system of the camera system, and K is the built-in initial matrix of the camera system:

[0036]

[0037] Among them, f x and f y is the focal length, c x and c x is the principal point coordinate, δ1~δ9 are the correction parameters of the initial matrix, among which, the values ​​of δ2, δ4, δ7 and δ8 are 0. This is to ensure that the data form of the built-in initial matrix is ​​not affected;

[0038] Scan the road environment in the real world based on the visible light camera, infrared camera and time-of-flight depth sensor of the head-mounted mixed reality device, divide the scanned real space into triangular meshes and model them, and convert them into a digital coordinate environment with the head-mounted mixed reality device as the world center, and store them in the cloud, deploy the three-dimensional coordinates of the digital world in the Unity game engine on the head-mounted mixed reality device, match the digital three-dimensional coordinates of the Unity game engine with the spatial coordinates after scanning the real environment, and map the digital objects to the real road engineering space through the head-mounted mixed reality device to serve as a digital environment for interaction between people, environment and computers;

[0039] Through the above spatial anchor point operation, the road digital asset information is anchored in the corresponding real road space, and the data information is visualized based on the real-time communication of sensors;

[0040] Based on the physical hand operation of digital objects, it provides on-site construction workers with real-time visual information and a computer operating system that facilitates human-computer interaction.

[0041] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0042] 1) By extending the reality technology and sensor system, and anchoring the road digital asset information and human-computer interaction system in the road physical environment based on spatial computing during the road operation and maintenance stage, the real-time response data and road inherent attribute data collected by the sensor system at the road engineering site can be effectively integrated and visualized in time and space, so as to provide on-site staff with road physical state information to assist in decision-making, realize effective interaction between people, environment and computers, reduce the time for staff to think and retrieve information, improve staff's perception, understanding and decision-making ability of road state information, and solve the problems of data integration and visualization, data time-space integration mapping and interaction between people, environment and computers at the engineering site in the operation and maintenance process of existing road projects;

[0043] 2. The extended reality human-computer interaction method for the operation and maintenance stage of road engineering provided in this embodiment has low development and operation costs, high practicality, strong maintainability, great promotion potential, can be used on a large scale, and can be industrialized.

[0044] 3. The extended reality human-computer interaction method for the operation and maintenance stage of road engineering provided in the example is not restricted by time and space, and can be integrated and implemented using head-mounted mixed reality devices. It has good engineering practicality and cutting-edge nature, and is conducive to promoting the development of basic technologies and basic tools for digital twins and intelligent construction of road engineering. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 It is a flow chart of an extended reality human-computer interaction method for the operation and maintenance stage of a road engineering project provided by an embodiment of the present invention;

[0047] Figure 2 is a schematic diagram of a temperature and humidity sensor arranged in layers based on road type, structure and function provided by an embodiment of the present invention;

[0048] Figure 3 It is a schematic diagram of anchoring road digital asset information and a human-computer interaction system in a road physical environment based on spatial computing provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0050] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0051] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0052] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0053] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0054] The embodiment of the present invention provides an augmented reality human-computer interaction method for the operation and maintenance stage of a road engineering project, such as Figure 1As shown, the processing flow of the method may include the following steps:

[0055] S1, during the road construction phase, the sensor system is arranged based on the road type, structure and function, and the spatial locations of various sensors are marked;

[0056] In this embodiment, the road types include expressways and municipal roads;

[0057] In this embodiment, the sensor system is arranged in layers as the roadbed and pavement engineering are constructed in layers; Figure 2 Schematic diagram of temperature and humidity sensors arranged in layers based on road type, structure and function.

[0058] The physical quantities monitored by the sensor system include one or more of stress, strain, load, displacement, oxygen content, temperature, humidity and ultraviolet radiation.

[0059] In this embodiment, marking the spatial positions of various sensors may specifically include the following steps:

[0060] Taking the road elevation as a reference, marking the depth of the road, the thickness of each layer of the roadbed and road surface, the length and width of the road surface, and taking the actual road engineering spatial information as a constraint, selecting any point on the road surface that is convenient for spatial marking as the origin O to construct a fixed global three-dimensional vector coordinate system, the global three-dimensional vector coordinate system takes the road length direction as the X direction, the road width direction as the Y direction, and the road depth direction as the Z direction;

[0061] In the global three-dimensional vector coordinate system, the layered road structure L along the road depth is taken as the reference. i All sensors in the group are marked with the digital three-dimensional coordinates of each sensor. Since the functions and usage methods of each type of sensor are different, the layout direction of the sensors is also different. The direction parallel to the road surface is 0°, and the angle θ between the two reference lines perpendicular to the road surface and the horizontal plane of the sensor is the layout direction of the sensor. The information of the spatial position of the sensor in the digital space is marked as S i,j (x digital ,y digital ,z digital ,θ), where S i,j Represents the i-th layer road structure L i The spatial position of the jth sensor in (x digital ,y digital ,z digital ,θ) represent the coordinate value and angle value of the sensor in the X direction, Y direction, and Z direction respectively.

[0062] S2, building a road digital asset system based on the physical information of the road structure and the real-time response data collected by the sensor system; specifically, the following steps may be included:

[0063] The physical information of the road structure is collected and a database of inherent road attributes is constructed. The inherent road attribute data includes: the spatial geometric dimensions of the layered road structure (i.e., the thickness, length and width of each layer of the road structure), the material information of each layer of the road structure and the engineering function;

[0064] Integrate the type, model, manufacturer, spatial location information of sensors in the road structure and the real-time response data collected by the sensor system to build a road structure response database;

[0065] The road digital asset system is constructed using BIM technology, including: a three-dimensional digital model of the road built at a scale of 1:100 containing road geometry information and materials, a road inherent attribute database, and a road structure response database; among them, BIM stands for Building Information Model.

[0066] In this embodiment, the real-time response data collected by the sensor system refers to the digital information collected by the sensors embedded in the road structure, that is, the monitoring signals of various types of sensors, including: stress, strain, load, displacement, oxygen content, temperature, humidity, ultraviolet radiation and other monitoring information.

[0067] In this embodiment, the constructed three-dimensional digital model of the road including road geometry information and materials, the road inherent attribute database and the road structure response database can also be stored in the private data management cloud platform of the road enterprise.

[0068] In this embodiment, the road inherent attribute database and the road structure response database are both relational databases built based on MySQL to store data information in relational road digital assets.

[0069] S3, based on the constructed road digital asset system, uses extended reality technology to visualize road digital asset information (including data in the road inherent attribute database and data in the road structure response database), and builds a distributed human-computer interaction system (i.e., an interaction system between people, environment, and computers) based on the industry regulations and processes of road engineering operation and maintenance;

[0070] In this embodiment, based on the constructed road digital asset system, the road digital asset information visualization is realized by using the extended reality technology, which may specifically include the following steps:

[0071] The 3D digital model of the road is exported as a file in .ifc format and imported into the Unity game engine; considering the problem that natural light affects the optical waveguide imaging, the transparency of the material of the 3D digital model of the road is set to 60% to 80%;

[0072] The multi-type data collected by various sensors in the road structure, including text, image and numerical data, are uniformly expressed and stored in the form of a matrix;

[0073] The road 3D digital model and sensor data visualization interface are deployed to the head-mounted mixed reality device through the Unity game engine.

[0074] In this embodiment, the sensor data visualization interface is a data visualization interface rewritten based on the TCP / IP communication interface of the data acquisition interface provided by various sensor manufacturers and is based on the C# language. The rewritten sensor data visualization interface can reflect the real-time dynamic expression of multiple types of data.

[0075] In this embodiment, a distributed human-computer interaction system is built based on the industry regulations and processes of road engineering operation and maintenance work, which may specifically include the following steps:

[0076] Import the .ifc file of the geometric information of the road digital assets in BIM into the Unity game engine as a digital physical object for human-computer interaction, and access the interface of the road inherent attribute database and the road structure response database;

[0077] Based on the industry regulations and processes of road engineering operation and maintenance, a human-computer interaction interface was built in the Unity game engine based on the front-end interface development platform Figma, and a human-computer interaction system was developed based on the mixed reality toolkit, including hand recognition and eye motion capture recognition, so as to realize the function of feedback of interactive information through physical hand operation of virtual digital interface in extended reality applications;

[0078] Based on the universal Windows platform, Visual Studio is used to deploy the road digital asset information and human-computer interaction system in the Unity game engine to the distributed application platform according to the application environment.

[0079] In this embodiment, the distributed application platform includes a head-mounted mixed reality device and a local background computer, wherein the head-mounted mixed reality device is used for round-the-clock operation and maintenance by on-site staff, and the background computer is used for round-the-clock auxiliary operation and maintenance and decision-making by background managers.

[0080] S4, in the road operation and maintenance stage, the road digital asset information and human-computer interaction system are anchored in the road physical environment (referring to the road environment in the real world, i.e., the road engineering site environment) based on spatial computing, such as Figure 3 As shown, the following steps may be specifically included:

[0081] The digital coordinates in the Mixed Reality Toolkit are set to the 3D coordinates of the digital world (X digital ,Y digital ,Z digital );

[0082] Convert the real-world data scanned by the camera system of the head-mounted mixed reality device into digital information that can be understood by the computer device, set spatial anchor points in the physical environment of the road, and map the digital physical objects at the anchor point positions to ensure that the digital information matches the real world;

[0083] Based on the 3D coordinates of the digital world, the 3D coordinates of the digital world are converted into the 3D coordinates of the real world through the camera system (X world ,Y world ,Z world ):

[0084] X world , Y world , Z world =K[R(X digital , Y digital , Z digital )+T]

[0085] Among them, R is the rotation matrix of the anchor coordinate system of the camera system, T is the translation matrix of the anchor coordinate system of the camera system, and K is the built-in initial matrix of the camera system:

[0086]

[0087] Among them, f x and f y is the focal length, c x and c x is the principal point coordinate, δ1~δ9 are the correction parameters of the initial matrix, among which, the values ​​of δ2, δ4, δ7 and δ8 are 0. This is to ensure that the data form of the built-in initial matrix is ​​not affected;

[0088] The visible light camera, infrared camera and time-of-flight depth sensor of the head-mounted mixed reality device are used to scan the road environment in the real world, and the single scanning space volume is specified to be 8m 3, divide the scanned real space into triangular meshes and model them, and convert them into a digital coordinate environment with the head-mounted mixed reality device as the world center, and store them in the cloud, deploy the three-dimensional coordinates of the digital world in the Unity game engine on the head-mounted mixed reality device, match the digital three-dimensional coordinates of the Unity game engine with the spatial coordinates after scanning the real environment, and map the digital objects to the real road engineering space through the head-mounted mixed reality device as a digital environment for interaction between people, environment and computers;

[0089] Through the above spatial anchor point operation, the road digital asset information is anchored in the corresponding real road space, and the data information is visualized based on the real-time communication of sensors;

[0090] Based on the physical hand operation of digital objects, it provides on-site construction personnel with real-time visual information and a computer operating system that facilitates human-computer interaction, and provides basic digital information expression methods and tools for subsequent road workers to perceive, understand and make decisions on information.

[0091] The augmented reality human-computer interaction method for the road engineering operation and maintenance stage provided in this embodiment has at least the following beneficial effects:

[0092] 1) By extending the reality technology and sensor system, and anchoring the road digital asset information and human-computer interaction system in the road physical environment based on spatial computing during the road operation and maintenance stage, the real-time response data and road inherent attribute data collected by the sensor system at the road engineering site can be effectively integrated and visualized in time and space, so as to provide on-site staff with road physical state information to assist in decision-making, realize effective interaction between people, environment and computers, reduce the time for staff to think and retrieve information, improve staff's perception, understanding and decision-making ability of road state information, and solve the problems of data integration and visualization, data time-space integration mapping and interaction between people, environment and computers at the engineering site in the operation and maintenance process of existing road projects;

[0093] 2. The extended reality human-computer interaction method for the operation and maintenance stage of road engineering provided in this embodiment has low development and operation costs, high practicality, strong maintainability, great promotion potential, can be used on a large scale, and can be industrialized.

[0094] 3. The extended reality human-computer interaction method for the operation and maintenance stage of road engineering provided in the example is not restricted by time and space, and can be integrated and implemented using head-mounted mixed reality devices. It has good engineering practicality and cutting-edge nature, and is conducive to promoting the development of basic technologies and basic tools for digital twins and intelligent construction of road engineering.

[0095] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0096] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0097] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0098] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0099] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0100] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0101] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0102] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0103] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0104] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0105] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An augmented reality human-computer interaction method for the operation and maintenance stage of a road project, characterized in that: The method comprises: During the road construction phase, the sensor system is arranged based on the road type, structure and function, and the spatial locations of various sensors are marked; Build a road digital asset system based on the physical information of the road structure and the real-time response data collected by the sensor system; Based on the constructed road digital asset system, the road digital asset information is visualized using extended reality technology, and a distributed human-computer interaction system is built based on the industry regulations and processes of road engineering operation and maintenance; During the road operation and maintenance stage, road digital asset information and human-computer interaction systems are anchored in the road physical environment based on spatial computing.

2. The augmented reality human-computer interaction method for the road engineering operation and maintenance stage according to claim 1, characterized in that: The road types include expressways and municipal roads; The sensor system is arranged in layers as the roadbed and pavement engineering are constructed in layers; The physical quantities monitored by the sensor system include one or more of stress, strain, load, displacement, oxygen content, temperature, humidity and ultraviolet radiation.

3. The augmented reality human-computer interaction method for the road engineering operation and maintenance stage according to claim 1 is characterized in that: The spatial positions of marking various sensors include: Taking the road elevation as a reference, marking the depth of the road, the thickness of each layer of the roadbed and road surface, the length and width of the road surface, and taking the actual road engineering spatial information as a constraint, selecting any point on the road surface that is convenient for spatial marking as the origin O to construct a fixed global three-dimensional vector coordinate system, the global three-dimensional vector coordinate system takes the road length direction as the X direction, the road width direction as the Y direction, and the road depth direction as the Z direction; In the global three-dimensional vector coordinate system, the layered road structure L along the road depth is taken as the reference. i All sensors in the group are marked with the digital three-dimensional coordinates of each sensor, with the direction parallel to the road surface as the 0° direction, and the angle θ between the two reference lines perpendicular to the road surface and the horizontal plane of the sensor as the arrangement direction of the sensor. The information of the spatial position of the sensor in the digital space is marked as S i,j (x digital ,y digital ,z digital ,θ), where S i,j Represents the i-th layer road structure L i The spatial position of the jth sensor in (x digital ,y digital ,z digital ,θ) represent the coordinate value and angle value of the sensor in the X direction, Y direction, and Z direction respectively.

4. The augmented reality human-computer interaction method for the road engineering operation and maintenance stage according to claim 1, characterized in that: The road digital asset system constructed based on the road structure physical information and the real-time response data collected by the sensor system includes: The physical information of the road structure is collected and a database of inherent road attributes is constructed. The inherent road attribute data includes: the spatial geometric dimensions of the layered road structure, the material information of each layer of the road structure, and the engineering function; Integrate the type, model, manufacturer, spatial location information of sensors in the road structure and the real-time response data collected by the sensor system to build a road structure response database; The road digital asset system is constructed using BIM technology, including: a three-dimensional digital model of the road built at a scale of 1:100 containing road geometry information and materials, a road inherent attribute database, and a road structure response database; among them, BIM stands for Building Information Model.

5. The augmented reality human-computer interaction method for the road engineering operation and maintenance stage according to claim 4, characterized in that: The road inherent attribute database and the road structure response database are both relational databases built based on MySQL to store data information in relational road digital assets.

6. The augmented reality human-computer interaction method for the road engineering operation and maintenance stage according to claim 4, characterized in that: The road digital asset system constructed based on the above method uses the augmented reality technology to realize the visualization of road digital asset information, including: Export the road 3D digital model as an .ifc file and import it into the Unity game engine; The multi-type data collected by various sensors in the road structure, including text, image and numerical data, are expressed and stored in the form of a matrix; The road 3D digital model and sensor data visualization interface are deployed to the head-mounted mixed reality device through the Unity game engine.

7. The augmented reality human-computer interaction method for the road engineering operation and maintenance stage according to claim 6, characterized in that: The sensor data visualization interface is a data visualization interface rewritten based on the TCP / IP communication interface of the data acquisition interface provided by various sensor manufacturers and is a data visualization interface rewritten based on the C# language. The rewritten sensor data visualization interface can reflect the real-time dynamic expression of multiple types of data.

8. The augmented reality human-computer interaction method for the road engineering operation and maintenance stage according to claim 6, characterized in that: The distributed human-computer interaction system based on the industry regulations and processes of road engineering operation and maintenance work includes: Import the .ifc file of the geometric information of the road digital assets in BIM into the Unity game engine as a digital physical object for human-computer interaction, and access the interface of the road inherent attribute database and the road structure response database; Based on the industry regulations and processes of road engineering operation and maintenance, a human-computer interaction interface was built in the Unity game engine based on the front-end interface development platform Figma, and a human-computer interaction system was developed based on the mixed reality toolkit, including hand recognition and eye motion capture recognition, so as to realize the function of feedback of interactive information through physical hand operation of virtual digital interface in extended reality applications; Based on the universal Windows platform, Visual Studio is used to deploy the road digital asset information and human-computer interaction system in the Unity game engine to the distributed application platform according to the application environment.

9. The augmented reality human-computer interaction method for the road engineering operation and maintenance stage according to claim 8, characterized in that: The distributed application platform includes a head-mounted mixed reality device and a local background computer, wherein the head-mounted mixed reality device is used for round-the-clock operation and maintenance by on-site staff, and the background computer is used for round-the-clock auxiliary operation and maintenance and decision-making by background managers.

10. The augmented reality human-computer interaction method for the road engineering operation and maintenance stage according to claim 8, characterized in that: In the road operation and maintenance stage, anchoring the road digital asset information and the human-computer interaction system in the road physical environment based on spatial computing includes: The digital coordinates in the Mixed Reality Toolkit are set to the 3D coordinates of the digital world (X digital ,Y digital ,Z digital ); Convert the real-world data scanned by the camera system of the head-mounted mixed reality device into digital information that can be understood by the computer device, set spatial anchor points in the physical environment of the road, and map the digital physical objects at the anchor point positions to ensure that the digital information matches the real world; Based on the 3D coordinates of the digital world, the 3D coordinates of the digital world are converted into the 3D coordinates of the real world through the camera system (X world ,Y world ,Z world ): X world ,Y world ,Z world =K[R(X digital ,Y digital ,Z digital )+T] Among them, R is the rotation matrix of the anchor coordinate system of the camera system, T is the translation matrix of the anchor coordinate system of the camera system, and K is the built-in initial matrix of the camera system: Among them, f x and f y is the focal length, c x and c x is the principal point coordinate, δ1~δ9 are the correction parameters of the initial matrix, among which, the values ​​of δ2, δ4, δ7 and δ8 are 0. This is to ensure that the data form of the built-in initial matrix is ​​not affected; Scan the road environment in the real world based on the visible light camera, infrared camera and time-of-flight depth sensor of the head-mounted mixed reality device, divide the scanned real space into triangular meshes and model them, and convert them into a digital coordinate environment with the head-mounted mixed reality device as the world center, and store them in the cloud, deploy the three-dimensional coordinates of the digital world in the Unity game engine on the head-mounted mixed reality device, match the digital three-dimensional coordinates of the Unity game engine with the spatial coordinates after scanning the real environment, and map the digital objects to the real road engineering space through the head-mounted mixed reality device to serve as a digital environment for interaction between people, environment and computers; Through the above spatial anchor point operation, the road digital asset information is anchored in the corresponding real road space, and the data information is visualized based on the real-time communication of sensors; Based on the physical hand operation of digital objects, it provides on-site construction workers with real-time visual information and a computer operating system that facilitates human-computer interaction.

Images