An AR virtual interaction system based on digital twins

Abstract

An AR virtual interaction system based on digital twins relates to the field of information transmission technology, including a monitoring center, wherein the monitoring center is communicatively connected to a data acquisition module, a data processing module, a data visualization module, a data analysis module and a voice interaction module; the data acquisition module collects data information; the data processing module divides the road traffic data display area on the AR glasses worn by the driver; the data visualization module visualizes the road traffic data; the data analysis module dynamically adjusts the road traffic data displayed on the AR glasses according to the driver's perspective information; the voice interaction module dynamically adjusts the visualization content of the AR glasses according to the driver's input voice information, determines the driver's status according to the driver's facial features and gives the driver a voice reminder; so that the driver can have a clearer and actual feeling of the vehicle's driving status during driving and reduce the risk of traffic accidents during driving.

Description

Technical Field

[0001] The present invention relates to the field of intelligent management and control technology, and specifically to an AR virtual interaction system based on digital twins. Background Art

[0002] AR technology combines virtual digital media information and real objects, and simulates more physical information (including images, sounds, text, etc.) within a certain time and space range through scientific and technological means, and then concentrates the processed information feedback and superimposes it on the visible real environment. It realizes the fusion of virtual objects and real environment through display devices, presenting a new environment with amplified sensory effects and more diversity to the experiencer, and ultimately achieves the purpose of enhancing the experiencer's sensory perception. With the help of tools, people can achieve the experience of strengthening their own perception. In the digital information age, people can use AR technology to receive as much information as possible from the Internet age.

[0003] The existing application of AR technology in the field of automobile driving is only to virtually display the street scenes around the car. However, how to make AR technology highly interactive with the driver during driving, and enable the driver to have a clearer and more practical feeling of the vehicle's driving status during driving and enhance the driver's perception of unknown dangers, break the limitations of time and space, and enable the driver to quickly obtain road conditions information with real reliability is an urgent problem that we need to solve. We now provide an AR virtual interaction system based on digital twins. Summary of the Invention

[0004] In order to solve the above technical problems, the purpose of the present invention is to provide an AR virtual interaction system based on digital twins, including a monitoring center, which is communicatively connected to a data acquisition module, a data processing module, a data visualization module, a data analysis module and a voice interaction module;

[0005] The data acquisition module is used to collect road traffic data and driver facial feature information within a preset range of the vehicle driving section and set a monitoring cycle;

[0006] The data processing module is used to construct an integrated visual map of urban road traffic and divide the road traffic data display area on the AR glasses worn by the driver;

[0007] The data visualization module visualizes the road traffic data within a preset range of the vehicle driving section in the integrated visual map of urban road traffic on the AR glasses worn by the driver;

[0008] The data analysis module is used to dynamically adjust the road traffic data displayed on the AR glasses according to the driver's perspective information;

[0009] The voice interaction module is used to dynamically adjust the visual content of the AR glasses according to the voice information input by the driver, determine the driver's status according to the driver's facial features, and provide voice reminders to the driver.

[0010] Furthermore, the process of the data acquisition module collecting road traffic data and driver facial feature information within a preset range of the vehicle driving section and setting a monitoring period includes:

[0011] Using a sensor device installed in the driver's vehicle, the driver's eye gaze position, gaze duration, eye movement trajectory, eye closure time, vehicle position information, vehicle speed, and distance to the vehicle ahead are obtained; the driver's eye gaze position, gaze duration, eye movement trajectory, and eye closure time are used as the driver's facial feature information;

[0012] A big data method is used to obtain road network data within a preset range of a vehicle driving section, and the obtained road network data is divided into several traffic sections in the form of road sections. The traffic flow of each traffic section is obtained, and the traffic flow, road conditions, position information of the driving vehicle, speed and distance information to the vehicle in front of each traffic section are used as the road traffic data.

[0013] Furthermore, the process of constructing the integrated visual map of urban road traffic by the data processing module includes:

[0014] Using GIS to obtain the location information of each traffic section and surrounding buildings in the city's physical space, construct a two-dimensional coordinate system, obtain a plan view representing each traffic section and surrounding buildings based on the location information of each traffic section and surrounding buildings, and map the plan view to the two-dimensional coordinate system to obtain a basic layer of urban road traffic;

[0015] A multi-source data heterogeneous set is constructed based on the traffic data of each traffic section in the city, and the data format of the multi-source data heterogeneous set is preprocessed. A twin data set is generated from the multi-source heterogeneous data set after data format preprocessing. Three-dimensional modeling is performed on each traffic section and surrounding buildings in the basic layer to obtain a three-dimensional model of each traffic section and surrounding buildings in the urban physical space. The twin data set is then matched with the three-dimensional model of each traffic section and surrounding buildings in the urban physical space to obtain a three-dimensional data twin model.

[0016] Different spatial scene information of each traffic section is obtained based on the surrounding buildings of each traffic section, the spatial scene information is processed into a scene sequence, and the scene sequence is stored in a three-dimensional data twin model. The three-dimensional models of each traffic section and surrounding buildings in the three-dimensional data twin model are combined with the scene sequence of each traffic section to generate an integrated visual map of urban road traffic.

[0017] Furthermore, the process of dividing the road traffic data display area on the AR glasses worn by the driver by the data processing module includes:

[0018] The driver's road attention field of view on the AR glasses is obtained based on the driver's eye gaze position, gaze duration, and eye movement trajectory. The four areas directly above, directly below, to the left, and to the right of the road attention field of view outside the display area of ​​the AR glasses are marked as areas to be measured. A big data method is used to obtain evaluation criteria for the driver's road attention level corresponding to the four areas directly above, directly below, to the left, and to the right of the driver's main field of view on the AR glasses. The road attention level is classified into alert, general, insufficiently alert, and distracted.

[0019] An evaluation standard matrix for road attention is established according to the evaluation standards of the road attention corresponding to each area in the area to be measured, an indicator weight matrix of the evaluation indicators is set, and the membership matrix of each area to be measured for road attention is obtained through fuzzy comprehensive evaluation; the road attention of each area to be measured is obtained according to the membership matrix and the indicator weight matrix.

[0020] Furthermore, the process of the data visualization module visually displaying the road traffic data within a preset range of the vehicle driving section in the integrated urban road traffic visual map on the AR glasses worn by the driver includes:

[0021] Transmitting road traffic data within a preset range of a vehicle's driving section in an integrated visual map of urban road traffic to the AR glasses, and displaying the road traffic data in a to-be-measured area of ​​the driver's AR glasses; obtaining traffic flow data for the driver's traffic section based on the road traffic data; setting traffic flow threshold intervals, each of which is assigned a different color; marking traffic flow levels with colors based on the traffic flow threshold intervals in which the traffic flow data falls, and displaying the traffic flow data in the form of arrows of different colors in the to-be-measured area of ​​the AR glasses with a normal road attention level; obtaining vehicle distance information between the driver's vehicle and the vehicle ahead based on the road traffic data; and displaying the vehicle distance information in digital form in the to-be-measured area of ​​the AR glasses with an alert road attention level;

[0022] Acquire a scene sequence of the traffic section traveled by the driver based on the integrated visual map of urban road traffic, and display the scene sequence in the form of a three-dimensional model in the area to be measured where the road attention level is insufficient;

[0023] When the driver inputs the destination information through the voice interaction module, the navigation information of the driver's driving section is obtained based on the integrated visual map of urban road traffic, the destination information input by the driver through the voice interaction module and the road traffic data, and the traffic section where the driver is driving and the surrounding buildings are displayed in the form of a three-dimensional model in the AR glasses as an area to be measured with a general road attention level.

[0024] Furthermore, the process of the data analysis module dynamically adjusting the road traffic data displayed on the AR glasses according to the driver's perspective information includes:

[0025] The position information and speed of the moving vehicle are obtained based on the road traffic data, and the vehicle driving status is obtained based on the position information and speed of the moving vehicle; when the vehicle driving status of the vehicle is uniform and straight during the monitoring period, the road traffic data displayed on the AR glasses is hidden, and only the vehicle distance information of the area to be measured with alert road attention is retained, and the driver can send voice information to the AR glasses through the voice interaction module to display the road traffic data hidden on the AR glasses.

[0026] Furthermore, the process of the voice interaction module dynamically adjusting the visual content of the AR glasses according to the driver's input voice information includes:

[0027] The display position and size of the road traffic data in the area to be measured of the AR glasses are adjusted in real time according to the voice information input by the driver.

[0028] Furthermore, the process of the voice interaction module determining the driver's status based on the driver's facial features and giving the driver a voice reminder includes:

[0029] A big data method is used to obtain the driver's eye gaze position, gaze duration, eye movement trajectory and eye closure time in different vehicle driving states during several historical detection cycles; and a facial feature prediction model of the driver in different vehicle driving states is constructed based on an RBF neural network. A historical facial feature dataset is constructed based on the eye gaze position, gaze duration, eye movement trajectory and eye closure time in different vehicle driving states during several historical detection cycles, and the historical facial feature dataset is divided into a training set and a test set. The facial feature prediction model is trained in real time using the training set until the loss function training is stable and the model parameters are saved. Then, the output data matrix of the iteratively trained facial feature prediction model is verified for similarity using the test set, and the facial feature dataset of the driver in different vehicle driving states is obtained based on the output layer of the facial feature prediction model verified by the test set.

[0030] Obtain the driver's facial feature information and vehicle driving status in the current monitoring period, compare the similarity between the driver's facial feature information in the current vehicle driving status and the facial feature dataset corresponding to the current vehicle driving status generated by the facial feature prediction model, set a similarity threshold, and if the similarity between the driver's facial feature information in the current vehicle driving status and the facial feature dataset corresponding to the current vehicle driving status generated by the facial feature prediction model does not meet the similarity threshold, mark the driver's status as a distracted state and give a voice warning to the driver.

[0031] Compared with the prior art, the beneficial effects of the present invention are: when determining the display position of data visualization, the present invention also takes into account the interaction method with the driver. The driver can use voice commands, touch panels and other interactive methods to control and adjust the display position of the data, which can provide a more convenient and safe user interface, allowing the driver to adjust and operate the position of the data display at any time, so that the driver can obtain the best user experience; at the same time, the driver's field of view is determined according to the driver's facial features, and the display area of ​​the road data is set outside the field of view. This avoids the impact of the display of road data on the driver's actual driving field of view, ensuring that when using AR glasses, the driver still needs to concentrate on the road, thereby improving the driver's driving safety. At the same time, the AR glasses worn by the driver can display and update road traffic data in real time, helping the driver to understand the road conditions more intuitively, provide navigation instructions, and reduce driving stress.

[0032] On the other hand, by real-time monitoring of the driver's facial features, the driver's driving status can be analyzed. When the driver is distracted or fatigued, the voice interaction module will give the driver timely voice reminders to reduce the risk of accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a schematic diagram of an AR virtual interaction system based on digital twins in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The following is a clear and complete description of the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0035] like Figure 1 As shown, an AR virtual interaction system based on digital twins includes a monitoring center, which is communicatively connected to a data acquisition module, a data processing module, a data visualization module, a data analysis module, and a voice interaction module;

[0036] The data acquisition module is used to collect road traffic data and driver facial feature information within a preset range of the vehicle driving section and set a monitoring cycle;

[0037] The data processing module is used to construct an integrated visual map of urban road traffic and divide the road traffic data display area on the AR glasses worn by the driver;

[0038] The data visualization module visualizes the road traffic data within a preset range of the vehicle driving section in the integrated visual map of urban road traffic on the AR glasses worn by the driver;

[0039] The data analysis module is used to dynamically adjust the road traffic data displayed on the AR glasses according to the driver's perspective information;

[0040] The voice interaction module is used to dynamically adjust the visual content of the AR glasses according to the voice information input by the driver, determine the driver's status according to the driver's facial features, and provide voice reminders to the driver.

[0041] It should be further explained that, in a specific implementation process, the process of the data acquisition module collecting road traffic data within a preset range of the vehicle driving section and the driver's facial feature information and setting a monitoring period includes:

[0042] Using a sensor device installed in the driver's vehicle, the driver's eye gaze position, gaze duration, eye movement trajectory, eye closure time, vehicle position information, vehicle speed, and distance to the vehicle ahead are obtained; the driver's eye gaze position, gaze duration, eye movement trajectory, and eye closure time are used as the driver's facial feature information;

[0043] A big data method is used to obtain road network data within a preset range of a vehicle driving section, and the obtained road network data is divided into several traffic sections in the form of road sections. The traffic flow of each traffic section is obtained, and the traffic flow, road conditions, position information of the driving vehicle, speed and distance information to the vehicle in front of each traffic section are used as the road traffic data.

[0044] It should be further explained that, in a specific implementation process, the process of the data processing module constructing an integrated visual map of urban road traffic includes:

[0045] Using GIS to obtain the location information of each traffic section and surrounding buildings in the city's physical space, construct a two-dimensional coordinate system, obtain a plan view representing each traffic section and surrounding buildings based on the location information of each traffic section and surrounding buildings, and map the plan view to the two-dimensional coordinate system to obtain a basic layer of urban road traffic;

[0046] A multi-source data heterogeneous set is constructed based on the traffic data of each traffic section in the city, and the data format of the multi-source data heterogeneous set is preprocessed. A twin data set is generated from the multi-source heterogeneous data set after data format preprocessing. Three-dimensional modeling is performed on each traffic section and surrounding buildings in the basic layer to obtain a three-dimensional model of each traffic section and surrounding buildings in the urban physical space. The twin data set is then matched with the three-dimensional model of each traffic section and surrounding buildings in the urban physical space to obtain a three-dimensional data twin model.

[0047] Different spatial scene information of each traffic section is obtained based on the surrounding buildings of each traffic section, the spatial scene information is processed into a scene sequence, and the scene sequence is stored in a three-dimensional data twin model. The three-dimensional models of each traffic section and surrounding buildings in the three-dimensional data twin model are combined with the scene sequence of each traffic section to generate an integrated visual map of urban road traffic.

[0048] It should be further explained that, in a specific implementation process, the process of the data processing module dividing the road traffic data display area on the AR glasses worn by the driver includes:

[0049] The driver's road attention field of view on the AR glasses is obtained based on the driver's eye gaze position, gaze duration, and eye movement trajectory. The four areas directly above, directly below, to the left, and to the right of the road attention field of view outside the display area of ​​the AR glasses are marked as areas to be measured. A big data method is used to obtain evaluation criteria for the driver's road attention level corresponding to the four areas directly above, directly below, to the left, and to the right of the driver's main field of view on the AR glasses. The road attention level is classified into alert, general, insufficiently alert, and distracted.

[0050] An evaluation standard matrix for road attention is established according to the evaluation standards of the road attention corresponding to each area in the area to be measured, an indicator weight matrix of the evaluation indicators is set, and the membership matrix of each area to be measured for road attention is obtained through fuzzy comprehensive evaluation; the road attention of each area to be measured is obtained according to the membership matrix and the indicator weight matrix.

[0051] It should be further explained that, in a specific implementation process, the data visualization module visualizes the road traffic data within a preset range of the vehicle driving section in the integrated urban road traffic visual map on the AR glasses worn by the driver, including:

[0052] Transmitting road traffic data within a preset range of a vehicle's driving section in an integrated visual map of urban road traffic to the AR glasses, and displaying the road traffic data in a to-be-measured area of ​​the driver's AR glasses; obtaining traffic flow data for the driver's traffic section based on the road traffic data; setting traffic flow threshold intervals, each of which is assigned a different color; marking traffic flow levels with colors based on the traffic flow threshold intervals in which the traffic flow data falls, and displaying the traffic flow data in the form of arrows of different colors in the to-be-measured area of ​​the AR glasses with a normal road attention level; obtaining vehicle distance information between the driver's vehicle and the vehicle ahead based on the road traffic data; and displaying the vehicle distance information in digital form in the to-be-measured area of ​​the AR glasses with an alert road attention level;

[0053] Acquire a scene sequence of the traffic section traveled by the driver based on the integrated visual map of urban road traffic, and display the scene sequence in the form of a three-dimensional model in the area to be measured where the road attention level is insufficient;

[0054] When the driver inputs the destination information through the voice interaction module, the navigation information of the driver's driving section is obtained based on the integrated visual map of urban road traffic, the destination information input by the driver through the voice interaction module and the road traffic data, and the traffic section where the driver is driving and the surrounding buildings are displayed in the form of a three-dimensional model in the AR glasses as an area to be measured with a general road attention level.

[0055] It should be further explained that, in a specific implementation process, the process in which the data analysis module dynamically adjusts the road traffic data displayed on the AR glasses according to the driver's perspective information includes:

[0056] The position information and speed of the moving vehicle are obtained based on the road traffic data, and the vehicle driving status is obtained based on the position information and speed of the moving vehicle; when the vehicle driving status of the vehicle is uniform and straight during the monitoring period, the road traffic data displayed on the AR glasses is hidden, and only the vehicle distance information of the area to be measured with alert road attention is retained, and the driver can send voice information to the AR glasses through the voice interaction module to display the road traffic data hidden on the AR glasses.

[0057] It should be further explained that, in a specific implementation, the process of the voice interaction module dynamically adjusting the visual content of the AR glasses according to the driver's input voice information includes:

[0058] The display position and size of the road traffic data in the area to be measured of the AR glasses are adjusted in real time according to the voice information input by the driver.

[0059] It should be further explained that, in a specific implementation process, the process in which the voice interaction module determines the driver's status based on the driver's facial features and gives the driver a voice reminder includes:

[0060] A big data method is used to obtain the driver's eye gaze position, gaze duration, eye movement trajectory and eye closure time in different vehicle driving states during several historical detection cycles; and a facial feature prediction model of the driver in different vehicle driving states is constructed based on an RBF neural network. A historical facial feature dataset is constructed based on the eye gaze position, gaze duration, eye movement trajectory and eye closure time in different vehicle driving states during several historical detection cycles, and the historical facial feature dataset is divided into a training set and a test set. The facial feature prediction model is trained in real time using the training set until the loss function training is stable and the model parameters are saved. Then, the output data matrix of the iteratively trained facial feature prediction model is verified for similarity using the test set, and the facial feature dataset of the driver in different vehicle driving states is obtained based on the output layer of the facial feature prediction model verified by the test set.

[0061] Obtain the driver's facial feature information and vehicle driving status in the current monitoring period, compare the similarity between the driver's facial feature information in the current vehicle driving status and the facial feature dataset corresponding to the current vehicle driving status generated by the facial feature prediction model, set a similarity threshold, and if the similarity between the driver's facial feature information in the current vehicle driving status and the facial feature dataset corresponding to the current vehicle driving status generated by the facial feature prediction model does not meet the similarity threshold, mark the driver's status as a distracted state and give a voice warning to the driver.

[0062] The above embodiments are only used to illustrate the technical method of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. An AR virtual interaction system based on digital twins, including a monitoring center, characterized in that: The monitoring center is communicatively connected to a data acquisition module, a data processing module, a data visualization module, a data analysis module and a voice interaction module; The data acquisition module is used to collect road traffic data and driver facial feature information within a preset range of the vehicle driving section and set a monitoring cycle: Using a sensor device installed in the driver's vehicle, the driver's eye gaze position, gaze duration, eye movement trajectory, eye closure time, vehicle position information, vehicle speed, and distance to the vehicle ahead are obtained; the driver's eye gaze position, gaze duration, eye movement trajectory, and eye closure time are used as the driver's facial feature information; using a big data method to obtain road network data within a preset range of a vehicle travel section, dividing the obtained road network data into a number of traffic sections in the form of road sections, obtaining the traffic volume of each traffic section, and using the traffic volume, road surface conditions, position information of the traveling vehicle, vehicle speed, and distance information to the preceding vehicle of each traffic section as the road traffic data; The data processing module is used to construct the process of the integrated visual map of urban road traffic: Using GIS to obtain the location information of each traffic section and surrounding buildings in the city's physical space, construct a two-dimensional coordinate system, obtain a plan view representing each traffic section and surrounding buildings based on the location information of each traffic section and surrounding buildings, and map the plan view to the two-dimensional coordinate system to obtain a basic layer of urban road traffic; A multi-source data heterogeneous set is constructed based on the traffic data of each traffic section in the city, and the data format of the multi-source data heterogeneous set is preprocessed. A twin data set is generated from the multi-source heterogeneous data set after data format preprocessing. Three-dimensional modeling is performed on each traffic section and surrounding buildings in the basic layer to obtain a three-dimensional model of each traffic section and surrounding buildings in the urban physical space. The twin data set is then matched with the three-dimensional model of each traffic section and surrounding buildings in the urban physical space to obtain a three-dimensional data twin model. Different spatial scene information of each traffic section is obtained based on the surrounding buildings of each traffic section, the spatial scene information is processed into a scene sequence, the scene sequence is stored in a three-dimensional data twin model, and the three-dimensional models of each traffic section and the surrounding buildings in the three-dimensional data twin model are combined with the scene sequence of each traffic section to generate an integrated visual map of urban road traffic; Divide the road traffic data display area on the AR glasses worn by the driver; The data visualization module is used to visualize the road traffic data within a preset range of the vehicle driving section in the urban road traffic integrated visual map on the AR glasses worn by the driver; The data analysis module is used to dynamically adjust the road traffic data displayed on the AR glasses according to the driver's perspective information; The voice interaction module is used to dynamically adjust the visual content of the AR glasses according to the voice information input by the driver, determine the driver's status according to the driver's facial features, and provide voice reminders to the driver.

2. The AR virtual interaction system based on digital twin according to claim 1, characterized in that: The process of dividing the road traffic data display area on the AR glasses worn by the driver by the data processing module includes: The driver's road attention field of view on the AR glasses is obtained based on the driver's eye gaze position, gaze duration, and eye movement trajectory. The four areas directly above, directly below, to the left, and to the right of the road attention field of view outside the display area of ​​the AR glasses are marked as areas to be measured. A big data method is used to obtain evaluation criteria for the driver's road attention level corresponding to the four areas directly above, directly below, to the left, and to the right of the driver's main field of view on the AR glasses. The road attention level is classified into alert, general, insufficiently alert, and distracted. An evaluation standard matrix for road attention is established according to the evaluation standards of the road attention corresponding to each area in the area to be measured, an indicator weight matrix of the evaluation indicators is set, and the membership matrix of each area to be measured for road attention is obtained through fuzzy comprehensive evaluation; the road attention of each area to be measured is obtained according to the membership matrix and the indicator weight matrix.

3. The AR virtual interaction system based on digital twin according to claim 2, characterized in that: The process of visually displaying the road traffic data within a preset range of the vehicle driving section in the integrated urban road traffic visual map by the data visualization module on the AR glasses worn by the driver includes: Transmitting road traffic data within a preset range of a vehicle's driving section in an integrated visual map of urban road traffic to the AR glasses, and displaying the road traffic data in a to-be-measured area of ​​the driver's AR glasses; obtaining traffic flow data for the driver's traffic section based on the road traffic data; setting traffic flow threshold intervals, each of which is assigned a different color; marking traffic flow levels with colors based on the traffic flow threshold intervals in which the traffic flow data falls, and displaying the traffic flow data in the form of arrows of different colors in the to-be-measured area of ​​the AR glasses with a normal road attention level; obtaining vehicle distance information between the driver's vehicle and the vehicle ahead based on the road traffic data; and displaying the vehicle distance information in digital form in the to-be-measured area of ​​the AR glasses with an alert road attention level; Acquire a scene sequence of the traffic section traveled by the driver based on the integrated visual map of urban road traffic, and display the scene sequence in the form of a three-dimensional model in the area to be measured where the road attention level is insufficient; When the driver inputs the destination information through the voice interaction module, the navigation information of the driver's driving section is obtained based on the integrated visual map of urban road traffic, the destination information input by the driver through the voice interaction module and the road traffic data, and the traffic section where the driver is driving and the surrounding buildings are displayed in the form of a three-dimensional model in the AR glasses as an area to be measured with a general road attention level.

4. The AR virtual interaction system based on digital twin according to claim 3, characterized in that: The process of the data analysis module dynamically adjusting the road traffic data displayed on the AR glasses according to the driver's perspective information includes: The position information and speed of the moving vehicle are obtained based on the road traffic data, and the vehicle driving status is obtained based on the position information and speed of the moving vehicle; when the vehicle driving status of the vehicle is uniform and straight during the monitoring period, the road traffic data displayed on the AR glasses is hidden, and only the vehicle distance information of the area to be measured with alert road attention is retained, and the driver can send voice information to the AR glasses through the voice interaction module to display the road traffic data hidden on the AR glasses.

5. The AR virtual interaction system based on digital twin according to claim 4, characterized in that: The process of the voice interaction module dynamically adjusting the visual content of the AR glasses according to the driver's input voice information includes: The display position and size of the road traffic data in the area to be measured of the AR glasses are adjusted in real time according to the voice information input by the driver.

6. The AR virtual interaction system based on digital twin according to claim 5, characterized in that: The process of the voice interaction module determining the driver's state based on the driver's facial features and giving the driver a voice reminder includes: A big data method is used to obtain the driver's eye gaze position, gaze duration, eye movement trajectory and eye closure time in different vehicle driving states during several historical detection cycles; and a facial feature prediction model of the driver in different vehicle driving states is constructed based on an RBF neural network. A historical facial feature dataset is constructed based on the eye gaze position, gaze duration, eye movement trajectory and eye closure time in different vehicle driving states during several historical detection cycles, and the historical facial feature dataset is divided into a training set and a test set. The facial feature prediction model is trained in real time using the training set until the loss function training is stable and the model parameters are saved. Then, the output data matrix of the iteratively trained facial feature prediction model is verified for similarity using the test set, and the facial feature dataset of the driver in different vehicle driving states is obtained based on the output layer of the facial feature prediction model verified by the test set. Obtain the driver's facial feature information and vehicle driving status in the current monitoring period, compare the similarity between the driver's facial feature information in the current vehicle driving status and the facial feature dataset corresponding to the current vehicle driving status generated by the facial feature prediction model, set a similarity threshold, and if the similarity between the driver's facial feature information in the current vehicle driving status and the facial feature dataset corresponding to the current vehicle driving status generated by the facial feature prediction model does not meet the similarity threshold, mark the driver's status as a distracted state and give a voice warning to the driver.

Images