AR-HUD display control method based on human intelligence and related equipment

Abstract

The invention provides an AR-HUD display control method based on human intelligence and related equipment. The method comprises the following steps: acquiring physiological feature data and identity information of a driver; determining the driving attention state of the driver according to the physiological feature data; wherein the driving attention state represents the attention degree of the driver when the driver drives the target vehicle; generating display parameters corresponding to the vehicle information according to the identity information and the driving attention state, and generating a corresponding display picture according to the vehicle information and the display parameters; and controlling the AR-HUD to project the display interface to the target area.

Description

Technical Field

[0001] The present invention relates to the field of information display technology, and in particular to an AR-HUD display control method based on human intelligence, an AR-HUD display control device based on human intelligence, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the rapid development of modern transportation, driving safety has become a vital issue. In order to meet the driver's need to obtain various important information in a timely manner to make accurate decisions during driving, Head Up Display (HUD) technology came into being. HUD technology improves driving convenience and safety by projecting key information directly into the driver's field of vision. In recent years, the combination of augmented reality (AR) technology and HUD technology has formed AR-HUD, which further enriches the way information is presented, integrates virtual information with real road scenes, and provides drivers with a more intuitive and rich visual experience.

[0003] However, most AR-HUD devices on the market are relatively fixed in terms of information presentation, lacking detailed consideration of factors such as individual differences among drivers and real-time driving status, thus affecting the reception of information. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems in the related art to a certain extent. To this end, the first purpose of the present invention is to propose an AR-HUD display control method based on human intelligence, which can accurately evaluate the driver's attention state by comprehensively applying biometric technology and augmented reality head-up display (AR-HUD), and realize personalized information display based on the evaluation results, thereby greatly improving the safety and efficiency of the driving process.

[0005] The second object of the present invention is to provide an AR-HUD display control device based on human intelligence.

[0006] A third objective of the present invention is to provide an electronic device.

[0007] A fourth object of the present invention is to provide a computer-readable storage medium.

[0008] To achieve the above objectives, the first embodiment of the present invention proposes an AR-HUD display control method based on human intelligence, comprising:

[0009] Obtain the driver's physiological characteristics data and identity information;

[0010] Determining the driver's driving attention state based on the physiological characteristic data; wherein the driving attention state represents the driver's attention level when driving the target vehicle;

[0011] Generate display parameters corresponding to vehicle information according to the identity information and the driving attention state, and generate a corresponding display screen according to the vehicle information and the display parameters;

[0012] Control AR-HUD to project the display interface to the target area.

[0013] In addition, the AR-HUD display control method based on human intelligence according to the above embodiment of the present invention may also have the following additional technical features:

[0014] As an optional embodiment, the physiological characteristic data includes at least one of the driver's electroencephalogram data, electrocardiogram data and electrodermal data.

[0015] As an optional embodiment, the identity information includes at least one of the driver's gender and age.

[0016] As an optional embodiment, determining the driving attention state of the driver according to the physiological characteristic data includes:

[0017] The physiological characteristic data is input into a pre-built attention state assessment model to obtain the driving attention state; wherein the driving attention state includes one of a concentration state, a fatigue state and a distraction state.

[0018] As an optional embodiment, the method further includes:

[0019] The basic neural network model is trained based on the pre-collected physiological characteristic data of the driver in a simulated driving situation, so that the basic neural network model learns the mapping relationship between identity information and physiological characteristic data and driving attention status, and obtains an attention status assessment model.

[0020] As an optional embodiment, display parameters corresponding to vehicle information are generated according to identity information and target attention state, including:

[0021] The identity information and the target attention state are input into a pre-built multimodal data fusion model to obtain display parameters; wherein the display parameters include color parameters, style parameters and presentation position coordinates for vehicle information, and the presentation position coordinates are coordinates relative to the driver's head-on field of view.

[0022] As an optional embodiment, the method further includes:

[0023] Based on the pre-collected different identity information of the driver in a simulated driving scenario, when various driving events occur and the display screen generated by using different display parameters prompts the driver, the driver's response speed, physiological characteristic data and driving attention status are used to train the basic neural network model, so that the basic neural network model learns the mapping relationship between the identity information, physiological characteristic data and driving attention status and the display parameters, and obtains a multimodal data fusion model; among which, the response speed is associated with the rationality of the display parameters.

[0024] As an optional embodiment, the method further includes:

[0025] Acquire driving data of a target vehicle driven by a driver and environmental data around the target vehicle;

[0026] Adjust the display screen according to the driving data and the environmental data to obtain an adjusted optimized display screen;

[0027] Control AR-HUD to project the optimized display image to the target area.

[0028] According to the human factor intelligence based AR-HUD display control method of the embodiment of the present invention, the driver's physiological characteristic data and identity information are first obtained; the driver's driving attention state is determined according to the physiological characteristic data; wherein the driving attention state characterizes the driver's attention level when driving the target vehicle; display parameters corresponding to the vehicle information are generated according to the identity information and the driving attention state, and a corresponding display screen is generated according to the vehicle information and the display parameters; the AR-HUD is controlled to project the display interface to the target area. The present invention uses a comprehensive application of biometrics technology and augmented reality head-up display (AR-HUD) to accurately evaluate the driver's attention state, and realizes personalized information display based on the evaluation results, which greatly improves the safety and efficiency of the driving process.

[0029] To achieve the above-mentioned object, a second embodiment of the present invention proposes an AR-HUD display control device based on human intelligence, the device comprising:

[0030] An acquisition module, configured to acquire physiological characteristic data and identity information of the driver;

[0031] An evaluation module is configured to determine a driving attention state of a driver based on the physiological characteristic data; wherein the driving attention state represents a degree of attention of the driver when driving the target vehicle;

[0032] A data processing module is configured to generate display parameters corresponding to vehicle information according to the identity information and the driving attention state, and to generate a corresponding display screen according to the vehicle information and the display parameters;

[0033] The projection module is configured to control the AR-HUD to project the display interface to the target area.

[0034] According to an embodiment of the present invention, an AR-HUD display control device based on human intelligence is provided. First, the driver's physiological characteristic data and identity information are obtained; the driver's driving attention state is determined according to the physiological characteristic data; wherein the driving attention state characterizes the driver's attention level when driving the target vehicle; display parameters corresponding to the vehicle information are generated according to the identity information and the driving attention state, and a corresponding display screen is generated according to the vehicle information and the display parameters; the AR-HUD is controlled to project the display interface to the target area. The present invention uses a combination of biometric technology and augmented reality head-up display (AR-HUD) to accurately evaluate the driver's attention state, and realizes personalized information display based on the evaluation results, which greatly improves the safety and efficiency of the driving process.

[0035] To achieve the above-mentioned objectives, the third aspect of the present invention proposes an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the AR-HUD display control method based on human factors intelligence as described above when executing the program.

[0036] In order to achieve the above-mentioned purpose, a computer-readable storage medium is proposed in an embodiment of the fourth aspect of the present invention, and the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the above-mentioned AR-HUD display control method based on human factors intelligence.

[0037] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 Schematic diagram of AR-HUD provided in an embodiment of the present invention.

[0040] Figure 2 A schematic diagram of a flow chart of an AR-HUD display control method based on human intelligence provided in an embodiment of the present invention.

[0041] Figure 3A schematic diagram of an optimized AR-HUD provided in an embodiment of the present invention.

[0042] Figure 4 Schematic diagram of an AR-HUD display control device based on human intelligence provided in an embodiment of the present invention.

[0043] Figure 5 A more specific schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0045] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0046] As mentioned in the background technology section, the rapid development of modern transportation has made driving safety a core issue that cannot be ignored. In driving, a complex and highly concentrated activity, drivers must quickly and accurately obtain various key information in order to make appropriate driving decisions. Head-up display (HUD) technology is an innovative solution born out of this demand. It projects important information directly into the driver's field of vision, greatly reducing the frequency and amplitude of the driver's line of sight, thereby improving the convenience and safety of driving.

[0047] In recent years, with the continuous advancement of technology, the integration of augmented reality (AR) and HUD technology has given rise to a new display technology, AR-HUD. AR-HUD not only has the function of displaying basic driving information like traditional HUD, but also seamlessly integrates virtual information with real road scenes through augmented reality technology, bringing drivers a richer, more intuitive and immersive visual experience. However, despite the broad prospects of AR-HUD technology, most AR-HUD devices on the market are still relatively rigid in terms of information presentation, and fail to fully consider the individual differences of drivers and the impact of real-time driving status on information reception.

[0048] In the process of implementing the present invention, the applicant found that some existing AR-HUD systems mainly focus on projecting driving-related information such as vehicle speed and navigation instructions into the driver's field of view in a fixed manner. These systems usually follow preset display rules and layouts, such as vehicle speed information is fixedly displayed in a certain position, and navigation arrows use specific colors and styles. Although this standardized information display method is convenient for system design and implementation, it ignores the subjectivity and diversity of the driver as the information receiver.

[0049] In terms of driver factors, although some AR-HUD systems will automatically adjust the display brightness according to the light difference between day and night, they lack detailed and targeted design for deeper individual characteristics of drivers (such as age, gender) and real-time attention status (such as concentration, fatigue, distraction, etc.). These systems fail to make full use of modern biometrics and physiological monitoring technologies, such as real-time monitoring of physiological data such as EEG, ECG, and skin electricity, to dynamically adjust the color, design style, and presentation position of the displayed content, and thus fail to provide drivers with the most optimized information display solution.

[0050] The technical solution of the present invention is further described in detail below through specific embodiments.

[0051] refer to Figure 1 , is a schematic diagram of an AR-HUD provided in an embodiment of the present invention.

[0052] like Figure 1 As shown, HUD projects important driving information such as speed and navigation onto the target area of ​​the windshield in front of the driver, allowing the driver to see important driving information such as speed and navigation without lowering or turning his head.

[0053] AR-HUD integrates AR technology into the HUD optical projection system, overlaying digital images on the real world seen by the driver, so that the information projected by the HUD is integrated with the real driving environment.

[0054] Vehicle information refers to information that can reflect the current status of the vehicle. HUD generally obtains it through the vehicle bus. Specifically, it can include: Core information refers to some important information displayed on traditional instruments, such as vehicle speed, speed, mileage, fuel level, etc. Prompt information refers to some prompt information that needs to be displayed during the driving process of the vehicle, such as gear position, turn signal, high beam, fog light, vehicle temperature, instantaneous fuel consumption, etc. Alarm information includes requesting the driver to take over, seat belt reminder, low fuel / battery, engine status, door status, parking status, remaining oil, tire pressure, airbag status, glass washer fluid inventory, etc. Additional information refers to special function information on some models, such as four-wheel drive mode, steering mode, driving mode, sunroof and sunroof opening status, seat status, chassis status, etc.

[0055] In addition, in addition to obtaining the vehicle's own status information, HUD can also obtain more external information through interaction with the outside world, including: travel information, which can include positioning, maps, navigation, pedestrian / obstacle detection, lane keeping assistance and other information. Safety information, mainly displays the real-time traffic conditions perceived by the intelligent driving system to assist the driver in safe driving, including: speeding warning, front vehicle collision warning, vehicle distance information prompt, traffic light prompt, road safety warning, remote fault diagnosis information, etc. Life information, mainly refers to traffic violation information, maintenance information, parking lot information, and driver reservations obtained through the Internet of Vehicles. Smart office information, such as WeChat, email, phone, voice / video conferencing, document processing, etc. Entertainment information, such as music and radio playback control, volume control, audio and video media playback and control, etc.

[0056] refer to Figure 2 , which is a flow chart of an AR-HUD display control method based on human intelligence provided in an embodiment of the present invention.

[0057] Step S201, obtaining the driver's physiological characteristic data and identity information.

[0058] As an optional embodiment, the physiological characteristic data includes at least one of the driver's electroencephalogram data, electrocardiogram data and electrodermal data.

[0059] Specifically, the driver's physiological data can be collected in real time through biometric sensors installed in the vehicle (such as eye trackers, facial recognition cameras, heart rate monitors, etc.), including but not limited to blinking frequency, gaze direction, facial changes, heart rate, etc.

[0060] Among them, eye tracking is based on optical principles to measure the position and movement of the eyeball. When the driver's eyes are illuminated by an infrared light source, the cornea of ​​the eyeball will reflect a bright point. The eye tracker tracks this reflection point through a high-speed camera, thereby recording the position and movement direction of the eye in real time. Facial recognition is a technology that quantifies a series of parameters for identification through the distance, angle, size and shape between facial features and facial organs. The application scenarios of facial recognition are very wide, including monitoring whether the driver is focused and driver identity recognition. By monitoring changes in heart rate, the driver's physiological stress level and health status can also be assessed. For example, when the driver's heart rate rises abnormally, the system can determine that the driver may be in a state of tension or anxiety and take corresponding safety measures.

[0061] As an optional embodiment, the identity information includes at least one of the driver's gender and age.

[0062] Specifically, the driver's identity can be verified and his identity information obtained through the driver's ID card, driver's license scanning, iris recognition or fingerprint recognition of the facial recognition system in the vehicle.

[0063] The driver's identity information may include basic information such as the driver's age, gender, and driving experience, which can reflect individual differences. Drivers of different ages and genders do have differences in information perception and processing. They also have different preferences and acceptance of information colors, styles, etc. For example, young drivers may be more inclined to accept information displays with bright and lively colors and strong dynamic effects. As they age, drivers' visual and cognitive abilities may gradually decline, and their sensitivity to information and processing speed will also decrease. Therefore, older drivers may need larger, clearer, and easier to understand information displays. In addition, male and female drivers may have differences in information perception and processing, such as sensitivity to color and how to interpret information. Female drivers may pay more attention to the details and overall presentation of information, while male drivers may pay more attention to the practicality and simplicity of information.

[0064] It should be noted that before using the technical solutions disclosed in the embodiments of the present invention, the type, scope of use, usage scenarios, etc. of the personal information involved in the present invention should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0065] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present invention according to the prompt message.

[0066] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0067] It is understandable that the above notification and user authorization process is only illustrative and does not constitute a limitation on the implementation of the present invention. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present invention.

[0068] Step S202, determining the driver's driving attention state based on the physiological characteristic data; wherein the driving attention state represents the driver's attention level when driving the target vehicle.

[0069] As an optional embodiment, determining the driver's driving attention state based on physiological characteristic data includes: inputting the physiological characteristic data into a pre-built attention state assessment model to obtain the driving attention state; wherein the driving attention state includes one of a concentration state, a fatigue state, and a distraction state.

[0070] In specific implementation, a driver attention state assessment model can be constructed based on a deep learning neural network model. For example, a long short-term memory network (LSTM) is used to input the feature vectors of preprocessed EEG, ECG, and skin conductance data. The LSTM network is trained on a large number of driver physiological data samples under different attention states (such as concentration, fatigue, distraction, etc.) to learn the complex mapping relationship between physiological data and attention state.

[0071] As an optional embodiment, the basic neural network model can be trained based on pre-collected physiological characteristic data of the driver in a simulated driving scenario, so that the basic neural network model learns the mapping relationship between identity information and physiological characteristic data and driving attention status, and obtains an attention status assessment model.

[0072] In the data collection stage, it is necessary to collect the EEG, ECG and GSR data of the driver in the simulated driving situation. In order to ensure the diversity and representativeness of the data, the collected data can cover driver groups of different ages, genders, driving experience, etc. Furthermore, it is necessary to pre-process the collected EEG, ECG and GSR data, for example, filter and denoise the EEG data, and extract features such as power spectrum density of different frequency bands (such as delta, theta, alpha, beta, gamma). Perform heart rate variability (HRV) analysis on the ECG data and extract parameters such as SDNN and RMSSD. Smooth the GSR data and extract features such as mean and variance of skin conductivity. Furthermore, based on the preliminary analysis, select features that have an important impact on the assessment of attention state, and use feature selection algorithms (such as recursive feature elimination, principal component analysis, etc.) to further optimize the feature set. Furthermore, standardize the feature data to make it have a unified dimension and distribution.

[0073] In the model building stage, we first need to design the LSTM network structure, including the input layer, LSTM layer (which may contain multiple stacked LSTM units), fully connected layer, and output layer. The input layer is used to receive the preprocessed feature vector, the LSTM layer captures the long-term dependencies of the time series data, the fully connected layer converts the output of the LSTM layer into the classification results of the attention state, and the output layer uses the softmax activation function to output the probability of each category. Furthermore, the cross entropy loss function can be selected as the loss function of the model, and the network parameters can be adjusted using optimization algorithms such as stochastic gradient descent (SGD) or Adam.

[0074] In the model training phase, the data set (i.e., the preprocessed feature vector) is divided into a training set, a validation set, and a test set. The training set is used to train the model, the validation set is used to adjust the model parameters (such as learning rate, number of LSTM layers, etc.), and the test set is used to evaluate the model performance. Furthermore, the LSTM model is trained using the training set data. During the training process, the accuracy, loss, and other indicators on the validation set are monitored to adjust the model parameters. Early stopping can be used to avoid overfitting. Finally, the performance of the model is evaluated on the test set, including indicators such as accuracy, precision, and recall, to analyze the performance of the model under different attention states and identify possible directions for improvement.

[0075] In the model optimization stage, grid search, random search and other methods can be used to adjust the model's hyperparameters (such as learning rate, batch size, number of LSTM layers, etc.). Evaluate the performance of different hyperparameter combinations on the validation set and select the optimal combination. Furthermore, try to fuse the features of different physiological signals to improve the performance of the model. Feature fusion can be achieved using methods such as feature concatenation and feature weighting. Furthermore, you can try to use model integration methods (such as voting, weighted averaging, etc.) to improve the stability and performance of the model. You can train multiple LSTM models and integrate their outputs. Finally, summarize the performance of the model on the test set, including indicators such as accuracy, precision, and recall. Analyze the performance differences of the model under different attention states, identify the advantages and disadvantages of the model, and select the model with the best expressiveness as the final attention state evaluation model.

[0076] Step S203, generating display parameters corresponding to the vehicle information according to the identity information and the driving attention state, and generating a corresponding display screen according to the vehicle information and the display parameters.

[0077] As an optional embodiment, display parameters corresponding to the vehicle information are generated according to the identity information and the target attention state, including: inputting the identity information and the target attention state into a pre-built multimodal data fusion model to obtain display parameters; wherein the display parameters include color parameters, style parameters and presentation position coordinates for the vehicle information, and the presentation position coordinates are coordinates relative to the driver's head-on field of view.

[0078] In this step, the driver's attention state is evaluated by acquiring his / her EEG, ECG, galvanic skin and other physiological data in real time, and the display color, design style, presentation position, etc. of the AR-HUD are dynamically changed based on factors such as the driver's age and gender.

[0079] As an optional embodiment, when various driving events occur in a simulated driving scenario and a display screen generated using different display parameters prompts the driver based on the driver's pre-collected different identity information, the basic neural network model is trained based on the driver's response speed, physiological characteristic data, and driving attention status, so that the basic neural network model learns the mapping relationship between the identity information, physiological characteristic data, driving attention status and the display parameters, and obtains a multimodal data fusion model; wherein the response speed is associated with the rationality of the display parameters.

[0080] In specific implementations, during driving, the setting of display parameters (such as display brightness, contrast, information density, etc.) has an important impact on the driver's attention state. Unreasonable display parameters may cause the driver to be distracted and slow down their reaction speed, thereby increasing the risk of traffic accidents. Therefore, it is necessary to dynamically adjust the display parameters based on the driver's identity information (such as age, gender, driving experience, etc.), physiological characteristic data (such as heart rate, eye movement trajectory, etc.) and attention state during driving to improve driving safety and comfort.

[0081] Specifically, driver data of different identity information can be collected in advance, including age, gender, driving experience, etc. In a simulated driving scenario, the driver's response speed, physiological characteristic data (such as heart rate changes, skin conductivity, etc.) and driving attention state are recorded when various driving events (such as emergency braking, lane departure, pedestrian crossing, etc.) occur. At the same time, the relevant data when the display screen generated by different display parameters is used to prompt the driver is recorded. The basic neural network model is trained using the collected data, and the training goal is to let the model learn the mapping relationship between identity information, physiological characteristic data, driving attention state and display parameters. Among them, the response speed is used as an important indicator to evaluate the rationality of display parameters, that is, reasonable display parameters should prompt the driver to respond faster. After training, a multimodal data fusion model is obtained, which can dynamically adjust the display parameters according to the driver's current identity information, physiological characteristic data and driving attention state to optimize the display effect and improve the driver's attention and reaction speed.

[0082] Multimodality refers to the use of information in multiple different forms or perceptual channels for expression, communication and understanding, usually including multiple sensory input and output methods such as vision, hearing, text, touch, etc. In the fields of computer science, artificial intelligence and machine learning, multimodal technology refers to the integration of data from different modalities (such as images, text, audio, video, etc.) to enhance the model's understanding and reasoning capabilities. This integration can improve the completeness and accuracy of information because each modality can provide unique information for specific tasks.

[0083] The multimodal data fusion model constructed above is used to determine the AR-HUD presentation mode under different ages, genders and attention states. The attention state category output by the age, gender and driver attention state assessment model is used as input, and the color parameters (such as RGB values) displayed by the AR-HUD, the design style category (such as simple type, detailed type, etc.), and the presentation position coordinates (relative to the driver's field of view) are used as outputs. The training is carried out using a multivariate linear regression model, a decision tree model or a neural network model.

[0084] As an optional embodiment, when using a neural network model, multiple hidden layers are set, and the number of neurons is reasonably determined according to the complexity of the input and output (such as 64, 128, 64, etc.). The training data can come from experimental data of participants of different genders and age groups in simulated driving scenarios, including vehicle data (such as speed changes, braking force, etc.) and eye movement data (such as AOI first gaze duration, gaze point distribution, etc.) under different color and style prompts when various driving events occur (such as pedestrians suddenly crossing the road, rear vehicles overtaking, etc.). Based on these two types of objective data, vehicle data and eye movement data, it is determined whether the prompt information can be quickly responded to by the driver. The faster the response, the better the display mode of the AR-HUD. Through the analysis and learning of these data, the model can establish a relationship between different input conditions and the optimal AR-HUD display mode.

[0085] Step S204, controlling the AR-HUD to project the display interface to the target area.

[0086] As an optional embodiment, the driving data of the target vehicle driven by the driver and the environmental data around the target vehicle can also be obtained; the display screen is adjusted according to the driving data and the environmental data to obtain an adjusted optimized display screen; and the AR-HUD is controlled to project the optimized display screen to the target area.

[0087] In a complex driving environment, the driver needs to process information from multiple sources at the same time, including the driving status of the vehicle (such as speed, acceleration, steering angle, etc.) and the surrounding environment (such as weather, road conditions, other vehicles and pedestrian dynamics, etc.). Therefore, the application scenarios and functions of AR-HUD (augmented reality head-up display) can be further expanded. By comprehensively considering the driving data of the target vehicle driven by the driver and the surrounding environment data, the display screen can be dynamically adjusted to improve driving safety and information communication efficiency. For example, on rainy or foggy days, the brightness and contrast of the display screen can be increased to improve the visibility of information; in traffic congestion or complex road conditions, key navigation information and potential danger areas can be highlighted.

[0088] By comprehensively considering driving data and environmental data, AR-HUD can provide more accurate and timely driving assistance information to help drivers better deal with complex road conditions and potential dangers. The optimized display screen not only improves the readability of information, but also improves driving comfort and pleasure by reducing the visual burden and cognitive load of the driver.

[0089] refer to Figure 3 , is a schematic diagram of an optimized AR-HUD provided in an embodiment of the present invention.

[0090] As drivers age, their vision, hearing and reaction speed may gradually decline, especially in distracted driving situations. Therefore, it is particularly important to provide more eye-catching and intuitive driving assistance information to older drivers.

[0091] For example, during the driver's driving process, the driver's age and other identity information can be obtained through the vehicle system or user registration information. Real-time monitoring of the driver's physiological characteristics data, such as heart rate, eye movement trajectory, facial expressions, etc., to evaluate the driver's driving attention state. According to the physiological characteristics data obtained in real time, the attention state evaluation model is used to identify whether the driver is in a distracted driving state. For example, abnormally high heart rate and frequent deviation of eye movement trajectory from the road ahead may indicate that the driver is distracted. For older drivers, when they are identified as being in a distracted driving state, the display parameters of the AR-HUD are dynamically adjusted according to the vehicle information and driving environment. For example, when encountering a turning intersection, the display font size of the turning prompt is enlarged, bolded, and even eye-catching colors or animation effects are used to more intuitively remind the driver of the road turning situation. At the same time, it is possible to consider adding auxiliary means such as voice prompts or vibration feedback to further enhance the transmission effect of driving assistance information.

[0092] Finally, based on the adjusted display parameters, the AR-HUD is controlled to project the optimized display image to the target area in front of the driver's field of vision. Ensure that the projection position, size and angle match the driver's line of sight and driving needs to improve the readability and ease of use of the information.

[0093] It should be noted that in actual applications, it is necessary to ensure the real-time and accuracy of data acquisition and processing to avoid misjudgment due to data delays or errors. The adjustment of display parameters should fully consider the driver's visual habits and attention distribution to avoid unnecessary interference or burden on the driver. System verification and optimization is an ongoing process that requires continuous collection of user feedback and data to continuously improve and optimize system performance.

[0094] From the above, it can be seen that according to the human intelligence-based AR-HUD display control method of an embodiment of the present invention, the driver's physiological characteristic data and identity information are first obtained; the driver's driving attention state is determined according to the physiological characteristic data; wherein the driving attention state characterizes the driver's attention level when driving the target vehicle; display parameters corresponding to the vehicle information are generated according to the identity information and the driving attention state, and a corresponding display screen is generated according to the vehicle information and the display parameters; the AR-HUD is controlled to project the display interface to the target area. The present invention comprehensively applies biometrics technology and augmented reality head-up display (AR-HUD) to accurately evaluate the driver's attention state, and realizes personalized information display based on the evaluation results, which greatly improves the safety and efficiency of the driving process.

[0095] It should be noted that the method of the embodiment of the present invention can be performed by a single device, such as a computer or a server. The method of this embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present invention, and the multiple devices will interact with each other to complete the described method.

[0096] It should be noted that some embodiments of the present invention are described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0097] Based on the same inventive concept, corresponding to the method provided in any of the above embodiments, the present invention also provides an AR-HUD display control device based on human factors intelligence.

[0098] refer to Figure 4 , which is a schematic diagram of an AR-HUD display control device based on human intelligence provided in an embodiment of the present invention.

[0099] The device comprises:

[0100] The acquisition module 401 is configured to acquire the physiological characteristic data and identity information of the driver;

[0101] The evaluation module 402 is configured to determine the driver's driving attention state according to the physiological characteristic data; wherein the driving attention state represents the driver's attention level when driving the target vehicle;

[0102] The data processing module 403 is configured to generate display parameters corresponding to the vehicle information according to the identity information and the driving attention state, and generate a corresponding display screen according to the vehicle information and the display parameters;

[0103] The projection module 404 is configured to control the AR-HUD to project the display interface to the target area.

[0104] Optionally, the physiological characteristic data includes at least one of the driver's electroencephalogram data, electrocardiogram data, and electrodermal data.

[0105] Optionally, the identity information includes at least one of the driver's gender and age.

[0106] Optionally, the evaluation module 402 is further configured to:

[0107] The physiological characteristic data is input into a pre-built attention state assessment model to obtain the driving attention state; wherein the driving attention state includes one of a concentration state, a fatigue state and a distraction state.

[0108] Optionally, the evaluation module 402 is further configured to:

[0109] The basic neural network model is trained based on the pre-collected physiological characteristic data of the driver in a simulated driving situation, so that the basic neural network model learns the mapping relationship between identity information and physiological characteristic data and driving attention status, and obtains an attention status assessment model.

[0110] Optionally, the data processing module 403 is further configured to:

[0111] The identity information and the target attention state are input into a pre-built multimodal data fusion model to obtain display parameters; wherein the display parameters include color parameters, style parameters and presentation position coordinates for vehicle information, and the presentation position coordinates are coordinates relative to the driver's head-on field of view.

[0112] Optionally, the data processing module 403 is further configured to:

[0113] Based on the pre-collected different identity information of the driver in a simulated driving scenario, when various driving events occur and the display screen generated by using different display parameters prompts the driver, the driver's response speed, physiological characteristic data and driving attention status are used to train the basic neural network model, so that the basic neural network model learns the mapping relationship between the identity information, physiological characteristic data and driving attention status and the display parameters, and obtains a multimodal data fusion model; among which, the response speed is associated with the rationality of the display parameters.

[0114] Optionally, the projection module 404 is further configured to:

[0115] Acquire driving data of a target vehicle driven by a driver and environmental data around the target vehicle;

[0116] Adjust the display screen according to the driving data and the environmental data to obtain an adjusted optimized display screen;

[0117] Control AR-HUD to project the optimized display image to the target area.

[0118] According to an embodiment of the present invention, an AR-HUD display control device based on human intelligence is provided. First, the driver's physiological characteristic data and identity information are obtained; the driver's driving attention state is determined according to the physiological characteristic data; wherein the driving attention state characterizes the driver's attention level when driving the target vehicle; display parameters corresponding to the vehicle information are generated according to the identity information and the driving attention state, and a corresponding display screen is generated according to the vehicle information and the display parameters; the AR-HUD is controlled to project the display interface to the target area. The present invention uses a combination of biometric technology and augmented reality head-up display (AR-HUD) to accurately evaluate the driver's attention state, and realizes personalized information display based on the evaluation results, which greatly improves the safety and efficiency of the driving process.

[0119] For the convenience of description, the above system is described as being divided into various modules according to their functions. Of course, when implementing the present invention, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0120] The system of the above embodiment is used to implement the corresponding method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0121] Based on the same inventive concept, corresponding to the method described in any of the above embodiments, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in any of the above embodiments when executing the program.

[0122] Figure 5 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 510, a memory 520, an input / output interface 530, a communication interface 540, and a bus 550. The processor 510, the memory 520, the input / output interface 530, and the communication interface 540 are connected to each other in communication within the device through the bus 550.

[0123] The processor 510 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0124] The memory 520 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 520 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 520 and are called and executed by the processor 510.

[0125] The input / output interface 530 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0126] The communication interface 540 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0127] The bus 550 comprises a pathway that transmits information between the various components of the device (eg, the processor 510 , the memory 520 , the input / output interface 530 , and the communication interface 540 ).

[0128] It should be noted that, although the above device only shows the processor 510, the memory 520, the input / output interface 530, the communication interface 540 and the bus 550, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0129] The electronic device of the above embodiment is used to implement the corresponding method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0130] Based on the same inventive concept, corresponding to the method described in any of the above embodiments, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method described in any of the above embodiments.

[0131] The above-mentioned computer-readable storage medium can be any available medium or data storage device that can be accessed by a computer, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NANDFLASH), solid-state drive (SSD)), etc.

[0132] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the method described in any embodiment of the above exemplary method part, and have the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0133] In addition, although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted in the flow chart can be performed in a different order. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0134] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0135] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the embodiments of the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0136] Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and this division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims. The scope of the attached claims conforms to the broadest interpretation, thereby including all such modifications and equivalent structures and functions.

Claims

1. An AR-HUD display control method based on human intelligence, characterized in that: include: Obtain the driver's physiological characteristics data and identity information; Determine the driving attention state of the driver according to the physiological characteristic data; wherein the driving attention state represents the degree of attention of the driver when driving the target vehicle; generating display parameters corresponding to vehicle information according to the identity information and the driving attention state, and generating a corresponding display screen according to the vehicle information and the display parameters; The AR-HUD is controlled to project the display interface to a target area.

2. The AR-HUD display control method based on human intelligence according to claim 1, characterized in that: The physiological characteristic data includes at least one of the driver's electroencephalogram data, electrocardiogram data, and electrodermal conductivity data.

3. The AR-HUD display control method based on human intelligence according to claim 2 is characterized in that: The identity information includes at least one of the gender and age of the driver.

4. The AR-HUD display control method based on human intelligence according to claim 3 is characterized in that: The determining the driving attention state of the driver according to the physiological characteristic data comprises: The physiological characteristic data is input into a pre-constructed attention state assessment model to obtain the driving attention state; wherein the driving attention state includes one of a concentration state, a fatigue state and a distraction state.

5. The AR-HUD display control method based on human intelligence according to claim 4 is characterized in that: The method further comprises: The basic neural network model is trained based on the pre-collected physiological characteristic data of the driver in a simulated driving situation, so that the basic neural network model learns the mapping relationship between the identity information and the physiological characteristic data and the driving attention state, and obtains the attention state evaluation model.

6. The AR-HUD display control method based on human intelligence according to claim 5 is characterized in that: The generating of display parameters corresponding to the vehicle information according to the identity information and the target attention state includes: The identity information and the target attention state are input into a pre-built multimodal data fusion model to obtain the display parameters; wherein the display parameters include color parameters, style parameters and presentation position coordinates for the vehicle information, and the presentation position coordinates are coordinates relative to the driver's head-up field of view.

7. The AR-HUD display control method based on human intelligence according to claim 6 is characterized in that: The method further comprises: When various driving events occur in a simulated driving scenario based on a driver with different identity information collected in advance and a display screen generated using different display parameters prompts the driver, the driver's response speed, physiological characteristic data and driving attention status are used to train the basic neural network model, so that the basic neural network model learns the mapping relationship between the identity information, physiological characteristic data and driving attention status and the display parameters, and obtains the multimodal data fusion model; wherein the response speed is associated with the rationality of the display parameters.

8. The AR-HUD display control method based on human intelligence according to claim 7 is characterized in that: The method further comprises: Acquiring driving data of a target vehicle driven by the driver and environmental data around the target vehicle; Adjusting the display picture according to the driving data and the environmental data to obtain an adjusted optimized display picture; The AR-HUD is controlled to project the optimized display image to the target area.

9. An AR-HUD display control device based on human intelligence, characterized in that: include: An acquisition module, configured to acquire physiological characteristic data and identity information of the driver; An evaluation module configured to determine the driver's driving attention state according to the physiological characteristic data; wherein the driving attention state represents the driver's attention level when driving the target vehicle; a data processing module configured to generate display parameters corresponding to vehicle information according to the identity information and the driving attention state, and to generate a corresponding display screen according to the vehicle information and the display parameters; The projection module is configured to control the AR-HUD to project the display interface to a target area.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the display control method of the AR-HUD based on human intelligence as described in any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable the computer to implement the display control method of the AR-HUD based on human intelligence as described in any one of claims 1 to 8.

Images