Augmented reality head-up display method and system considering driver's emotions

Abstract

The present invention discloses an augmented reality head-up display method and system that takes into account the driver's emotions, including a TOF module that obtains a target image in RGB format of the current scene; an image recognition module that recognizes the eyeballs and faces in the target image and generates a recognition result; the recognition result is input into an image processing module for processing, including obtaining the depth information of the image, outputting the eyeball position information, and obtaining the facial expression image to analyze the driver's current emotions; the eyeball position information is transmitted to the AR‑HUD system for enhanced display real-time tracking; and the corresponding content result display adjustment is performed according to the driver's emotional perception data transmitted to the AR‑HUD system. The beneficial effects of the present invention: by using the TOF solution, the eyeball tracking module integrated with the ARHUD adds the algorithm of emotion recognition, shares a set of systems and can be used in conjunction, reduces costs, improves interactive performance, and after the emotion recognition is completed, acts better on the ARHUD, and makes corresponding early warning improvements, thereby reducing the occurrence of accidents.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent cockpit display, and in particular to a method for taking driver emotions into consideration and acting on augmented reality head-up display, and a system for taking driver emotions into consideration and acting on augmented reality head-up display. Background Art

[0002] In recent years, with the development of society, the progress of the times and the vigorous development of automobile intelligence, the control system of automobiles has become increasingly complex, and the contradiction between drivers and automobile control systems has become increasingly prominent. At the same time, "road rage" and the driver's driving tendency have a vital impact on driving safety. According to relevant information, most traffic accidents can be attributed to abnormal driving emotions and excessive driving tendencies. How to timely identify such abnormal driving emotions and excessive driving tendencies is an important way to avoid traffic accidents and ensure the safe driving of vehicles.

[0003] Existing car safety warning methods rarely take driving emotions into consideration, but driving emotions play an important role in driving safety and cannot be ignored. Emotion recognition is an important means of analyzing driving emotions. Currently, the most effective emotion recognition is an image recognition algorithm based on a convolutional neural network. It takes a photo of the driver's facial expression, recognizes the photo, and then determines what kind of emotion the driver is in. Therefore, when issuing safety warnings for cars, the driver's driving emotions must be considered. Summary of the invention

[0004] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0005] In view of the above existing problems, the present invention is proposed.

[0006] Therefore, a technical problem solved by the present invention is to propose an augmented reality head-up display method that takes the driver's emotions into consideration, and improve safety performance in combination with driving emotions.

[0007] To solve the above technical problems, the present invention provides the following technical solutions: an augmented reality head-up display method taking into account the driver's emotions, comprising a TOF module acquiring a target image in RGB format of the current scene; an image recognition module identifying eyeballs and faces in the target image and generating a recognition result; the recognition result is input into an image processing module for processing, including acquiring image depth information, outputting eyeball position information, and acquiring facial expression images to analyze the driver's current emotions; the eyeball position information is transmitted to an AR-HUD system for enhanced display real-time tracking; and corresponding content result display adjustments are made according to the driver's emotion perception data transmitted to the AR-HUD system.

[0008] As a preferred solution of the augmented reality head-up display method taking the driver's emotions into consideration as described in the present invention, the content result display adjustment includes changing the AR-HUD content display according to the driver's emotional perception results, including bolding the warning lines and deepening the color in the case of anger.

[0009] As a preferred solution of the augmented reality head-up display method taking into account the driver's emotions described in the present invention, the TOF module includes a 3D camera module and a TOF sensor, including using the 3D camera module to obtain RGB image information of the target image; obtaining corresponding depth information through the TOF sensor and in combination with a near-infrared light array; and transmitting the obtained RGB image information and depth information to the image recognition module.

[0010] As a preferred solution of the augmented reality head-up display method taking the driver's emotions into consideration as described in the present invention, the image processing module includes: image recognition corresponding to the position of the eyeball in the image; obtaining the corresponding depth information according to the position recognized by the image and the depth information obtained; and analyzing and processing the image of the facial expression to obtain the emotional state of the driver.

[0011] As a preferred solution of the augmented reality head-up display method taking into account the driver's emotions described in the present invention, the acquisition of eyeball position information includes determining the pupil area and position in a local area image containing a human eye through Haar features; establishing a differential Gaussian pyramid model of the human eye image to calculate the regional extreme points; establishing an adaptive pupil fitting template in the pupil area image, and fitting the pupil edge and center with the regional extreme points as the template center.

[0012] As a preferred embodiment of the augmented reality head-up display method considering the driver's emotion according to the present invention, wherein: the image recognition module includes selecting different Haar rectangular features according to face samples, and training a weak classifier for each Haar feature template; constructing M weak classifiers according to face features; selecting the optimal N weak classifiers; setting an error rate and a threshold to generate M cascaded strong classifiers; inputting the target image, using the M generated strong classifiers to obtain a recognition result, and merging adjacent sub-regions to obtain a face region.

[0013] As a preferred embodiment of the augmented reality head-up display method considering the driver's emotion according to the present invention, wherein: the target image is subjected to histogram equalization processing to increase the local contrast of the target image, including the steps of histogram equalization, counting the number of occurrences of each gray level in the histogram and saving it as an array; calculating the cumulative distribution function value of each gray level in the histogram and saving it as another array; calculating the gray level value after histogram equalization of the original image according to the following formula;

[0014]

[0015] As a preferred embodiment of the augmented reality head-up display method considering the driver's emotion according to the present invention, wherein: the image processing module 300 uses a convolutional neural network to extract abstract features of facial expressions, which includes an input layer, three alternating convolutional layers and pooling layers, a fully connected layer and an output layer. The input of the network is a 100*100 facial expression image, the size of the convolutional kernel is 3*3, the convolutional step is 1, the activation function uses the rectified linear unit function, and the classification function uses the Softmax function of the normalized exponential function.

[0016] As a preferred embodiment of the augmented reality head-up display method considering the driver's emotion according to the present invention, wherein: the convolutional neural network includes the following training steps: initialize i = 0, randomly initialize each weight and parameter value in the convolutional neural network; calculate the feature maps of each layer according to the designed convolutional neural network structure; calculate the loss term of the softmax layer; calculate the network parameter matrix W; select relevant parameters J; calculate the overall loss term L; update the loss parameters, and adjust each weight and bias value parameter by backpropagation; until L < s or the maximum number of iterations is reached, jump out of the loop; use the Softmax activation function to determine the category to which the entire facial expression image belongs.

[0017] Another technical problem solved by the present invention is to propose an augmented reality head-up display system considering the driver's emotion, which improves the safety performance in combination with the driving emotion.

[0018] To solve the above technical problems, the present invention provides the following technical solutions: an augmented reality head-up display system taking driver emotions into consideration, comprising a TOF module, an image recognition module, an image processing module and an AR-HUD module; the TOF module is used to obtain a target image in RGB format of the current scene; the image recognition module is connected to the TOF module, and is used to identify eyeballs and faces in the target image and generate recognition results; the depth information of the acquired image, the output eyeball position information and the acquired facial expression image are used to analyze the driver's current emotions; the image processing module is connected to the image recognition module, and is used to analyze and obtain the eyeball position information and transmit it to the AR-HUD module for enhanced display real-time tracking; the AR-HUD module is used to make corresponding content result display adjustments according to the driver's emotional perception data.

[0019] The beneficial effects of the present invention are as follows: by using the TOF solution, the eye tracking module of the integrated ARHUD is added with the algorithm of emotion recognition, a common system is shared and can be used in combination, thus reducing costs and improving interactive performance. After the emotion recognition is completed, it can better act on the ARHUD and make corresponding early warning improvements, thereby reducing the occurrence of accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0021] Figure 1 This is a schematic diagram of the flow structure of the augmented reality head-up display method taking into account the driver's emotions according to the first embodiment of the present invention;

[0022] Figure 2 This is a schematic diagram of the structure principle of the TOF system described in the first embodiment of the present invention;

[0023] Figure 3 This is a schematic diagram of the pupil center positioning process described in the first embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of positioning the candidate pupil centers according to the first embodiment of the present invention;

[0025] Figure 5 This is a schematic diagram of valid candidate pupil center positioning according to the first embodiment of the present invention;

[0026] Figure 6 This is a schematic diagram of using the AdaBoost weak classification cascade algorithm to perform face detection according to the first embodiment of the present invention;

[0027] Figure 7 This is a schematic diagram of the structure of a convolutional neural network according to the first embodiment of the present invention;

[0028] Figure 8 This is a schematic diagram of the overall principle structure of the augmented reality head-up display system taking into account the driver's emotions according to the first embodiment of the present invention;

[0029] Fig. 9 It is a schematic diagram of the effect of the prior art described in the present invention;

[0030] Fig.10 It is a schematic diagram of the effect of adopting the present invention. DETAILED DESCRIPTION

[0031] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0032] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0033] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0034] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0035] At the same time, in the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "upper, lower, inner and outer" are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0036] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0037] Example 1

[0038] HUD (Head up Display) is a device used on aircraft, designed to help pilots perceive the surrounding environment more easily. With the promotion of the technology field, it is now also used in automobiles. The development of HUD in the automotive industry has gone through C-HUD, W-HUD, and is currently developing towards AR-HUD. C-HUD (Combiner HUD) was initially used in the aftermarket and was relatively cheap. Products from companies such as Navdy and Cheluobo belong to this category. Now it has also entered the pre-installed market. C-HUD is mostly installed in non-premium cars, such as Jiangling Yizhi E300, W-HUD (Windshield HUD), and high-end cars are mostly equipped with pre-installed W-HUD, such as the BMW 3 Series. AR-HUD (Augmented Reality HUD is called augmented reality HUD, which mainly uses AR technology to project display content to the front of the car, allowing the driver to observe the actual road conditions more intuitively.

[0039] In order to achieve a better AR-HUD effect, an eye tracking module is required. Using an eye tracking module alone will result in a high cost waste. The driver emotion recognition module also needs to be equipped with a dedicated camera module.

[0040] Reference Figure 1 The schematic diagram is a schematic diagram of the overall process of an augmented reality head-up display method considering the driver's emotions proposed in this embodiment, which specifically includes the following steps:

[0041] S1: The TOF module 100 acquires a target image of the current scene in RGB format; the TOF module 100 includes a 3D camera module 101 and a TOF sensor 102, including:

[0042] Acquire RGB image information of the target image using the 3D camera module 101;

[0043] Obtain corresponding depth information through TOF sensor 102 in combination with near infrared light array;

[0044] The acquired RGB image information and depth information are transmitted to the image recognition module 200 .

[0045] S2: The image recognition module 200 recognizes the eyeballs and the face in the target image and generates a recognition result;

[0046] S3: The recognition result is input to the image processing module 300 for processing, including obtaining the depth information of the image, outputting the eyeball position information and obtaining the facial expression image to analyze the current emotion of the driver; the image processing module 300 includes,

[0047] The image recognizes the position of the corresponding eye in the image;

[0048] The corresponding depth information is obtained based on the position recognized by the image and the depth information obtained; it should be noted that the first depth information mentioned above is the depth information in the picture, and the second depth information refers to the position information of the eyeball corresponding to the camera, so it can also be called position information.

[0049] Based on the facial expression images, the driver's emotional state is obtained through analysis and processing.

[0050] S4: The eye position information is transmitted to the AR-HUD system for enhanced display and real-time tracking;

[0051] S5: According to the driver's emotional perception data transmitted to the AR-HUD system, the corresponding content result display adjustment is performed. The content result display adjustment includes:

[0052] According to the driver's emotional perception results, the AR-HUD content display changes, including bold warning lines and darker colors in angry situations. For example, when the mood is good, the overall color can be lighter, and the font can be diversified without affecting safety. In addition, the overall interface layout can be adjusted according to different moods, or when the driver is calm, the content display can be reduced.

[0053] Reference Figure 2As shown in the figure, TOF is the abbreviation of Time off light, which literally means time of flight. The so-called time-of-flight 3D imaging is to continuously send light pulses to the target, and then use the sensor to receive the light returned from the object, and detect the flight (round-trip) time of the light pulse to obtain the distance of the target object. This technology is basically similar to the principle of 3D laser sensors, except that 3D laser sensors scan point by point, while TOF cameras obtain the depth information of the entire image at the same time. TOF cameras are also similar to ordinary machine vision imaging processes, and are composed of several units such as light sources, optical components, sensors, control circuits, and processing circuits. Compared with binocular measurement systems that belong to the same non-invasive three-dimensional detection and have very similar application fields, TOF cameras have fundamentally different 3D imaging mechanisms. TOF cameras can obtain color RGB image depth map information.

[0054] Reference Figures 3 to 5 As shown in FIG. 1 , the acquisition of eyeball position information includes the following steps:

[0055] The first step is pupil area positioning. The pupil area and position are determined in the local area image containing the human eye through Haar features, and the differential Gaussian pyramid model of the human eye image is established to calculate the regional extreme points. The second step is pupil center fitting. An adaptive pupil fitting template is established in the pupil area image, and the pupil edge and center are fitted with the regional extreme point as the template center.

[0056] Haar feature is a rectangular feature template used to extract simple edge feature information in an image. The feature template is composed of two or more black and white rectangular areas of different sizes. The feature value of the template is the difference between the sum of the grayscale values ​​of the pixels in the white rectangular area and the sum of the grayscale values ​​of the pixels in the black rectangular area. The feature value of the template is defined as follows:

[0057]

[0058] is the sum of the grayscale values ​​of the pixels in the white rectangular area. It is the sum of the grayscale values ​​of the pixels in the black rectangular area.

[0059] In Haar feature templates, due to the large number of Haar feature templates, calculating the sum of the grayscale values ​​of pixels in the black and white areas will generate a huge amount of calculation. In order to speed up the calculation, the concept of integral graph can be used to quickly calculate the sum of the grayscale values ​​of pixels in any area of ​​the feature template. The integral graph can quickly calculate any pixel point (x, y) in the image. The sum of the grayscale values ​​of pixels in the area to the left of the x coordinate and above the y coordinate of the pixel point is defined as:

[0060]

[0061] Where i(x`,y`) is the pixel value of the image at point p(x`,y`), then the value ii(x`,y`) of a point (x,y) in the integral image is defined as the sum of the pixel values ​​of the rectangular area with the origin (0, 0) and point P as the diagonal in the original image.

[0062] Therefore, the sum of the grayscale values ​​of each column of pixels in the image is:

[0063] When the integral image is used to calculate the Haar template, the time required to calculate the Haar template feature value for any area of ​​the image is the same and the calculation is simple.

[0064] Furthermore, the rough positioning of the pupil center specifically includes the following steps:

[0065] The pupil area is located using the HAAR template. At the same time, the eye image area is downsampled to a quarter of the original image size and convolved with four Gaussian kernels of different scales G n (n=1,2,3,4) Then the adjacent Gaussian convolution images G n and G n+1 Subtract to establish the difference Gaussian pyramid model D n (n=1,2,3), G n Defined as:

[0066]

[0067] If the grayscale value of the pixel is the minimum grayscale value of the pixels in the neighborhood, then the pixel point is taken as a candidate feature point of the pupil center. In the human eye area image, through the above steps, we have obtained the pupil area and the candidate pupil center. We superimpose the pupil area with the human eye area image with the candidate pupil center, and take the pixel grayscale extreme point in the pupil area as a valid candidate pupil center.

[0068] In order to locate the pupil center more accurately, the pupil area and sclera area of ​​the human eye are analyzed as concentric circles. Therefore, a human eye matching template is further established to fit the pupil center according to the geometric structure of the pupil area and the sclera area. The black area in the pupil center fitting template represents the pupil area, and the white area represents the sclera area.

[0069] Define the human eye response function:

[0070]

[0071] The left side of the equation is the response function, and the last term on the right side is the change of the pupil radius within a certain range. The center of the valid candidate pupil is used as the matching template center, and the pupil center fitting template is convolved with the extracted pupil area. The convolution process is similar to the Haar template. Finally, the pixel point and fitting radius that minimize the response function R value are selected, and the coordinates of the pixel point are considered to be the pupil center. The specific iterative process is as follows:

[0072]

[0073] In the binary pupil area image, a clearer pupil outline is obtained. Next, the Canny edge detection operator is used to detect the pupil edge, and five different feature points are randomly selected on the obtained pupil edge. Finally, the ellipse fitting equation is established:

[0074] Q(x i ,y i )=Ax 2 +Bxy+Cy 2 +Dx+Ey+F

[0075] Substitute the selected feature point coordinates into the ellipse equation respectively, and consider the center of the ellipse to be the pupil center. After confirming the data of the pupil center, transfer it to the depth map to obtain the corresponding depth coordinate Z value (the depth coordinate value is the Z value of the eyeball relative to the camera coordinate system).

[0076] Reference Figures 5-6 In this embodiment, the AdaBoost weak classification cascade algorithm is used to perform face detection, which specifically includes the following steps:

[0077] (1) Select different Haar rectangle features according to the face samples, and train a weak classifier for each Haar feature template;

[0078] (2) construct M weak classifiers based on facial features;

[0079] (3) Repeat step (2) and select the best N weak classifiers using the following formula;

[0080]

[0081] Repeat steps (2) and (3), and set the error rate and threshold to generate M cascaded strong classifiers.

[0082] Input a face image, use the M strong classifiers that have been generated, obtain the detection results, and merge adjacent sub-regions to obtain the face region.

[0083] In this embodiment, the image processing module 300 is an important part of facial expression recognition. Good image preprocessing will directly improve the representation ability of the extracted features, thereby improving the accuracy of expression classification. Due to the differences in lighting and equipment during acquisition, as well as the different degrees of damage and various noise pollution during transmission and transformation, the input image often suffers from image degradation, edge blur, noise increase, etc.

[0084] AdaBoost algorithm is used for face detection and location. AdaBoost algorithm is a classic iterative algorithm proposed by Schapire and Freund in 1995. Its basic principle is to learn many different weak classifiers by repeatedly training the same training set. For samples that were misclassified by the previous round of weak classifiers, its weight is increased so that they can be correctly classified in the next round of training. In addition, the weight of weak classifiers with small classification error rates is increased, and the weight of weak classifiers with large classification error rates is reduced, so that weak classifiers with different weights play different roles in the final weighted majority vote, and finally the weak classifiers are combined to form a strong classifier.

[0085] Image grayscale refers to converting a color image into a grayscale image. Most color images are in RGB format, and the RGB component values ​​of each pixel are generally different. A grayscale image is a special color image in which the three RGB component values ​​are the same. In order to facilitate the subsequent image calculations and to ensure that the overall and local chromaticity distribution and brightness characteristics of the image are not destroyed during the image grayscale process, the color image is first converted into a grayscale image.

[0086] The weighted average method is to perform weighted average of the three RGB components according to different weights. Since the human eye has different sensitivities to different colors, the three RGB components are often weighted averaged according to the following formula.

[0087] Gray(i,j)=0.299*R(i,j)+0.587*G(i,j)+0.114*B(i,j)

[0088] The grayscale values ​​of the pixel values ​​of the three components of RGB in the color image are R(i,j), G(i,j), B(i,j), and Gray(i,j) to represent the grayscale values ​​of the image after conversion. In the process of facial image acquisition, due to the influence of equipment and lighting background, the lighting in the facial image is often uneven, resulting in low grayscale values ​​in some areas of the face in the image, which interferes with face detection and positioning. It is necessary to perform histogram equalization on the image to increase the local contrast of the image. The steps of histogram equalization are as follows:

[0089] (1) Count the number of occurrences of each grayscale in the histogram and save it as an array;

[0090] (2) Calculate the cumulative distribution function value of each grayscale in the histogram and save it as another array;

[0091] (3) Calculate the grayscale value of the original image after histogram equalization according to the following formula.

[0092]

[0093] Median filtering is a commonly used spatial domain filtering method that can effectively filter out noise and pulse interference. Median filtering is to sort all pixels in the neighborhood centered on the point (x, y) according to the grayscale value, and then replace the output of this point with the statistical median value in this neighborhood. When the number of pixels in the neighborhood of point (x, y) is an even number, the average of the two middle grayscale values ​​in the area is taken as the grayscale value of (x, y). When the number of pixels in the neighborhood of point (x, y) is an odd number, the middle value of the area is taken as the new grayscale value at (x, y). The formula expression can be written as:

[0094] g(m,n)=med{f(mi,nj),(i,j)∈W}

[0095] Where f(m, n), g(m, n) are the original image and the processed image respectively, W is a two-dimensional template, and the local mean method and bilinear interpolation method are used to scale the facial expression image.

[0096] The local mean method is an improvement on the equal-interval sampling method. The original image is divided into sub-blocks, and the pixel value in the reduced image is the mean of the corresponding sub-block pixels.

[0097] Suppose the original image is F(i, j), the size is W*H, and the reduced image is G(x, y).

[0098] (1) Find the local sub-block as follows:

[0099]

[0100] (2) Find the mean value of the reduced image G(x, y) = F(x, y).

[0101] Bilinear interpolation is currently the most widely used method for image magnification. Its core principle is: according to the address and magnification ratio of a certain pixel point in the target image, find the corresponding address in the original image, and perform bilinear interpolation based on the grayscale values ​​of the pixels in the four neighborhoods of the address. The specific steps of the algorithm are as follows:

[0102] (1) Create a new image based on the original image and the enlarged ratio.

[0103] (2) Map the address (x, y) of a pixel in the new image to the corresponding address (x', y') in the original image.

[0104] (3) The calculated (x', y') is not an integer. We round (x', y') to get (x", y"), and get another three values: (x"+1, y"), (x", y"+1), and (x"+1, y"+1).

[0105] (4) Use bilinear interpolation to obtain the pixel value of address (x", y").

[0106] (5) Repeat steps (2) to (4) until all pixel values ​​of all new images are obtained.

[0107] Furthermore, in this embodiment, the image processing module 300 uses a convolutional neural network to extract abstract features of facial expressions. Figure 7 The structure of the network is shown in Figure 1, which includes an input layer, three alternating convolutional layers and pooling layers, a fully connected layer, and an output layer. The input of the network is a 100*100 facial expression picture, the size of the convolution kernel is 3*3, the convolution step is 1, the activation function uses a linear rectification function, and the classification function uses a normalized exponential function Softmax function.

[0108] The specific steps include:

[0109] (1) In the convolution layer, the local area of ​​the output feature map (or input image) of the previous layer is convolved with the convolution kernel, and then a feature map of this layer is obtained through the activation function:

[0110]

[0111] (2) The maximum pooling operation is used, and the size is 2*2. After the pooling operation, the size of the input feature map is reduced to half of the original size, but the number of feature maps does not change:

[0112]

[0113] In a convolutional neural network, after multiple convolutional layers and pooling layers, one or more fully connected layers are usually connected. This layer is connected in the same way as the neurons in a traditional neural network, that is, all neurons between the two layers are connected to each other. It is mainly used to organize the feature information extracted by the previous layers for classification, as expressed as follows:

[0114]

[0115] (3) The output layer of the traditional LeNet-5 network uses an RBF classifier. Since facial expression features are more complex than digital features and there is no unified template for the classification of facial expressions, the classification layer of the network uses the Softmax function, which is widely used in the field of machine learning and has stronger classification capabilities, as a classifier.

[0116] The Softmax function model is a generalization of the Logistics model in multi-classification. Its essence is to map a K-dimensional vector z containing any real number into another K-dimensional real number vector, where the value of each element in the vector is between (0, 1), and the sum of all element values ​​is 1, which is defined as:

[0117]

[0118] A supervised local preservation projection loss term is added to the penultimate Softmax layer of the CNN structure to make full use of intra-class and inter-class information, improve the feature learning ability of the convolutional neural network, and increase the discriminative ability of the extracted features while maintaining the local manifold structure of the sample features in the high-dimensional space, that is, to increase the similarity of features of samples of the same type and reduce the similarity of samples of different types.

[0119] Define the objective function:

[0120] The K nearest neighbor method is used to define:

[0121]

[0122] So the objective function becomes

[0123]

[0124] Consider inter-class information:

[0125] in:

[0126]

[0127]

[0128]

[0129]

[0130]

[0131]

[0132] Finally, the losses of softmax and SLPP are combined to consider both global information and utilize the intra-class and inter-class difference information, improving the feature learning ability of the convolutional neural network. Then the overall loss function is: L = L s + λL slpp

[0133] The neural network training in this embodiment includes the following:

[0134] Input: Training samples The size n of the min-batch and the maximum number of iterations Maxlter.

[0135] Output: The feature map of the last pooling layer

[0136] Procedure:

[0137] 1: Initialization: i = 0, randomly initialize the weights and parameter values in the convolutional neural network;

[0138] 2: Calculate the feature maps of each layer according to the designed convolutional neural network structure;

[0139] 3: Calculate the loss term of the softmax layer;

[0140] 4: Calculate the network parameter matrix W;

[0141] 5: Select the relevant parameter J;

[0142] 6: Calculate the overall loss term L;

[0143] 7: Update the loss parameters, and adjust the weights and bias values of each parameter through backpropagation;

[0144] 8: Repeat steps 2 to 7 until L < s or the maximum number of iterations is reached, then jump out of the loop;

[0145] 9: Use the Softmax activation function to determine the category to which the entire facial expression image belongs.

[0146] Refer to Figure 9-10 For the schematic illustration, due to the relationship of the gray scale of the attached drawings, it should be particularly noted in this embodiment that the actual effect cannot be seen, so it is particularly described. Fig. 9 It shows that in the case of pleasure, the displayed color and brightness of the vehicle in front are recognized to be lower, while in the case of anger, the recognized color and brightness are adjusted higher, playing a warning role. Fig.10 It shows that in the case of pleasure, the lines of the vehicle in front are recognized to be thinner, while in the case of anger, the lines of the vehicle in front are recognized to be wider, playing a warning role.

[0147] Embodiment 2

[0148] Reference Figure 8 The schematic diagram shows the overall principle structure of an augmented reality head-up display system that takes driver emotions into consideration, including a TOF module 100, an image recognition module 200, an image processing module 300 and an AR-HUD module 400; specifically, the TOF module 100 is used to obtain a target image in RGB format of the current scene; the image recognition module 200 is connected to the TOF module 100, and is used to identify the eyeballs and faces in the target image and generate recognition results; the depth information of the image is obtained, the eyeball position information is output, and the facial expression image is obtained to analyze the driver's current emotions; the image processing module 300 is connected to the image recognition module 200, and is used to analyze and obtain the eyeball position information and transmit it to the AR-HUD module 400 for enhanced display real-time tracking; the AR-HUD module 400 is used to make corresponding content result display adjustments according to the driver's emotional perception data.

[0149] It should be appreciated that embodiments of the present invention may be implemented or enforced by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The method may be implemented in a computer program using standard programming techniques - including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner - according to the methods and drawings described in the specific embodiments. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program may be implemented in an assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed ASIC for this purpose.

[0150] Furthermore, the operations of the processes described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors, by hardware, or a combination thereof. The computer program includes a plurality of instructions that may be executed by one or more processors.

[0151] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or part thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described herein includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques described in the present invention, the present invention also includes the computer itself. The computer program can be applied to input data to perform the functions described herein, thereby converting the input data to generate output data stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents a physical and tangible object, including a specific visual depiction of the physical and tangible object produced on a display.

[0152] As used in this application, the terms "component", "module", "system", etc. are intended to refer to a computer-related entity, which can be hardware, firmware, a combination of hardware and software, software, or software in operation. For example, a component can be, but is not limited to: a process running on a processor, a processor, an object, an executable file, a thread in execution, a program, and / or a computer. As an example, an application running on a computing device and the computing device can both be components. One or more components can exist in a process and / or thread in execution, and a component can be located in a computer and / or distributed between two or more computers. In addition, these components can be executed from various computer-readable media having various data structures thereon. These components can communicate in a local and / or remote process manner, such as according to a signal having one or more data packets (e.g., data from a component that interacts with another component in a local system, a distributed system, and / or interacts with other systems in a signal manner through a network such as the Internet).

[0153] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An augmented reality head-up display method considering driver emotions, characterized in that: include, The TOF module (100) obtains a target image of the current scene in RGB format; The image recognition module (200) recognizes the eyeballs and the face in the target image and generates a recognition result; The recognition result is input into an image processing module (300) for processing, the processing comprising acquiring depth information of the image, outputting eyeball position information and acquiring a facial expression image to analyze the current emotion of the driver; Specifically, the method includes: identifying the position of the eyeball in the image corresponding to the image recognition; obtaining the corresponding depth information according to the position identified by the image and the depth information obtained; the first depth information is the depth information in the image, and the second depth information refers to the position information of the eyeball corresponding to the camera; analyzing and processing the image of the facial expression to obtain the emotional state of the driver; The eyeball position information is transmitted to the AR-HUD system for enhanced display and real-time tracking; According to the driver's emotional perception data transmitted to the AR-HUD system, the corresponding content result display adjustment is made; The content result display adjustment includes: According to the driver's emotional perception results, the AR-HUD content display is changed, including bolding the warning lines and deepening the color in the case of anger; when in a good mood, the overall color becomes lighter, and the font becomes more diverse without affecting safety; in addition, the overall interface layout is adjusted according to different moods. When the driver is in a relatively calm mood, the content display is reduced; specifically, in a good mood, the display color and brightness of the vehicle ahead are lower, while in an angry mood, the color and brightness of the vehicle ahead are adjusted higher to serve as a warning; in a good mood, the lines of the vehicle ahead are thinner, while in an angry mood, the lines of the vehicle ahead are wider to serve as a warning; The acquisition of eyeball position information includes: The pupil area and position are determined in the local area image containing the human eye through Haar features; a differential Gaussian pyramid model of the human eye image is established to calculate the regional extreme points; An adaptive pupil fitting template is established in the pupil region image, and the pupil edge and center are fitted with the regional extreme point as the template center; In the binary pupil area image, an ellipse fitting equation is established, the coordinates of the selected feature points are substituted into the ellipse equation, and the center of the ellipse is considered to be the center of the pupil. After the data of the obtained pupil center is confirmed, it is transferred to the depth map to obtain the corresponding depth coordinate Z value, which is the Z value of the eyeball relative to the camera coordinate system; The target image is processed by histogram equalization to increase the local contrast of the target image; the step of histogram equalization specifically includes: Count the number of times each grayscale appears in the histogram and save it as an array; calculate the cumulative distribution function value of each grayscale in the histogram and save it as another array; calculate the grayscale value of the original image after histogram equalization according to the following formula; The image processing module (300) uses a convolutional neural network to extract the abstract features of facial expressions. Among them, the convolutional neural network includes an input layer, three alternating convolutional layers and pooling layers, a fully connected layer, and an output layer. The input of the network is a 100*100 facial expression image. The size of the convolutional kernel is 3*3, the convolutional stride is 1, the activation function uses the rectified linear unit function, and the classification function uses the Softmax function of the normalized exponential function; The convolutional neural network includes the following training steps. Initialize i = 0, and randomly initialize the weights and parameter values in the convolutional neural network; calculate the feature maps of each layer according to the designed convolutional neural network structure; Calculate the loss term of the softmax layer; calculate the network parameter matrix W; Select the relevant parameter J; Calculate the overall loss term L; Update the loss parameters, and adjust the weights and bias values of each parameter through backpropagation; until L < s or the maximum number of iterations is reached, then jump out of the loop; Use the Softmax activation function to determine the category to which the entire facial expression image belongs.

2. The augmented reality head-up display method considering the driver's emotions as claimed in claim 1, characterized in that: The TOF module (100) includes a 3D camera module (101) and a TOF sensor (102); Use the 3D camera module (101) to obtain the RGB map information of the target image.

3. An augmented reality head-up display system that takes into account the driver's emotions, characterized by: It includes a TOF module (100), an image recognition module (200), an image processing module (300), and an AR-HUD module (400); The TOF module (100) is used to obtain the target image in the current scene in RGB format; The image recognition module (200) is connected to the TOF module (100) and is used to recognize the eyeballs and faces in the target image and generate recognition results; The image processing module (300) is connected to the image recognition module (200) and is used to analyze and obtain the eyeball position information and transmit it to the AR-HUD module (400) for enhanced display and real-time tracking; Identify the position of the corresponding eyeball in the image; according to the position identified in the image and the obtained depth information, obtain the corresponding depth information; the first depth information above is the depth information in the picture, and the second depth information refers to the position information of the eyeball relative to the camera; according to the image of the facial expression, analyze and process to obtain the emotional state of the driver; The AR-HUD module (400) is used to make corresponding content result display adjustments according to the driver's emotional perception data; the content result display adjustment includes, according to the driver's emotional perception results, making changes to the AR-HUD content display, including making the warning lines bolder and the color darker in the case of anger; when the mood is good, the overall color becomes lighter, and the font becomes more diverse without affecting safety; in addition, the overall interface layout is adjusted according to different moods, and when the driver is in a relatively calm mood, the content display is reduced; specifically, in the case of good mood, the display color and brightness of the identified vehicle in front are lower, while in the case of anger, the identified color and brightness are adjusted higher to serve as a warning; in the case of good mood, the lines of the identified vehicle in front are thinner, while in the case of anger, the lines of the identified vehicle in front are wider to serve as a warning.

Images