Intelligent vehicle-mounted prompt system and method based on expression recognition

By adopting an intelligent on-board prompt system based on expression recognition in the car cockpit, using a monocular camera and a car server to identify the driver's facial expressions, the problems of low recognition accuracy and efficiency in traditional technology are solved, and the driver's fatigue and Effective monitoring and reminding of emotions improve driving safety.

CN114360001BActive Publication Date: 2024-12-03JIANGSU UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111502820.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-10
Publication Date
2024-12-03
Estimated Expiration
2041-12-10

AI Technical Summary

Technical Problem

The lack of expression recognition technology in the existing car cockpits makes it difficult to monitor driver fatigue and emotional changes, affecting driving safety. In addition, traditional facial expression recognition technology requires artificial design features, resulting in low recognition accuracy and efficiency.

Method used

Design an intelligent vehicle-mounted prompt system based on expression recognition, using a monocular camera to collect driver pictures, extract facial expression features through the vehicle server, and determine whether it is a dangerous expression through the facial expression recognition model. If so, it will be done through the instrument display disk. It is suggested that the system does not need to design features artificially, and combines optimized feature extraction operators and grayscale processing to improve recognition accuracy and efficiency.

Benefits of technology

It has achieved driving safety guarantees without manpower costs, improved the accuracy and efficiency of driver expression recognition, and enhanced drivers' awareness of safe driving.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114360001B_ABST
    Figure CN114360001B_ABST
Patent Text Reader

Abstract

The present invention provides an intelligent vehicle-mounted prompting system and method based on expression recognition, the system comprising: a monocular camera, a vehicle-mounted server and an instrument display panel; the monocular camera is used to collect a driver's picture at the current moment and transmit the driver's picture to the vehicle-mounted server; the vehicle-mounted server is used to extract a facial expression picture containing the driver's current facial expression from the driver's picture, extract features from the facial expression picture to obtain a facial feature map, and determine whether the current facial expression contained in the facial feature map belongs to a dangerous expression through a facial expression recognition model, and when the current facial expression belongs to a dangerous expression, the driver is prompted through the instrument display panel. Thus, the driver can be prompted based on expression recognition without artificially designing features, which can not only ensure driving safety while saving labor costs, but also help improve recognition accuracy and efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of automobile intelligent cockpit equipment, and in particular to an intelligent vehicle-mounted prompting system based on expression recognition and an intelligent vehicle-mounted prompting method based on expression recognition. Background Art

[0002] At present, due to the increase in the number of cars, the pressure of urban commuting has increased, which has prolonged the travel time and caused drivers to be prone to fatigue driving. Some drivers with bad tempers have forced to change lanes and say goodbye to other cars because of anger, which not only endangers their own driving safety, but also causes immeasurable harm to other car drivers and passengers. The current facial expression recognition technology requires artificially designed features, which consumes a lot of manpower, and the feature extraction is separated from the expression classification, resulting in low recognition accuracy and low recognition efficiency. Therefore, this expression recognition technology is not used in traditional car cockpits. Summary of the invention

[0003] The present invention aims to solve one of the above technical problems at least to a certain extent.

[0004] To this end, the present invention proposes the following technical solutions:

[0005] The first embodiment of the present invention proposes an intelligent vehicle prompt system based on expression recognition, comprising: a monocular camera, a vehicle computer server and an instrument display panel, wherein the vehicle computer server is connected to the monocular camera and the instrument display panel respectively;

[0006] The monocular camera is used to collect the driver's picture at the current moment and transmit the driver's picture to the vehicle computer server;

[0007] The vehicle computer server is used to extract a facial expression picture containing the driver's current facial expression from the driver picture, perform feature extraction on the facial expression picture to obtain a facial feature map, and judge whether the current facial expression contained in the facial feature map belongs to a dangerous expression through a facial expression recognition model, and when the current facial expression belongs to the dangerous expression, prompt the driver through the instrument display panel.

[0008] In addition, the intelligent vehicle-mounted prompt system based on expression recognition proposed in the above embodiment of the present invention may also have the following additional technical features:

[0009] According to one embodiment of the present invention, when extracting a facial expression picture containing the driver's current facial expression from the driver picture, the vehicle computer server is specifically used to: determine the facial position in the driver picture; and extract the facial expression picture containing the current facial expression based on the facial position.

[0010] According to one embodiment of the present invention, when the vehicle server extracts features from the facial expression picture to obtain a facial feature map, it is specifically used to: select a square block from the facial expression picture, wherein the square block includes a central pixel point and eight neighborhood pixel points; determine a central pixel value corresponding to the central pixel point and a neighborhood pixel value corresponding to each of the neighborhood pixel points; determine a square value of the central pixel value and a square value of each neighborhood pixel value; determine a local feature description sequence of the facial expression picture based on the square value of the central pixel value and the square values ​​of each of the neighborhood pixel values; and generate the facial feature map based on the local feature description sequence.

[0011] According to one embodiment of the present invention, when the vehicle computer server determines the local feature description sequence of the facial expression image according to the square value of the central pixel value and the square values ​​of each of the neighborhood pixel values, it is specifically used to: select a first neighborhood pixel point and a second neighborhood pixel point from each of the neighborhood pixel points, wherein the position of the second neighborhood pixel point is different from the position of the first neighborhood pixel point; starting from the first neighborhood pixel point in the square block according to a preset direction, sequentially determine the first difference between the square value of the central pixel value and the square values ​​of each of the neighborhood pixel values ​​to obtain a first difference sequence arranged in the order of generation; starting from the second neighborhood pixel point in the square block according to the preset direction, sequentially determine the central pixel value The second difference between the square value of the pixel value and the square value of each of the neighborhood pixels is used to obtain a second difference sequence arranged in the order of generation; the first difference sequence and the second difference sequence are arranged up and down in a one-to-one correspondence; starting from the first first difference in the first difference sequence, each of the first difference values ​​is compared with the corresponding second difference in turn, and if the first difference value is greater than or equal to the corresponding second difference value, the generated value is determined to be a first preset value; if the first difference value is less than the corresponding second difference value, the generated value is determined to be a second preset value, wherein the second preset value is different from the first preset value; and each of the generated values ​​is arranged in sequence in the order of generation to obtain a local feature description sequence of the facial expression image.

[0012] According to one embodiment of the present invention, the vehicle server is further used to: perform grayscale processing on the driver image before extracting the facial expression image containing the driver's current facial expression from the driver image; and perform rotation correction on the driver image.

[0013] According to one embodiment of the present invention, when performing rotation correction on the driver image, the vehicle computer server is specifically used to: determine the original coordinates of the left eye and the right eye in the driver image; determine the angle between the centers of the two eyes in the driver image and the horizontal direction; determine the rotation angle according to the angle between the centers of the two eyes and the horizontal direction; determine the rotation matrix according to the rotation angle; and perform rotation correction on the driver image based on the original coordinates of the left eye and the right eye and the rotation matrix.

[0014] According to one embodiment of the present invention, the vehicle server corrects the driver image by the following formula:

[0015]

[0016] Among them, f' x 、f' y represents the corrected left and right eye coordinates, f x 、f y represents the original coordinates of the left eye and the right eye before correction, θ represents the rotation angle, Represents a rotation matrix.

[0017] According to an embodiment of the present invention, the second neighborhood pixel point is adjacent to the first neighborhood pixel point.

[0018] According to an embodiment of the present invention, the first preset value is 1, and the second preset value is 0.

[0019] The second embodiment of the present invention proposes an intelligent vehicle-mounted prompt method based on expression recognition, comprising: collecting a picture of the driver at the current moment;

[0020] Extracting a facial expression picture including the driver's current facial expression in the driver picture;

[0021] Performing feature extraction on the facial expression picture to obtain a facial feature map;

[0022] The facial feature map is judged by a facial expression recognition model to determine whether the current facial expression contained therein belongs to a dangerous expression, and when the current facial expression belongs to the dangerous expression, the driver is prompted through an instrument display panel.

[0023] The technical solution of the present invention can prompt the driver based on facial expression recognition without artificially designed features, which can not only ensure driving safety while saving labor costs, but also help to improve recognition accuracy and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The figure is a schematic diagram of the structure of an intelligent vehicle-mounted prompt system based on expression recognition according to an embodiment of the present invention.

[0025] Figure 2 The figure is a schematic diagram of an application scenario of an embodiment of the present invention.

[0026] Figure 3 The figure is a schematic diagram of performing feature extraction by using an optimized feature extraction operator according to a specific example of the present invention.

[0027] Figure 4 This is a schematic diagram of a vehicle server performing expression recognition through a facial expression recognition model according to an embodiment of the present invention.

[0028] Figure 5 The present invention is a flowchart of an intelligent vehicle-mounted prompting method based on expression recognition according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] With the development of artificial intelligence, all aspects of automobiles are also developing in the direction of intelligence. The adaptive cruise function currently available in some models is the best embodiment of the application of artificial intelligence in smart cars. As an important branch of human-computer interaction, expression recognition plays an important role in promoting the intelligent upgrade of automobiles and travel safety. Therefore, the embodiment of the present invention proposes an intelligent vehicle prompt system based on expression recognition, which recognizes the driver's expression and prompts the driver according to the recognition result.

[0031] Figure 1 The figure is a schematic diagram of the structure of an intelligent vehicle-mounted prompt system based on expression recognition according to an embodiment of the present invention.

[0032] like Figure 1 As shown, the system includes: a monocular camera 1, a vehicle computer server 2 and an instrument display panel 3. The vehicle computer server 2 is connected to the monocular camera 1 and the instrument display panel 3 respectively. The monocular camera 1 is installed between the instrument display panel 3 and the steering wheel (not shown in the figure).

[0033] The monocular camera 1 is used to collect the driver's picture at the current moment and transmit the driver's picture to the vehicle server 2; the vehicle server 2 is used to extract the facial expression picture containing the driver's current facial expression from the driver's picture, perform feature extraction on the facial expression picture to obtain a facial feature map, and judge whether the current facial expression contained in the facial feature map belongs to a dangerous expression through a facial expression recognition model, and when the current facial expression belongs to a dangerous expression, prompt the driver through the instrument display panel 3.

[0034] Among them, the driver picture can be understood as a picture including but not limited to the driver's face, the facial expression picture refers to a picture that only contains the driver's face, that is, the facial expression picture only contains the driver's facial expression, and the facial feature map refers to the feature map corresponding to the facial expression picture.

[0035] Among them, the facial expression recognition model is a pre-trained neural network model for identifying the driver's facial expressions. Its input is the driver's facial feature map, and the output is the expression category corresponding to the facial expression. Dangerous expressions refer to expressions that may cause traffic accidents or may endanger public transportation safety, such as anger expressions, irritation expressions, fatigue expressions, etc.

[0036] Specifically, in practical applications, during vehicle driving, such as Figure 2 As shown, the monocular camera 1 can collect the driver's picture at the current moment (the driver is sitting in the driver's seat) in real time, and transmit the driver's picture to the vehicle computer server 2, and then the vehicle computer server 2 extracts the facial expression picture in the driver's picture to obtain a facial expression picture containing only the driver's face, thereby reducing the influence of background information irrelevant to facial expressions (such as neck information) on facial recognition during subsequent facial recognition. Afterwards, the vehicle computer server 2 generates a facial feature map by extracting features from the facial expression picture, such as texture features, and then the vehicle computer server 2 calls the facial expression recognition model and inputs the facial feature map into the facial expression recognition model, and then the facial expression recognition model recognizes the facial feature map. Output the expression category corresponding to the facial feature map, then the car server 2 determines whether the expression category belongs to a dangerous expression. If it is (for example, the current expression category belongs to an angry expression), the instrument display panel 3 is controlled to prompt the driver. The prompt method can be, for example, the instrument display panel 3 displays text (currently fatigued driving, please drive safely, etc.), patterns or numbers, or it can emit different brightness to prompt the driver, or it can emit voice to prompt the driver. At the same time, the car server 2 emits a warning sound when the current facial expression belongs to a dangerous expression, so as to further remind the driver to drive safely. The driver knows that the current driving is in a dangerous state through the prompt of the instrument display panel 3, and takes corresponding measures.

[0037] When a dangerous expression is recognized, the vehicle server 3 can also generate an alarm code and send it to the instrument display panel 3, so that the alarm code is displayed on the instrument display panel 3 to remind the driver to drive safely.

[0038] The intelligent vehicle prompting system of the embodiment of the present invention extracts the driver's facial expression picture from the driver's picture, extracts features from the facial expression picture to obtain a facial feature map, uses a facial expression recognition model to recognize the driver's expression based on the facial feature map, and when it is determined according to the recognition result that the driver's current facial expression belongs to a dangerous expression, prompts the driver on the instrument display panel. Among them, facial expression recognition is performed on the basis of obtaining the facial expression picture and performing feature extraction, which is conducive to improving recognition efficiency. When performing expression recognition, feature extraction and expression classification are not separated, which is conducive to improving recognition accuracy.

[0039] Therefore, the intelligent vehicle-mounted prompt system based on expression recognition in the embodiment of the present invention can prompt the driver based on expression recognition without artificially designed features, which can not only ensure driving safety while saving manpower costs, but also help to improve recognition accuracy and efficiency.

[0040] In one embodiment of the present invention, the vehicle server 2, when extracting a facial expression picture containing the driver's current facial expression from the driver's picture, can be specifically used to: determine the facial position in the driver's picture; and extract the facial expression picture containing the current facial expression based on the facial position.

[0041] Specifically, the vehicle server 2 calls a SSD (Single Shot MultiBox Detector)-Lite model to locate the target, so as to locate the facial position in the driver's picture. This model is adopted because its network has the advantages of fast detection speed and high positioning accuracy in target detection tasks. Afterwards, the vehicle server 2 extracts the picture at the facial position according to the facial position, that is, obtains the facial expression picture, which only contains the driver's facial expression, thereby reducing the influence of background information irrelevant to the facial expression.

[0042] When extracting facial features, the related art uses a local binary pattern (LBP) feature extraction operator to extract features, while the embodiment of the present invention optimizes the local binary pattern operator to obtain an optimized feature extraction operator, and extracts features from the facial expression image according to the optimized feature extraction operator to obtain a facial feature map. The following is a specific description:

[0043] In one embodiment of the present invention, the vehicle computer server, when performing feature extraction on a facial expression picture to obtain a facial feature map, can be specifically used to: select a square block from the facial expression picture, wherein the square block includes a central pixel point and eight neighborhood pixel points; determine a central pixel value corresponding to the central pixel point and a neighborhood pixel value corresponding to each neighborhood pixel point; determine the square value of the central pixel value and the square value of each neighborhood pixel value; determine a local feature description sequence of the facial expression picture based on the square value of the central pixel value and the square values ​​of each neighborhood pixel value; and generate a facial feature map based on the local feature description sequence.

[0044] The local feature description sequence refers to a binary sequence of image features used to represent the facial feature map.

[0045] Specifically, first, a square block with a length and width of three pixels is selected from the facial expression picture, that is, the square block contains a central pixel point and eight neighboring pixel points, and the central pixel value of the central pixel point and the neighboring pixel values ​​of the neighboring pixel points are determined, and then the square value of each pixel value is determined, and the local feature description sequence of the facial expression picture is determined according to each square value, and then a facial feature map is generated according to the local feature description sequence.

[0046] In one example, when the vehicle computer server determines the local feature description sequence of the facial expression picture according to the square value of the central pixel value and the square value of each neighboring pixel value, it can be specifically used to: select a first neighborhood pixel point and a second neighborhood pixel point from each neighborhood pixel point, wherein the position of the second neighborhood pixel point is different from the position of the first neighborhood pixel point; starting from the first neighborhood pixel point in a preset direction in the square block, determine in sequence the first difference between the square value of the central pixel value and the square value of each neighboring pixel value to obtain a first difference sequence arranged in the order of generation; starting from the second neighborhood pixel point in a preset direction in the square block, determine in sequence the square value of the central pixel value The second difference between the square value and the square value of each neighboring pixel value is obtained to obtain a second difference sequence arranged in the order of generation; the first difference sequence and the second difference sequence are arranged up and down in a one-to-one correspondence; starting from the first first difference in the first difference sequence, each first difference is compared with the corresponding second difference in turn, if the first difference is greater than or equal to the corresponding second difference, the generated value is determined to be a first preset value; if the first difference is less than the corresponding second difference, the generated value is determined to be a second preset value, wherein the second preset value is different from the first preset value; each generated value is arranged in sequence according to the order of generation to obtain a local feature description sequence of the facial expression image.

[0047] Furthermore, the second neighborhood pixel point is adjacent to the first neighborhood pixel point.

[0048] Furthermore, the preset direction is clockwise or counterclockwise, the first preset value is 1, and the second preset value is 0.

[0049] Specifically, after obtaining the pixel values ​​of each pixel point in the square block (a central pixel value and eight neighboring pixel values), a first neighboring pixel point and a second neighboring pixel point located at adjacent positions are selected from each neighboring pixel value, and the first difference between the square value of the central pixel value and the square value of each neighboring pixel value is determined in sequence from the first neighboring pixel point in a clockwise direction (or counterclockwise direction) to obtain a first difference sequence; the second difference between the square value of the central pixel value and the square value of each neighboring pixel value is determined in sequence from the second neighboring pixel point in a clockwise direction (or counterclockwise direction) to obtain a second difference sequence. In the first difference sequence, each first difference value is arranged in sequence according to the order in which it is generated, and in the second difference sequence, each second difference value is arranged in sequence according to the order in which it is generated.

[0050] After obtaining the first difference sequence and the second difference sequence, the first difference sequence and the second difference sequence are arranged up and down in a one-to-one correspondence, that is, the first difference sequence is arranged in the first row, and the second difference sequence is arranged in the second row, and the first difference value of the first difference sequence corresponds to the first difference value of the second difference sequence, until the last difference value of the first difference sequence corresponds to the last difference value of the second difference sequence.

[0051] After the arrangement, starting from the first difference, the corresponding first difference is compared with the second difference. If the first difference is greater than or equal to the corresponding second difference, the generated value is determined to be 1; if the first difference is less than the corresponding second difference, the generated value is determined to be 0. The generated values ​​are arranged in order of generation to obtain a local feature description sequence of the facial expression image.

[0052] That is to say, the feature extraction operator optimized by the embodiment of the present invention is: subtract the square of the pixel value of the center point of the LBP operator from the square of the pixel values ​​of each neighboring point around it in clockwise order, and then, in a way of arranging and comparing them up and down, it can be defined that if the upper value (first difference) is greater than or equal to the lower value (second difference), a value of 1 is generated, otherwise a value of 0 is generated. The local feature description sequence generated by this method. Compared with the traditional local binarization, its advantage is that it not only compares the pixel values ​​of the adjacent points and the center point in the square block, but also indirectly compares the pixel values ​​between each adjacent point. This has the advantage of not only ensuring the connection between the pixel values ​​of the adjacent points and the center point, but also taking into account the relationship between the pixel values ​​of each adjacent point.

[0053] For example, Figure 3As shown in the figure, the central pixel value in the square block is 5, and its square value is 25. The pixel values ​​in each area are: 1, 4, 3, 7, 1, 3, 5 and 6, and their square values ​​are: 1, 16, 9, 49, 1, 9, 25 and 36. When the first neighborhood pixel point is selected as the pixel point in the upper left corner of the square and the second neighborhood pixel point is the pixel point adjacent to the upper left corner pixel point in the clockwise direction, starting from the upper left corner (pixel value 1), the square value of the central pixel value 25 and each area pixel value are determined in the clockwise direction. The difference between the square values ​​of the neighboring pixel values ​​1, 16, 9, 49, 1, 9, 25 and 36 is used to obtain the first difference sequence: 24, 9, 16, -24, 24, 16, 0 and -11. Starting from pixel value 4, the difference between the square value 25 of the central pixel value and the square values ​​16, 9, 49, 1, 9, 25, 36 and 1 of each neighboring pixel value is determined in a clockwise direction to obtain the second difference sequence: 9, 16, -24, 24, 16, 0, -11 and 24. The first difference sequence and the second difference sequence are compared one by one, and according to the rule: when the upper value is greater than or equal to the lower value, the generated value is determined to be 1, and when the upper value is less than the lower value, the generated value is determined to be 0, thereby obtaining the sequence 10101110, which is the local feature description sequence of the facial expression image, and the driver's facial feature map can be generated according to the sequence.

[0054] Therefore, the embodiment of the present invention not only compares the pixel values ​​of adjacent points and the center point in the square block, but also indirectly compares the pixel values ​​between each adjacent point. The advantage of this is that it not only ensures the connection between the pixel values ​​of adjacent points and the center point, but also takes into account the relationship between the pixel values ​​of each adjacent point. On the basis of retaining the sensitivity of the feature extraction operator to tiny features, the ability to extract facial texture features is further improved. Compared with the traditional feature map obtained by local binarization, the embodiment of the present invention can use the optimized feature extraction operator to ensure that the system does not occupy too much vehicle server memory during operation, while ensuring high efficiency of operation.

[0055] It should be noted that in order to ensure that the system does not occupy too much memory of the vehicle server during operation and to ensure efficient operation, in addition to optimizing the feature extraction operator, the preprocessing optimization of the acquired image can also be performed.

[0056] That is, in one embodiment of the present invention, the vehicle server 2 can also be used to: perform grayscale processing on the driver image before extracting the facial expression image containing the driver's current facial expression; and perform rotation correction on the driver image.

[0057] In one example, the vehicle computer server 2, when performing rotation correction on the driver's picture, is specifically used to: determine the original coordinates of the left eye and the right eye in the driver's picture; determine the angle between the centers of the two eyes in the driver's picture and the horizontal direction; determine the rotation angle according to the angle between the centers of the two eyes and the horizontal direction; determine the rotation matrix according to the rotation angle; and perform rotation correction on the driver's picture based on the original coordinates of the left eye and the right eye and the rotation matrix.

[0058] Furthermore, the vehicle server 2 corrects the driver's image using the following formula:

[0059]

[0060] Among them, f' x 、f' y represents the corrected left and right eye coordinates, f x 、f y represents the original coordinates of the left eye and the right eye before correction, and θ represents the rotation angle; Represents a rotation matrix.

[0061] f' x represents the corrected left eye coordinates, f' y represents the corrected right eye coordinates, f x represents the original coordinates of the left eye before correction, f y Represents the original coordinates of the right eye before correction.

[0062] It should be noted that in expression recognition, the factors that affect expression classification are mainly facial contour features, facial features, facial texture features, and other information that are not related to color. In order to verify the embodiment of the present invention, an experiment was conducted to input the collected high-dimensional RGB (red, green, and blue) images and the low-dimensional grayscale images after grayscale processing into the feature extraction network for expression recognition, and it was found that the recognition accuracy was almost the same; at the same time, after testing an entire data set, it was found that the grayscale image recognition time was only two-thirds of the RGB image recognition time. Therefore, it is concluded that grayscale preprocessing of the collected images not only speeds up the recognition speed, but also has almost no effect on the recognition accuracy.

[0063] Specifically, before extracting the facial expression picture, the vehicle computer server 2 uses a grayscale dimension reduction method to grayscale the driver's picture, which not only speeds up the recognition speed, but also has almost no effect on the recognition accuracy.

[0064] Since in actual driving scenarios, the driver's face cannot be facing the camera in real time (for example, the driver is facing the camera diagonally), the vehicle server 2 needs to perform facial positioning and rotation correction on the driver's image before extracting the facial expression image. First, the input image is tested for facial position determination to reduce the influence of background information unrelated to facial expression; secondly, since the input facial image may have angle differences, the driver's image is rotated to avoid such problems affecting feature extraction.

[0065] When performing rotation correction, in order to obtain the rotation angle, the angle between the center of the two eyes in the driver's picture and the horizontal direction can be obtained, and the rotation angle can be determined based on the angle. Then, the rotation matrix is ​​determined based on the rotation angle, and the driver's picture is rotated based on the rotation matrix to rotate the driver's face to a position facing the camera. After correction, the angle between the center point between the driver's two eyes and the horizontal direction is 0.

[0066] Therefore, by optimizing the driver's picture and the feature extraction operator, it can be ensured that the system does not occupy too much memory of the vehicle server during operation, while ensuring efficient operation.

[0067] It should be noted that the facial expression recognition technology in the related art requires artificially designed features, which consumes a lot of manpower. In addition, feature extraction and expression classification are separated and cannot be integrated into an end-to-end model, resulting in low recognition accuracy and low recognition efficiency.

[0068] After the vehicle server 2 obtains the facial feature map, it inputs the facial feature map into a pre-trained facial expression recognition model, such as Figure 4 As shown in the figure, the facial expression picture is first subjected to a deeper feature extraction by the backbone network. The backbone network is essentially a feature extraction network to obtain more and more robust features of the picture. Then the backbone network outputs the feature map after convolution calculation, performs spatial symmetry processing, and generates a feature map of a fixed size; it is then divided into two branches. Branch one is the image segmentation branch, which divides the previous feature map into small images for local feature recognition; branch two performs full convolution and then connects to the fully connected layer to locate the target frame, and merges it with the classification output by branch one to determine whether the current input server picture is anger, fatigue, or other types of expressions, and gives different reminders to the driver based on different judgments.

[0069] Figure 4 The network structure is modified on Mask-RCNN, reducing three convolutional layers, adding two upsampling layers, optimizing the model, reducing the overfitting probability, and improving the classification efficiency.

[0070] The embodiment of the present invention realizes the input of expression images and the rapid output of expression results, and determines whether the expression at the current moment belongs to the category of dangerous expressions. For the driver's picture, it is first preprocessed, and then the optimized feature operator is used to extract features, and the extracted features are input into the convolutional neural network to extract image features, and then nonlinear classification is performed through the fully connected layer. At this time, the fully connected layer is similar to MLP (Multilayer Perceptron), and Dropout (random inactivation) and Batch Normalization (batch normalization) operations are also added to prevent overfitting.

[0071] In summary, the embodiment of the present invention uses an improved feature extraction operator, followed by a deep neural network for data training and predictive recognition of a facial expression recognition model. This model implements end-to-end expression recognition and improves model robustness, thereby achieving classification and recognition of multiple expressions; on this basis, based on the recognized expression, it is determined whether the current expression is a dangerous expression that is likely to cause an accident, and accordingly, a driver fatigue reminder or a reminder to drive safely is given on the dashboard display, overcoming the defects of low recognition accuracy and low recognition efficiency, and ensuring safe driving of the vehicle.

[0072] Figure 5 The present invention is a flowchart of an intelligent vehicle-mounted prompting method based on expression recognition according to an embodiment of the present invention.

[0073] like Figure 5 As shown, the intelligent vehicle-mounted prompt method based on expression recognition includes the following steps:

[0074] S1, collect the driver's picture at the current moment.

[0075] S2, extracting a facial expression picture containing the driver's current facial expression from the driver picture.

[0076] S3, extracting features from the facial expression image to obtain a facial feature map.

[0077] S4, judging whether the current facial expression contained in the facial feature map belongs to a dangerous expression through the facial expression recognition model, and prompting the driver through the instrument display panel when the current facial expression belongs to a dangerous expression.

[0078] It should be noted that other specific implementation methods of the intelligent in-vehicle prompt method based on expression recognition in the embodiment of the present invention can refer to the specific implementation methods of the intelligent in-vehicle prompt system based on expression recognition mentioned above. In order to avoid redundancy, they will not be repeated here.

[0079] The intelligent in-vehicle prompt method based on expression recognition in the embodiment of the present invention can prompt the driver based on expression recognition without artificially designed features, which can not only ensure driving safety while saving manpower costs, but also help to improve recognition accuracy and efficiency.

[0080] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. "Multiple" means two or more, unless otherwise clearly and specifically defined.

[0081] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0082] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0083] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0084] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0085] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0086] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0087] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent vehicle-mounted prompt system based on expression recognition, characterized in that: include: A monocular camera, a vehicle computer server and an instrument display panel, wherein the vehicle computer server is connected to the monocular camera and the instrument display panel respectively; The monocular camera is used to collect the driver's picture at the current moment and transmit the driver's picture to the vehicle computer server; The vehicle computer server is used to extract a facial expression picture containing the driver's current facial expression from the driver picture, perform feature extraction on the facial expression picture to obtain a facial feature map, and determine whether the current facial expression contained in the facial feature map belongs to a dangerous expression through a facial expression recognition model, and when the current facial expression belongs to the dangerous expression, prompt the driver through the instrument display panel, The vehicle computer server, when extracting features from the facial expression picture to obtain a facial feature map, is specifically used to: Selecting a square block from the facial expression picture, wherein the square block includes a central pixel point and eight neighboring pixel points; Determine a central pixel value corresponding to the central pixel point and a neighborhood pixel value corresponding to each of the neighborhood pixel points; Determine the square value of the central pixel value and the square value of each neighborhood pixel value; Determine a local feature description sequence of the facial expression picture according to the square value of the central pixel value and the square values ​​of each of the neighborhood pixel values; generating the facial feature map according to the local feature description sequence, The vehicle computer server, when determining the local feature description sequence of the facial expression picture according to the square value of the central pixel value and the square value of each of the neighborhood pixel values, is specifically used to: Selecting a first neighborhood pixel point and a second neighborhood pixel point from each of the neighborhood pixel points, wherein a position of the second neighborhood pixel point is different from a position of the first neighborhood pixel point; Starting from a first neighborhood pixel point in a preset direction in the square block, first differences between the square value of the central pixel value and the square values ​​of each of the neighborhood pixel values ​​are sequentially determined to obtain a first difference sequence arranged in a generation order, wherein the preset direction is a clockwise direction or a counterclockwise direction; Starting from a second neighborhood pixel point in the square block along the preset direction, sequentially determine second differences between the square value of the central pixel value and the square values ​​of each of the neighborhood pixel values ​​to obtain a second difference sequence arranged in a generation order; Arrange the first difference sequence and the second difference sequence up and down in a one-to-one correspondence; Starting from the first first difference in the first difference sequence, each first difference is compared with the corresponding second difference in sequence, and if the first difference is greater than or equal to the corresponding second difference, a generated value is determined to be a first preset value; if the first difference is less than the corresponding second difference, a generated value is determined to be a second preset value, wherein the second preset value is different from the first preset value; The generated values ​​are arranged in sequence according to the order in which they are generated to obtain a local feature description sequence of the facial expression image.

2. The intelligent vehicle-mounted prompt system based on expression recognition according to claim 1 is characterized in that: The vehicle computer server, when extracting the facial expression picture containing the driver's current facial expression from the driver picture, is specifically used to: Determining a facial position in the driver image; A facial expression picture including the current facial expression is extracted according to the facial position.

3. The intelligent vehicle-mounted prompt system based on expression recognition according to claim 1 is characterized in that: The vehicle computer server is also used for: Before extracting the facial expression picture containing the driver's current facial expression from the driver picture, performing grayscale processing on the driver picture; The driver image is rotationally corrected.

4. The intelligent vehicle-mounted prompt system based on expression recognition according to claim 3 is characterized in that: The vehicle computer server, when performing rotation correction on the driver image, is specifically used to: Determining original coordinates of the left eye and the right eye in the driver image; Determine the angle between the centers of the two eyes in the driver image and the horizontal direction; Determine the rotation angle according to the angle between the centers of the two eyes and the horizontal direction; Determine a rotation matrix according to the rotation angle; The driver image is rotationally corrected based on the original coordinates of the left eye and the right eye and the rotation matrix.

5. The intelligent vehicle-mounted prompt system based on expression recognition according to claim 4 is characterized in that: The vehicle server corrects the driver image using the following formula: Among them, f' x 、f' y represents the corrected left and right eye coordinates, f x 、f y represents the original coordinates of the left eye and the right eye before correction, θ represents the rotation angle, Represents a rotation matrix.

6. The intelligent vehicle-mounted prompt system based on expression recognition according to claim 1 is characterized in that: The second neighborhood pixel point is adjacent to the first neighborhood pixel point.

7. The intelligent vehicle-mounted prompt system based on expression recognition according to claim 1 is characterized in that: The first preset value is 1, and the second preset value is 0.

8. An intelligent vehicle-mounted prompting method based on expression recognition using the intelligent vehicle-mounted prompting system based on expression recognition according to any one of claims 1 to 7, characterized in that: include: Collect the current driver's picture; Extracting a facial expression picture including the driver's current facial expression in the driver picture; Performing feature extraction on the facial expression picture to obtain a facial feature map; The facial feature map is judged by a facial expression recognition model to determine whether the current facial expression contained therein belongs to a dangerous expression, and when the current facial expression belongs to the dangerous expression, the driver is prompted through an instrument display panel.