Driver expression recognition method and system in combination with attention mechanism and residual network

Abstract

The invention relates to the technical field of facial recognition of image processing and deep learning, and discloses a driver expression recognition method and system combining an attention mechanism and a residual network, and the method comprises the steps: generating a texture image through extracting the local texture of an original image; fusing the original image and the texture image, inputting the fused image into a residual network, and performing feature extraction through convolution operation; a predefined channel attention mechanism is introduced into the residual network to construct an identification network; and based on a recognition network, performing expression recognition on the extracted feature image to obtain an expression recognition result. According to the method, a channel attention mechanism is introduced into the residual network to perform expression recognition, so that the recognition accuracy and generalization ability are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of facial recognition of image processing and deep learning, and specifically to a driver expression recognition method, system, computer-readable storage medium and electronic device combining an attention mechanism and a residual network. Background Art

[0002] The driver's facial expression is an important indicator reflecting his driving status and psychological state, and is of great significance for preventing traffic accidents and improving driving safety.

[0003] However, existing driver expression recognition systems are often limited by factors such as lighting changes, facial occlusion, and complex expressions, resulting in low recognition accuracy. Summary of the invention

[0004] The purpose of the present invention is to provide a driver expression recognition method and system combining an attention mechanism and a residual network to address the problem of low algorithm accuracy in driver emotion recognition. LBP is used to extract texture information of a face image, and the original image is combined with the texture image and input into a residual network for feature extraction. Expression recognition is performed in a network combined with a channel attention mechanism, thereby improving recognition accuracy and generalization ability.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] In a first aspect, the present application provides a method for driver expression recognition combining an attention mechanism and a residual network, the method comprising:

[0007] Generate a texture image by extracting the local texture of the original image;

[0008] The original image is fused with the texture image and input into the residual network to extract features through convolution operation. A predefined channel attention mechanism is introduced into the residual network to construct a recognition network.

[0009] Based on the recognition network, expression recognition is performed on the extracted feature image to obtain the expression recognition result.

[0010] Optionally, generating a texture image by extracting local texture of the original image includes: collecting image data from the vehicle-mounted terminal, selecting several images as original images to perform LBP local binary processing, and generating a texture image.

[0011] Optionally, the selecting of several images as original images for LBP local binary processing includes: using LBP as an operator for extracting local texture features of the original image, calculating the LBP value of each pixel in the image, and describing the local texture information of the image based on the LBP value.

[0012] Optionally, the recognition network is composed of residual blocks; each residual block includes two convolutional layers and one fully connected layer;

[0013] The fused texture image and the original image are convolved through the convolution layer to obtain a feature image; the residual block has a predefined channel attention mechanism, which is placed after the convolution layer of the residual network. The channel attention weight is calculated using the feature image obtained through the convolution operation of the convolution layer, and the channel attention weight is applied to the feature image to obtain a weighted feature map. The output of the residual block is formed by adaptively adjusting the attention weight.

[0014] Optionally, performing expression recognition on the extracted feature image based on the recognition network to obtain the expression recognition result includes:

[0015] The feature image obtained by the convolution operation is input into the recognition network established by combining the residual network and the predefined channel attention mechanism;

[0016] Use global average pooling and global maximum pooling to obtain the spatial features of each layer channel of the feature image respectively;

[0017] By mapping adaptive one-dimensional convolution, the information interaction between each layer channel and the adjacent layer channel is performed, so that the spatial features output by each layer channel and the adjacent layer channel are added to obtain the spatial feature information of the feature image;

[0018] The channel attention weight is calculated based on the extracted spatial feature information; the channel attention weight is trained to obtain the expression recognition result.

[0019] Optionally, the calculating of the channel attention weight according to the extracted spatial feature information; and the obtaining of the expression recognition result by training the channel attention weight comprises:

[0020] The spatial feature information is normalized by a Sigmoid function to obtain a channel attention weight in a channel domain;

[0021] Applying the channel attention weight to the feature image, and obtaining a weighted feature map by multiplying the channel attention weight and the feature image;

[0022] The weighted feature map is converted into a one-dimensional vector through global average pooling, and the classification output is performed using a fully connected layer, and the probability distribution of each category is generated based on the SoftMax activation function;

[0023] Obtain the driver's expression image set and divide it into a training set and a test set; input the test set into the recognition network to obtain the probability distribution of the true label;

[0024] The cross entropy loss is defined as the difference between the probability distribution of each category and the probability distribution of the true label;

[0025] With the goal of minimizing the cross entropy loss, the channel attention weight is trained based on a preset learning rate until a channel attention weight that meets the goal after training is obtained, and the optimizer Adam is used to update the parameters of the recognition network;

[0026] The test set is input into the updated recognition network, and the trained channel attention weights are loaded to obtain the recognition accuracy.

[0027] Optionally, the acquiring the driver expression image set and dividing the image set into a training set and a test set comprises:

[0028] Acquire a driver expression image set; wherein the driver expression image set includes driving scene image data and driver expression image data;

[0029] Identifying a face region of the driver expression image set, and after cropping the face region, dividing the images corresponding to the face region into a training set and a test set based on a preset division ratio;

[0030] The image data in the training set is resized to match the image size of the recognition network input.

[0031] In a second aspect, the present application provides a driver expression recognition system combining an attention mechanism and a residual network, the system comprising:

[0032] A texture feature generation module is used to generate a texture image by extracting local texture of the original image;

[0033] The information extraction module is used to fuse the original image with the texture image, input it into the residual network, and extract features through convolution operations; and introduce a predefined channel attention mechanism into the residual network to build a recognition network;

[0034] The recognition module is used to perform expression recognition on the extracted feature image based on the recognition network to obtain the expression recognition result.

[0035] In a third aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method steps as described in any one of the first aspects.

[0036] In a fourth aspect, the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the method steps as described in any one of the first aspects.

[0037] The beneficial effects of the present invention are embodied in:

[0038] The above-mentioned driver expression recognition method and system combining attention mechanism and residual network are original in the field of driver expression recognition. Specifically, the local texture of the original image is extracted to generate a texture image; the original image and the texture image are fused and input into the residual network, and feature extraction is performed through convolution operation; and a predefined channel attention mechanism is introduced into the residual network to construct a recognition network; based on the recognition network, expression recognition is performed on the extracted feature image to obtain the expression recognition result. The above uses LBP to describe the local texture information of the face image, and combines the original image with the texture feature image to enhance the extracted feature information; in addition, the difference network combined with the channel attention mechanism is used for expression recognition, which improves the recognition effect, as well as the accuracy and generalization ability of recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the description of the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.

[0040] Figure 1 This is a flow chart of the driver expression recognition method combining the attention mechanism and the residual network of the present invention;

[0041] Figure 2 A network diagram combining the attention mechanism and the residual network provided by the present invention;

[0042] Figure 3 Schematic diagram of the channel attention mechanism based on ECA attention provided by the present invention;

[0043] Figure 4 A schematic diagram of the structure of a driver expression recognition system combining an attention mechanism and a residual network provided by the present invention;

[0044] Figure 5 The present invention provides a schematic structural diagram of an electronic device. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0046] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.

[0047] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0048] It should be understood that although the terms first, second, third, etc. may be used to describe in the embodiments of the present application, these descriptions should not be limited to these terms. These terms are only used to distinguish the descriptions. For example, without departing from the scope of the embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first.

[0049] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0050] It should also be noted that the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the commodity or device including the elements.

[0051] It should be particularly noted that any symbols and / or numbers in the specification that are not marked in the accompanying drawings are not drawing marks.

[0052] The embodiment of the present application provides a driver expression recognition method and system combining an attention mechanism and a residual network, which is mainly used in vehicle-mounted devices. The method can be applied to a terminal, a server, or a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, etc.

[0053] Reference Figure 1 , an embodiment of the present invention provides a driver expression recognition method combining an attention mechanism and a residual network, the method comprising:

[0054] S101 generates a texture image by extracting local texture of the original image;

[0055] S102 fuses the original image with the texture image, inputs them into the residual network, and extracts features through convolution operation; and introduces a predefined channel attention mechanism into the residual network to construct a recognition network;

[0056] S103 performs expression recognition on the extracted feature image based on the recognition network to obtain an expression recognition result.

[0057] In one embodiment, generating a texture image by extracting local texture of an original image in step S101 includes: collecting image data from a vehicle-mounted terminal, selecting several images as original images, performing LBP local binary processing, and generating a texture image.

[0058] In the above embodiment, the selection of several images as original images for LBP local binary processing includes: using LBP as an operator to extract local texture features of the original image, calculating the LBP value of each pixel in the image, and describing the local texture information of the image based on the LBP value.

[0059] In the above-mentioned embodiment, LBP is an operator for describing the local features of an image. It has grayscale invariance and rotation invariance, so that it can still extract stable features after illumination changes or image rotation, and can capture the local texture information of the image. It is very effective for tasks such as expression recognition that rely on facial texture changes. By calculating the LBP value of each pixel, an LBP feature map can be generated. Among them, LBP (Local Binary Patterns) is a feature extraction method widely used in the field of image processing, and is particularly suitable for describing the local texture information of an image. LBP can provide texture information about the neighborhood of each pixel in the image, and generates features by comparing the grayscale values ​​of each pixel with the pixels around it. This method is widely used because it is simple, fast and can effectively express image texture features, especially in the fields of face recognition and texture classification.

[0060] The basic idea of ​​LBP is to use the grayscale value of the central pixel as the threshold value in the 3x3 neighborhood of a pixel, and compare the grayscale values ​​of the surrounding 8 pixels with it. If the surrounding pixel value is greater than the central pixel value, the pixel position is marked as 1, otherwise it is 0, thus obtaining an 8-bit binary number, namely LBP value 6. This value reflects the texture information of the area around the central pixel, and different LBP values ​​correspond to different texture features.

[0061] In order to enhance the extraction of feature information, innovative methods can be used to combine the original image with the LBP feature image. One possible method is to perform LBP feature extraction on the original image, generate a texture feature image, and then fuse or combine this texture feature image with the original image. For example, this can be achieved through simple superposition, averaging, or other more complex image fusion techniques. The fused image not only retains the detail information of the original image, but also adds a description of the texture features, thereby providing richer information when performing image analysis.

[0062] In addition, there are many variants of LBP, such as circular LBP (using circular neighborhoods instead of square neighborhoods), rotation-invariant LBP (rotation-invariant LBP is achieved by rotating the LBP pattern), and uniform pattern LBP (Uniform Pattern LBP, which reduces the number of pattern types and improves statistics by limiting the number of jumps in the binary sequence). These variants can provide more adaptive feature descriptions in different application scenarios.

[0063] In practical applications, LBP feature extraction can be implemented through the Python programming language, such as using the local_binary_pattern function in the skimage library or the Open-CV library to calculate the LBP feature. The extracted LBP features can be further used to train classifiers, perform image matching, or serve as input for other image analysis tasks.

[0064] In short, LBP feature extraction provides an effective means for image texture analysis. By innovatively combining the original image with the texture feature image, the expressiveness of the feature can be further improved and the performance of image processing tasks can be enhanced.

[0065] In one embodiment, step S102 fuses the original image with the texture image to enhance the texture information of the original image. The image is then input into a residual network for feature extraction; the residual network uses a ResNet18 residual network; wherein ResNet18 is a deep convolutional neural network, i.e., a residual network; residual learning is introduced to solve the gradient vanishing or exploding problem that occurs during the training of a deep neural network.

[0066] In one embodiment, step S103 performs expression recognition on the extracted feature image based on a recognition network, and obtaining the expression recognition result includes:

[0067] The feature image obtained by the convolution operation is input into the recognition network established by combining the residual network and the predefined channel attention mechanism;

[0068] Use global average pooling and global maximum pooling to obtain the spatial features of each layer channel of the feature image respectively;

[0069] By mapping adaptive one-dimensional convolution, the information interaction between each layer channel and the adjacent layer channel is performed, so that the spatial features output by each layer channel and the adjacent layer channel are added to obtain the spatial feature information of the feature image;

[0070] The channel attention weight is calculated based on the extracted spatial feature information; the channel attention weight is trained to obtain the expression recognition result.

[0071] In the above embodiment, the recognition network established by combining the residual network and the predefined channel attention mechanism, that is, the ResNet18 residual network combined with the attention mechanism, is as follows: Figure 2 As shown. The recognition network constructed is composed of residual blocks; each residual block contains two convolutional layers and a fully connected layer; so that the input can be directly added to the output of the residual block, which helps the network to better learn the identity mapping during the training process, thereby making it easier to optimize the deep network. In each residual block, a channel attention module is combined and placed after the convolutional layer of ResNet to calculate the channel weights using the feature maps extracted by the convolutional layer, and these weights are applied to the feature maps. Each residual block can adaptively adjust the weights between its channels according to the characteristics of the input data, thereby improving the performance of the network model.

[0072] Specifically, the fused texture image and the original image are convolved through a convolution layer to obtain a feature image; the residual block has a predefined channel attention mechanism, which is placed after the convolution layer of the residual network, and the channel attention weight is calculated using the feature image obtained through the convolution operation of the convolution layer, and the channel attention weight is applied to the feature image to obtain a weighted feature map, and the output of the residual block is formed by adaptively adjusting the attention weight.

[0073] In the above embodiment, the predefined channel attention mechanism is improved based on ECA attention, as follows: Figure 3As shown in the figure. The input feature image is processed using global average pooling and global maximum pooling to extract richer spatial feature information. The channels of each layer are allowed to interact with the channels of the adjacent layers through adaptive one-dimensional convolution and share weights. The features of the two outputs are added and fused and normalized by the Sigmoid function. At this time, the attention weight vector in the channel domain dimension direction can be obtained. Finally, the calculated channel weight is multiplied by the original feature map to obtain the weighted feature map. This process is a channel-level re-weighting of the feature map, which highlights the features of important channels and suppresses the features of unimportant channels.

[0074] Optionally, calculating a channel attention weight according to the extracted spatial feature information; the channel attention weight is trained to obtain an expression recognition result, including:

[0075] The spatial feature information is normalized by a Sigmoid function to obtain a channel attention weight in a channel domain;

[0076] Applying the channel attention weight to the feature image, and obtaining a weighted feature map by multiplying the channel attention weight and the feature image;

[0077] The weighted feature map is converted into a one-dimensional vector through global average pooling, and the classification output is performed using a fully connected layer, and the probability distribution of each category is generated based on the SoftMax activation function;

[0078] Obtain the driver's expression image set and divide it into a training set and a test set; input the test set into the recognition network to obtain the probability distribution of the true label;

[0079] The cross entropy loss is defined as the difference between the probability distribution of each category and the probability distribution of the true label;

[0080] With the goal of minimizing the cross entropy loss, the channel attention weight is trained based on a preset learning rate until a channel attention weight that meets the goal after training is obtained, and the optimizer Adam is used to update the parameters of the recognition network;

[0081] The test set is input into the updated recognition network, and the trained channel attention weights are loaded to obtain the recognition accuracy.

[0082] In the above embodiment, the cross entropy loss function is one of the commonly used loss functions in machine learning and deep learning, especially in classification problems. It is used to measure the difference between the probability distribution predicted by the model and the probability distribution of the true label. The smaller the cross entropy loss, the closer the model's prediction result is to the actual label, and the better the model's performance.

[0083] Optionally, in the network, downsampling is performed in four stages, and the number of channels of the feature image is set to 64, 128, 256, and 512 respectively, and the size of the feature map is reduced and the perception is increased. Finally, the feature map is globally averaged pooled and converted into a one-dimensional vector. A fully connected layer with a size of 1000 is used for classification output. The softmax activation function is used to generate a probability distribution of 1000 categories.

[0084] Optionally, the acquiring the driver expression image set and dividing the image set into a training set and a test set comprises:

[0085] Acquire a driver expression image set; wherein the driver expression image set includes driving scene image data and driver expression image data;

[0086] Identifying a face region of the driver expression image set, and after cropping the face region, dividing the images corresponding to the face region into a training set and a test set based on a preset division ratio;

[0087] The image data in the training set is resized to match the image size of the recognition network input.

[0088] Optionally, the detailed training process is given below:

[0089] 1. The KMU-FED driver expression dataset includes 1,106 images of 6 expressions in driving scenarios, namely anger, fear, contempt, happiness, sadness, and surprise. The face area is obtained by face recognition and cropped, and the data is divided into training set and test set with a division ratio of 7:3.

[0090] The training set is resized to 224×224 to match the network input data augmentation (such as rotation, cropping, flipping, etc.) and normalization to preprocess the data.

[0091] 2. The loss function uses cross entropy loss to measure the difference between the model prediction and the true label. The optimizer selects Adam to update the network parameters. The learning rate is set to 0.1, and the learning rate is ×0.3 for every 20 rounds of training. The batch size is set to 64. When the network model completes 200 training cycles, the training is terminated and the training model is kept.

[0092] 3. Input the test set into the network and load the trained weights to obtain the recognition accuracy.

[0093] For the method steps disclosed in the above embodiments, the method steps are described as a series of action combinations for the purpose of simple description, but those skilled in the art should know that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0094] Based on the same inventive concept, the embodiment of the present application also provides an expression recognition system for implementing the above-mentioned method for driver expression recognition combining attention mechanism and residual network. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the driver expression recognition system combining attention mechanism and residual network provided below can be found in the above limitations on the above method for driver expression recognition combining attention mechanism and residual network, which will not be repeated here. It can be understood that the above description of its method is also applicable to the description of the system.

[0095] In one embodiment, a driver expression recognition system combining an attention mechanism and a residual network is also provided. The embodiments of the present invention are described below with reference to the accompanying drawings. Figure 4 As shown, the system specifically includes:

[0096] The texture feature generation module 210 is used to generate a texture image by extracting local texture of the original image;

[0097] The information extraction module 220 is used to fuse the original image with the texture image, input them into the residual network, extract features through convolution operation, and introduce a predefined channel attention mechanism into the residual network to construct a recognition network;

[0098] The recognition module 230 is used to perform expression recognition on the extracted feature image based on the recognition network to obtain an expression recognition result.

[0099] It is worth noting that although only some basic functional modules are disclosed in the embodiments of the present invention, it does not mean that the composition of the present system is limited to the above basic functional modules. On the contrary, what the present embodiment wants to express is that on the basis of the above basic functional modules, those skilled in the art can arbitrarily add one or more functional modules in combination with the prior art to form an infinite number of embodiments or technical solutions. In other words, the present system is open rather than closed. Just because the present embodiment only discloses individual basic functional modules, it cannot be considered that the protection scope of the claims of the present invention is limited to the disclosed basic functional modules. At the same time, for the convenience of description, the above devices are described in terms of functions and are described separately in various units and modules. Of course, when implementing the present invention, the functions of each unit and module can be implemented in the same or one or more software and / or hardware.

[0100] like Figure 5 As shown, the present invention also provides an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of a driver expression recognition method combining an attention mechanism and a residual network.

[0101] Figure 5 Schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 5 The electronic device provided in the embodiment of the present invention includes: one or more processors 310 and a storage device 320; the processor 310 in the electronic device may be one or more, Figure 3 A processor 310 is taken as an example; the storage device 320 is used to store one or more programs; the one or more programs are executed by the one or more processors 310, so that the one or more processors 310 implement the method for processing node setting conflicts as described in any one of the embodiments of the present invention.

[0102] The electronic device may further include: an input device 330 and an output device 340 .

[0103] The processor 310, storage device 320, input device 330 and output device 340 in the electronic device can be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0104] The storage device 320 in the electronic device is a computer-readable storage medium that can be used to store one or more programs, which can be software programs, computer executable programs and modules, such as program instructions / modules corresponding to the driver expression recognition method combined with the attention mechanism and the residual network provided in the embodiment of the present invention. The processor 310 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the storage device 320, that is, the method for processing node setting conflicts in the above method embodiment is implemented.

[0105] The storage device 320 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the storage device 320 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the storage device 320 may further include a memory remotely arranged relative to the processor 310, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0106] The input device 330 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device. The output device 340 may include a display device such as a display screen.

[0107] The present invention also provides a computer-readable storage medium storing a computer program executable by an electronic device. When the computer program runs on the electronic device, the electronic device executes the steps of a driver expression recognition method combining an attention mechanism and a residual network.

[0108] Specifically, the computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A driver expression recognition method combining attention mechanism and residual network, It is characterized in that The method comprises: Generate a texture image by extracting the local texture of the original image; The original image is fused with the texture image and input into the residual network to extract features through convolution operation. A predefined channel attention mechanism is introduced into the residual network to construct a recognition network. Based on the recognition network, expression recognition is performed on the extracted feature image to obtain the expression recognition result.

2. The method according to claim 1, It is characterized in that The method of generating a texture image by extracting local texture of the original image includes: collecting image data from the vehicle-mounted terminal, selecting several images as original images for LBP local binary processing, and generating a texture image.

3. The method according to claim 2, It is characterized in that The selecting of several images as original images for LBP local binary processing includes: using LBP as an operator to extract local texture features of the original image, calculating the LBP value of each pixel in the image, and describing the local texture information of the image based on the LBP value.

4. The method according to claim 1, It is characterized in that The recognition network is composed of residual blocks; each residual block contains two convolutional layers and one fully connected layer; The fused texture image and the original image are convolved through the convolution layer to obtain a feature image; the residual block has a predefined channel attention mechanism, which is placed after the convolution layer of the residual network. The channel attention weight is calculated using the feature image obtained through the convolution operation of the convolution layer, and the channel attention weight is applied to the feature image to obtain a weighted feature map. The output of the residual block is formed by adaptively adjusting the attention weight.

5. The method according to claim 4, It is characterized in that The method of performing expression recognition on the extracted feature image based on the recognition network to obtain the expression recognition result includes: The feature image obtained by the convolution operation is input into the recognition network established by combining the residual network and the predefined channel attention mechanism; Use global average pooling and global maximum pooling to obtain the spatial features of each layer channel of the feature image respectively; By mapping adaptive one-dimensional convolution, the information interaction between each layer channel and the adjacent layer channel is performed, so that the spatial features output by each layer channel and the adjacent layer channel are added to obtain the spatial feature information of the feature image; The channel attention weight is calculated based on the extracted spatial feature information; the channel attention weight is trained to obtain the expression recognition result.

6. The method according to claim 5, It is characterized in that The channel attention weight is calculated according to the extracted spatial feature information; The channel attention weights are trained to obtain expression recognition results including: The spatial feature information is normalized by a Sigmoid function to obtain a channel attention weight in a channel domain; Applying the channel attention weight to the feature image, and obtaining a weighted feature map by multiplying the channel attention weight and the feature image; The weighted feature map is converted into a one-dimensional vector through global average pooling, and the classification output is performed using a fully connected layer, and the probability distribution of each category is generated based on the SoftMax activation function; Obtain the driver's expression image set and divide it into a training set and a test set; input the test set into the recognition network to obtain the probability distribution of the true label; The cross entropy loss is defined as the difference between the probability distribution of each category and the probability distribution of the true label; With the goal of minimizing the cross entropy loss, the channel attention weight is trained based on a preset learning rate until a channel attention weight that meets the goal after training is obtained, and the optimizer Adam is used to update the parameters of the recognition network; The test set is input into the updated recognition network, and the trained channel attention weights are loaded to obtain the recognition accuracy.

7. The method according to claim 6, It is characterized in that The obtaining of the driver expression image set and dividing the image set into a training set and a test set comprises: Acquire a driver expression image set; wherein the driver expression image set includes driving scene image data and driver expression image data; Identifying a face region of the driver expression image set, and after cropping the face region, dividing the images corresponding to the face region into a training set and a test set based on a preset division ratio; The image data in the training set is resized to match the image size of the recognition network input.

8. A driver expression recognition system combining attention mechanism and residual network, It is characterized in that The system comprises: A texture feature generation module is used to generate a texture image by extracting local texture of the original image; The information extraction module is used to fuse the original image with the texture image, input it into the residual network, and extract features through convolution operations; and introduce a predefined channel attention mechanism into the residual network to build a recognition network; The recognition module is used to perform expression recognition on the extracted feature image based on the recognition network to obtain the expression recognition result.

9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the method steps described in any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory and a processor, It is characterized in that A computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the method steps as claimed in any one of claims 1 to 7.

Images