A method for constructing a micro-expression detection model and application thereof

Abstract

The application discloses a micro-expression detection model construction method and application, and belongs to the technical field of integrated circuits; the application considers that the marginal benefit of convolution depth is obvious, the number of convolution layers is set to be less than 16, the number of convolution layer parameters is greatly reduced, and therefore the calculation efficiency of the model is improved; meanwhile, the application adds a GAP layer before the fully connected layer, respectively performs average pooling operations on micro-expression features under each channel of the output of the last convolution layer, can greatly relieve the parameter quantity of the fully connected layer, avoids overfitting phenomenon caused by too many parameters, and therefore improves the accuracy of the model. In addition, the input and output channel numbers of each convolution layer in the application are the same, so that the information quantity and the operation quantity are further balanced, hardware resource reuse is more convenient, the application is suitable for SOC architecture, and is easy to integrate in embedded devices. Based on this, the application can detect micro-expression in real time, efficiently and accurately.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of integrated circuits, and more particularly relates to a micro-expression detection model construction method and application. BACKGROUND

[0002] Micro-expression is a facial expression that flashes in an instant and lasts for 1 / 25s, which can reveal the true emotions and moods of a person, and is often applied to identify lies and non-lies, and judge the true emotional state of a person, and can be further applied in the fields of depression prevention and detection, psychology, medicine, etc.

[0003] The micro-expression of a person is mainly divided into 7 categories, including happy, sad, scared, angry, disgusted, surprised and contempt, each of which has relatively obvious facial features. Among them, happy: facial movements include: mouth corners are raised, cheekbones are raised, eyelids are contracted, and "fish tail wrinkles" are formed at the tail of the eyes; sad: facial features include squinting, frowning eyebrows, mouth corners are pulled down, and chin is raised or tightened; scared: when scared, the mouth and eyes are open, the eyebrows are raised, and the nostrils are flared; angry: the eyebrows are lowered, the forehead is wrinkled, and the eyelids and lips are tense; disgusted: the disgusted expression includes snorting, lifting the upper lip, lowering the eyebrows, and squinting. Surprised: when surprised, the lower jaw is lowered, the lips and mouth are relaxed, the eyes are wide open, and the eyelids and eyebrows are slightly raised. Contempt: the side of the contemptuous mouth is raised, making a sneer or smug smile.

[0004] In order to achieve high accuracy, the existing micro-expression detection method is to establish a relatively complex deep neural network such as VGG network to detect micro-expression; but because the micro-expression lasts for a short time and the expression changes quickly, there are certain requirements for accuracy, and there are also high requirements for detection efficiency and real-time performance. The above method has high accuracy, but because the network structure is complex, the parameter quantity is large, the processing speed is slow, the detection time required is long, the detection efficiency is low, and the frame rate and delay requirements cannot be met, the micro-expression cannot be detected in real time, and it cannot be applied to SOC architecture integrated in embedded devices; therefore, designing a light, efficient, accurate and embedded device integrated micro-expression detection model is a technical problem to be solved in the field of micro-expression recognition and detection. SUMMARY

[0005] In view of the above defects or improvement needs of the prior art, the present application provides a micro-expression detection model construction method and application to solve the technical problem that the existing system cannot detect micro-expression in real time, efficiently and accurately.

[0006] In order to achieve the above purpose, in a first aspect, the present application provides a micro-expression detection model construction method, comprising:

[0007] S1, a micro-expression detection model is built; the micro-expression detection model comprises:

[0008] N cascaded convolution layers for extracting micro-expression features of the input image; the number of channels of each convolution layer is the same; N < 16;

[0009] a GAP layer for performing average pooling operation on the micro-expression features under each channel output by the last convolution layer;

[0010] a fully connected layer for mapping the features after the average pooling operation, thereby obtaining the micro-expression type of the input image;

[0011] S2, a pre-acquired training set is input into the micro-expression detection model for training; the training set comprises micro-expression images of different types and corresponding micro-expression type labels.

[0012] Further preferably, N is 8; the input and output channel numbers of each convolution layer are 32.

[0013] Further preferably, the convolution kernel sliding step length in the first N1 convolution layers is greater than the convolution kernel sliding step length in the last N-N1 convolution layers; 1 ≤ N1 < N.

[0014] In a second aspect, the present application provides a micro-expression detection method, comprising: inputting a micro-expression image to be detected into a micro-expression detection model to obtain the micro-expression type of the micro-expression image.

[0015] The micro-expression detection model is constructed by using the construction method of the micro-expression detection model provided in the first aspect of the present application.

[0016] In a third aspect, the present application provides a micro-expression detection system, comprising: a memory and a processor, the memory stores a computer program, and the processor executes the computer program to execute the micro-expression detection method provided in the second aspect of the present application.

[0017] In a fourth aspect, the present application provides an embedded SOC hardware system for micro-expression detection, comprising: a model parameter storage module, M parallel convolution operation circuits, and an output module; M ≥ 2.

[0018] The storage module is used for storing the micro-expression image to be detected, the parameters of the micro-expression detection model, and the operation results of the convolution operation circuit; the micro-expression detection model is constructed by using the construction method of the micro-expression detection model provided in the first aspect of the present application.

[0019] M parallel convolution operation circuits are used to implement convolution operation in M channels in parallel during each layer of convolution operation; the convolution operation circuit is used to read the micro-expression image and the convolution kernel parameters corresponding to the channel of the first convolution layer to perform convolution operation, and store the obtained results; when the i-th layer of convolution operation is performed, the convolution operation results corresponding to the channel obtained by the last convolution operation and the convolution kernel parameters corresponding to the channel of the i-th convolution layer are read to perform convolution operation, and the obtained results are stored; i = 2, 3, …, N; N < 16;

[0020] The output module is used to read the results obtained by the last layer of convolution operation and the parameters of the full connection layer; the average value of the convolution operation results of each channel obtained by the last layer of convolution operation is calculated to obtain a feature vector with a dimension of M; and full connection calculation is performed on the feature vector and the parameters of the full connection layer, so that the micro-expression type of the micro-expression image is obtained.

[0021] Further preferably, the storage module is used to convert the parameters of the micro-expression detection model from floating-point numbers to fixed-point numbers for storage, so as to improve the operation efficiency.

[0022] Further preferably, the storage module includes M embedded block random memories, which are used to store the convolution operation results in M channels respectively; when the i-th layer of convolution operation is performed, the M parallel convolution operation circuits simultaneously read the convolution operation results in M channels obtained by the last convolution operation from the corresponding block random memories, so as to ensure the continuity of parallel operation.

[0023] Further preferably, the embedded SOC hardware system further includes a camera and an image preprocessing module; the image preprocessing module is used to perform grayscale on each frame of micro-expression image collected by the camera;

[0024] The storage module includes a dual-port memory connected to the output end of the image preprocessing module, which is used to simultaneously read the current frame of micro-expression image and store the next frame of micro-expression image, so as to complete the ping-pong operation of storage, and ensure that the next frame of micro-expression image does not affect the processing of the current frame of micro-expression image.

[0025] Further preferably, the embedded SOC hardware system further includes a display module, which is used to display the micro-expression image in real time.

[0026] Further preferably, the embedded SOC hardware system further includes an upper computer training module, which is used to execute the construction method of the micro-expression detection model provided in the first aspect of the application, obtain the micro-expression detection model, and store the parameters of the micro-expression detection model in the storage module.

[0027] In general, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0028] 1. The present application provides a micro-expression detection model construction method, which constructs a lightweight, efficient, accurate and SOC architecture applicable micro-expression detection model. Specifically, considering the obvious marginal benefit of convolution depth, the number of convolution layers is set to less than 16 layers, which greatly reduces the number of convolution layer parameters, thereby improving the calculation efficiency of the model. At the same time, the present application adds a GAP layer before the fully connected layer, which can greatly alleviate the parameter quantity of the fully connected layer and avoid overfitting phenomenon caused by too many parameters, thereby improving the accuracy of the model. In addition, the input and output channel numbers of each convolution layer in the present application are the same, which further balances the information quantity and computational complexity, and is more convenient for hardware resource reuse, suitable for SOC architecture and easy to integrate in embedded devices. Based on this, the present application can efficiently and accurately detect micro-expression in real time.

[0029] 2. Further, the micro-expression detection model construction method provided by the present application, the convolution kernel sliding step in the first N1 convolution layer is greater than the convolution kernel sliding step in the last N-N1 convolution layer, which can quickly reduce the feature map size and expand the receptive field, further improve the accuracy of detection under the premise of improving the calculation efficiency.

[0030] 3. The present application provides an embedded SOC hardware system for micro-expression detection, which is used to realize the function of the micro-expression detection model provided by the present application. Thanks to the lightweight of the micro-expression detection model, the multi-channel convolution operation circuit composed of M parallel convolution operation circuits can be used as the basic unit of the convolution layer, and the convolution operation in M channels can be realized in parallel during each convolution operation process, which greatly improves the convolution operation speed and can process micro-expression with a duration of ms level, thereby achieving the purpose of real-time, efficient and accurate detection of micro-expression. In this process, a multi-channel convolution operation circuit can realize the function of multiple cascaded convolution layers, greatly saving resource overhead. In addition, the present application can integrate the storage module used in the system on a chip, which can be applied in embedded devices.

[0031] 4. Further, the embedded SOC hardware system provided by the present application stores the parameters of the micro-expression detection model in the model parameter storage unit in the form of fixed-point numbers, which can further reduce the parameter quantity under the premise of meeting the detection accuracy, thereby further improving the calculation efficiency.

[0032] 5. Further, the embedded SOC hardware system provided by the present application adopts an independent dual-port memory to realize reading of a current frame of micro-expression image and storage of a next frame of micro-expression image at the same time, so that the image data stored in the next frame will not affect the processing of the current frame. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 A structure diagram of a micro-expression detection model provided for Embodiment 1 of the present application;

[0034] Figure 2 A structure diagram of an embedded SOC hardware system for micro-expression detection provided for Embodiment 4 of the present application;

[0035] Figure 3 A convolution operation principle diagram provided for Embodiment 4 of the present application;

[0036] Figure 4 A multi-channel convolution operation principle block diagram provided for Embodiment 4 of the present application;

[0037] Figure 5 A principle block diagram of an output module provided for Embodiment 4 of the present application;

[0038] Figure 6 A VGA display process block diagram provided for Embodiment 4 of the present application. DETAILED DESCRIPTION

[0039] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as there is no conflict.

[0040] Embodiment 1,

[0041] A method for constructing a micro-expression detection model, comprising:

[0042] S1, building a micro-expression detection model;

[0043] The micro-expression detection model comprises:

[0044] N cascaded convolution layers for extracting micro-expression features of an input image; the number of channels of each convolution layer is the same; N < 16; preferably, N is 8, and the number of channels of each convolution layer is set to 32;

[0045] a GAP layer for performing average pooling operation on the micro-expression features under each channel output by the last convolution layer;

[0046] a full connection layer, configured to map the features after the average pooling operation, so as to obtain the micro-expression type of the input image;

[0047] S2, inputting the pre-acquired training set into the micro-expression detection model for training; the training set comprises micro-expression images of different types and corresponding micro-expression type labels.

[0048] In an optional embodiment, according to the actual application scenario, the size of the collected micro-expression data set picture is adjusted when training the neural network, the 100*100 size picture is interpolated without affecting the picture quality, and the picture is expanded to 200*200 for training, which is consistent with the size of the picture collected by the camera in the actual micro-expression recognition system, and the accuracy in the actual application scenario is ensured.

[0049] In an optional embodiment, the existing international general micro-expression data set is expanded, and some micro-expression images of Chinese people are added, so that the training effect is more conducive to the prediction of Chinese people.

[0050] Specifically, as Figure 1 As shown in FIG. 1, which is a schematic diagram of a micro-expression detection model in an optional embodiment, the parameters of the corresponding convolutional layers are shown in Table 1; wherein the number of convolutional layers is set to 12; the number of convolutional channels of each convolutional layer is set to 32; and the size of the convolutional kernel of each convolutional layer is set to 3*3.

[0051] Table 1

[0052] CONV1 CONV2 CONV3 CONV4 CONV5 CONV6 CONV7 CONV8 input image length 201 101 51 26 26 26 26 26 input image width 201 101 51 26 26 26 26 26 input channel number 1 32 32 32 32 32 32 32 kernel size 3 3 3 3 3 3 3 3 kernel number 32 32 32 32 32 32 32 32 kernel channel number 1 32 32 32 32 32 32 32 padding 1 1 1 1 1 1 1 1 stride 2 2 2 1 1 1 1 1 output image length 101 51 26 26 26 26 26 26 output image width 101 51 26 26 26 26 26 26 output channel number 32 32 32 32 32 32 32 32

[0053] Further, the convolution kernel sliding step length in the first N1 convolutional layers is greater than the convolution kernel sliding step length in the last N-N1 convolutional layers; 1≤N1<N. In the above embodiment, the convolution kernel sliding step length in the last 5 convolutional layers is set to 1, and the convolution kernel sliding step length in the first three convolutional layers is set to 2, so as to quickly reduce the feature map size and expand the receptive field, which can quickly reduce the image size through convolution operation under the premise of ensuring convolution processing, and the size of each image in the last five layers is already small, so the sliding step length is set to 1 to perform sufficient convolution operation. Based on this, the application can further improve the detection accuracy under the premise of improving the calculation efficiency.

[0054] Further, the micro-expression picture mainly identifies the texture features of the expression, and does not need a color picture, and the RGB image is not easy to process compared with the grayscale image, because not only the RGB three channels need to be processed, but also the amount of parameters to be processed is greatly increased. Preferably, the micro-expression images in the input image and the training set are grayscale images.

[0055] It should be noted that the micro-expression detection model provided by the present application is light, efficient, and has high accuracy. The model has simple structure, small parameter quantity, high calculation efficiency, and good accuracy. Specifically, considering that the marginal benefit of convolution depth is obvious, the number of convolution layers is set to less than 16 layers in the present application, which greatly reduces the number of convolution layer parameters, thereby improving the calculation efficiency of the model. At the same time, the GAP layer is added before the fully connected layer, and the micro-expression features under each channel output by the last convolution layer are respectively subjected to average pooling operation, which can greatly alleviate the parameter quantity of the fully connected layer and avoid overfitting phenomenon caused by too many parameters, thereby improving the accuracy of the model. In addition, the input and output channel numbers of each convolution layer in the present application are the same, which further balances the information quantity and operation quantity, and is more convenient for reuse of hardware resources, suitable for SOC architecture, and easy to integrate in embedded devices.

[0056] In order to further illustrate the performance of the micro-expression detection model provided by the present application, the performance of the micro-expression detection model provided by the present application is compared with the performance of the deep neural network model VGG-11 in the art under the same general micro-expression dataset as follows:

[0057] Table 2

[0058] model VGG-16 expression detection model depth 16 8 parameter quantity 130M <1M portability weak strong training parameter sensitivity high low extensiveness weak strong universal set accuracy 80％ 81.7％

[0059] As can be seen from the table, under the same dataset, the accuracy of the expression detection model provided by the present application can reach 81.7%, while the accuracy of the traditional VGG-16 can only reach 80%. In addition, the parameter quantity of the expression detection model provided by the present application is less than 1M, which is much smaller than the 130M of VGG-11. It can be seen that the present application provides a micro-expression detection model which is light, efficient, accurate, and suitable for SOC architecture, and can detect micro-expression in real time, efficiently and accurately.

[0060] Further, in an optional embodiment, the convolution parameters in the trained micro-expression detection model are clipped to a fixed point number to further improve the generalization ability and detection accuracy of the model. The clipped model is used to detect the micro-expression verification set again, and the accuracy is 81.6%, which is basically not reduced.

[0061] Embodiment 2,

[0062] A micro-expression detection method, comprising: inputting a micro-expression image to be detected into a micro-expression detection model to obtain a micro-expression type of the micro-expression image.

[0063] The micro-expression detection model is constructed by using the construction method of the micro-expression detection model provided by the present application.

[0064] It should be noted that when the micro-expression detection model is a model trained using a grayscale image, the micro-expression image input here is also a grayscale image.

[0065] The related technical solution is the same as that of Embodiment 1, and will not be repeated here.

[0066] Embodiment 3,

[0067] A micro-expression detection system includes a memory and a processor, the memory stores a computer program, and the processor executes the computer program to execute the micro-expression detection method provided in Embodiment 2 of the application.

[0068] The related technical solution is the same as that of Embodiment 2, and will not be repeated here.

[0069] Embodiment 4,

[0070] An embedded SOC hardware system for micro-expression detection includes a model parameter storage module, M parallel convolution operation circuits, and an output module; where M >= 2, the same as the number of channels in the micro-expression detection model; in this embodiment, M is 32;

[0071] The storage module is used to store the micro-expression image to be detected, the parameters of the micro-expression detection model, and the operation results of the convolution operation circuit; where the micro-expression detection model is constructed using the construction method of the micro-expression detection model provided in Embodiment 1 of the application; the related technical solution is the same as that of Embodiment 1, and will not be repeated here;

[0072] The M parallel convolution operation circuits are used to perform convolution operations in M channels in parallel during each convolution operation; the convolution operation circuit is used to read the micro-expression image and the convolution kernel parameters in the corresponding channel of the first convolution layer to perform convolution operation, and store the obtained results; during the i-th convolution operation, read the convolution operation results in the corresponding channel obtained by the previous convolution operation and the convolution kernel parameters in the corresponding channel of the i-th convolution layer to perform convolution operation, and store the obtained results; i = 2, 3, …, N; N < 16; the number of N is the same as the number of convolution layers in the micro-expression detection model;

[0073] The output module is used to read the results obtained by the last convolution operation and the parameters of the fully connected layer; calculate the average value of the convolution operation results in each channel obtained by the last convolution operation to obtain a feature vector with a dimension of M; and perform fully connected calculation on the feature vector and the parameters of the fully connected layer to obtain the micro-expression type of the input image.

[0074] In an optional embodiment, the storage module includes a model parameter storage unit configured to store the parameters of the micro-expression detection model in fixed-point numbers from floating-point numbers to improve the operation efficiency. In a preferred embodiment, the model parameter storage unit includes a plurality of block RAMs configured to store the convolution kernel parameters of each channel of each convolution layer and the fully connected layer parameters of the fully connected layer. It should be noted that the block RAM has a high access speed, can be accessed in parallel without interference, and can be integrated on a chip with a circuit, thereby ensuring the storage and reading speed of the convolution operation results.

[0075] In an optional embodiment, the storage module includes an intermediate result storage unit, specifically, M embedded block RAMs configured to store the convolution operation results of the M channels. When performing the i-th layer convolution operation, the M parallel convolution operation circuits simultaneously read the convolution operation results of the M channels obtained by the previous layer convolution operation from the corresponding block RAMs to ensure the continuity of the parallel operation.

[0076] In an optional embodiment, the embedded SOC hardware system described above further includes a camera and an image preprocessing module. The image preprocessing module is configured to perform grayscale on each frame of micro-expression image collected by the camera. Further, the storage module includes an image storage unit, specifically, a dual-port memory (preferably, a dual-port DDR memory) connected to the output end of the image preprocessing module, configured to simultaneously read the current frame of micro-expression image and store the next frame of micro-expression image, thereby completing the ping-pong operation of the stored micro-expression image and ensuring that the next frame of micro-expression image does not affect the processing of the current frame of micro-expression image.

[0077] In an optional embodiment, the embedded SOC hardware system described above further includes a display module configured to display the micro-expression image in real time.

[0078] In an optional embodiment, the embedded SOC hardware system described above further includes an upper computer training module configured to perform the construction method of the micro-expression detection model provided in Embodiment 1 to obtain the micro-expression detection model and store the parameters of the micro-expression detection model in the storage module.

[0079] To further illustrate the embedded SOC hardware system for micro-expression detection provided in the present embodiment, a specific embodiment is described in detail as follows:

[0080] As Figure 2As shown, the embedded SOC hardware system provided by the present embodiment includes a camera, an image preprocessing module, an image storage unit, a multi-channel convolution operation circuit (M parallel convolution operation circuits, M being the number of channels), a model parameter storage unit, an intermediate result storage unit, an output module, a hard disk, and an upper computer training module.

[0081] The upper computer training module is configured to execute the construction method of the micro-expression detection model provided in Embodiment 1 of the present application, obtain the micro-expression detection model, and store the parameters of the micro-expression detection model in the model parameter storage unit. In the present embodiment, the obtained parameters are extracted and stored in the model parameter storage unit in the form of 16-bit or 32-bit fixed-point numbers. Preferably, the model parameters are converted into 16-bit fixed-point numbers for storage, which has less precision loss (less than 1%) compared to 32-bit floating point, but the parameter amount is reduced by 50%.

[0082] When the embedded SOC hardware system starts to work, the camera is turned on, and the sampling rate of the camera is 60 frames. The size of the collected image meets the requirement of the duration of 1 / 25s of the expression required by the micro-expression. In the present embodiment, the camera ov_5640 (but not limited to this camera) is used. The camera ov_5640 collects images with a size of 200*200 at a sampling rate of 60 frames (the image size is increased as much as possible under the premise of fitting the hardware overhead). The format of the image is a 16-bit RGB565 color image. When a picture is collected, the camera sends the data RGB image data to the image preprocessing module. The image preprocessing module converts the RGB image data into gray image data with a format of 8 bits. The picture size is still 200*200. The image preprocessing module is connected with a special memory interface circuit required by the image storage unit (in the present embodiment, a DDR memory is used) and writes the processed image data into the image storage unit for storage. The image storage unit includes a separate DDR memory and a special memory interface circuit required by the DDR memory. The size of the DDR memory is not less than 512MB, and it is a dual-port memory that can simultaneously read and write at two ports.

[0083] When image processing is performed, first layer convolution is performed, the multi-channel convolution operation circuit reads a 200*200 grayscale image from the image storage unit (DDR memory) for processing, and simultaneously reads the corresponding convolution operation parameters (i.e., the corresponding convolution kernel parameters) of the first layer from the model parameter storage unit for first layer convolution operation processing, to obtain the results after multi-channel convolution operation; the convolution operation results of different channels are stored in different block Rams of the intermediate result storage unit, until all pixel points of the picture are calculated, and the next layer of convolution operation is performed. Specifically, in the first layer convolution, a 200*200 grayscale picture and a plurality of convolution kernels corresponding to the first layer simultaneously perform convolution in the multi-channel convolution operation circuit to obtain a plurality of pictures after convolution operation, which are respectively stored in a plurality of block Rams of the intermediate result storage unit. As shown in Figure 3 The convolution operation principle schematic diagram provided by the embodiment is shown, taking a 3*3 size convolution kernel as an example, when convolution operation is performed, the nine convolution parameters corresponding to the 3*3 size convolution kernel are respectively multiplied with the pixel points corresponding to the image, and then the nine results after multiplication are added, and then one convolution operation of the 3*3 convolution kernel is completed. It should be noted that the multi-channel parallel convolution operation circuit simultaneously performs convolution operation, and the processing rate of one frame of image is about 10ms, which can meet the 60 frame sampling rate of the camera.

[0084] After the first layer convolution layer is calculated, the second layer convolution is calculated, as shown in Figure 4 The multi-channel convolution operation circuit simultaneously reads the image data stored in the different block Rams of the intermediate result storage unit to obtain the results stored by the first layer convolution operation, and simultaneously reads the corresponding convolution kernel parameters of the second layer from the model parameter storage unit and puts them into the plurality of channels of the multi-channel convolution operation circuit to simultaneously perform operation, and then all results of the second layer convolution operation can be obtained, and the multi-channel calculation results are stored in the different block Rams of the intermediate result storage unit until all pixel points of the layer are calculated; when all results of the second layer are calculated, the third layer convolution operation is started, and the data is taken out from the intermediate result storage unit and simultaneously calculated in the multi-channel convolution operation circuit, and the obtained results are stored in the intermediate result storage unit until all convolution layers are calculated, and the results are stored in the intermediate result storage unit. In this way, until all convolution layers are calculated.

[0085] After all convolution layers are calculated, the data of the intermediate result storage unit is sent to the result output module for final result calculation, as shown in Figure 5As shown, the output module first averages the data stored in all block rams to obtain a plurality of single average values; the average values are fully connected to calculate the probability of the current frame micro-expression image belonging to each micro-expression type by the full connection layer, and the micro-expression type corresponding to the maximum probability is expressed as a binary number and written into the hard disk to complete the detection of the current frame micro-expression image and send a processing completion signal to the display module. The display module starts to read the grayscale image data stored in the image storage unit and displays it on the screen. In this embodiment, the micro-expression types include 7 micro-expression types: happy, sad, scared, angry, disgusted, surprised and contemptuous, and the 7 micro-expression types correspond to binary 1 to 7.

[0086] It should be noted that the display module described above can adopt any display, and in this embodiment, a VGA display is adopted to display the image captured by the camera so that the face is in the screen and the correct micro-expression recognition result is obtained. Figure 6 As shown in the process block diagram of the VGA display provided by this embodiment, after the camera data acquisition is completed, the image is preprocessed, the processed grayscale image is stored in BANK1 of the DDR memory, and when the VGA display receives the processing completion signal, the image is read from BANK1 of the DDR memory and displayed on the VGA display. It is worth noting that the total time from the start of processing by the multi-channel convolution operation circuit to the display by the VGA display can be close to 60ms, i.e. close to the 60 frame sampling rate of the camera, so during the detection and display of the current frame micro-expression image, the next frame of micro-expression image captured by the camera may have already started to be processed and stored. To avoid data conflict, the next frame of micro-expression image is stored in BANK2 of the DDR memory, and the storage is alternated in this way to realize the ping-pong operation of storage. The use of independent dual-port DDR memory enables the current frame micro-expression image to be read while the next frame micro-expression image is stored, which can ensure that the image data stored in the next frame does not affect the processing of the current frame.

[0087] The block ram used in the main circuit and system is made on one chip, so that the system can be used in embedded devices, and the defect that micro-expression detection cannot be performed on embedded hardware systems is compensated. The peripherals such as display, camera and hard disk are used for human-computer interaction and result storage, which improves the integrity and usability of the entire system, and the peripherals used can be adjusted according to actual conditions.

[0088] The application provides a SOC hardware system capable of real-time micro-expression recognition, which can process micro-expression with a duration of ms level, has high processing speed, complete functions and can be further applied to embedded devices, and solves defects in the field to some extent and fills a blank in the field of real-time micro-expression detection.

[0089] Those skilled in the art can easily understand that the above description is only preferred embodiments of the present application, and is not used to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for constructing a micro-expression detection model, characterized in that, The method comprises the following steps: S1, constructing a micro-expression detection model; The micro-expression detection model comprises: N cascaded convolutional layers for extracting micro-expression features of the input image; the number of channels of each convolutional layer is the same; ; the sliding step of the convolution kernel in the first N1 convolutional layers is greater than the sliding step of the convolution kernel in the last N-N1 convolutional layers; ; a GAP layer, configured to perform average pooling operation on the micro-expression features under each channel of the output of the last convolution layer; a full connection layer, configured to map the features after the average pooling operation, so as to obtain the micro-expression type of the input image; S2, inputting a pre-acquired training set into the micro-expression detection model for training; the training set comprises micro-expression images of different types and corresponding micro-expression type labels; The M parallel convolution operation circuit is used to perform convolution operation in M channels in parallel during each layer convolution operation; the convolution operation circuit is used to read the micro-expression image and the convolution kernel parameter corresponding to the channel of the first convolution layer to perform convolution operation, and store the obtained result when performing the first layer convolution operation; read the convolution operation result corresponding to the channel obtained by the last layer convolution operation and the convolution kernel parameter corresponding to the channel of the i-th convolution layer to perform convolution operation, and store the obtained result when performing the i-th layer convolution operation; i=2, 3, …, N; read the result obtained by the last layer convolution operation and the full connection layer parameter when implementing the operation of the GAP layer and the full connection layer; calculate the average value of the convolution operation result of each channel obtained by the last layer convolution operation to obtain a feature vector with a dimension of M; and perform full connection calculation on the feature vector and the full connection layer parameter to obtain the micro-expression type of the micro-expression image. . 2.The method of claim 1, wherein, N is 8; the input and output channel numbers of each convolution layer are 32.

3. A micro-expression detection method, characterized in that, The method comprises the following steps: inputting a micro-expression image to be detected into the micro-expression detection model to obtain the micro-expression type of the micro-expression image; The micro-expression detection model is constructed by using the construction method of the micro-expression detection model in claim 1 or 2.

4. A micro-expression detection system characterized by, The method comprises the following steps: a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to execute the micro-expression detection method in claim 3.

5. An embedded SOC hardware system for micro-expression detection, characterized in that, The storage module is configured to store the micro-expression image to be detected, the parameters of the micro-expression detection model, and the operation result of the convolution operation circuit; the micro-expression detection model is constructed by using the construction method of the micro-expression detection model in claim 1 or 2. The model parameter storage module, the M parallel convolution operation circuits, and the output module; ; The output module is configured to read the result obtained by the last convolution operation and the parameters of the full connection layer; calculate the average value of the convolution operation result under each channel obtained by the last convolution operation to obtain a feature vector with a dimension of M; The M parallel convolution operation circuits are configured to implement convolution operations in M channels in parallel during each layer convolution operation; the convolution operation circuit is configured to, when performing the first layer convolution operation, read the micro-expression image and the convolution kernel parameters in the corresponding channel of the first convolution layer to perform convolution operation, and store the obtained result; when performing the i-th layer convolution operation, read the convolution operation result in the corresponding channel obtained by the (i-1)-th layer convolution operation and the convolution kernel parameters in the corresponding channel of the i-th convolution layer to perform convolution operation, and store the obtained result; and when performing the last layer convolution operation, read the convolution operation result in the corresponding channel obtained by the (i-1)-th layer convolution operation and the convolution kernel parameters in the corresponding channel of the i-th convolution layer to perform convolution operation, and store the obtained result. i i ;​​​ perform full connection calculation on the feature vector and the parameters of the full connection layer, so as to obtain the micro-expression type of the micro-expression image. The storage module is configured to convert the parameters of the micro-expression detection model from floating-point numbers to fixed-point numbers for storage.

6. The system of claim 5, wherein, Further comprising:

7. The hardware system of claim 5, wherein, The storage module includes M embedded block random memories for respectively storing the convolution operation results of M channels. i When performing the layer convolution operation, the M parallel convolution operation circuits simultaneously read the convolution operation results of M channels obtained by the previous layer convolution operation from the corresponding block random memories.

8. The hardware system of claim 5, wherein, a camera and an image preprocessing module; The image preprocessing module is configured to perform grayscale processing on each frame of micro-expression image collected by the camera; The storage module comprises a dual-port memory connected to the output end of the image preprocessing module, configured to simultaneously read the current frame of micro-expression image and store the next frame of micro-expression image, so as to complete the ping-pong operation of storage. Further comprising:

9. The system of any of claims 5-8, wherein, a display module, configured to display the micro-expression image in real time. Further comprising:

10. The system of any of claims 5-8, wherein, a host computer training module, configured to execute the construction method of the micro-expression detection model in claim 1 or 2, obtain the micro-expression detection model, and store the parameters of the micro-expression detection model into the storage module. ​

Images