Micro-expression recognition method based on message queue telemetry transmission

Abstract

The present invention relates to the technical field of micro-expression recognition, and discloses a micro-expression recognition method based on message queue telemetry transmission, comprising the following steps: S1: establishing a connection, first establishing a TCP connection between a micro-expression recognition algorithm service client and a proxy server; S2: determining the connection, after receiving a request to establish a connection, the proxy server checks whether the account password connection is allowed in its own connection authority configuration, and if allowed, establishes and saves a session identifier, binds the client ID and the session, and records the heartbeat interval time and the will message; S3: subscribes to a topic; S4: parses data; S5: sends data; S6: processes a topic message. The present invention realizes timely algorithm data communication, integrates a micro-expression recognition algorithm module into the entire system in a decoupled form, plays an intelligent processing role from the underlying data source to the upper-layer application, and provides algorithm data support for micro-expression modality for the upper-layer multimodal emotion analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of micro-expression recognition, and in particular to a micro-expression recognition method based on message queue telemetry transmission. Background Art

[0002] Micro-expression is a term in psychology. People express their inner feelings to others by making some expressions. Between different expressions or in a certain expression, the face will "leak" other information. "Micro-expression" can last for as short as 1 / 25 second. Although a subconscious expression may only last for a moment, it sometimes expresses the opposite emotion.

[0003] In the multimodal emotion analysis and judgment system, real-time micro-expression recognition is one of the algorithm service modules, which undertakes the underlying video stream data and the upper emotion analysis and judgment application layer. It needs to quickly receive the real-time video stream data transmitted from the underlying layer, and quickly calculate and analyze the micro-expressions of each frame, and send the micro-expression data to the upper-level application in time. Therefore, it is necessary to adopt a lightweight, low-overhead, low-bandwidth instant communication protocol to implement the above algorithm communication process. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the shortcomings of the prior art, the present invention provides a micro-expression recognition method based on message queue telemetry transmission, which is mainly used to solve the problem that the existing real-time micro-expression recognition algorithm service module cannot quickly receive the real-time video stream data transmitted from the bottom layer, and quickly calculate and analyze the micro-expressions of each frame, and send the micro-expression data to the upper-level application in time.

[0006] (II) Technical solution

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] The micro-expression recognition method based on message queue telemetry transmission includes the following steps:

[0009] S1: Establish a connection. First, establish a TCP connection between the micro-expression recognition algorithm service client and the proxy server.

[0010] S2: Confirm the connection. After receiving the request to establish a connection, the proxy server checks whether the account and password are allowed to connect in its own connection permission configuration. If allowed, it creates and saves the session identifier, binds the client ID to the session, and records the heartbeat interval time and will message. Then, it sends a connection success confirmation message back to the algorithm client. After the algorithm client receives the connection success confirmation message, it proceeds to the next step.

[0011] S3: Subscribe to the topic. The algorithm client subscribes to the base_topic topic sent by the underlying video stream data layer client and sends it to the proxy server through the SUBSCRIBE message.

[0012] S4: Parsing data. After receiving the message from base_topic, the micro-expression recognition algorithm service client parses the video stream data and processes the frame data in the video stream.

[0013] S5: Send data. The micro-expression recognition algorithm service client sends the micro-expression analysis results in real time in the form of emotion_topic to the proxy server.

[0014] S6: Process the topic message. The proxy server receives and processes the topic message. The upper-layer multimodal emotion analysis application layer client subscribes to the emotion_topic topic, completing the link process of the micro-expression recognition algorithm service receiving the underlying data, processing the underlying data, and sending it to the upper-layer application.

[0015] As a further solution of the present invention, the S1 includes a system module, and the system module includes an underlying video stream data layer client, an MQTT proxy server, and an upper multimodal emotion analysis application layer client.

[0016] Furthermore, the MQTT proxy server is connected with a connection monitoring processing module, a subscription monitoring processing module and a publishing monitoring processing module. The connection monitoring processing module first configures the account password, and then calls the connect method to start subscribing to the topic after confirming that the parsed message connection is successful.

[0017] On the basis of the above scheme, the subscription monitoring processing module first prepares the base_topic to be subscribed when working, and then calls the Subscribe method to monitor the message of the proxy end after the parsing message subscription is successful.

[0018] In a further solution of the present invention, the publishing monitoring processing module first prepares the emotion_topic to be published when working, and then calls the Publish method, and monitors the message of the proxy end after the parsed message is published successfully.

[0019] Furthermore, the process of establishing a connection in S1 is actively initiated by the algorithm client, and the proxy service is always in a listening state of the designated port. When it detects that a client wants to access, it will process it immediately. When the algorithm client initiates a connection request, it carries data such as the client ID, account number, password, and heartbeat interval.

[0020] Based on the above solution, the proxy server in S3 records the topic under the algorithm client ID, and then replies with a confirmation message SUBACK message. The algorithm client receives the SUBACK message and the subscription is successful.

[0021] In a further embodiment of the present invention, the processing in S4 includes face detection, facial micro-movement detection, facial dynamic micro-movement analysis, and micro-expression analysis.

[0022] (III) Beneficial effects

[0023] Compared with the prior art, the present invention provides a micro-expression recognition method based on message queue telemetry transmission, which has the following beneficial effects:

[0024] 1. The present invention can support the rapid iteration and rollback of the micro-expression recognition algorithm module. The iteration process does not affect the normal operation of the functions of other modules. The base_topic is published through the underlying video stream data layer client, and the emotion_topic is subscribed through the upper multimodal emotion analysis application layer client.

[0025] 2. The present invention can decouple the micro-expression recognition algorithm service from other modules, and take over the underlying data flow and application analysis and display layer, quickly read the underlying data, process the data, and send the algorithm result data, so as to realize the instant circulation of the algorithm service data.

[0026] 3. The present invention realizes timely algorithm data communication, integrates the micro-expression recognition algorithm module into the entire system in a decoupled form, plays the role of intelligent processing from the underlying data source to the upper-level application, and provides algorithm data support for the micro-expression modality for the upper-level multimodal emotion analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a schematic diagram of the process structure of the micro-expression recognition method based on message queue telemetry transmission proposed in the present invention.

[0028] Figure 2 This is a schematic diagram of the system module structure of the micro-expression recognition method based on message queue telemetry transmission proposed in the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] Example 1

[0031] Reference Figure 1-2 ,The micro-expression recognition method based on message queue telemetry transmission includes the following steps:

[0032] S1: Establish a connection. First, establish a TCP connection between the micro-expression recognition algorithm service client and the proxy server.

[0033] S2: Confirm the connection. After receiving the request to establish a connection, the proxy server checks whether the account and password are allowed to connect in its own connection permission configuration. If allowed, it creates and saves the session identifier, binds the client ID to the session, and records the heartbeat interval time and will message. Then, it sends a connection success confirmation message back to the algorithm client. After the algorithm client receives the connection success confirmation message, it proceeds to the next step.

[0034] S3: Subscribe to the topic. The algorithm client subscribes to the base_topic topic sent by the underlying video stream data layer client and sends it to the proxy server through the SUBSCRIBE message.

[0035] S4: Parsing data. After receiving the message from base_topic, the micro-expression recognition algorithm service client parses the video stream data and processes the frame data in the video stream.

[0036] S5: Send data. The micro-expression recognition algorithm service client sends the micro-expression analysis results in real time to the proxy server in the form of emotion_topic, which can support the rapid iteration and rollback of the micro-expression recognition algorithm module. The iteration process does not affect the normal operation of other modules.

[0037] S6: Process the topic message. The proxy server receives and processes the topic message. The upper-layer multimodal emotion analysis application layer client subscribes to the emotion_topic topic, completing the link process of the micro-expression recognition algorithm service receiving the underlying data, processing the underlying data, and sending it to the upper-layer application. The service architecture designed by this solution can decouple the micro-expression recognition algorithm service from other modules, and take over the underlying data flow and application analysis display layer, quickly read the underlying data, process the data, and send the algorithm result data, so as to realize the instant circulation of algorithm service data.

[0038] S1 of the present invention includes a system module, which includes an underlying video stream data layer client, an MQTT proxy server and an upper-layer multimodal emotion analysis application layer client. The base_topic is published through the underlying video stream data layer client, and the emotion_topic is subscribed to through the upper-layer multimodal emotion analysis application layer client. The MQTT proxy server is connected to a connection monitoring processing module, a subscription monitoring processing module and a publishing monitoring processing module. The connection monitoring processing module first configures an account password, and then calls a connect method to start subscribing to a topic after determining that the parsed message connection is successful. When working, the subscription monitoring processing module first prepares a base_topic to be subscribed, and then calls a Subscibe method to monitor messages from the proxy end after the parsed message subscription is successful. When working, the publishing monitoring processing module first prepares an emotion_topic to be published, and then calls a Publish method to monitor messages from the proxy end after the parsed message is published successfully.

[0039] It should be noted that the process of establishing a connection in S1 is actively initiated by the algorithm client, and the proxy service is always in the listening state of the designated port. When it listens to a client wanting to access, it will immediately process it. When the algorithm client initiates a connection request, it carries data such as the client ID, account, password, and heartbeat interval, realizing timely algorithm data communication, and integrating the micro-expression recognition algorithm module into the entire system in a decoupled form, playing the role of intelligent processing from the underlying data source to the upper-level application, and providing algorithm data support for the micro-expression mode for the upper-level multimodal emotion analysis. In S3, the proxy server records the topic under the algorithm client ID, and then replies with a confirmation message SUBACK message. The algorithm client receives the SUBACK message and the subscription is successful. The processing in S4 includes face detection, facial micro-motion detection, facial dynamic micro-motion analysis, and micro-expression analysis.

[0040] Example 2

[0041] Reference Figure 1-2 ,The micro-expression recognition method based on message queue telemetry transmission includes the following steps:

[0042] S1: Establish a connection. First, establish a TCP connection between the micro-expression recognition algorithm service client and the proxy server.

[0043] S2: Confirm the connection. After receiving the request to establish a connection, the proxy server checks whether the account and password are allowed to connect in its own connection permission configuration. If allowed, it creates and saves the session identifier, binds the client ID to the session, and records the heartbeat interval time and will message. Then, it sends a connection success confirmation message back to the algorithm client. After the algorithm client receives the connection success confirmation message, it proceeds to the next step.

[0044] S3: Subscribe to the topic. The algorithm client subscribes to the base_topic topic sent by the underlying video stream data layer client and sends it to the proxy server through the SUBSCRIBE message.

[0045] S4: Parsing data. After receiving the message from base_topic, the micro-expression recognition algorithm service client parses the video stream data and processes the frame data in the video stream.

[0046] S5: Send data. The micro-expression recognition algorithm service client sends the micro-expression analysis results in real time to the proxy server in the form of emotion_topic, which can support the rapid iteration and rollback of the micro-expression recognition algorithm module. The iteration process does not affect the normal operation of other modules.

[0047] S6: Process the topic message. The proxy server receives and processes the topic message. The upper-layer multimodal emotion analysis application layer client subscribes to the emotion_topic topic, completing the link process of the micro-expression recognition algorithm service receiving the underlying data, processing the underlying data, and sending it to the upper-layer application. The service architecture designed by this solution can decouple the micro-expression recognition algorithm service from other modules, and take over the underlying data flow and application analysis display layer, quickly read the underlying data, process the data, and send the algorithm result data, so as to realize the instant circulation of algorithm service data.

[0048] S1 of the present invention includes a system module, which includes an underlying video stream data layer client, an MQTT proxy server and an upper multimodal emotion analysis application layer client. The base_topic is published through the underlying video stream data layer client, and the emotion_topic is subscribed through the upper multimodal emotion analysis application layer client. The MQTT proxy server is connected with a connection monitoring processing module, a subscription monitoring processing module and a publishing monitoring processing module. The connection monitoring processing module first configures an account password, and then calls a connect method. After determining that the parsed message connection is successful, it starts to subscribe to the topic. When the subscription monitoring processing module is working, it first prepares the base_topic to be subscribed, and then calls the Subscibe method. After the parsed message subscription is successful, it monitors the message of the proxy end. When the publishing monitoring processing module is working, it first prepares the emotion_topic to be published, and then calls the Publish method. After the parsed message is published successfully, it monitors the message of the proxy end. When the system module is working, the initialized model is loaded and then stored in the memory for calling. After parsing the base_topic data, data preprocessing is performed. After preprocessing, the micro-expression recognition model is retrieved and compared with the stored micro-expression recognition model.

[0049] It should be noted that the process of establishing a connection in S1 is actively initiated by the algorithm client, and the proxy service is always in the listening state of the designated port. When it listens to a client wanting to access, it will immediately process it. When the algorithm client initiates a connection request, it carries data such as the client ID, account, password, and heartbeat interval, realizing timely algorithm data communication, and integrating the micro-expression recognition algorithm module into the entire system in a decoupled form, playing the role of intelligent processing from the underlying data source to the upper-level application, and providing algorithm data support for the micro-expression mode for the upper-level multimodal emotion analysis. In S3, the proxy server records the topic under the algorithm client ID, and then replies with a confirmation message SUBACK message. The algorithm client receives the SUBACK message and the subscription is successful. The processing in S4 includes face detection, facial micro-motion detection, facial dynamic micro-motion analysis, and micro-expression analysis.

[0050] In the description of this article, it should be noted that relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0051] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A micro-expression recognition method based on message queue telemetry transmission, characterized in that: The following steps are involved: S1: Establish a connection. First, establish a TCP connection between the micro-expression recognition algorithm service client and the MQTT proxy server. S2: Confirm the connection. After receiving the request to establish a connection, the MQTT proxy server checks whether the connection is allowed in its own connection permission configuration. If allowed, it creates and saves the session identifier, binds the algorithm service client ID to the session, and records the heartbeat interval time and will message. Then, it sends a connection success confirmation message back to the algorithm service client. After the algorithm service client receives the connection success confirmation message, it proceeds to the next step. S3: Subscribe to the topic. The algorithm service client subscribes to the base_topic topic sent by the underlying video stream data layer client and sends it to the MQTT proxy server through a SUBSCRIBE message. S4: Parsing data. After receiving the message from base_topic, the micro-expression recognition algorithm service client parses the video stream data and processes the frame data in the video stream. S5: Send data. The micro-expression recognition algorithm service client sends the micro-expression analysis results in real time in the form of emotion_topic to the MQTT proxy server. S6: Process the topic message. The MQTT proxy server receives and processes the topic message. The upper-layer multimodal emotion analysis application layer client subscribes to the emotion_topic topic, completing the link process of the micro-expression recognition algorithm service receiving the underlying data, processing the underlying data, and sending it to the upper-layer application.

2. The micro-expression recognition method based on message queue telemetry transmission according to claim 1, characterized in that: The S1 includes a system module, which includes an underlying video stream data layer client, an MQTT proxy server, and an upper multimodal emotion analysis application layer client.

3. The micro-expression recognition method based on message queue telemetry transmission according to claim 2 is characterized in that: The MQTT proxy server is connected with a connection monitoring processing module, a subscription monitoring processing module and a publishing monitoring processing module. The connection monitoring processing module first configures the account password, and then calls the connect method to confirm that the parsed message connection is successful and then starts to subscribe to the topic.

4. The micro-expression recognition method based on message queue telemetry transmission according to claim 3 is characterized in that: The subscription monitoring processing module first prepares the base_topic to be subscribed when working, and then calls the Subscribe method to monitor the message of the proxy end after the parsing message subscription is successful.

5. The micro-expression recognition method based on message queue telemetry transmission according to claim 3 is characterized in that: The publishing monitoring processing module first prepares the emotion_topic to be published when working, and then calls the Publish method, and monitors the message of the proxy end after the parsed message is published successfully.

6. The micro-expression recognition method based on message queue telemetry transmission according to claim 1, characterized in that: The process of establishing a connection in S1 is actively initiated by the algorithm service client. The MQTT proxy server is always in the listening state of the specified port. When it detects that an algorithm service client wants to access, it will process it immediately. When the algorithm service client initiates a connection request, it carries the algorithm service client ID, account, password, and heartbeat interval time data.

7. The micro-expression recognition method based on message queue telemetry transmission according to claim 1, characterized in that: The MQTT proxy server in S3 records the topic under the algorithm service client ID, and then replies with a confirmation message SUBACK message. The algorithm service client receives the SUBACK message and the subscription is successful.

8. The micro-expression recognition method based on message queue telemetry transmission according to claim 1, characterized in that: The processing in S4 includes face detection, facial micro-motion detection, facial dynamic micro-motion analysis, and micro-expression analysis.

Images