A method and device for detecting micro-expression video sequences in video streams
By adopting stream processing method in the processing framework of the graphics processing unit group, using multiple GPUs for batch preprocessing and feature extraction, the problem of low micro-expression detection efficiency in real-time video streams is solved, and efficient and accurate micro- Expression detection.
Patent Information
- Application Number
- CN202210021817.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-10
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-01-10
AI Technical Summary
The prior art is difficult to accurately detect micro-expressions in real-time video streams, and traditional computing frameworks cannot meet the real-time processing requirements, the detection time is long and the calculation complexity is high.
Using at least two graphics processing unit groups, micro-expression detection is performed through stream processing, multiple GPUs are used for batch preprocessing and feature extraction, so as to achieve maximum utilization of each GPU computing resource and device I/O resource Rate.
It improves the efficiency and accuracy of micro-expression detection, meets the needs of real-time processing, and is suitable for micro-expression detection of high-frame-rate video streams.
Smart Images

Figure CN114463809B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for detecting a micro-expression video sequence in a video stream. Background Art
[0002] Micro-expressions are facial expressions that people show when they try to suppress or hide their true emotions. They are not controlled by themselves and can truly reflect a person's psychological activities. They play an important role in criminal investigation, interrogation, business negotiations, clinical diagnosis, etc. Therefore, micro-expression detection technology has attracted cross-field cooperative research by researchers from various industries and has become a research hotspot for human expression detection and recognition.
[0003] Micro-expressions last for a short time and involve small facial muscle movements, making them difficult to pinpoint and accurately detect in real-time video streams. Usually, video sequences for detecting micro-expressions are mostly from videos shot by high-speed cameras, which contain several times more video frames per second than ordinary cameras. Micro-expression detection refers to detecting image sequences with micro-expressions in the video. Most micro-expression detection methods have high computational complexity and take a long time to detect. Traditional computing frameworks cannot meet real-time processing requirements. Summary of the invention
[0004] An embodiment of the present invention provides a method and device for detecting micro-expression video sequences in a video stream, which are used to complete micro-expression detection of a large number of video frames by utilizing a processing framework of at least two graphics processing unit groups, and achieve maximum utilization of each GPU computing resource and device I / O resource by stream processing.
[0005] The embodiment of the present invention provides a method for detecting a micro-expression video sequence in a video stream, which is applied to a processing framework including at least two graphics processing unit groups, wherein the at least two graphics processing unit groups are cascaded, and each graphics processing unit group includes multiple graphics processing units GPU. The detection method includes:
[0006] Acquire a video stream, and generate multiple batches of video frame sequences from the video stream in chronological order, wherein the number of video frames included in the batch of video frame sequences is the same as the number of GPUs of the first graphics processing unit group;
[0007] Using each GPU of the first graphics processing unit group, loading each video frame in the first batch of video frame sequences, and preprocessing the batch of video frames;
[0008] Inputting at least part of the preprocessed batch of video frame sequences into a second graphics processing unit group;
[0009] Utilizing each GPU of the second graphics processing unit group, each video frame of the at least part of the video frame sequence after preprocessing is loaded, and the micro-expression feature of each video frame is extracted through a preset algorithm to realize micro-expression detection.
[0010] In some embodiments, the detection method is applied to a processing framework including two graphics processing unit groups, the two graphics processing unit groups are cascaded, and each graphics processing unit group includes a plurality of graphics processing units GPU;
[0011] The number of GPUs in the first graphics processing unit group is greater than or equal to the number of GPUs in the second graphics processing unit group.
[0012] In some embodiments, generating a plurality of batches of video frame sequences from the video stream in time order includes:
[0013] According to a fixed time length, multiple batches of video frame sequences are generated in time order based on the video stream.
[0014] In some embodiments, at least a portion of the video frame sequence input to the second graphics processing unit group is determined by:
[0015] Provide sliding windows;
[0016] Based on the preprocessed video frames, a partial video frame sequence of the length of the sliding window is intercepted.
[0017] In some embodiments, the sliding window is slid according to a preset step size; and
[0018] A specified number of video frame sequences captured by the sliding window is input into the second graphics processing unit group as a batch, wherein the specified number is the same as the number of GPUs of the second graphics processing unit group.
[0019] In some embodiments, the duration of the sliding window is less than or equal to 200 ms.
[0020] In some embodiments, extracting micro-expression features of each video frame using a preset algorithm to implement micro-expression detection includes:
[0021] Extract the micro-expression features of each video frame using a preset algorithm, and classify the extracted micro-expression features;
[0022] The types of micro-expressions in the video stream are determined based on the classification of the outputs of the GPUs of the second graphics processing unit group.
[0023] The embodiment of the present invention further proposes a detection device for a micro-expression video sequence in a video stream, which is applied to a processing framework including at least two graphics processing unit groups, at least two graphics processing unit groups are cascaded, and each graphics processing unit group includes multiple graphics processing units GPU, and the detection device includes a processor, which is configured as follows:
[0024] Acquire a video stream, and generate multiple batches of video frame sequences from the video stream in chronological order, wherein the number of video frames included in the batch of video frame sequences is the same as the number of GPUs of the first graphics processing unit group;
[0025] Using each GPU of the first graphics processing unit group, loading each video frame in the first batch of video frame sequences, and preprocessing the batch of video frames;
[0026] Inputting at least part of the preprocessed batch of video frame sequences into a second graphics processing unit group;
[0027] Utilizing each GPU of the second graphics processing unit group, each video frame of the at least part of the video frame sequence after preprocessing is loaded, and the micro-expression feature of each video frame is extracted through a preset algorithm to realize micro-expression detection.
[0028] An embodiment of the present invention further proposes a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for detecting a micro-expression video sequence in a video stream as described in the embodiments of the present disclosure are implemented.
[0029] The embodiment of the present invention utilizes the processing framework of at least two graphics processing unit groups to complete micro-expression detection of a large number of video frames. It is particularly suitable for micro-expression detection of video streams by performing stream processing on the video frame sequence in layers and batches, and achieves maximum utilization of each GPU computing resources and device I / O resources through stream processing.
[0030] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0032] Figure 1is a basic flow chart of a method for detecting micro-expression video sequences according to an embodiment of the present invention;
[0033] Figure 2 Schematic diagram of the basic framework of the method for detecting micro-expression video sequences according to an embodiment of the present invention;
[0034] Figure 3 It is a schematic diagram of inputting the second GPU group in batches using a sliding window according to an embodiment of the present invention. DETAILED DESCRIPTION
[0035] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.
[0036] An embodiment of the present invention provides a method for detecting a micro-expression video sequence in a video stream, which is applied to a processing framework including at least two graphics processing unit groups, wherein the at least two graphics processing unit groups are cascaded, and each graphics processing unit group includes a plurality of graphics processing units (GPUs). In this example, at least two graphics processing unit groups are cascaded. Specifically, a first-level graphics processing unit group can receive an image to be processed, and after processing, can output the image to a second-level graphics processing unit group. Figure 1 As shown, the detection method comprises:
[0037] In step S101, a video stream is acquired, and the video stream is used to generate multiple batches of video frame sequences in chronological order, wherein the number of video frames included in the batch of video frame sequences is the same as the number of GPUs in the first graphics processing unit group.
[0038] In step S102, each GPU of the first graphics processing unit group is used to load each video frame in the first batch of video frame sequences, and pre-process the video frames of the batch. That is, each frame image of each batch can be sent to a GPU for processing, so that the operation of each GPU is relatively independent and does not interfere with each other, and because one GPU processes one frame of image, the computing amount of the GPU of the first graphics processing unit group is similar. Therefore, the processing time for completing the video frame sequence of the batch is also similar, so that the acquired image frames can be processed in batches, thereby greatly improving the processing efficiency of image data.
[0039] In step S103, at least a portion of the pre-processed batch of video frame sequences is input into the second graphics processing unit group.
[0040] In step S104, each GPU of the second graphics processing unit group is used to load each video frame of the at least part of the video frame sequence after preprocessing, and the micro-expression features of each video frame are extracted by a preset algorithm to achieve micro-expression detection.
[0041] The embodiment of the present invention utilizes the processing framework of at least two graphics processing unit groups to complete the micro-expression detection of a large number of video frames by performing stream processing on the video frame sequence in layers and batches. That is, each GPU of the first graphics processing unit group is responsible for preprocessing the video frame sequence in batches, and the GPU of the second graphics processing unit group is responsible for detecting the preprocessed video frames through a preset algorithm, and the maximum utilization rate of each GPU computing resource and device I / O resource is achieved through stream processing, which is particularly suitable for micro-expression detection of high frame rate video streams.
[0042] In some embodiments, the detection method is applied to a processing framework including two graphics processing unit groups, the two graphics processing unit groups are cascaded, and each graphics processing unit group includes multiple graphics processing units GPU; wherein the number of GPUs in the first graphics processing unit group is greater than or equal to the number of GPUs in the second graphics processing unit group.
[0043] Specifically, two graphics processing unit groups can be cascaded, and one graphics processing unit group can use one graphics card or multiple graphics cards in a specific application. Figure 2 As shown, in the embodiment of the present invention, one group is GPU1n, with a total of n GPUs; the second group is GPU2m, with a total of m GPUs. GPU1n can be used to preprocess the video frame image, and GPU2m can be used to detect and classify micro-expressions on the video frame sequence. In a specific application, the first graphics processing unit group preprocesses the video frame sequence input in batches, including image operations such as sequential grayscale marking of each video frame.
[0044] In some embodiments, generating multiple batches of video frame sequences from the video stream in time sequence includes: generating multiple batches of video frame sequences based on the video stream in time sequence according to a fixed time length. Specifically, Figure 2 As shown, the video frame stream is divided into video frame batch 1, video frame batch 2, etc. according to a fixed time length or a fixed number of frames, and both video frame batch 1 and video frame batch 2 contain n frames of video images. Therefore, for each batch, n frames of video images are input to GPU11-GPU1n of the first graphics processing unit group, which can improve the utilization rate of the GPU.
[0045] In some embodiments, at least a portion of the video frame sequence input to the second graphics processing unit group is determined by:
[0046] Provide sliding windows;
[0047] Based on the preprocessed video frames, a partial video frame sequence of the length of the sliding window is intercepted.
[0048] In some embodiments, the sliding window is slid according to a preset step size; and
[0049] A specified number of video frame sequences captured by the sliding window is input into the second graphics processing unit group as a batch, wherein the specified number is the same as the number of GPUs of the second graphics processing unit group.
[0050] like Figure 2 The sliding step size set in is 1, that is, the image frame sequence output by the first graphics processing unit group is slide-cut to capture a corresponding number of image frames using the sliding window. Figure 3 As shown, multiple image frame sequences can be intercepted according to the sliding step size, and the intercepted multiple image frame sequences can be sent to GPU21-GPU2m of the second graphics processing unit group. By setting the specified number to be the same as the number of GPUs of the second graphics processing unit group, each GPU of GPU21-GPU2m of the second graphics processing unit group can process an image sequence intercepted by a sliding window, thereby improving the utilization rate of the second graphics processing unit group. That is, each GPU in the second graphics processing unit group is responsible for acquiring the video sequence data in the corresponding sliding window and storing it in the GPU video memory, and performing feature extraction, micro-expression detection and other steps on the video frame sequence. The operation of each GPU is relatively independent and does not interfere with each other. After completing the calculation, each GPU releases its video memory resources and computing resources, waiting to process the next batch of sliding window video frame sequence data.
[0051] In some embodiments, the duration of the sliding window is less than or equal to 200ms. Since micro-expressions are mostly involuntary and occur in a very short period of time, by setting the sliding time window to be less than or equal to 200ms, it is possible to focus on the recognition of micro-expressions and improve the recognition effect of micro-expressions.
[0052] In some embodiments, micro-expression features of each video frame are extracted using a preset algorithm to achieve micro-expression detection, including: extracting micro-expression features of each video frame using a preset algorithm, and classifying the extracted micro-expression features; determining the type of micro-expression in the video stream based on the classification output by each GPU of the second graphics processing unit group. The specific preset algorithm for feature extraction and micro-expression detection classification in the embodiment of the present invention can be implemented by, for example, a deep learning model, and is not specifically limited here. The image frame sequence of the sliding window length output by each GPU of the second graphics processing unit group is the corresponding type of micro-expression.
[0053] By processing the real-time video stream in batches, each GPU computing processing unit forms a pipeline operation of data reading, writing and computing, so that the method of this embodiment can achieve the maximum utilization of each GPU computing resource and device I / O resource.
[0054] The computing framework is easily scalable. By expanding GPU computing resources, a multi-GPU computing framework is realized, which is particularly suitable for real-time processing requirements in computing scenarios such as micro-expression recognition, video sequence computing, and feature extraction. The method of this embodiment is also suitable for real-time video sequence batch processing scenarios with high computational complexity, and has a high value for promotion and utilization.
[0055] The embodiment of the present invention further proposes a detection device for a micro-expression video sequence in a video stream, which is applied to a processing framework including at least two graphics processing unit groups, at least two graphics processing unit groups are cascaded, and each graphics processing unit group includes multiple graphics processing units GPU, and the detection device includes a processor, which is configured as follows:
[0056] Acquire a video stream, and generate multiple batches of video frame sequences from the video stream in chronological order, wherein the number of video frames included in the batch of video frame sequences is the same as the number of GPUs of the first graphics processing unit group;
[0057] Using each GPU of the first graphics processing unit group, loading each video frame in the first batch of video frame sequences, and preprocessing the batch of video frames;
[0058] Inputting at least part of the preprocessed batch of video frame sequences into a second graphics processing unit group;
[0059] Utilizing each GPU of the second graphics processing unit group, each video frame of the at least part of the video frame sequence after preprocessing is loaded, and the micro-expression feature of each video frame is extracted through a preset algorithm to realize micro-expression detection.
[0060] An embodiment of the present invention further proposes a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for detecting a micro-expression video sequence in a video stream as described in the embodiments of the present disclosure are implemented.
[0061] The embodiment of the present invention utilizes the processing framework of at least two graphics processing unit groups to complete micro-expression detection of a large number of video frames. It is particularly suitable for micro-expression detection of video streams by performing stream processing on the video frame sequence in layers and batches, and achieves maximum utilization of each GPU computing resources and device I / O resources through stream processing.
[0062] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0063] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0064] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for a terminal (which can be a mobile phone, computer, server or network equipment, etc.) to execute the methods described in each embodiment of the present invention.
[0065] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.
Claims
1. A method for detecting micro-expression video sequences in a video stream, It is characterized in that Applied to a processing framework including at least two graphics processing unit groups, the at least two graphics processing unit groups are cascaded, and each graphics processing unit group includes a plurality of graphics processing units GPU, the detection method includes: Acquire a video stream, and generate multiple batches of video frame sequences from the video stream in chronological order, wherein the number of video frames included in the first batch of video frame sequences is the same as the number of GPUs of the first graphics processing unit group; Using each GPU of the first graphics processing unit group, loading each video frame in the first batch of video frame sequences, and preprocessing the first batch of video frames; Inputting at least part of the preprocessed first batch of video frame sequences into the second graphics processing unit group; Using each GPU of the second graphics processing unit group, each video frame of at least a portion of the preprocessed video frame sequence is loaded, and the micro-expression features of each video frame are extracted through a preset algorithm to achieve micro-expression detection.
2. The method for detecting micro-expression video sequences in a video stream as claimed in claim 1, It is characterized in that The detection method is applied to a processing framework including two graphics processing unit groups, the two graphics processing unit groups are cascaded, and each graphics processing unit group includes a plurality of graphics processing units GPU; The number of GPUs in the first graphics processing unit group is greater than or equal to the number of GPUs in the second graphics processing unit group.
3. The method for detecting micro-expression video sequences in a video stream as claimed in claim 1, It is characterized in that Generating multiple batches of video frame sequences from the video stream in time order comprises: According to a fixed time length, multiple batches of video frame sequences are generated in time order based on the video stream.
4. The method for detecting micro-expression video sequences in a video stream as claimed in claim 1, It is characterized in that At least a portion of the video frame sequence input to the second graphics processing unit group is determined by: Provide sliding windows; Based on the preprocessed video frames, a partial video frame sequence of the length of the sliding window is intercepted.
5. The method for detecting micro-expression video sequences in a video stream as claimed in claim 4, It is characterized in that The detection method further comprises: Sliding the sliding window according to a preset step size; and A specified number of video frame sequences captured by the sliding window is input into the second graphics processing unit group as a batch, wherein the specified number is the same as the number of GPUs of the second graphics processing unit group.
6. The method for detecting micro-expression video sequences in a video stream as claimed in claim 4, It is characterized in that The duration of the sliding window is less than or equal to 200 ms.
7. The method for detecting micro-expression video sequences in a video stream as claimed in claim 1, It is characterized in that The micro-expression features of each video frame are extracted using a preset algorithm to achieve micro-expression detection including: Extract the micro-expression features of each video frame using a preset algorithm, and classify the extracted micro-expression features; The types of micro-expressions in the video stream are determined based on the classification of the outputs of the GPUs of the second graphics processing unit group.
8. A device for detecting micro-expression video sequences in a video stream, It is characterized in that Applied to a processing framework including at least two graphics processing unit groups, the at least two graphics processing unit groups are cascaded, each graphics processing unit group includes a plurality of graphics processing units GPU, the detection device includes a processor, which is configured to: Acquire a video stream, and generate multiple batches of video frame sequences from the video stream in chronological order, wherein the number of video frames included in the first batch of video frame sequences is the same as the number of GPUs of the first graphics processing unit group; Using each GPU of the first graphics processing unit group, loading each video frame in the first batch of video frame sequences, and preprocessing the first batch of video frames; Inputting at least part of the preprocessed first batch of video frame sequences into the second graphics processing unit group; Using each GPU of the second graphics processing unit group, each video frame of at least a portion of the preprocessed video frame sequence is loaded, and the micro-expression features of each video frame are extracted through a preset algorithm to achieve micro-expression detection.
9. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for detecting a micro-expression video sequence in a video stream according to any one of claims 1 to 7 are implemented.
