Face detection method, face recognition method, behavior recognition method and system

Through embedded artificial intelligence technology and optimized face detection and recognition algorithms, the existing examination room monitoring system has been solved, efficient and low power consumption monitoring has been achieved, and exam fairness and security are improved.

CN118942137APending Publication Date: 2024-11-12SOUTH CHINA NORMAL UNIV
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410996157.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing examination room monitoring system relies on manual inspection, resulting in low monitoring efficiency, low detection accuracy, poor real-time performance, and strong dependence on network signals, resulting in limited monitoring functions in a network-free environment, affecting the candidates' examination status.

Method used

Using embedded artificial intelligence technology, through the YOLO5-Dface lightweight face detection model and Mobi-FaceNeXt face recognition network, combining coordinate attention mechanism and joint loss function, efficient face detection and recognition without networking is achieved, combined with CNN Behavior identification is performed with a single-stage multi-person pose estimation algorithm, reducing monitoring costs and interference from candidates.

Benefits of technology

It realizes efficient, low power consumption and strong real-time examination room monitoring, improves identity recognition efficiency and accuracy, reduces the interference of examiners' inspections on candidates, creates a fairer and quieter examination environment, and provides reliable evidence support.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942137A_ABST
    Figure CN118942137A_ABST
Patent Text Reader

Abstract

The invention discloses a face detection method, a face recognition method, a behavior recognition method and a system. The system comprises an image acquisition module for acquiring images in an examination room in real time by using an embedded camera; the lightweight face detection module is used for performing rapid face detection and positioning on the image by adopting the optimized convolutional neural network; the face recognition module is used for constructing a Mobile-FaceNeXt network containing an attention mechanism; the behavior recognition module is used for designing the behavior recognition module based on a CNN and a single-stage multi-person posture estimation Baseline algorithm, and recognizing the behaviors of the examinees by analyzing the detected human faces and the surrounding environment of the human faces; and the alarm module is used for giving an alarm in real time and recording related images and videos when abnormal behaviors are detected. The system has the characteristics of high efficiency, low power consumption and high real-time performance, and can be widely applied to the technical fields of image processing, examination room invigilation and the like.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a face detection method, a face recognition method, a behavior recognition method and a system. Background Art

[0002] With the continuous improvement of education level and the improvement of examination system, examination room monitoring system plays an increasingly important role in ensuring the fairness and security of examination. However, the existing examination room monitoring system generally has the following problems: manual monitoring is still the mainstream, the necessary means to achieve invigilation is still on-site patrol, identity recognition efficiency is low, detection accuracy is not high, real-time performance is poor, and network signal shielding. These problems and invigilation mode not only make the invigilation cost high, but also easily affect the on-site status of candidates. The current daily use of artificial intelligence is realized through networking. For example, the voice recognition function of mobile phones transmits information to the background server through the network, and then transmits the text to the mobile phone after calculation. In this mode, the use scenario is limited. In an environment without network and the upgrade of the background server may cause the terminal to be unable to use certain functions. Therefore, embedded artificial intelligence becomes more and more important. Embedded artificial intelligence refers to the device without networking, relying on the computing power of the device itself to perform local calculations, and is completely independent of the cloud server. Summary of the invention

[0003] The main purpose of the embodiments of the present invention is to provide an efficient and low-power face detection method, face recognition method, behavior recognition method and system, which can reduce costs, improve efficiency and ensure reliability.

[0004] To achieve the above object, an embodiment of the present invention provides a face detection method, comprising the following steps:

[0005] Acquire the image to be detected;

[0006] Transmitting the image to be detected to a face detection network model for face detection, and outputting a face detection result;

[0007] The detection process of the face detection network model includes:

[0008] Add hole convolution structure and residual connection to the YOLO5-Dface lightweight face detection model for detection processing;

[0009] The posture of the face is optimized according to the result of the detection processing to determine the face detection result.

[0010] In some embodiments, the adding of a dilated convolution structure and a residual connection to the YOLO5-Dface lightweight face detection model for detection processing includes the following steps:

[0011] Four dilation convolutions with different expansion rates are added to extract feature information of the image to be detected, and through a residual connection operation, information fusion between the input image and the convolved image is completed to output a feature map;

[0012] Performing maximum pooling operations of different scales on the feature maps respectively, and retaining the largest pixel in the convolution area;

[0013] Perform feature concatenation on the output of the maximum pooling operation and fuse it with the input features of the CBS convolution block;

[0014] The fused features are convolutionally restored to complete the detection process.

[0015] In some embodiments, performing face posture optimization according to the result of the detection processing to determine the face detection result includes the following steps:

[0016] According to the results of the detection process, five key points of YOLO5-Dface network regression are selected as reference points for posture optimization;

[0017] By comparing the relative positions of the left eye, right eye, nose tip, left corner of the mouth, and right corner of the mouth, the detected face posture is determined, faces with large deflection angles are eliminated, and faces with small angles that meet the preset conditions are retained.

[0018] In some embodiments, the method of comparing the relative positions of the five points of the left eye, the right eye, the nose tip, the left corner of the mouth, and the right corner of the mouth to determine the detected face posture, eliminating faces with large deflection angles, and retaining faces with small angles that meet preset conditions, includes the following steps:

[0019] The five key points of the face are marked with X and Y, namely, the left eye (x1, y1), the right eye (x2, y2), the nose tip (x3, y3), the left corner of the mouth (x4, y4), and the right corner of the mouth (x5, y5). Based on the five key points of the face, the degree of left-right and up-down deflection of the face posture is calculated;

[0020] The deflection position of the face is calculated by the distance between the eyes and the corners of the mouth, and the left and right deflection posture of the face is determined by calculating the ratio of the distance between the left eye and the left corner of the mouth, and the distance between the right eye and the right corner of the mouth to the tip of the nose;

[0021] The calculation formula for the degree of left and right deflection is:

[0022] The calculation formula for the degree of up and down deflection is:

[0023] Another aspect of the embodiment of the present invention further provides a face recognition method, comprising:

[0024] Obtaining a face detection result for the image to be detected; wherein the face detection result is obtained according to the face detection method as described above;

[0025] Downsampling the face detection result using a fast downsampling strategy to reduce the input feature map in the face detection result;

[0026] A facial feature extraction network of a novel inverted residual structure is constructed to process the result of the downsampling process, wherein the construction process of the novel inverted residual structure is as follows: the convolution kernel size in the inverted residual structure is changed to 7×7; the number of channels set in the multi-head self-attention module is reduced; a PReLU activation function is used in a shallower network layer, and a HardSwi sh activation function is used to replace the PReLU activation function in a deeper network layer;

[0027] Applying a coordinate attention mechanism that takes into account both channel information and spatial position to the face feature extraction network to perform face feature extraction processing;

[0028] The coordinate attention mechanism is combined with the novel inverted residual structure, and three network structures are constructed according to the insertion position of the coordinate attention, namely the deep structure, the deep point-by-point structure and the final point-by-point structure;

[0029] A new joint loss function is formed based on MagFace Loss and Center Loss, and face recognition is performed according to the joint loss function to obtain a face recognition result.

[0030] In some embodiments, the application process of the coordinate attention mechanism includes the following steps:

[0031] Using two global pooling operations, each channel is encoded horizontally and vertically along the horizontal and vertical directions respectively;

[0032] The horizontal and vertical pooling layers are concatenated and output in cascade, and then a 1×1 convolution operation is performed on the cascade output result to interact information in two directions, followed by a BN operation and a HardSwish nonlinear activation function operation;

[0033] The resulting tensor is split into two separate tensors and subjected to convolution and sigmoid operations respectively to produce attention vectors in the horizontal and vertical directions with the same number of channels.

[0034] Another aspect of the embodiment of the present invention further provides a behavior recognition method, including:

[0035] Extract features from the image to be identified through the backbone network model;

[0036] The output features are further processed by multiple detection heads, and the coordinates of the human body center point and the offsets of the human body key points output by the multiple detection heads are combined to obtain the final coordinates of the human body key points; wherein the detection head is embedded with a feature alignment module based on human body parts.

[0037] Combining the face recognition result and the coordinates of the key points of the human body, a behavior recognition method is generated; wherein the face recognition result is obtained according to the face detection method as described above.

[0038] In some embodiments, the application process of the human body part-based feature alignment module includes the following steps:

[0039] Use the position P of the center point of the human body center The feature vector at predicts the offset u from the position near the human body part to the center point of the human body part , and calculate the position Ppart of the human body part;

[0040] Using P part The feature vector at predicts the offset u of a key point to the position near the human body part keypoint , and calculate the position P of the key point keypoint ;

[0041] According to the position of the center point of the human body P center , the position of the human body part Ppart, the position of the key point P keypoint , calculate the coordinates of the key point.

[0042] Another aspect of the embodiment of the present invention further provides an examination room monitoring system, including:

[0043] An image acquisition module, used for acquiring images to be detected;

[0044] A lightweight face detection module, used to implement the face detection method as described above;

[0045] A face recognition module, used to implement the face recognition method as described above;

[0046] A behavior recognition module, used to implement the behavior recognition method as described above;

[0047] The alarm module is used to issue alarm prompts based on the face recognition results and behavior recognition results.

[0048] Another aspect of an embodiment of the present invention further provides an electronic device, including a processor and a memory;

[0049] The memory is used to store programs;

[0050] The processor executes the program to implement the method described above.

[0051] Another aspect of the embodiments of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method described above.

[0052] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0053] The embodiments of the present invention include at least the following beneficial effects: The present invention provides a face detection method, a face recognition method, a behavior recognition method and a system. In the face detection method of the scheme, an image to be detected is obtained; the image to be detected is transmitted to a face detection network model for face detection, and a face detection result is output; wherein the detection process of the face detection network model includes: adding a hole convolution structure and a residual connection to the YOLO5-Dface lightweight face detection model for detection processing; performing face posture optimization according to the result of the detection processing to determine the face detection result.

[0054] In the face recognition method of the scheme, a face detection result for an image to be detected is obtained; a fast downsampling strategy is used to downsample the face detection result to reduce the input feature map in the face detection result; a face feature extraction network with a new inverted residual structure is constructed to process the result of the downsampling process, wherein the construction process of the new inverted residual structure is as follows: the convolution kernel size in the inverted residual structure is changed to 7×7; the number of channels set by the multi-head self-attention module is reduced; a PReLU activation function is used in a shallower network layer, and a HardSwi sh activation function is used to replace the PReLU activation function in a deeper network layer; a coordinate attention mechanism that takes into account both channel information and spatial position is applied to the face feature extraction network to perform face feature extraction processing; the coordinate attention mechanism is combined with the new inverted residual structure, and three network structures are constructed according to the insertion position of the coordinate attention, namely, a deep structure, a deep point-by-point structure and a final point-by-point structure; a new joint loss function is formed based on MagFace Loss and Center Loss, and face recognition is performed according to the joint loss function to obtain a face recognition result.

[0055] In the behavior recognition method of this solution, the image to be recognized is subjected to a backbone network model to extract features; the output features are further processed by multiple detection heads, and the coordinates of the human body center point and the offset of the human body key points output by the multiple detection heads are combined to obtain the final coordinates of the human body key points; wherein the detection head is embedded with a feature alignment module based on human body parts. Combining the face recognition results and the coordinates of the human body key points, a behavior recognition method is generated.

[0056] The present invention can reduce costs, improve efficiency and ensure reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a flow chart of the overall steps provided by an embodiment of the present invention;

[0058] Figure 2 It is a network structure diagram of the YOLOv5 algorithm provided by an embodiment of the present invention;

[0059] Figure 3 Schematic diagram of the YOLO5-Dface network structure provided by an embodiment of the present invention;

[0060] Figure 4 is a schematic diagram of a convolution receptive field provided by an embodiment of the present invention;

[0061] Figure 5 is a schematic diagram comparing conventional convolution and dilated convolution provided by an embodiment of the present invention;

[0062] Figure 6 Schematic diagram of the change of the receptive field of the dilated convolution provided by an embodiment of the present invention;

[0063] Figure 7 is a schematic diagram of the SPP module structure provided by an embodiment of the present invention;

[0064] Figure 8 is a schematic diagram of a DSPP module network structure provided by an embodiment of the present invention;

[0065] Fig. 9 It is a schematic diagram of the inverted residual structure of MobileNetV2 and MobileFaceNet provided by an embodiment of the present invention;

[0066] Fig.10 is a schematic diagram of the Block structure and detailed parameters of Mobi-FaceNeXt provided in an embodiment of the present invention;

[0067] Fig.11 is a schematic diagram of a coordinate attention module framework diagram provided by an embodiment of the present invention;

[0068] Fig.12Schematic diagram of three different structures of coordinate attention in Block provided by an embodiment of the present invention;

[0069] Fig.13 Schematic diagrams of three common human body modeling methods provided by embodiments of the present invention;

[0070] Fig.14 is a schematic diagram of a Baseline algorithm framework provided by an embodiment of the present invention;

[0071] Fig.15 is a schematic diagram of a Baseline framework + FAMBP module provided in an embodiment of the present invention;

[0072] Fig.16 Schematic diagram of a method for calculating coordinates of key points of a human body provided by an embodiment of the present invention;

[0073] Fig.17 It is a schematic diagram of a FAMBP module provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0075] It is understood that the terms "first", "second", etc. used in the present invention can be used to describe various concepts in the present invention, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determination".

[0076] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0077] Unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the present invention are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0078] The face detection method, face recognition method, behavior recognition method and system provided in the embodiment of the present invention relate to the field of image processing technology. The face detection method, face recognition method and behavior recognition method provided in the embodiment of the present invention can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the face detection method, face recognition method, and behavior recognition method, but is not limited to the above forms.

[0079] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0080] Reference Figure 1 , Figure 1The flow chart of the face detection method, face recognition method and behavior recognition method applied to the server provided by the embodiment of the present invention, the execution subject of the method can be any of the aforementioned computer devices (including servers or terminals). Exemplarily, the method of the present invention can be applied in the invigilation system, and the present invention can solve many problems existing in the existing examination room monitoring system, and provide a lightweight face detection, face recognition and behavior recognition examination room invigilation system based on embedded devices. The current examination room monitoring mainly relies on manual inspections, which not only has high invigilation costs, but also easily affects the on-site status of the examinees. The existing identity recognition efficiency is low, the detection accuracy and real-time performance of the monitoring system are not high, and the dependence on the network is strong. The network signal shielding problem further limits the application scope of the system. In order to solve these problems, the present invention proposes a new type of invigilation system, which combines embedded artificial intelligence technology to realize real-time monitoring and analysis without networking and relying on the local computing power of the device. By combining advanced embedded artificial intelligence technology and optimized algorithms, an efficient, low-power, and real-time examination room monitoring system is realized, which significantly improves the fairness and security of the examination and has broad application prospects.

[0081] For example, in the specific application scenario of invigilation in an examination room, the present invention can solve the following problems:

[0082] (1) Improve monitoring efficiency: The embedded camera collects images in real time and uses the optimized YOLO5-Dface convolutional neural network to perform rapid face detection and recognition, thereby improving the efficiency and accuracy of identity recognition in the examination room.

[0083] (2) Enhanced real-time performance: Through localized computing, efficient and real-time examination room monitoring can be achieved to detect and handle abnormal behaviors of examinees in a timely manner.

[0084] (3) Improve system reliability: The application of embedded artificial intelligence technology enables the system to operate stably in an environment without a network or with a limited network, ensuring that the monitoring function is not affected.

[0085] (4) Reduce disruption to examinees: The automated monitoring system reduces disruption to examinees caused by examiners’ patrolling, creating a fairer and quieter examination environment.

[0086] (5) Comprehensive behavior recognition: The behavior recognition module based on CNN, Transformer and single-stage multi-person pose estimation baseline algorithm can accurately identify various behaviors of candidates, including cheating behaviors such as eye wandering, communicating with others and using electronic devices.

[0087] (6) Alarm and Recording: The system integrates an alarm module that issues real-time alarms when abnormal behavior is detected and records relevant images and videos, providing strong evidence support for examination room managers.

[0088] Reference Figure 1 The method can include the following three parts:

[0089] 1. Face detection algorithm based on embedded system:

[0090] 2. Design of lightweight face recognition algorithm based on embedded system;

[0091] 3. Design of single-stage multi-person behavior recognition algorithm based on embedded system.

[0092] Specifically, the face detection process in the first part includes the following:

[0093] The YOLOv5 algorithm is an end-to-end single-stage target detection algorithm. The model regards target detection as a regression problem of the target detection box. During detection, the image is transmitted to the network model at one time, and the output result is the detection box, target category and other information after regression processing. Its network structure is as follows: Figure 2 As shown:

[0094] YOLO5Face is improved on the basis of YOLOv5, such as Figure 3 As shown, a key point regression strategy is added therein. While the model regresses the face position, the coordinate information of the five key points on the face is regressed. In order to overcome the problem that the loss function in v5 is insensitive to small error data, a wing loss function is introduced. Similarly, YOLO5Face also designs detection models of different sizes, from large models that can achieve optimal performance to super small models for real-time detection on embedded or mobile devices, and as the model size is reduced, the reasoning speed gradually increases, but the detection performance will be affected to a certain extent. Therefore, for the application of face detection on embedded systems in the present invention, the lightweight face detection model YOLO5Face is selected as the baseline algorithm detected by the present invention after comparison. The main advantage of YOLO5Face is that it can be realized to run smoothly on embedded systems and maintain a high reasoning speed. At the same time, YOLO5Face can regress the detected face key points, but the less receptive field range in the network leads to the insufficient ability of the model to obtain larger feature information.

[0095] In order to solve the above problems, the present invention improves and designs the YOLO5-Dface lightweight face detection model based on YOLO5Face, mainly introducing a hole convolution structure and residual connection to increase the receptive field of the network and further improve the network accuracy. At the same time, the present invention also uses the regressed face key points to optimize the face posture and screen out small posture faces suitable for the next step of face recognition, thereby reducing the time cost spent by the system for face recognition and making the model more suitable for running on embedded systems.

[0096] The following is an introduction to the contents of the improved DSPP based on dilated convolution according to an embodiment of the present invention:

[0097] Dilated convolution: Dilated convolution and residual connection operations are introduced into the YOLO5Face network. By improving and optimizing the SPP in the model, the receptive field of the model is further expanded, and the identity mapping when extracting image features is achieved. The DSPP module is designed.

[0098] Receptive field is one of the most important basic concepts in convolutional neural networks. It describes the area where the pixel points on the output feature map are projected on the input feature map. Each area on the input image corresponds to a pixel point in the output according to the size of the convolution kernel. The larger the area corresponding to the output pixel point, the larger the receptive field. In the convolution layer, each pixel point on the output feature map depends on an area of ​​the input image and does not intersect with any input information outside this area. This dependent area is the receptive field. Therefore, the receptive field can also be understood as the mapping of an area of ​​the input feature to a pixel point of the output feature. The size relationship between the input and output feature maps is shown in Formula 3.1:

[0099]

[0100] where n out Indicates the size of the output feature map, n in represents the size of the input feature map, p represents the size of the feature map border padding, different sizes of convolution kernels have different outputs, f represents the size of the convolution sum, and s represents the step size of the convolution kernel after each convolution operation. When continuous convolution operations are performed on an image, the size of the output of each convolution operation will be reduced accordingly, such as Figure 4 As shown in Figure 1, the size of the output feature map changes after continuous convolution operations.

[0101] Figure 4The left side shows the change process of the first convolution, and the right side shows the change process of two consecutive convolution operations. In the first convolution operation, one pixel of the output feature corresponds to a 3×3 area on the input feature map. The size of this area is the size of the receptive field. The second convolution operation is shown in the yellow part. At this time, each yellow pixel is mapped to the input feature map with a size of 7×7. Compared with the first convolution operation, the range of the receptive field will be increased on the original basis. Formula 3.2 shows the calculation formula for the receptive field size:

[0102]

[0103] Among them, RF l+1 Indicates the size of the receptive field on the feature map after the convolution operation, RF l It represents the size of the receptive field of the previous convolution operation corresponding to the input feature map, F l+1 It represents the size of the convolution kernel of the convolution layer. Represents the product of the stride of each previous convolution operation.

[0104] Dilated convolution is also called expanded convolution or dilated convolution. The commonly used operation is to insert one or more spaces between each adjacent element of the convolution kernel. The size of the convolution kernel is expanded by the inserted spaces, so that each convolution operation maps a larger area to the feature map, thereby expanding the convolution kernel's perception range of the network in each convolution and increasing the size of the receptive field. Figure 5 The figure shows the convolution process of conventional convolution and dilated convolution. Compared with the convolution kernels in conventional convolution, which are closely adjacent, there are more intervals between the convolution kernels in dilated convolution. The receptive field range of the convolution kernel is consistent with the size of the convolution kernel, while the convolution kernels of dilated convolution are not close to each other. Adjacent elements are blocked by some spaces, which adds intervals between adjacent elements of the convolution kernel, thereby increasing the coverage of each convolution operation.

[0105] The size of the convolution kernel of the dilated convolution is related to the size of the original convolution kernel and the dilation rate of the convolution. The larger the dilation rate, the larger the convolution kernel of the dilated convolution and the larger the corresponding receptive field. Formula 3.3 represents the size of the convolution kernel of the dilated convolution kernel after the original size convolution kernel is expanded:

[0106] K=K1+(K1-1)(x-1)#(3.3)

[0107] In the formula, K represents the size of the convolution kernel of the dilated atrous convolution, K1 represents the size of the convolution kernel of the regular convolution, and x represents the dilation rate of the atrous convolution, also called the dilation coefficient. Except for the dilation of the convolution kernel size, the rest of the convolution calculation process of the atrous convolution is the same as that of the regular convolution, which is the multiplication and addition of corresponding elements. Figure 6The following shows the changes in the convolution kernel size and receptive field range when different expansion rates are used:

[0108] The dilated convolution effectively expands the range of the receptive field, and because the relevant parameters of the convolution operation are consistent with those of the conventional convolution, the receptive field of all dilated convolutions is larger and does not incur a greater computational cost due to the expansion of the convolution kernel. Therefore, the use of dilated convolution can effectively expand the range of the receptive field without incurring additional computational costs. Usually, a large number of pooling layer operations are used in the neural network structure for downsampling. The result of using a large number of pooling layers is that the network will lose some relevant information, which will have a certain impact when decoding upsampling and reconstructing the resolution. The use of dilated convolution can effectively solve this problem. Adding dilated convolution operations with different expansion rates to the detection model further expands the range of the network model's receptive field, enhances the model's ability to obtain contextual information and large-size feature information, and completes the information fusion between the upper and lower layers through the residual connection operation, realizing the identity mapping when extracting image features, thereby enhancing the ability of YOLO5-Dface to detect faces.

[0109] The following is a detailed description of the DSPP module design (depth-separable pyramid pooling):

[0110] The main function of the spatial pyramid pooling layer (SPP) is to expand the receptive field, fuse the information of feature maps of different scales, and complete feature fusion. Figure 7 The figure shows the structure of SPP in YOLO5Face, which consists of three pooling layers Maxp-ppl of different scales and two convolution blocks CBS. The input features are first extracted by the convolution block CBS to extract the feature information of the image. The number of channels of the output feature map is halved after the feature extraction. Then, the input is subjected to the maximum pooling operation of different scales, and the largest pixel in the convolution area is retained. After that, the output of the pooling layer is subjected to feature splicing operation and fused with the input features of the CBS convolution block. Finally, the fused features are convolutionally restored. Although SPP can effectively process input images of different sizes and resolutions, the process of processing images also causes the model to lose a large number of original features. At the same time, the small range of the receptive field also makes the model insufficient in extracting the overall image features.

[0111] like Figure 8As shown, based on the original SPP, the present invention adds four hole convolutions with different expansion rates to perform convolution operations on the input, and finally performs a fusion operation. Through the convolution operation with different expansion coefficients, the expansion of the network receptive field is completed, and then through a residual connection operation, the information fusion between the upper and lower layers is completed to achieve the identity mapping when extracting image features. The input feature map completes the feature extraction within the range of different receptive fields through four hole convolutions with different expansion rates, and is fused with the original input features after the convolution operation, enhancing the ability of the hole convolution operation to extract complete image feature information, realizing the fusion of feature information between the front and rear layers after the hole convolution operation, and finally fused with the input features of the SPP module to complete the model's extraction of feature information.

[0112] The following is a detailed description of the improved detection network based on posture optimization according to an embodiment of the present invention:

[0113] As an important part of the face recognition system, the accuracy of face detection results will directly affect the accuracy and reasoning speed of subsequent face recognition. In order to further improve the efficiency of the next step of face recognition, a face posture optimization strategy is introduced after face detection to remove large posture faces that do not meet the recognition criteria in advance and reduce the number of faces to be identified. The present invention optimizes the posture of the face on the basis of YOLO5-Dface, selects five key points of YOLO5-Dface network regression as reference points of the posture optimization method of the present invention, and roughly judges the detected face posture by comparing the relative positions of the five points of the left eye, right eye, nose tip, left corner of the mouth, and right corner of the mouth, thereby eliminating faces with larger deflection angles, and only retaining small-angle faces that can be correctly recognized by the subsequent face recognition model within the specified angle range.

[0114] In an embodiment of the present invention, the five key points of the face can be marked with X, Y, the left eye (x1, y1), the right eye (x2, y2), the nose tip (x3, y3), the left corner of the mouth (x4, y4), and the right corner of the mouth (x5, y5). The degree of left-right deflection and up-down deflection of the face posture is calculated according to the five key points of the face and the following formula. Formula 3.4 shows the calculation of the left-right deflection of the face posture, and the approximate deflection position is calculated by the distance between the eyes and the corners of the mouth:

[0115]

[0116] The left-right deflection posture of the face is determined by calculating the ratio of the distance between the left eye and the left corner of the mouth and the right eye and the right corner of the mouth to the tip of the nose. Due to the symmetry of the face, when there is no left-right deflection, the left and right distances of the face are roughly equal. At this time, the value of the left-right deflection is equal to 1. When the left-right deflection occurs, the value obtained will swing left and right with 1 as the center. When the left-right deflection value is between 0.3 and 3, it is most suitable for the subsequent face recognition network to complete face recognition.

[0117]

[0118] The up and down deflection posture of the face is determined by calculating the ratio of the distance between the left eye and the right eye to the distance between the left eye and the left corner of the mouth. In the absence of up and down deflection, the up and down deflection coefficient of most faces is about 0.72. When up and down deflection occurs, the obtained value will gradually increase. When the up and down deflection coefficient is between 0.65 and 1.3, it is most suitable for the subsequent face recognition network to complete face recognition.

[0119] The experimental results and analysis of the face detection model algorithm provided by the embodiment of the present invention are as follows:

[0120] (a) Experimental results and analysis of face detection model algorithm

[0121] In order to verify the feasibility and effectiveness of the face detection model proposed in the present invention, a comparative verification experiment was conducted on the WiderFace face dataset with commonly used popular algorithms, wherein the comparative algorithms include: RetinaFace, DSFD, HAMBox, TinaFace, SCRFD-10GF, S3FD, YOLOv5 and the baseline algorithm YOLO5FACE. In order to better test the performance of the face detection model, the present invention verifies the detection accuracy on three subsets of different standard difficulties in the WiderFace face detection dataset, wherein the training set in the dataset widerface is used as the training set of the model, and the performance of the model is tested in its verification set. The detection comparison effect of the model is shown in Table 1:

[0122] Table 1

[0123]

[0124] From the results in Table 1, it can be seen that compared with the baseline algorithm, the improved YOLO5-Dface of the present invention has achieved good results. Compared with the baseline algorithm, Easy has been improved by 0.16, Medium has been improved by 0.65, and Hard has been improved by 0.32, while the size of the model has only increased by less than 1M. This is mainly because the improved YOLO5-Dface designed by the present invention further expands the receptive field of the model and strengthens the model's overall extraction ability of the feature map. The performance of the improved model exceeds that of the TinaFace algorithm, the SCRFD-10G algorithm, the DSFD algorithm, and other algorithms, proving that the improvement of the present invention is practical and effective.

[0125] (b) Experimental results and analysis of face pose optimization strategy

[0126] In order to verify the feasibility of the face posture optimization strategy proposed in the present invention, the detection results of the model before optimization and the detection results of the model after optimization were compared. The detection videos of the two models are consistent.

[0127] Table 2

[0128]

[0129] Table 2 shows the test results of the posture optimization experiment. This experiment tested 8 videos of different lengths. Due to the changes in video scenes and the movement of pedestrians, the number of faces contained in each frame of each video is different. From the experimental data, it can be seen that the number of detections of the detection model after optimization is greatly reduced compared with before, and the number of faces to be recognized input into the face recognition network is also less, which can effectively reduce the recognition time of the system and improve the recognition efficiency of the system.

[0130] The following is a detailed description of the design of a lightweight face recognition algorithm based on an embedded system of the present invention:

[0131] 1. Network structure design:

[0132] Based on the MobileNet series of networks and the MobileFaceNet network, combined with the advantages of current excellent lightweight networks, and referring to excellent networks such as Swin-Transformer that have good performance in the field of computer vision, the network is optimized and designed to obtain the lightweight neural network Mobi-FaceNeXt specifically for facial feature extraction tasks.

[0133] Both MobileNetV2 and MobileFaceNet use the same inverted residual structure (with different activation functions), such as Fig. 9As shown, more information features are retained, the performance of the model is guaranteed, and the depth separable convolution mentioned above is more suitable, so as to achieve the purpose of reducing the amount of parameters and the amount of calculation while completing feature extraction. The Mobi-FaceNeXt network designed by the present invention also adopts a similar bottleneck structure. In addition, Mobi-FaceNeXt adopts a fast downsampling strategy at the beginning of the network, which can quickly reduce the size of the input feature map, thereby reducing the amount of calculation of the subsequent convolution layer. At the same time, the dimension reduction strategy is adopted earlier in the subsequent convolution layer, which can further reduce the amount of calculation and the amount of parameters of the network.

[0134] In the face recognition task, using a larger convolution kernel can improve the model's perception of the face area, thereby improving the accuracy. Therefore, the present invention changes the convolution kernel size in the inverted residual structure to 7×7 to improve the accuracy of the face recognition algorithm. Although the accuracy is improved, the problem brought about is the increase in the number of parameters and the amount of calculation. The more complex and inefficient multi-head self-attention module is set with fewer channels, and the more efficient and more numerous 1×1 convolution layers complete more complex and arduous tasks. The multi-head self-attention module is similar to the deep convolution (Depthwi se conv), so in the network structure design, the deep convolution is moved up, and the deep convolution layer can extract higher-level features. Moving it up can make the model better learn these features, and more parameters can be placed in a shallower layer, thereby reducing the complexity of the model. In the selection of the activation function, the present invention adopts the PReLU activation function. The PReLU activation function is an optimized version of the ReLU activation function. Compared with the ReLU function, the PReLU activation function parameter amount is almost increased, which can reduce the risk of overfitting and improve the fitting ability of the network model. In addition, in deeper network layers, the present invention uses the HardSwish activation function to replace the PReLU function, which makes training easier and improves the convergence speed of training. The HardSwish function is a lightweight version of the Swish function, which can effectively reduce the amount of calculation of the Swish function. It is very suitable for lightweight neural networks and will not affect the accuracy of the model.

[0135] After the above optimization and design, we can get Fig.10 The new inverted residual structure shown in Figure 1, where H, W, and C represent the height, width, and number of channels of the input, respectively, K is the size of the convolution kernel, S is the stride, Exp Size is the number of expansion channels, BN is batch normalization, and the activation function is PReLU or HardSwish.

[0136] 2. Attention mechanism design:

[0137] Attention Mechanism is a mechanism that simulates human sensory systems such as vision or hearing. It enables the neural network to assign different importance or attention to different parts of the input data when processing information, thereby improving the efficiency and performance of the network in processing information. The present invention uses the coordinate attention mechanism (Coordinate Attention.CA) that takes into account both channel information and spatial position in the selection of the attention mechanism, which is used for the face feature extraction network Mobi-FaceNeXt introduced above. The coordinate attention mechanism embeds the position information into the channel attention, so that the network can focus on a larger important area at a lower computational cost. The coordinate attention mechanism operation is mainly divided into three steps: the first step is to use two global pooling operations to encode each channel horizontally and vertically along the horizontal (X) direction and the vertical (Y) direction respectively; the second step is to concatenate the pooling layers in the above two directions, cascade the output, and then perform a 1×1 convolution transformation operation on it to enable information interaction in two directions, and then use the BN operation and the HardSwish nonlinear activation function operation; the third step is to divide the obtained tensor into two separate tensors, and perform convolution and Sigmoid operations on them respectively to generate attention vectors in the horizontal and vertical directions with the same number of channels. The specific structure is as follows: Fig.11 shown.

[0138] Combining the novel inverted residual structure mentioned above with the coordinate attention module, the present invention proposes three structures according to the insertion position of the coordinate attention module: deep structure (DCA), deep point-by-point structure (DPCA) and final point-by-point structure (LPCA), such as Fig.12 shown.

[0139] 3. Design of joint loss function:

[0140] Based on MagFaceLoss and CenterLoss, a new joint loss function L is formed:

[0141]

[0142] λ is the weight factor. The first part of the new joint loss function uses the MagFaceLoss loss function to evaluate the quality of the image based on the modulus of the image feature, pulls the easy-to-identify simple samples closer to the center of the class, and pushes the difficult samples away from the center of the class, while maintaining the distance between classes and maintaining a good distribution structure within the class. The second part uses Center Loss to reduce the distance between edge samples and the center of the class, making the distribution within the class more compact. Through the joint loss function, while taking into account a good distribution within the class, the class is made more compact and a good distance between classes is maintained.

[0143] 4 Experimental results and analysis

[0144] The present invention uses the Mobi-FaceNeXt network without the attention mechanism as the baseline, and uses the designed joint loss function to obtain the recognition results in different data sets as shown in Table 3 below:

[0145] Table 3

[0146]

[0147] Combining the three structures mentioned above, namely the deep structure (DCA), the deep point-by-point structure (DPCA) and the final point-by-point structure (LPCA), corresponding training and testing experiments were carried out respectively, and the results are shown in Table 4.

[0148] Table 4

[0149]

[0150] It can be seen that based on excellent lightweight networks such as the MobileNet series, and referring to the design ideas of excellent network models with good performance in the field of computer vision, the network is optimized and designed, and by embedding a coordinate attention mechanism module that takes into account both channel information and spatial position, a lightweight neural network Mobi-FaceNeXt network specifically for face feature extraction tasks is obtained. Combined with the joint loss function, the amount of calculation and parameters meet the requirements of embedded devices, can run in embedded devices, and the accuracy of the model can be guaranteed.

[0151] The following is a detailed description of the design of a single-stage multi-person behavior recognition algorithm based on an embedded system provided by an embodiment of the present invention:

[0152] 1. Introduction to the single-stage multi-person behavior recognition algorithm framework:

[0153] Many backbone networks have been designed according to the characteristics of multi-person behavior recognition tasks. High-Resolution Network (HRNet), Deep Layer Aggression Network (DLA) and Residual Neural Network (ResNet) are three backbone networks with better effects and more applications.

[0154] like Fig.13 Shown are three common ways of modeling the human body:

[0155] Each human body instance is labeled with 17 key points. These 17 key point annotations correspond to 17 key parts of the human body. The specific annotations and the corresponding human body parts are as follows:

[0156] a) nose(nose), b) left_eye(left eye), c) right_eye(right eye), d) left_ear(left ear), e) right_ear(right ear), f) left_shoulder(left shoulder), g) right_shoulder(right shoulder), h) left_elbow(left elbow), i) right_elbow(right elbow), j) left_wrist(left wrist), k) right_wrist(right wrist), l) left_hip(left hip), m) right_hip(right hip), n) left_knee(left knee), o) right_knee(right knee), p) left_ankle(left ankle), q) right_ankle(right ankle).

[0157] like Fig.14 The figure shows the multi-person posture estimation baseline algorithm framework selected by the present invention based on the idea of ​​OneStage multi-person posture estimation method. The overall process of the algorithm framework is: the input image is extracted through the backbone network model, and the output features are further processed by multiple heads. The coordinates of the human body center point output by head1 and the offset of the human body key points output by head2 can be combined to obtain the final coordinates of the human body key points. Taking into account the size and effect of network parameters, the backbone network of the present invention adopts HRNet, because it guarantees accuracy while ensuring a small amount of parameters. This is an algorithm framework that is more suitable for embedded devices.

[0158] The backbone network in the Baseline algorithm framework is funnel-shaped, which indicates that there will be an upsampling process after downsampling. This is because human target positioning is required, so the resolution of the output feature map needs to be large to distinguish multiple people in the picture. The feature map finally output by the backbone network is usually 1 / 4 of the input image resolution at a specific resolution (for example, when the resolution of the input image is 512×512, the resolution of the feature map output by the backbone network is 128×128). The Baseline algorithm framework designs four heads to further process the features extracted by the backbone network to make predictions of human key points. The algorithm framework is relatively simple, making it easier to deploy the algorithm later, and the algorithm can be efficiently migrated and applied on different platforms.

[0159] 2. Optimization of the single-stage multi-person pose estimation algorithm framework:

[0160] Feature Alignment Module based on Body Parts: The Baseline algorithm framework uses a feature vector to regress and predict 17 key points of the human body. It is very difficult to achieve such a large number of coordinate offsets, and the receptive field size of the feature vector at the center of the human body is limited. It can only perceive the spatial feature information near it, so its ability to predict the coordinates of these positions is limited, which affects the final prediction effect. In order to alleviate the above problems, the present invention adopts a Feature Alignment Module based on Body Parts (FAMBP). After adding FAMBP, the overall framework of the algorithm is as follows: Fig.15 As shown in the figure, the green FAMBP module is embedded in the position before head2 outputs the key point offset. Before head2 outputs the human body key point offset, it is optimized by the FAMBP module.

[0161] The FAMBP module mainly utilizes the characteristics of the human body's skeletal structure. Although the human body can present a variety of postures, it is limited by the rigid body characteristics of the skeleton and local parts cannot undergo large deformations. For example, the structure of the facial features of each person's head is similar and fixed. Therefore, the feature vector near the head position can be responsible for predicting the offset of the key points of the facial features. In this way, the feature vector responsible for prediction must be close to the key points in space, and its receptive field can cover the position of the key points well, thereby improving the prediction effect. The specific principle is as follows Fig.16 As shown, the red circle indicates the position P of the center point of the human body center , the blue circle indicates the position near a certain body part (such as near the right arm)P part , the green circle indicates the position P of a key point of the human body contained in this part keypoint (e.g. right wrist in right arm).

[0162] The first step is to use P center The feature vector at predicts the offset u from the nearby position of the body part to the center point part , use formula 3.7 to calculate the position Ppart of the part.

[0163] P part =P center +u part #3.7

[0164] The second step is to use P part The feature vector at predicts the offset u of a key point to the position near the body part keypoint , use formula 3.8 to calculate the position P of the key point keypoint .

[0165] P keypoint =Ppart +u keypoint #3.8

[0166] Combining Formula 3.7 and Formula 3.8, we can get a method for calculating the coordinates of a key point based on the center point, as shown in Formula 3.9.

[0167] P keypoint =P center +u part +u keypoint #3.9

[0168] The human body structure-based feature alignment module FAMBP changes the human body key point coordinate prediction from one step to two steps, which can alleviate the serious deviation between the receptive field of the output feature and the actual predicted position.

[0169] FAMBP changes the prediction of human key point coordinates from one step to two steps. Fig.17 As shown on the left, each channel of the key point offset feature map F output by head2 is no longer directly responsible for predicting the offset of each human key point relative to the center point of the human body, but is responsible for predicting the offset of the position near the human body part relative to the center point of the human body. i 1 (where 1≤i≤17 represents the i-th key point). Then the second step is to remove the off of the feature map F. i 1 Take out the offset of the key point of the human body relative to the position of the nearby human body part off i 2 , use formula 3.10 to calculate the offset of a key point relative to the center point off i , and turn off i As the final output result.

[0170] off i =off i 1 +off i 2 #3.10

[0171] The present invention uses a bilinear interpolation operation to take the weighted average of four feature vectors near the predicted coordinate position as the actual value.

[0172] The present invention proposes a lightweight face detection, face recognition and behavior recognition system based on embedded devices suitable for examination rooms. By integrating image acquisition module, lightweight face detection module, face recognition module, behavior recognition module and alarm module, high-efficiency, low-power consumption and strong real-time monitoring of examination rooms is realized. The system uses the optimized YOLO5-Dface algorithm for fast face detection, combines MobileNet and MobileFaceNet networks for accurate face recognition, and uses CNN and single-stage multi-person posture estimation Baseline algorithm for comprehensive behavior recognition, effectively detecting abnormal behaviors of examinees, such as eye drifting, communicating with others, using electronic devices, etc.

[0173] Through embedded artificial intelligence technology, the present invention not only solves the problem of existing examination room monitoring systems relying on the network and cloud computing, significantly reducing monitoring costs, but also reduces the interference of examiners' patrols on examinees, creating a more fair and quiet examination environment. The real-time alarm and image and video recording functions of the alarm module provide reliable evidence support for examination room managers, further improving the fairness and security of the examination.

[0174] In summary, the present invention has made significant progress in the intelligentization and automation of the examination room monitoring system, and has broad application prospects and important practical value. Through the application of this system, the efficiency and accuracy of examination room monitoring can be effectively improved, providing a strong technical guarantee for the construction of a modern examination system.

[0175] The embodiment of the present invention further provides an examination room monitoring system, comprising:

[0176] An image acquisition module, used for acquiring images to be detected;

[0177] A lightweight face detection module, used to implement the face detection method as described above;

[0178] A face recognition module, used to implement the face recognition method as described above;

[0179] A behavior recognition module, used to implement the behavior recognition method as described above;

[0180] The alarm module is used to issue alarm prompts based on the face recognition results and behavior recognition results.

[0181] It should be noted that the system provided by the embodiment of the present invention includes an image acquisition module, a lightweight face detection module, a face recognition module, a behavior recognition module and an alarm module. The image acquisition module uses an embedded camera to collect images in the examination room in real time. The lightweight face detection module uses the optimized convolutional neural network YOLO5-Dface to quickly detect and locate the face of the image. The face recognition module is based on the MobileNet series network and the MobileFaceNet network, and includes the Mobi-FaceNeXt network of the attention mechanism. The behavior recognition module designs a behavior recognition module based on CNN and a single-stage multi-person posture estimation Baseline algorithm to identify the behavior of the examinee by analyzing the detected face and its surrounding environment, including but not limited to whether there is cheating behavior, such as eye drifting, communicating with others, using electronic devices, etc. When the alarm module detects abnormal behavior, it issues an alarm in real time and records relevant images and videos. All of the above modules are deployed in an embedded system. The system has the characteristics of high efficiency, low power consumption, and strong real-time performance. It can be implemented on resource-constrained embedded devices, is suitable for examination room monitoring, and improves the fairness and security of the examination.

[0182] The embodiment of the present invention further provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned face detection method, face recognition method, and behavior recognition method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0183] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0184] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned face detection method, face recognition method, and behavior recognition method are implemented.

[0185] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0186] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0187] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art can appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0188] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0189] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0190] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0191] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0192] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0193] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0194] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0195] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0196] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store programs.

[0197] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the embodiments of the present invention is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present invention shall be within the scope of the rights of the embodiments of the present invention.

Claims

1. A face detection method, characterized in that: The following steps are involved: Acquire the image to be detected; Transmitting the image to be detected to a face detection network model for face detection, and outputting a face detection result; The detection process of the face detection network model includes: Add hole convolution structure and residual connection to the YOLO5-Dface lightweight face detection model for detection processing; The posture of the face is optimized according to the result of the detection processing to determine the face detection result.

2. A face detection method according to claim 1, characterized in that: The method of adding a dilated convolution structure and a residual connection to the YOLO5-Dface lightweight face detection model for detection processing includes the following steps: Four dilation convolutions with different expansion rates are added to extract feature information of the image to be detected, and through a residual connection operation, information fusion between the input image and the convolved image is completed to output a feature map; Performing maximum pooling operations of different scales on the feature maps respectively, and retaining the largest pixel in the convolution area; Perform feature concatenation on the output of the maximum pooling operation and fuse it with the input features of the CBS convolution block; The fused features are convolutionally restored to complete the detection process.

3. A face detection method according to claim 1, characterized in that: The step of performing face posture optimization according to the result of the detection processing and determining the face detection result comprises the following steps: According to the results of the detection process, five key points of YOLO5-Dface network regression are selected as reference points for posture optimization; By comparing the relative positions of the left eye, right eye, nose tip, left corner of the mouth, and right corner of the mouth, the detected face posture is determined, faces with large deflection angles are eliminated, and faces with small angles that meet the preset conditions are retained.

4. A face detection method according to claim 3, characterized in that: The method comprises the following steps: judging the detected facial posture by comparing the relative positions of the five points of the left eye, the right eye, the nose tip, the left corner of the mouth, and the right corner of the mouth, eliminating the faces with large deflection angles, and retaining the faces with small angles that meet the preset conditions: The five key points of the face are marked with X and Y, namely, the left eye (x1, y1), the right eye (x2, y2), the nose tip (x3, y3), the left corner of the mouth (x4, y4), and the right corner of the mouth (x5, y5). Based on the five key points of the face, the degree of left-right and up-down deflection of the face posture is calculated; The deflection position of the face is calculated by the distance between the eyes and the corners of the mouth, and the left and right deflection posture of the face is determined by calculating the ratio of the distance between the left eye and the left corner of the mouth, and the distance between the right eye and the right corner of the mouth to the tip of the nose; The calculation formula for the degree of left and right deflection is: The calculation formula for the degree of up and down deflection is:

5. A face recognition method, characterized in that: include: Obtaining a face detection result for the image to be detected; wherein the face detection result is obtained according to the face detection method according to any one of claims 1 to 4; Downsampling the face detection result using a fast downsampling strategy to reduce the input feature map in the face detection result; A facial feature extraction network of a novel inverted residual structure is constructed to process the result of the downsampling process, wherein the construction process of the novel inverted residual structure is as follows: the convolution kernel size in the inverted residual structure is changed to 7×7; the number of channels set in the multi-head self-attention module is reduced; a PReLU activation function is used in a shallower network layer, and a HardSwish activation function is used to replace the PReLU activation function in a deeper network layer; Applying a coordinate attention mechanism that takes into account both channel information and spatial position to the face feature extraction network to perform face feature extraction processing; The coordinate attention mechanism is combined with the novel inverted residual structure, and three network structures are constructed according to the insertion position of the coordinate attention, namely the deep structure, the deep point-by-point structure and the final point-by-point structure; A new joint loss function is formed based on MagFace Loss and Center Loss, and face recognition is performed according to the joint loss function to obtain a face recognition result.

6. A face recognition method according to claim 5, characterized in that: The application process of the coordinate attention mechanism includes the following steps: Using two global pooling operations, each channel is encoded horizontally and vertically along the horizontal and vertical directions respectively; The horizontal and vertical pooling layers are concatenated and output in cascade, and then a 1×1 convolution operation is performed on the cascade output result to interact information in two directions, followed by a BN operation and a HardSwish nonlinear activation function operation; The resulting tensor is split into two separate tensors and subjected to convolution and sigmoid operations respectively to produce attention vectors in the horizontal and vertical directions with the same number of channels.

7. A behavior recognition method, characterized in that: include: Extract features from the image to be identified through the backbone network model; The output features are further processed by multiple detection heads, and the coordinates of the human body center point and the offsets of the human body key points output by the multiple detection heads are combined to obtain the final coordinates of the human body key points; wherein the detection head is embedded with a feature alignment module based on human body parts. Combining the face recognition result and the coordinates of the key points of the human body, a behavior recognition method is generated; wherein the face recognition result is obtained according to the face detection method as described in claim 5 or 6.

8. A face recognition method according to claim 7, characterized in that: The application process of the feature alignment module based on human body parts includes the following steps: Use the position P of the center point of the human body center The feature vector at predicts the offset u from the position near the human body part to the center point of the human body part , and calculate the position Ppart of the human body part; Using P part The feature vector at predicts the offset u of a key point to the position near the human body part keypoint , and calculate the position P of the key point keypoint ; According to the position of the center point of the human body P center , the position of the human body part Ppart, the position of the key point P keypoint , calculate the coordinates of the key point.

9. An examination room monitoring system, characterized in that: include: An image acquisition module, used for acquiring images to be detected; A lightweight face detection module, used to implement the face detection method according to any one of claims 1 to 4; A face recognition module, used to implement the face recognition method according to any one of claims 5-6; A behavior recognition module, used to implement the behavior recognition method according to any one of claims 7-8; The alarm module is used to issue alarm prompts based on the face recognition results and behavior recognition results.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 8.

Citation Information

Cited By

  • Human body posture feature extraction model, method and system and posture estimation network, method and system

    CN120299061A