Lip shielding detection method and device, readable storage medium and electronic equipment
The lip area is extracted by the face key points and the lip occlusion is detected using a pre-trained neural network. The lip semantic guidance assistance network is used for auxiliary training, which solves the problem of insufficient lip occlusion detection in the existing technology and realizes the lip occlusion Effective detection of partial occlusion.
Patent Information
- Application Number
- CN202410823929.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-24
- Publication Date
- 2024-11-08
AI Technical Summary
The prior art lacks an effective lip occlusion detection method, and it is difficult to determine whether the lip is blocked in a specific scenario.
The key points of the face are obtained through face detection, the lip area image is extracted, and the lip occlusion detection is used using a pre-trained neural network, and the lip semantic guidance auxiliary network is combined for auxiliary training to generate lip mask images to improve detection accuracy sex.
Effective detection of lip occlusion can be realized, and it is possible to accurately determine whether the lip is blocked in a specific scenario, improving the accuracy and reliability of lip occlusion detection.
Smart Images

Figure CN118918622A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of image processing technology, and in particular, relates to a lip occlusion detection method, device, computer-readable storage medium and electronic device. Background Art
[0002] In the prior art, there are many mature face occlusion detection methods, that is, to determine whether an object blocks the face. However, in some special scenarios, for example, when determining whether a user is speaking, it is not necessary to perform occlusion detection on the entire face of the user, but it is more desirable to focus on the user's lips for occlusion detection, that is, to determine whether an object blocks the lips. However, compared with face occlusion detection, the prior art still lacks an effective lip occlusion detection method. Summary of the invention
[0003] In view of this, embodiments of the present application provide a lip occlusion detection method, device, computer-readable storage medium, and electronic device to solve the problem of lack of effective lip occlusion detection method in the prior art.
[0004] A first aspect of an embodiment of the present application provides a lip occlusion detection method, which may include:
[0005] Acquire the target image to be detected;
[0006] Performing face detection in the target image to obtain face key points;
[0007] Extracting a lip region image in the target image according to the facial key points;
[0008] Using a preset lip occlusion detection network to perform lip occlusion detection on the lip area image to obtain a lip occlusion detection result;
[0009] The lip occlusion detection network is a pre-trained neural network for lip occlusion detection.
[0010] In a specific implementation of the first aspect, before using a preset lip occlusion detection network to perform lip occlusion detection on the lip region image to obtain a lip occlusion detection result, the method may further include:
[0011] Based on a preset lip semantics guided auxiliary network, the lip occlusion detection network is auxiliary trained to obtain the trained lip occlusion detection network;
[0012] Among them, the lip semantic guidance auxiliary network is used to generate a lip mask image according to the feature data of each layer of the lip occlusion detection network, so as to provide lip semantic guidance for the lip occlusion detection network.
[0013] In a specific implementation of the first aspect, the auxiliary training of the lip occlusion detection network based on the preset lip semantic guidance auxiliary network to obtain the trained lip occlusion detection network may include:
[0014] Using the lip occlusion detection network to perform lip occlusion detection on the sample image of the lip area, and obtain an actual output of the lip occlusion detection;
[0015] Calculating a first training loss according to the actual output of the lip occlusion detection and a preset expected output of the lip occlusion detection;
[0016] Using the lip semantic guidance auxiliary network to generate a lip mask image for each layer feature data of the lip occlusion detection network, and obtaining an actual output of the lip mask image;
[0017] Calculating a second training loss according to the actual output of the lip mask image and a preset expected output of the lip mask image;
[0018] The lip occlusion detection network is trained according to the first training loss and the second training loss to obtain the trained lip occlusion detection network.
[0019] In a specific implementation of the first aspect, before using a preset lip occlusion detection network to perform lip occlusion detection on the lip region image to obtain a lip occlusion detection result, the method may further include:
[0020] Performing image synthesis on the first type of sample images and the lip occlusion template image to obtain the second type of sample images;
[0021] Among them, the first type of sample images are sample images of the lip area without lip occlusion, the second type of sample images are sample images of the lip area with lip occlusion, and the first type of sample images and the second type of sample images are used to train the lip occlusion detection network.
[0022] In a specific implementation of the first aspect, synthesizing the first type of sample image with the lip occlusion template image to obtain the second type of sample image may include:
[0023] Determining a hand mask image corresponding to the lip occlusion template image;
[0024] The first type of sample images and the lip occlusion template image are synthesized according to the hand mask image to obtain the second type of sample images.
[0025] In a specific implementation of the first aspect, extracting the lip region image in the target image according to the facial key points may include:
[0026] Determine the lip center position according to the lip key points among the facial key points;
[0027] Determining lip width and lip height according to the facial key points;
[0028] The lip area image in the target image is extracted according to the lip center position, the lip width and the lip height.
[0029] In a specific implementation of the first aspect, determining the lip width and the lip height according to the facial key points may include:
[0030] Determine the face width and face height according to the face key points;
[0031] Determining the lip width according to the face width and a preset width ratio coefficient;
[0032] The lip height is determined according to the face height and a preset height ratio coefficient.
[0033] A second aspect of an embodiment of the present application provides a lip occlusion detection device, which may include:
[0034] A target image acquisition module is used to acquire a target image to be detected;
[0035] A face detection module, used to perform face detection in the target image to obtain face key points;
[0036] A lip region image extraction module, used to extract the lip region image in the target image according to the facial key points;
[0037] A lip occlusion detection module, used to perform lip occlusion detection on the lip area image using a preset lip occlusion detection network to obtain a lip occlusion detection result;
[0038] The lip occlusion detection network is a pre-trained neural network for lip occlusion detection.
[0039] In a specific implementation of the second aspect, the lip occlusion detection device may further include:
[0040] An auxiliary training module, used to perform auxiliary training on the lip occlusion detection network based on a preset lip semantic guidance auxiliary network to obtain the trained lip occlusion detection network;
[0041] Among them, the lip semantic guidance auxiliary network is used to generate a lip mask image according to the feature data of each layer of the lip occlusion detection network, so as to provide lip semantic guidance for the lip occlusion detection network.
[0042] In a specific implementation of the second aspect, the auxiliary training module can be specifically used to: use the lip occlusion detection network to perform lip occlusion detection on a sample image of the lip area to obtain an actual output of the lip occlusion detection; calculate a first training loss based on the actual output of the lip occlusion detection and a preset expected output of the lip occlusion detection; use the lip semantics-guided auxiliary network to generate a lip mask image for the feature data of each layer of the lip occlusion detection network to obtain an actual output of the lip mask image; calculate a second training loss based on the actual output of the lip mask image and the preset expected output of the lip mask image; train the lip occlusion detection network based on the first training loss and the second training loss to obtain the trained lip occlusion detection network.
[0043] In a specific implementation of the second aspect, the lip occlusion detection device may further include:
[0044] An image synthesis module, used for synthesizing the first type of sample image with the lip occlusion template image to obtain a second type of sample image;
[0045] Among them, the first type of sample images are sample images of the lip area without lip occlusion, the second type of sample images are sample images of the lip area with lip occlusion, and the first type of sample images and the second type of sample images are used to train the lip occlusion detection network.
[0046] In a specific implementation of the second aspect, the image synthesis module can be specifically used to: determine a hand mask image corresponding to the lip occlusion template image; and perform image synthesis on the first type of sample image and the lip occlusion template image according to the hand mask image to obtain the second type of sample image.
[0047] In a specific implementation of the second aspect, the lip area image extraction module may include:
[0048] A lip center position determination unit, used to determine the lip center position according to the lip key points among the facial key points;
[0049] A lip width and height determination unit, used to determine the lip width and lip height according to the key points of the face;
[0050] A lip area image extraction unit is used to extract the lip area image in the target image according to the lip center position, the lip width and the lip height.
[0051] In a specific implementation of the second aspect, the lip width and height determination unit may be specifically used to: determine the face width and face height according to the face key points;
[0052] Determining the lip width according to the face width and a preset width ratio coefficient;
[0053] The lip height is determined according to the face height and a preset height ratio coefficient.
[0054] A third aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of any one of the above-mentioned lip occlusion detection methods are implemented.
[0055] A fourth aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above-mentioned lip occlusion detection methods when executing the computer program.
[0056] A fifth aspect of an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the steps of any one of the above-mentioned lip occlusion detection methods.
[0057] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the embodiments of the present application obtain a target image to be detected; perform face detection in the target image to obtain face key points; extract the lip area image in the target image based on the face key points; use a preset lip occlusion detection network to perform lip occlusion detection on the lip area image to obtain a lip occlusion detection result; wherein the lip occlusion detection network is a pre-trained neural network for lip occlusion detection. Through the embodiments of the present application, the lip area image can be extracted based on the face key points obtained by face detection, and the lip occlusion detection result can be obtained therefrom using a pre-trained neural network, so that effective detection can be performed on whether there is lip occlusion. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0059] Figure 1 This is a flow chart of an embodiment of a lip occlusion detection method in an embodiment of the present application;
[0060] Figure 2 A schematic diagram of key points of a human face;
[0061] Figure 3A schematic flow chart of extracting a lip region image in a target image based on facial key points;
[0062] Figure 4 This is a schematic diagram of the hand segmentation effect;
[0063] Figure 5 A schematic diagram of a lip occlusion template image and a corresponding hand mask image;
[0064] Figure 6 This is a schematic diagram of the effect of image synthesis;
[0065] Figure 7 This is a schematic diagram of the network structure of the lip occlusion detection network;
[0066] Figure 8 Schematic diagram of the overall network structure of the lip occlusion detection network and the lip semantic guidance auxiliary network;
[0067] Fig. 9 A schematic flowchart of auxiliary training of a lip occlusion detection network based on lip semantics-guided auxiliary network;
[0068] Fig.10 This is a structural diagram of an embodiment of a lip occlusion detection device in an embodiment of the present application;
[0069] Fig.11 This is a schematic block diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION
[0070] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0071] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.
[0072] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0073] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0074] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.
[0075] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0076] In the prior art, there are many mature face occlusion detection methods, that is, to determine whether an object blocks the face. However, in some special scenarios, for example, when determining whether a user is speaking, it is not necessary to perform occlusion detection on the entire face of the user, but it is more desirable to focus on the user's lips for occlusion detection, that is, to determine whether an object blocks the lips. However, compared with face occlusion detection, the prior art still lacks an effective lip occlusion detection method.
[0077] In view of this, embodiments of the present application provide a lip occlusion detection method, device, computer-readable storage medium, and electronic device to solve the problem of lack of effective lip occlusion detection method in the prior art.
[0078] In an embodiment of the present application, the lip area image can be extracted based on the facial key points obtained by face detection, and a pre-trained neural network can be used to obtain the lip occlusion detection result therefrom, so as to effectively detect whether there is lip occlusion.
[0079] The execution subject of the embodiments of the present application may be an electronic device, which may include but is not limited to computing devices such as mobile phones, tablet computers, desktop computers, notebooks, PDAs, robots, and servers.
[0080] See also Figure 1 In the embodiment of the present application, an embodiment of a lip occlusion detection method may include:
[0081] Step S101: Acquire a target image to be detected.
[0082] The target image may be an image containing a face.
[0083] In a specific implementation of the embodiment of the present application, the target image can be directly acquired by the electronic device through its visual sensor.
[0084] In another specific implementation of the embodiment of the present application, the target image can be acquired by other devices and sent to the electronic device through a preset data transmission link.
[0085] Step S102: perform face detection in the target image to obtain face key points.
[0086] In the embodiments of the present application, any face detection method in the prior art can be used to perform face detection according to actual conditions, including but not limited to active shape models (ASM), active appearance models (AAM), cascaded pose regression (CPR), deep convolutional networks (DCNN), task-constrained deep convolutional networks (TCDCN), multi-task cascaded convolutional networks (MTCNN), tweaked convolutional neural networks (TCNN), DAN (Deep Alignment Networks) and other methods, which are not specifically limited in the embodiments of the present application.
[0087] Figure 2 FIG. 1 is a schematic diagram of facial key points, in which a total of 98 facial key points are shown. It should be noted that under different face detection methods, the number of facial key points obtained may vary. Generally, the number of facial key points may include but is not limited to 21 points, 29 points, 68 points, 98 points, 106 points, and other numbers, which are not specifically limited in the present embodiment. For the sake of simplicity, the present embodiment is based on Figure 2 The 98 facial key points shown are used as an example for explanation, and for the case of other numbers of facial key points, the processing method in the embodiment of the present application can also be referred to, and the embodiment of the present application will not be repeated here.
[0088] Step S103: extracting the lip area image in the target image according to the facial key points.
[0089] In a specific implementation method of an embodiment of the present application, the upper lip key point (i.e., key point 79), the lower lip key point (i.e., key point 85), the left lip corner key point (i.e., key point 76) and the right lip corner key point (i.e., key point 82) can be selected from the facial key points respectively, and the image of the area defined by these key points can be directly captured as the lip area image.
[0090] However, considering that the shape and size of each person's lips are different, directly intercepting and scaling them will easily cause lip deformation, making it more difficult to learn features. Therefore, in another specific implementation of the embodiment of the present application, the following method can be used: Figure 3 The process shown extracts the lip region image in the target image:
[0091] Step S1031, determining the lip center position according to the lip key points among the facial key points.
[0092] Specifically, the positions of the lip key points in the facial key points may be averaged and used as the lip center position.
[0093] by Figure 2 Taking the 98 facial key points shown in the figure as an example, key points 76 to 95 are lip key points, and the lip center position can be determined according to the following formula:
[0094]
[0095] Among them, P i is the position of key point i, Lip center The center of the lips.
[0096] Step S1032: determine the lip width and lip height according to the key points of the face.
[0097] Specifically, the face width and face height may be determined first according to the face key points. For example, the minimum bounding rectangle of the face key points may be constructed, and the width and height of the minimum bounding rectangle may be used as the face width and face height, respectively.
[0098] Then, the lip width can be determined according to the face width and the preset width ratio coefficient, and the lip height can be determined according to the face height and the preset height ratio coefficient. For example, the lip width and lip height can be determined according to the following formula:
[0099] Lip_H=K_H×Face_H
[0100] Lip_W=K_W×Face_W
[0101] Among them, Face_H is the face height, K_H is the height ratio coefficient, and its specific value can be set according to the actual situation. For example, it can be set to 0.3 or other values, and the embodiment of the present application does not make specific limitations on this. Lip_H is the lip height, Face_W is the face width, K_W is the width ratio coefficient, and its specific value can be set according to the actual situation. For example, it can be set to 0.7 or other values, and the embodiment of the present application does not make specific limitations on this. Lip_W is the lip width.
[0102] Step S1033: extract the lip area image in the target image according to the lip center position, lip width and lip height.
[0103] After determining the lip center position, lip width and lip height, a rectangular area with the lip center position as the center, the lip width as the width and the lip height as the height in the target image can be used as the lip area, and the lip area image in the target image can be extracted.
[0104] After extracting the lip area image, it can also be scaled to adjust it to a preset standard size. The standard size can be set according to actual conditions, for example, it can be set to a size of 48 pixels (width) × 32 pixels (height) or other sizes, which is not specifically limited in the embodiments of the present application.
[0105] Step S104: Use a preset lip occlusion detection network to perform lip occlusion detection on the lip area image to obtain a lip occlusion detection result.
[0106] The lip occlusion detection network is a pre-trained neural network for lip occlusion detection.
[0107] In an embodiment of the present application, the lip occlusion detection network can be trained in advance through a large number of lip area sample images. The lip area sample images can include lip area sample images with no lip occlusion (referred to as the first type of sample images) and lip area sample images with lip occlusion (referred to as the second type of sample images).
[0108] In practical applications, the first type of sample images with unobstructed lips are often more numerous and easier to collect, while the second type of sample images with obstructed lips are fewer and difficult to collect enough to meet the training requirements. Therefore, in a specific implementation of the embodiment of the present application, the first type of sample images can be synthesized with the lip obstructed template image to obtain the second type of sample images.
[0109] As an example, we can construct hand segmentation data based on the cross-modal segmentation technology Grounded-SAM, and set the prompt word (Prompt) to hand (Hand), then we can get the following Figure 4 The effect of hand segmentation is shown.
[0110] On this basis, if Figure 5 As shown, the hand image can be extracted and used as the lip occlusion template image, and the corresponding hand mask image can be determined. Then, images with the same aspect ratio as the standard size can be cropped from top to bottom, scaled to the standard size, and aligned with the lip area image.
[0111] Next, the first type of sample images and the lip occlusion template image can be synthesized according to the hand mask image to obtain the second type of sample images. In order to ensure that only the occluded hand information is introduced, only a number of lip occlusion template images (e.g., 100 or other numbers) whose hand segmentation ratio is less than a preset threshold (e.g., 75% or other ratio) can be selected, and then corresponding to a large number of first type sample images that need to be enhanced (e.g., 20,000 or other numbers), lip occlusion template images are randomly selected for image synthesis, as shown in the following formula:
[0112] I_new(x,y)=I_1(x,y)if Mask(x,y)=0else I_2(x,y)
[0113] Among them, I_1 is the lip occlusion template image, I_2 is the first type of sample image, Mask is the hand mask image, I_new is the second type of sample image synthesized by the first type of sample image and the lip occlusion template image, and (x, y) is the pixel coordinate.
[0114] This formula indicates that for any pixel coordinate (x, y), if its pixel value in the hand mask image is 0 (that is, indicating that the pixel belongs to the hand), then its pixel value in the second category sample image takes the corresponding pixel value of the lip occlusion template image; if its pixel value in the hand mask image is 1 (that is, indicating that the pixel does not belong to the hand), then its pixel value in the second category sample image takes the corresponding pixel value of the first category sample image.
[0115] In a specific implementation method of an embodiment of the present application, in order to make the image synthesis effect more natural, the synthesized second-type sample image can also be smoothed. The specific smoothing method can be set according to actual conditions. For example, a 3×3 Gaussian operator can be used for smoothing. The embodiment of the present application does not make specific limitations on this.
[0116] Figure 6 The figure shows the effect of image synthesis, where the upper part is the first type of sample image before synthesis, and the lower part is the corresponding second type of sample after synthesis. By means of image synthesis, a sufficient number of second type samples can be quickly obtained to meet the training requirements.
[0117] In the embodiment of the present application, any neural network structure in the prior art can be used as the lip occlusion detection network according to actual conditions.
[0118] As an example, in a specific implementation of the embodiment of the present application, a lip occlusion detection network can be constructed based on the lightweight network shufflenetv2. However, considering that the last feature dimension directly obtained by shufflenetv2 is 1×1×256, the aspect ratio will be destroyed to a certain extent. Therefore, the maximum pooling layer (maxpooling) after the convolution layer (Conv) in the original shufflenetv2 network can be removed to obtain the following: Figure 7 The network structure of the lip occlusion detection network is shown in FIG. Among them, the input image size of the lip occlusion detection network can be 48×32×3. After being processed by the convolution layer (Conv), the output feature dimension is 24×16×64. After being processed by stage 1 (stage1), the output feature dimension is 12×8×128. After being processed by stage 2 (stage2), the output feature dimension is 6×4×128. After being processed by stage 3 (stage3), the output feature dimension is 3×2×256. Finally, after being processed by the global maximum pooling layer (GMP) and the fully connected layer (fc), the lip occlusion detection result with an output dimension of 2 is output, which respectively represents the probability of lip occlusion and the probability of lip non-occlusion.
[0119] In a specific implementation of an embodiment of the present application, the lip occlusion detection network can also be structurally re-parameterized. For example, batch normalization (BN) and 1×1 separable convolution can be added to the lip occlusion detection network to enhance the feature extraction capability without increasing computing power, wherein BN retains the original features, and 1×1 separable convolution is similar to rotating and translating the image, which can obtain more diverse and richer features. The convolution parameters mainly include weights (w) and biases (b). The parameters corresponding to the BN branch before re-parameterization can be defined as w1 and b1, the parameters corresponding to the 1×1 convolution branch can be defined as w2 and b2, and the parameters corresponding to the 3×3 convolution branch can be defined as w3 and b3. These parameters can be merged into new convolution parameters w_new and b_new during re-parameterization. The complex network before re-parameterization is used during network training, and the simple network after re-parameterization is used during actual network reasoning, thereby further improving the performance of the lip occlusion detection network.
[0120] In a specific implementation of an embodiment of the present application, considering that image color disturbances and uncommon occlusions are prone to cause misrecognition, the lip occlusion detection network can be assisted in training based on a preset lip semantics-guided auxiliary network (aux-decoder-guide) during network training to obtain a trained lip occlusion detection network. The lip semantics-guided auxiliary network is used to generate lip mask images based on the feature data of each layer of the lip occlusion detection network to perform lip semantics guidance on the lip occlusion detection network. By introducing the lip semantics-guided auxiliary network for auxiliary training to enhance feature capabilities, when the lips are not occluded, it is expected that the extracted features can reconstruct a lip mask image through the lip semantics-guided auxiliary network. When the lips are occluded, it is expected that the lip semantics-guided auxiliary network predicts an empty black image. By establishing such constraints, the task of lip occlusion can be fundamentally learned.
[0121] Figure 8The figure shows the overall network structure of the lip occlusion detection network and the lip semantic guidance auxiliary network. As shown in the figure, the feature dimension of the stage 3 output of the lip occlusion detection network is 3×2×256. After the convolution layer (Conv_aux) in the lip semantic guidance auxiliary network, the output feature dimension is 3×2×128. After unpooling, the output feature dimension is 6×4×128. It is concatenated with the feature data of the stage 2 output of the lip occlusion detection network with a feature dimension of 6×4×128 to obtain concatenated data with a feature dimension of 6×4×256. After the convolution processing, the feature dimension of the 6×4×256 concatenated data becomes 6×4×128. After unpooling, the output feature dimension is 12×8×128. It is concatenated with the feature data of the stage 1 output of the lip occlusion detection network with a feature dimension of 12×8×128 to obtain concatenated data with a feature dimension of 12×8×256. The 12×8×256 spliced data is processed by convolution, and the feature dimension becomes 12×8×64. After unpooling, the output feature dimension is 24×16×64, which is spliced with the feature data of the first Conv output of the lip occlusion detection network with a feature dimension of 24×16×64 to obtain spliced data with a feature dimension of 24×16×128. The 24×16×128 spliced data is processed by convolution, and the feature dimension becomes 24×16×64. After unpooling, the output feature dimension is 48×32×64, which is spliced with the input image convolution data with a feature dimension of 48×32×64 to obtain spliced data with a feature dimension of 48×32×128, where the input image convolution data is the input image of the lip occlusion detection network (feature dimension is 48×32×3) processed by the convolution layer in the lip semantic guidance auxiliary network. The 48×32×128 spliced data undergoes one convolution process, and the feature dimension becomes 48×32×64. After another convolution process, the result is the actual output of the lip mask image, and its feature dimension is 48×32×2.
[0122] like Fig. 9 As shown, in a specific implementation of the embodiment of the present application, auxiliary training of the lip occlusion detection network based on the lip semantics guided auxiliary network may include the following process:
[0123] Step S901: Use the lip occlusion detection network to perform lip occlusion detection on the sample image of the lip area to obtain the actual output of the lip occlusion detection.
[0124] The specific processing process of the lip occlusion detection network can be found in the previous Figure 7 The corresponding detailed description will not be repeated here.
[0125] Step S902: Calculate a first training loss according to the actual output of the lip occlusion detection and the preset expected output of the lip occlusion detection.
[0126] In a specific implementation of the embodiment of the present application, the annotation data corresponding to the sample image of the lip area can be pre-set and used as the expected output of the lip occlusion detection (i.e., groundtruth), and the error between the actual output of the lip occlusion detection and the expected output of the lip occlusion detection is processed using a preset first loss function to obtain a first training loss. The first loss function can adopt any loss function in the prior art according to actual conditions, and the embodiment of the present application does not specifically limit this.
[0127] Step S903: Use the lip semantics guided auxiliary network to generate a lip mask image for each layer of feature data of the lip occlusion detection network to obtain the actual output of the lip mask image.
[0128] The specific processing process of the lip semantic guidance auxiliary network can be found in the above Figure 8 The corresponding detailed description will not be repeated here.
[0129] Step S904: Calculate the second training loss according to the actual output of the lip mask image and the preset expected output of the lip mask image.
[0130] Among them, the expected output of the lip mask image can be generated based on the facial key points of the lip area sample image. For example, the key points 76 to 87 can be connected into a closed contour through a Bezier curve, and then the expected output of the lip mask image can be generated based on the contour filling.
[0131] In a specific implementation of the embodiment of the present application, a preset second loss function can be used to process the error between the actual output of the lip mask image and the expected output of the lip mask image to obtain a second training loss. The second loss function can adopt any loss function in the prior art according to actual conditions, which may include but is not limited to a celoss function or a diceloss function, and the embodiment of the present application does not specifically limit this.
[0132] Step S905: train the lip occlusion detection network according to the first training loss and the second training loss to obtain a trained lip occlusion detection network.
[0133] In a specific implementation method of an embodiment of the present application, the first training loss and the second training loss can be summed to obtain a superimposed training loss, and network training can be performed on the overall network structure of the lip occlusion detection network and the lip semantic guidance auxiliary network based on the superimposed training loss.
[0134] After the network training is completed, the trained lip occlusion detection network can be put into practical application. It should be noted that the lip semantic guidance auxiliary network is only used in the training process, aiming to help the lip occlusion detection network learn lip features, and will not be used in actual reasoning, so it will not affect the reasoning speed in actual application.
[0135] In summary, the embodiment of the present application obtains a target image to be detected; performs face detection in the target image to obtain face key points; extracts a lip area image in the target image based on the face key points; uses a preset lip occlusion detection network to perform lip occlusion detection on the lip area image to obtain a lip occlusion detection result; wherein the lip occlusion detection network is a pre-trained neural network for performing lip occlusion detection. Through the embodiment of the present application, the lip area image can be extracted based on the face key points obtained by face detection, and the lip occlusion detection result can be obtained therefrom using a pre-trained neural network, so that effective detection can be performed on whether there is lip occlusion.
[0136] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0137] It should be noted that the information collection process involved in this application (such as the facial image collection process) is carried out with the user's knowledge and permission, that is, the information collection process complies with relevant requirements and does not constitute an act that harms the public interest.
[0138] Corresponding to the lip occlusion detection method described in the above embodiment, Fig.10 A structural diagram of an embodiment of a lip occlusion detection device provided in an embodiment of the present application is shown.
[0139] In this embodiment, a lip occlusion detection device may include:
[0140] The target image acquisition module 1001 is used to acquire the target image to be detected;
[0141] A face detection module 1002 is used to perform face detection in the target image to obtain face key points;
[0142] A lip region image extraction module 1003 is used to extract a lip region image in the target image according to the facial key points;
[0143] A lip occlusion detection module 1004 is used to perform lip occlusion detection on the lip region image using a preset lip occlusion detection network to obtain a lip occlusion detection result;
[0144] The lip occlusion detection network is a pre-trained neural network for lip occlusion detection.
[0145] In a specific implementation of the embodiment of the present application, the lip occlusion detection device may further include:
[0146] An auxiliary training module, used to perform auxiliary training on the lip occlusion detection network based on a preset lip semantic guidance auxiliary network to obtain the trained lip occlusion detection network;
[0147] Among them, the lip semantic guidance auxiliary network is used to generate a lip mask image according to the feature data of each layer of the lip occlusion detection network, so as to provide lip semantic guidance for the lip occlusion detection network.
[0148] In a specific implementation method of an embodiment of the present application, the auxiliary training module can be specifically used to: use the lip occlusion detection network to perform lip occlusion detection on a sample image of the lip area to obtain an actual output of the lip occlusion detection; calculate a first training loss based on the actual output of the lip occlusion detection and a preset expected output of the lip occlusion detection; use the lip semantics-guided auxiliary network to generate a lip mask image for the feature data of each layer of the lip occlusion detection network to obtain an actual output of the lip mask image; calculate a second training loss based on the actual output of the lip mask image and a preset expected output of the lip mask image; train the lip occlusion detection network based on the first training loss and the second training loss to obtain the trained lip occlusion detection network.
[0149] In a specific implementation of the embodiment of the present application, the lip occlusion detection device may further include:
[0150] An image synthesis module, used for synthesizing the first type of sample image with the lip occlusion template image to obtain a second type of sample image;
[0151] Among them, the first type of sample images are sample images of the lip area without lip occlusion, the second type of sample images are sample images of the lip area with lip occlusion, and the first type of sample images and the second type of sample images are used to train the lip occlusion detection network.
[0152] In a specific implementation method of an embodiment of the present application, the image synthesis module can be specifically used to: determine a hand mask image corresponding to the lip occlusion template image; and perform image synthesis on the first type of sample image and the lip occlusion template image according to the hand mask image to obtain the second type of sample image.
[0153] In a specific implementation of the embodiment of the present application, the lip area image extraction module may include:
[0154] A lip center position determination unit, used to determine the lip center position according to the lip key points among the facial key points;
[0155] A lip width and height determination unit, used to determine the lip width and lip height according to the key points of the face;
[0156] A lip area image extraction unit is used to extract the lip area image in the target image according to the lip center position, the lip width and the lip height.
[0157] In a specific implementation of the embodiment of the present application, the lip width and height determination unit may be specifically used to: determine the face width and face height according to the face key points;
[0158] Determining the lip width according to the face width and a preset width ratio coefficient;
[0159] The lip height is determined according to the face height and a preset height ratio coefficient.
[0160] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0161] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0162] Fig.11 A schematic block diagram of an electronic device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.
[0163] like Fig.11 As shown, the electronic device 11 of this embodiment includes: a processor 110, a memory 111, and a computer program 112 stored in the memory 111 and executable on the processor 110. When the processor 110 executes the computer program 112, the steps in the above-mentioned lip occlusion detection method embodiments are implemented, for example Figure 1 Alternatively, when the processor 110 executes the computer program 112, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Fig.10 The functions of modules 1001 to 1004 are shown.
[0164] Exemplarily, the computer program 112 may be divided into one or more modules / units, which are stored in the memory 111 and executed by the processor 110 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 112 in the electronic device 11.
[0165] The electronic device 11 may include but is not limited to computing devices such as mobile phones, tablet computers, desktop computers, notebooks, PDAs, robots, and servers. Those skilled in the art will appreciate that Fig.11 It is only an example of the electronic device 11 and does not constitute a limitation of the electronic device 11. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 11 may also include input and output devices, network access devices, buses, etc.
[0166] The processor 110 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0167] The memory 111 may be an internal storage unit of the electronic device 11, such as a hard disk or memory of the electronic device 11. The memory 111 may also be an external storage device of the electronic device 11, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 11. Further, the memory 111 may also include both an internal storage unit of the electronic device 11 and an external storage device. The memory 111 is used to store the computer program and other programs and data required by the electronic device 11. The memory 111 may also be used to temporarily store data that has been output or is to be output.
[0168] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0169] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0170] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0171] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0172] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0173] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0174] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.
[0175] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A lip occlusion detection method, characterized in that: include: Acquire the target image to be detected; Performing face detection in the target image to obtain face key points; Extracting a lip region image in the target image according to the facial key points; Using a preset lip occlusion detection network to perform lip occlusion detection on the lip area image to obtain a lip occlusion detection result; The lip occlusion detection network is a pre-trained neural network for lip occlusion detection.
2. The lip occlusion detection method according to claim 1, characterized in that: Before using a preset lip occlusion detection network to perform lip occlusion detection on the lip region image to obtain a lip occlusion detection result, the method further includes: Based on the preset lip semantics guided auxiliary network, the lip occlusion detection network is assisted in training to obtain the trained lip occlusion detection network; Among them, the lip semantic guidance auxiliary network is used to generate a lip mask image according to the feature data of each layer of the lip occlusion detection network, so as to provide lip semantic guidance for the lip occlusion detection network.
3. The lip occlusion detection method according to claim 2, characterized in that: The method of performing auxiliary training on the lip occlusion detection network based on the preset lip semantic guidance auxiliary network to obtain the trained lip occlusion detection network includes: Using the lip occlusion detection network to perform lip occlusion detection on the sample image of the lip area, and obtain an actual output of the lip occlusion detection; Calculating a first training loss according to the actual output of the lip occlusion detection and a preset expected output of the lip occlusion detection; Using the lip semantic guidance auxiliary network to generate a lip mask image for each layer feature data of the lip occlusion detection network, and obtaining an actual output of the lip mask image; Calculating a second training loss according to the actual output of the lip mask image and a preset expected output of the lip mask image; The lip occlusion detection network is trained according to the first training loss and the second training loss to obtain the trained lip occlusion detection network.
4. The lip occlusion detection method according to claim 1, characterized in that: Before using a preset lip occlusion detection network to perform lip occlusion detection on the lip region image to obtain a lip occlusion detection result, the method further includes: Performing image synthesis on the first type of sample images and the lip occlusion template image to obtain the second type of sample images; Among them, the first type of sample images are sample images of the lip area without lip occlusion, the second type of sample images are sample images of the lip area with lip occlusion, and the first type of sample images and the second type of sample images are used to train the lip occlusion detection network.
5. The lip occlusion detection method according to claim 4, characterized in that: The step of synthesizing the first type of sample image with the lip occlusion template image to obtain the second type of sample image includes: Determining a hand mask image corresponding to the lip occlusion template image; The first type of sample images and the lip occlusion template image are synthesized according to the hand mask image to obtain the second type of sample images.
6. The lip occlusion detection method according to any one of claims 1 to 5, characterized in that: The step of extracting the lip region image in the target image according to the facial key points includes: Determine the lip center position according to the lip key points among the facial key points; Determining lip width and lip height according to the facial key points; The lip area image in the target image is extracted according to the lip center position, the lip width and the lip height.
7. The lip occlusion detection method according to claim 6, characterized in that: The determining the lip width and the lip height according to the key points of the face includes: Determine the face width and face height according to the face key points; Determining the lip width according to the face width and a preset width ratio coefficient; The lip height is determined according to the face height and a preset height ratio coefficient.
8. A lip occlusion detection device, characterized in that: include: A target image acquisition module is used to acquire a target image to be detected; A face detection module, used to perform face detection in the target image to obtain face key points; A lip region image extraction module, used to extract the lip region image in the target image according to the facial key points; A lip occlusion detection module, used to perform lip occlusion detection on the lip area image using a preset lip occlusion detection network to obtain a lip occlusion detection result; The lip occlusion detection network is a pre-trained neural network for lip occlusion detection.
9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the lip occlusion detection method according to any one of claims 1 to 7 are implemented.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the lip occlusion detection method according to any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Deep convolutional neural network-based human face occlusion detection method
CN106485215A
Lip movement detection method and device, terminal equipment and computer readable storage medium
CN112966654A
Method for detecting face occlusion based on image segmentation
CN115131843A
Occlusion detection model training method and facial image beautification method
WO2022078041A1
Cited By
Digital human video generation method, device, equipment, medium and program product
CN121357387A
