Biological face shielding degree detection method and device, equipment and storage medium
By training the picture to generate segmentation network and occlusion prediction network, combined with the output results, the accuracy problem of biological face occlusion degree detection in the prior art is solved, efficient occlusion degree detection is achieved, and the dependence of manual labeling is reduced.
Patent Information
- Application Number
- CN202410981324.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-12
AI Technical Summary
The existing biological face occlusion detection method cannot accurately identify the occlusion area and occlusion degree of the picture. Relying on manual annotation leads to high subjectivity, low accuracy and long-term consumption.
By determining the image generation network and labeling pictures, the image generation segmentation network and the occlusion prediction network are trained, and the occlusion degree detection result of the image to be predicted is determined based on the output results of the two, reducing labor costs and improving detection accuracy.
It saves labor costs, improves the accuracy of biological face occlusion detection methods, and can accurately identify the occlusion areas and occlusion levels of pictures.
Smart Images

Figure CN118942113A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image detection technology, and in particular to a method, device, equipment and storage medium for detecting the occlusion degree of a biological face. Background Art
[0002] In the financial insurance field, many people will buy insurance for themselves or other living creatures. For example, in cattle farming, many larger farmers will buy insurance for their cattle. When the cattle get sick or have other problems, they will make claims. During the claims process, it is necessary to distinguish whether the insured cattle and the claimed cattle are the same, or whether the farmer has used a sick cattle to make multiple claims. In order to solve this problem, it is necessary to recognize the cow's face. When recognizing the cow's face, it is necessary to determine whether the quality of the cow's face meets the requirements. One important thing is that the key parts of the cow's face cannot be blocked.
[0003] Generally speaking, if the key parts of a cow's face, such as the eyes, nose, and mouth, are blocked, it will affect the recognition of the cow's face. If the parts other than these key points are blocked and the degree of occlusion is not large, it will basically not affect the recognition of the cow's face. Therefore, it is also very important to correctly distinguish the position of the cow's face.
[0004] Currently, the determination of whether an image is occluded is done manually. The occluded images are manually tagged, and then the neural network is trained based on the manually annotated images as training samples. This method requires manual annotation to determine whether the face is occluded, which is highly subjective, resulting in low accuracy of the face occlusion results. The addition of the alignment step increases the time consumption, and it is not possible to accurately distinguish which part is occluded and the degree of occlusion. Summary of the invention
[0005] The main purpose of the present invention is to provide a method, device, equipment and storage medium for detecting the degree of occlusion of a biological face, aiming to solve the technical problem that the existing method technology for detecting the degree of occlusion of a biological face cannot accurately identify the occluded part and the degree of occlusion in the image.
[0006] To achieve the above object, the present invention provides a method for detecting the degree of occlusion of a biological face, the method comprising the following steps:
[0007] Determine an image generation network, and obtain an image generation segmentation network according to the image generation network and the labeled image;
[0008] Generate a segmentation network based on the image to obtain a segmentation network, and obtain a picture occlusion prediction network based on the segmentation network generated by the image;
[0009] The picture to be predicted is respectively input into the picture segmentation network and the picture occlusion prediction network to obtain the occlusion degree detection result of the biological face in the picture to be predicted.
[0010] Optionally, before the step of obtaining the image generation segmentation network according to the image generation network and the labeled image, the step further includes:
[0011] The segmentation training image is obtained through the image generation network, and the segmentation training image is marked according to a preset area to obtain the labeled image.
[0012] Optionally, the step of obtaining a picture generation segmentation network according to the picture generation network and the labeled picture includes:
[0013] Inputting the annotated image into the image generation network, obtaining intermediate features of the image generation network, and performing an upsampling operation on the intermediate features;
[0014] The upsampled image and the segmentation annotation calculate the cross entropy loss loss, so that the cross entropy loss loss is less than a preset value, and the parameters of the image generation network are updated to obtain the image generation segmentation network.
[0015] Optionally, the step of generating a segmentation network according to the picture to obtain a picture segmentation network includes:
[0016] A first initial segmentation network is determined, a normal picture and a segmented picture are generated by the picture generation segmentation network, and the normal picture and the segmented picture are used as training sets to train the first initial segmentation network to obtain the picture segmentation network.
[0017] Optionally, the step of obtaining the picture occlusion prediction network according to the picture segmentation network includes:
[0018] Determine a second initial segmentation network, generate a normal picture through the picture generation segmentation network, occlude the normal picture, obtain an occluded picture training set, train the second initial segmentation network according to the occluded picture training set, and obtain the picture occlusion prediction network.
[0019] Optionally, the step of blocking the ordinary image and obtaining a blocked image training set includes:
[0020] The ordinary picture is masked by random pasting or random color block masking to obtain the masked picture training set.
[0021] Optionally, the step of inputting the picture to be predicted into the picture segmentation network and the picture occlusion prediction network respectively to obtain the diagnosis degree result of the picture to be predicted includes:
[0022] Inputting the picture to be predicted into the picture segmentation network to obtain a first output;
[0023] Inputting the to-be-predicted picture into the picture occlusion prediction network to obtain a second output of a binary occlusion mask;
[0024] The first output is multiplied by the second output to obtain a detection result of the occlusion degree of the biological face in the image to be predicted.
[0025] Furthermore, in order to achieve the above-mentioned purpose, the present invention also provides a biological face occlusion degree detection method and device, and the biological face occlusion degree detection method and device comprises:
[0026] A determination module is used to determine a picture generation network, and obtain a picture generation and segmentation network according to the picture generation network and the annotated picture;
[0027] A training module, used to obtain a picture segmentation network according to the picture generation segmentation network, and obtain a picture occlusion prediction network according to the picture generation segmentation network;
[0028] The detection module is used to input the image to be predicted into the image segmentation network and the image occlusion prediction network respectively to obtain the occlusion degree detection result of the biological face in the image to be predicted.
[0029] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a biological face occlusion degree detection method device, which includes a memory, a processor, and a biological face occlusion degree detection method program stored in the memory and executable on the processor, and when the biological face occlusion degree detection method program is executed by the processor, the steps of the biological face occlusion degree detection method as described above are implemented.
[0030] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a storage medium, on which a biological face occlusion degree detection method program is stored, and when the biological face occlusion degree detection method program is executed by a processor, the steps of the biological face occlusion degree detection method as described above are implemented.
[0031] The present invention determines a picture generation network, performs training according to the picture generation network and annotated pictures, obtains a picture generation and segmentation network that can automatically generate and segment pictures, then generates training pictures according to the picture generation and segmentation network, obtains a picture segmentation network through training, and simultaneously obtains a picture occlusion prediction network according to the picture generation and segmentation network, respectively inputs the picture to be predicted into the picture segmentation network and the picture occlusion prediction network, and determines the occlusion degree detection result of the picture to be predicted by combining the output results of the two, thereby saving labor costs and improving the accuracy of the biological face occlusion degree detection method. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1It is a schematic diagram of the structure of the device hardware operating environment involved in the embodiment of the device for detecting the degree of occlusion of a biological face of the present invention;
[0033] Figure 2 It is a flowchart of a first embodiment of a biological face occlusion degree detection method of the present invention;
[0034] Figure 3 Schematic diagram of functional modules of a preferred embodiment of a biological face occlusion degree detection method and device of the present invention.
[0035] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0036] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0037] It should be noted that in the field of financial insurance, many people will purchase insurance for themselves or other living creatures. For example, in cattle farming, many larger farmers will purchase insurance for their cattle. When the cattle have diseases and other problems, they will make claims. During the claims process, it is necessary to distinguish whether the insured cattle and the claimed cattle are the same, or whether the farmer has used a sick cattle for multiple claims. In order to solve this problem, it is necessary to recognize the cow's face. When recognizing the cow's face, it is necessary to determine whether the quality of the cow's face meets the requirements. One important thing is that the key parts of the cow's face cannot be blocked.
[0038] Generally speaking, if the key parts of a cow's face, such as the eyes, nose, and mouth, are blocked, it will affect the recognition of the cow's face. If the parts other than these key points are blocked and the degree of occlusion is not large, it will basically not affect the recognition of the cow's face. Therefore, it is also very important to correctly distinguish the position of the cow's face.
[0039] Currently, the determination of whether an image is occluded is done manually. The occluded images are manually tagged, and then the neural network is trained based on the manually annotated images as training samples. This method requires manual annotation to determine whether the face is occluded, which is highly subjective, resulting in low accuracy of the face occlusion results. The addition of the alignment step increases the time consumption, and it is not possible to accurately distinguish which part is occluded and the degree of occlusion.
[0040] Based on the above-mentioned defects, the present invention provides a method and device for detecting the degree of occlusion of a biological face, referring to Figure 1 , Figure 1 It is a structural schematic diagram of the device hardware operating environment involved in the device embodiment of the biological face occlusion degree detection method of the present invention.
[0041] like Figure 1As shown, the biological face occlusion degree detection method and device may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may optionally be a storage device independent of the aforementioned processor 1001.
[0042] Those skilled in the art will understand that Figure 1 The hardware structure of the biological face occlusion degree detection method and device shown in does not constitute a limitation of the biological face occlusion degree detection method and device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0043] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a biological face occlusion degree detection method program. Among them, the operating system is a program for managing and controlling the biological face occlusion degree detection method device and software resources, supporting the operation of the network communication module, the user interface module, the biological face occlusion degree detection method program and other programs or software; the network communication module is used to manage and control the network interface 1004; the user interface module is used to manage and control the user interface 1003.
[0044] exist Figure 1 In the hardware structure of the biological face occlusion degree detection method device shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the client and communicate data with the client; the processor 1001 can call the biological face occlusion degree detection method program stored in the memory 1005 and perform the following operations:
[0045] Determine an image generation network, and obtain an image generation segmentation network according to the image generation network and the labeled image;
[0046] Generate a segmentation network based on the image to obtain a segmentation network, and obtain a picture occlusion prediction network based on the segmentation network generated by the image;
[0047] The picture to be predicted is respectively input into the picture segmentation network and the picture occlusion prediction network to obtain the occlusion degree detection result of the biological face in the picture to be predicted.
[0048] Furthermore, before the step of obtaining the image generation segmentation network according to the image generation network and the labeled image, the step further includes:
[0049] The segmentation training image is obtained through the image generation network, and the segmentation training image is marked according to a preset area to obtain the labeled image.
[0050] Furthermore, the step of obtaining a picture generation segmentation network based on the picture generation network and the labeled picture includes:
[0051] Inputting the annotated image into the image generation network, obtaining intermediate features of the image generation network, and performing an upsampling operation on the intermediate features;
[0052] The upsampled image and the segmentation annotation calculate the cross entropy loss loss, so that the cross entropy loss loss is less than a preset value, and the parameters of the image generation network are updated to obtain the image generation segmentation network.
[0053] Furthermore, the step of generating a segmentation network according to the picture to obtain a picture segmentation network comprises:
[0054] A first initial segmentation network is determined, a normal picture and a segmented picture are generated by the picture generation segmentation network, and the normal picture and the segmented picture are used as training sets to train the first initial segmentation network to obtain the picture segmentation network.
[0055] Furthermore, the step of obtaining the picture occlusion prediction network according to the picture segmentation network includes:
[0056] Determine a second initial segmentation network, generate a normal picture through the picture generation segmentation network, occlude the normal picture, obtain an occluded picture training set, train the second initial segmentation network according to the occluded picture training set, and obtain the picture occlusion prediction network.
[0057] Furthermore, the step of blocking the ordinary image and obtaining a blocked image training set includes:
[0058] The ordinary picture is masked by random pasting or random color block masking to obtain the masked picture training set.
[0059] Furthermore, the step of inputting the picture to be predicted into the picture segmentation network and the picture occlusion prediction network respectively to obtain the diagnosis degree result of the picture to be predicted includes:
[0060] Inputting the picture to be predicted into the picture segmentation network to obtain a first output;
[0061] Inputting the to-be-predicted picture into the picture occlusion prediction network to obtain a second output of a binary occlusion mask;
[0062] The first output is multiplied by the second output to obtain a detection result of the occlusion degree of the biological face in the image to be predicted.
[0063] The specific implementation of the biological face occlusion degree detection method and device of the present invention is basically the same as the various embodiments of the biological face occlusion degree detection method described below, and will not be repeated here.
[0064] The present invention also provides a biological face occlusion degree detection method based on the above-mentioned biological face occlusion degree detection method device.
[0065] Reference Figure 2 , Figure 2 Schematic diagram of the flow of the first embodiment of the biological face occlusion degree detection method of the present invention.
[0066] The embodiment of the present invention provides an embodiment of a method for detecting the degree of occlusion of a biological face. It should be noted that although a logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in an order different from that shown here.
[0067] In various embodiments of the biological face occlusion degree detection method, the execution subject is a biological face occlusion degree detection method device.
[0068] The biological face occlusion degree detection method is applied to a biological face occlusion degree detection method device, and the biological face occlusion degree detection method comprises:
[0069] Step S10, determining a picture generation network, and obtaining a picture generation and segmentation network according to the picture generation network and the annotated picture;
[0070] In the financial insurance field, many people will buy insurance for themselves or other living creatures. For example, in cattle farming, many larger farmers will buy insurance for their cattle. When the cattle get sick or have other problems, they will make claims. During the claims process, it is necessary to distinguish whether the insured cattle and the claimed cattle are the same, or whether the farmer has used a sick cattle to make multiple claims. In order to solve this problem, it is necessary to recognize the cow's face. When recognizing the cow's face, it is necessary to determine whether the quality of the cow's face meets the requirements. One important thing is that the key parts of the cow's face cannot be blocked.
[0071] Generally speaking, if the key parts of a cow's face, such as the eyes, nose, and mouth, are blocked, it will affect the recognition of the cow's face. If the parts other than these key points are blocked and the degree of occlusion is not large, it will basically not affect the recognition of the cow's face. Therefore, it is also very important to correctly distinguish the position of the cow's face.
[0072] Currently, classification models are mostly used to determine whether a cow face is occluded. However, it is not possible to accurately distinguish which part is occluded and the degree of occlusion.
[0073] In order to solve the technical problem that the existing biological face occlusion degree detection method technology cannot accurately identify the occlusion part and occlusion degree of the picture, a biological face occlusion degree detection method is proposed in an embodiment of the present invention, which aims to determine a picture generation network, train it according to the picture generation network and the labeled picture, obtain a picture generation and segmentation network that can automatically generate and segment the picture, then generate training pictures according to the picture generation and segmentation network, train to obtain the picture segmentation network, and at the same time obtain the picture occlusion prediction network according to the picture generation and segmentation network, respectively input the picture to be predicted into the picture segmentation network and the picture occlusion prediction network, and determine the occlusion degree detection result of the picture to be predicted by combining the output results of the two, thereby saving labor costs and improving the accuracy of the biological face occlusion degree detection method.
[0074] The biological face occlusion degree detection method in this embodiment is applicable to a device having a biological face occlusion degree detection method function.
[0075] In this embodiment, the image generation network is first determined. The image generation network is used to automatically generate images for subsequent network model training to reduce the cost of manual screening. The image generation network can be trained using an adversarial generation network, such as styleGAN2, or a diffusion network. The generation process is a process of generating biological faces through random vectors. During the training process, sample images of the same biological type as the image to be predicted are required, and the biological faces are detected and aligned. At this time, there is no need to specially annotate the images.
[0076] Secondly, use the image generation network to generate images containing biological faces, select the generated images for annotation, and the number of selected images is not specifically limited. For example, 20 biological face images can be selected. The positions of the image annotation include the characteristic parts of the biological face and the background. For example, the face and background of the biological in the image are first marked separately, and then the face of the biological is divided and marked according to the upper jaw, lower jaw, eyes, ears, horns and other parts. The marked parts are not limited here, and the marked parts can be customized as needed during the actual operation. At the same time, in order to distinguish the annotations of different parts, each part can be marked in a different color, that is, each part and the background correspond to different colors, which is convenient for identifying the specific occluded parts and for users to confirm whether the occluded parts are important. For example, after identifying the background other than the face in the picture, the background part is marked as black, the upper jaw is marked as red, the lower jaw is marked as blue, the eyes are marked as pink, the ears are marked as green, and so on.
[0077] In addition, when labeling, it can be done manually or automatically through software. During the initial training, training samples can be created by manual labeling, which can train the image network model more accurately.
[0078] The annotated image is then input into the image generation network to obtain the intermediate features of the network, perform upsampling, calculate the cross entropy loss loss between the upsampled image and the segmentation annotation, minimize the loss, update the network parameters, and obtain the image generation segmentation network. When training the image generation segmentation network, the weights of the previously trained image generation network are fixed and no longer participate in the gradient update. After the training is completed, a picture segmentation network model that can simultaneously generate biological faces and segment according to the corresponding biological face annotations is obtained.
[0079] Step S20, obtaining a picture segmentation network according to the picture generation segmentation network, and obtaining a picture occlusion prediction network according to the picture generation segmentation network;
[0080] In this embodiment, after obtaining a segmentation network that can generate biological faces and automatically segment facial parts according to a picture generation segmentation network, it is also necessary to obtain a picture segmentation network through the picture generation segmentation network. The purpose of the picture generation segmentation network is to automatically generate a large number of training data sets required by the picture segmentation network. A large number of biological faces and biological face pictures that have been segmented are required in the training process of the picture segmentation network.
[0081] Generally speaking, if this part of the training data set is obtained manually, it will take a lot of time and cost. Therefore, by using a pre-trained image to generate a segmentation network model, the image generation segmentation network model is used to automatically generate the required biological faces and segmentation images as training data sets, and the training data set is used to quickly train the image segmentation network, which greatly saves time costs and greatly improves detection efficiency.
[0082] For example, first determine the basic network required for the image segmentation network. The basic network can use network structures such as unet and AFFormer, without limiting the specific structure; secondly, use the image to generate the segmentation network to generate a biological face and a biological face segmentation picture set. At this time, the generated pictures include biological faces and segmented pictures after the biological faces are segmented and labeled accordingly. These pictures are used as training sets to train the determined basic network, and finally the image segmentation network is obtained.
[0083] While obtaining the image segmentation network, the ordinary image generated by the image generation segmentation network is also used. The ordinary image is a biological face image that has not been labeled. Occlusion objects are added to the ordinary image, and then the image occlusion prediction network is trained. First, the basic network model of the image occlusion prediction network before training is determined. The main structure of the image occlusion prediction network is still the segmentation network structure. The specific structure is not limited, but the image occlusion prediction network finally outputs a binary mask matrix. At the same time, the ordinary image generated by the image generation segmentation network is occluded. The occlusion method can cover other images to part of the ordinary image to form an occluded image; or a color block can be used to occlude part of the ordinary image. For example, random pasting is used. Random pasting refers to randomly pasting a target subset of an image at a random position of another image, such as using pictures of obstructing objects such as hands, mobile phones, and watches to block them. Secondly, random color blocking can be used to block ordinary images. Random color blocking refers to color blocking of a random matrix, circle or polygon, regular or irregular shape, etc. in an image. Random color blocking can be automatically performed through software to block the corresponding area in the ordinary image.
[0084] At the same time, the basic network is trained using ordinary images generated by the image generation and segmentation network and occluded images after occluding the ordinary images to obtain the final image occlusion prediction network.
[0085] Step S30, inputting the picture to be predicted into the picture segmentation network and the picture occlusion prediction network respectively, to obtain the occlusion degree detection result of the biological face in the picture to be predicted.
[0086] In this embodiment, after the image segmentation network and the image occlusion prediction network are trained, the image to be predicted is input into the image segmentation network and the image occlusion prediction network respectively to obtain the detection result of the degree of occlusion of the biological face in the image to be predicted. The image to be predicted refers to the biological face image that needs to identify the occluded part and the degree of occlusion of the biological face in the image, wherein the biological face in the image to be predicted is consistent with the biological face type in the training data used when training the image segmentation network and the image occlusion prediction network, and the consistency of the biological face type means that the organisms displayed by the two belong to the same family, genus or species. The output of the image segmentation network is the occluded part of the biological face in the image to be predicted, and the output result of the image occlusion prediction network can reflect the degree of occlusion of the biological face in the image to be predicted, and then detect the occlusion part and degree of the biological face in the image to be predicted.
[0087] Among them, after the image to be predicted is input into the image segmentation network, a tensor of h*w*7 will be output, and then softmax will be performed in the depth direction to obtain a part segmentation map of h*w*1. After the image to be predicted is input into the image occlusion prediction network, a binary occlusion mask will be obtained. The h*w*1 part segmentation map is multiplied by the binary occlusion mask to obtain the occlusion area and occlusion degree of the biological face in the image to be predicted. For example, taking the image to be predicted as a cow face, the cow face annotation is segmented into cow face, upper jaw, lower jaw, eyes, ears, horns when training the image segmentation network. If the predicted cow face picture is input into the image segmentation network, the result is that the sum of the pixel values of the 6 parts of the cow face is X = [x1, x2, x3, x4, x5, x6], and the sum of the pixels of each part after multiplying by the occlusion mask output by the occlusion prediction network is Y = [y1, y2, y3, y4, y5, y6]. If a part is not occluded, the corresponding yi value is 0, and the final occlusion ratio of each part is: R = X / Y = [x1 / y1, x2 / y2, x3 / y3, x4 / y4, x5 / y5, x6 / y6].
[0088] The present invention determines a picture generation network, performs training according to the picture generation network and annotated pictures, obtains a picture generation and segmentation network that can automatically generate and segment pictures, then generates training pictures according to the picture generation and segmentation network, obtains a picture segmentation network through training, and simultaneously obtains a picture occlusion prediction network according to the picture generation and segmentation network, respectively inputs the picture to be predicted into the picture segmentation network and the picture occlusion prediction network, and determines the occlusion degree detection result of the picture to be predicted by combining the output results of the two, thereby saving labor costs and improving the accuracy of the biological face occlusion degree detection method.
[0089] Furthermore, a second embodiment of the biological face occlusion degree detection method of the present invention is proposed. In the above step S10, before the step of obtaining the image generation segmentation network according to the image generation network and the annotated image, the method further includes:
[0090] Step a1, obtaining a segmentation training image through the image generation network, and marking the segmentation training image according to a preset area to obtain the labeled image.
[0091] In this embodiment, a pre-trained image generation network model is used to automatically generate the required biological face ordinary image, i.e., the segmentation training image, using the image generation network model. The training image is then segmented according to preset segmentation area markers to obtain a labeled image.
[0092] Specifically, the biological face areas in the usual sense include the forehead, eyelids, corners of the eyes, eye sockets, nose bridge, cheeks, lips, upper jaw and lower jaw, etc., and in the biological face recognition technology, recognition can be based on all of the above-mentioned biological face areas, or based on part of the above-mentioned biological face areas. Therefore, by training a good biological face segmentation model, at least partial biological face area segmentation, at least partial occluded area segmentation within the biological face area, or at least partial non-occluded area segmentation within the biological face area can be obtained.
[0093] The preset segmentation area refers to the parts divided according to the shape of the face set by the user or at the factory, such as eyes, nose, mouth, etc., which can be automatically set according to user needs, and the preset segmentation area can also be actively modified before or during training the network model.
[0094] The positions annotated in the image include the characteristic parts of the creature's face and the background. For example, the face and background of the creature in the image are first annotated separately, and then the face of the creature is divided and annotated according to the upper jaw, lower jaw, eyes, ears, horns and other parts. The annotated parts are not limited here, and the annotated parts can be customized as needed during the actual operation. At the same time, in order to distinguish the annotations of different parts, each part can be annotated in a different color, that is, each part and the background have different colors, which is convenient for identifying the specific occluded parts and for users to confirm whether the occluded parts are important. For example, after identifying the background other than the face in the picture, the background part is marked as black, the upper jaw is marked as red, the lower jaw is marked as blue, the eyes are marked as pink, the ears are marked as green, and so on.
[0095] This embodiment uses the image generation network to quickly and massively generate training data sets, and uses the training data sets to quickly train the image segmentation network, which greatly saves time costs and greatly improves detection efficiency.
[0096] Furthermore, a third embodiment of the biological face occlusion degree detection method of the present invention is proposed. In the above step S10, the step of obtaining the image generation segmentation network according to the image generation network and the annotated image includes:
[0097] Step b1, inputting the annotated image into the image generation network, obtaining intermediate features of the image generation network, and performing upsampling operations on the intermediate features;
[0098] Step b2, calculating the cross entropy loss loss of the upsampled image and the segmentation annotation, making the cross entropy loss loss less than a preset value, updating the parameters of the image generation network, and obtaining the image generation segmentation network.
[0099] In this embodiment, the annotated image is first input into the image generation network to obtain the intermediate feature map, and the intermediate feature map is upsampled. The cross entropy loss loss is calculated based on the upsampled image and the segmentation annotation to minimize the cross entropy loss loss. A preset value can be set. When the cross entropy loss loss is less than the preset value, it can be determined that the cross entropy loss loss has been minimized, and the parameters of the image generation network are updated to obtain the image generation segmentation network.
[0100] For example, the training set of common images and annotated images is repeatedly input into the generative network for iteration and parameter update, and the iteration termination condition may be that the difference between the maximum loss value and the minimum loss value in the total loss values of the most recent preset number of determined values is less than the first preset value. For example, the iteration termination condition is that the difference between the maximum loss value and the minimum loss value in the total loss values obtained after the most recent 10 iterations is less than 10-16.
[0101] The condition for terminating the iteration may also be that the difference between the most recent total loss value and the last total loss value is less than a second preset value. For example, the condition for terminating the iteration is that the difference between the most recent total loss value and the last total loss value is less than 10-17.
[0102] The condition for terminating the iteration may also be that the size of the most recent total loss value determined is less than a third preset value. For example, the condition for terminating the iteration may also be that the size of the most recent total loss value determined is less than 10-2. It should be noted that there is no connection between the first preset value, the second preset value and the third preset value.
[0103] This embodiment minimizes the cross entropy loss by using the above method to calculate the cross entropy loss with segmentation annotation, and then updates the network parameters to obtain an accurate image generation and segmentation network, thereby improving the accuracy of image occlusion degree detection.
[0104] Furthermore, a fourth embodiment of the biological face occlusion degree detection method of the present invention is proposed. In the above step S20, the step of generating a segmentation network according to the image to obtain an image segmentation network includes:
[0105] Step c1, determining a first initial segmentation network, generating a normal picture and a segmented picture through the picture generation segmentation network, using the normal picture and the segmented picture as training sets to train the first initial segmentation network, and obtaining the picture segmentation network.
[0106] In this embodiment, the first initial segmentation network is first determined, and the first initial segmentation network refers to the basic network structure used to train the image segmentation network. For example, the basic network required for the image segmentation network is first determined, and the basic network can use a network structure such as unet, AFFormer, etc., without limiting the specific structure.
[0107] Secondly, ordinary pictures refer to original pictures of biological faces without annotations generated by the picture generation and segmentation network, and segmented pictures refer to pictures after the picture generation and segmentation network divides the original pictures into regions according to the annotations. At this time, the generated pictures include biological faces and segmented pictures after the biological faces are segmented and annotated accordingly. These pictures are used as training sets to train the determined basic network and finally obtain the picture segmentation network.
[0108] For example, a segmentation network is generated using trained images to generate a large number of cow face and cow face segmentation image pairs. 10,000 pairs of data can be generated. These generated images are used to create training sets and test sets to train an image segmentation network, wherein the number of samples used as training data is not limited.
[0109] This embodiment uses a pre-trained image generation segmentation network model, uses the image generation segmentation network model to automatically generate the required biological faces and segmentation images as a training data set, and uses the training data set to quickly train the image segmentation network, which greatly saves time costs and greatly improves detection efficiency.
[0110] Furthermore, a fifth embodiment of the biological face occlusion degree detection method of the present invention is proposed. In the above step S20, the step of obtaining the image occlusion prediction network according to the image segmentation network includes:
[0111] Step d1, determine the second initial segmentation network, generate a normal picture through the picture generation segmentation network, occlude the normal picture, obtain the occluded picture training set, train the second initial segmentation network according to the occluded picture training set, and obtain the picture occlusion prediction network.
[0112] In this embodiment, while obtaining the image segmentation network, an ordinary image generated by the image segmentation network is also used. The ordinary image is a biological face image that has not been labeled. Occlusion objects are added to the ordinary image, and then the image occlusion prediction network is trained.
[0113] First, determine the second initial segmentation network for training the image occlusion prediction network. The second initial segmentation network is the basic network model. The main structure of the image occlusion prediction network is still the segmentation network structure. The specific structure is not limited, but the image occlusion prediction network finally outputs a binary mask matrix. At the same time, the ordinary image generated by the image generation segmentation network is occluded. The occlusion method can cover other images to part of the ordinary image to form an occluded image; or color blocks can be used to occlude part of the ordinary image.
[0114] For example, random pasting is used. Random pasting refers to randomly pasting a target subset of an image at a random position of another image, such as using pictures of obstructing objects such as hands, mobile phones, and watches to block them. Secondly, random color blocking can be used to block ordinary images. Random color blocking refers to color blocking of a random matrix, circle or polygon, regular or irregular shape, etc. in an image. Random color blocking can be automatically performed through software to block the corresponding area in the ordinary image.
[0115] At the same time, the occluded image training data set is obtained by using the ordinary images generated by the image generation segmentation network and the occluded images after the ordinary images are occluded, and the basic network is trained to obtain the final image occlusion prediction network.
[0116] This embodiment uses a pre-trained image generation and segmentation network model, uses the image generation and segmentation network model to automatically generate the required biological face, then adds occlusion objects to the biological face to obtain an occlusion image, obtains an occlusion image training data set, and uses the training data set to quickly train the image segmentation network, which greatly saves time cost and greatly improves detection efficiency.
[0117] Furthermore, a sixth embodiment of the biological face occlusion degree detection method of the present invention is proposed. In the above step d1, the step of occluding the ordinary picture and obtaining the occluded picture training set includes:
[0118] Step d11, masking the ordinary image by random pasting or random color block masking to obtain the masked image training set.
[0119] In this embodiment, the ordinary picture generated by the image generation and segmentation network is blocked. The blocking method can cover other images to part of the area in the ordinary picture to form a blocked picture; or a color block can be used to block part of the area in the ordinary picture. For example, a random pasting method is used. Random pasting refers to randomly pasting a target subset of an image at a random position of another image, such as using a hand, mobile phone, watch or other blocking object image for blocking; secondly, the ordinary picture can be blocked by random color blocking. Random color blocking refers to color blocking of a random matrix, circle or polygon, regular or irregular shape, etc. of an image. Random color blocking can be automatically performed by software to block the corresponding area in the ordinary picture.
[0120] This embodiment obtains a large number of random occlusion picture training sets by occluding ordinary pictures in a random manner such as random pasting or random color blocks, thereby improving the accuracy of the training picture occlusion prediction network.
[0121] Furthermore, a seventh embodiment of the biological face occlusion degree detection method of the present invention is proposed, wherein the above step S30 comprises:
[0122] Step e1, inputting the picture to be predicted into the picture segmentation network to obtain a first output;
[0123] Step e2, inputting the image to be predicted into the image occlusion prediction network to obtain a second output of a binary occlusion mask;
[0124] Step e3, multiplying the first output by the second output to obtain a detection result of the degree of occlusion of the biological face in the image to be predicted.
[0125] In this embodiment, the image to be predicted refers to an image of a biological face in which the occluded part and occlusion degree of the biological face in the image need to be identified, wherein the biological face in the image to be predicted is consistent with the biological face type in the training data used when training the image segmentation network and the image occlusion prediction network, and the consistency of the biological face type means that the organisms displayed by the two belong to the same family, genus or species. The output of the image segmentation network is the occluded part of the biological face in the image to be predicted, and the output result of the image occlusion prediction network can reflect the occlusion degree of the biological face in the image to be predicted, thereby detecting the occlusion part and degree of the biological face in the image to be predicted.
[0126] After training the image segmentation network and the image occlusion prediction network, the image to be predicted is input into the image segmentation network and the image occlusion prediction network respectively. After the image to be predicted is input into the image segmentation network, a first output will be obtained, which can reflect the occluded part of the biological face in the image; after the image to be predicted is input into the image occlusion prediction network, a second output will be obtained, which will reflect the degree of occlusion of the biological face in the image to be predicted. The combination of the two can reflect the occluded part and the degree of occlusion of the biological face in the image to be predicted.
[0127] Secondly, after the image to be predicted is input into the image segmentation network, a tensor of h*w*7 will be obtained. Then softmax is performed in the depth direction to obtain a part segmentation map of h*w*1, which is the first output. After the image to be predicted is input into the image occlusion prediction network, a binary occlusion mask, which is the second output, is obtained. The first output is multiplied by the second output to obtain the occlusion area and occlusion degree of the biological face in the image to be predicted. For example, taking the image to be predicted as a cow face, the cow face annotation is segmented into cow face, upper jaw, lower jaw, eyes, ears, horns, etc. when training the image segmentation network. If the predicted cow face picture is input into the image segmentation network, the result is that the sum of the pixel values of the 6 parts of the cow face is X = [x1, x2, x3, x4, x5, x6], and the sum of the pixels of each part after multiplying by the occlusion mask output by the occlusion prediction network is Y = [y1, y2, y3, y4, y5, y6]. If a part is not occluded, the corresponding yi value is 0, and the final occlusion ratio of each part is: R = X / Y = [x1 / y1, x2 / y2, x3 / y3, x4 / y4, x5 / y5, x6 / y6].
[0128] This embodiment inputs the image to be predicted into the image segmentation network and the image occlusion prediction network respectively, and combines the output results of the two to determine the occlusion degree detection result of the image to be predicted, thereby saving labor costs and improving the accuracy of the biological face occlusion degree detection method.
[0129] The invention also provides a method and device for detecting the occlusion degree of a biological face.
[0130] Reference Figure 3 , Figure 3 This is a functional module diagram of the first embodiment of the biological face occlusion degree detection method and device of the present invention. The biological face occlusion degree detection method and device comprises:
[0131] A determination module is used to determine a picture generation network, and obtain a picture generation and segmentation network according to the picture generation network and the annotated picture;
[0132] A training module, used to obtain a picture segmentation network according to the picture generation segmentation network, and obtain a picture occlusion prediction network according to the picture generation segmentation network;
[0133] The detection module is used to input the image to be predicted into the image segmentation network and the image occlusion prediction network respectively to obtain the occlusion degree detection result of the biological face in the image to be predicted.
[0134] In addition, an embodiment of the present invention further provides a storage medium.
[0135] The storage medium stores a biological face occlusion degree detection method program, and when the biological face occlusion degree detection method program is executed by the processor, the steps of the biological face occlusion degree detection method as described above are implemented.
[0136] The specific implementation of the storage medium of the present invention is basically the same as the embodiments of the above-mentioned biological face occlusion degree detection method, and will not be repeated here.
[0137] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims. All equivalent structures or equivalent process changes made using the contents of the specification and drawings of the present invention, or directly or indirectly used in other related technical fields, are protected by the present invention.
[0138] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0139] In addition, each functional module in each embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. If the function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk.
[0140] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0141] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A method for detecting the degree of occlusion of a biological face, characterized in that: The biological face occlusion degree detection comprises the following steps: Determine an image generation network, and obtain an image generation segmentation network according to the image generation network and the labeled image; Generate a segmentation network based on the image to obtain a segmentation network, and obtain a picture occlusion prediction network based on the segmentation network generated by the image; The picture to be predicted is respectively input into the picture segmentation network and the picture occlusion prediction network to obtain the occlusion degree detection result of the biological face in the picture to be predicted.
2. The biological face occlusion degree detection method according to claim 1, characterized in that: Before the step of obtaining a picture generation segmentation network according to the picture generation network and the labeled picture, the method further includes: The segmentation training image is obtained through the image generation network, and the segmentation training image is marked according to a preset area to obtain the labeled image.
3. The biological face occlusion degree detection method according to claim 1, characterized in that: The step of obtaining a picture generation segmentation network based on the picture generation network and the labeled picture includes: Inputting the labeled image into the image generation network, obtaining intermediate features of the image generation network, and performing upsampling and intersection operations on the intermediate features; The upsampled image and the segmentation annotation are used to calculate the cross entropy loss loss, so that the cross entropy loss loss is less than a preset value, and the parameters of the image generation network are updated to obtain the image generation segmentation network.
4. The biological face occlusion degree detection method according to claim 1, characterized in that: The step of generating a segmentation network according to the picture to obtain a picture segmentation network comprises: A first initial segmentation network is determined, a normal picture and a segmented picture are generated by the picture generation segmentation network, and the normal picture and the segmented picture are used as training sets to train the first initial segmentation network to obtain the picture segmentation network.
5. The biological face occlusion degree detection method according to claim 1, characterized in that: The step of obtaining the picture occlusion prediction network according to the picture segmentation network comprises: Determine a second initial segmentation network, generate a normal picture through the picture generation segmentation network, occlude the normal picture, obtain an occluded picture training set, train the second initial segmentation network according to the occluded picture training set, and obtain the picture occlusion prediction network.
6. The biological face occlusion degree detection method as claimed in claim 5, characterized in that: The step of blocking the ordinary image and obtaining a blocked image training set comprises: The ordinary picture is masked by random pasting or random color block masking to obtain the masked picture training set.
7. The biological face occlusion degree detection method as claimed in claim 1, characterized in that: The step of inputting the picture to be predicted into the picture segmentation network and the picture occlusion prediction network respectively to obtain the diagnosis degree result of the picture to be predicted comprises: Inputting the picture to be predicted into the picture segmentation network to obtain a first output; Inputting the to-be-predicted picture into the picture occlusion prediction network to obtain a second output of a binary occlusion mask; The first output is multiplied by the second output to obtain a detection result of the occlusion degree of the biological face in the image to be predicted.
8. A method and device for detecting the degree of occlusion of a biological face, characterized in that: The biological face occlusion degree detection method and device comprises: A determination module is used to determine a picture generation network, and obtain a picture generation and segmentation network according to the picture generation network and the annotated picture; A training module, used to obtain a picture segmentation network according to the picture generation segmentation network, and obtain a picture occlusion prediction network according to the picture generation segmentation network; The detection module is used to input the image to be predicted into the image segmentation network and the image occlusion prediction network respectively to obtain the occlusion degree detection result of the biological face in the image to be predicted.
9. A method and device for detecting the degree of occlusion of a biological face, characterized in that: The biological face occlusion degree detection method device includes a memory, a processor, and a biological face occlusion degree detection method program stored in the memory and executable on the processor. When the biological face occlusion degree detection method program is executed by the processor, the steps of the biological face occlusion degree detection method as described in any one of claims 1 to 7 are implemented.
10. A storage medium, characterized in that: The storage medium stores a biological face occlusion degree detection method program, and when the biological face occlusion degree detection method program is executed by the processor, the steps of the biological face occlusion degree detection method as described in any one of claims 1-7 are implemented.
Citation Information
Patent Citations
Semantic segmentation-based face integrity measurement method, system and device and storage medium
CN113111817A
