A method and device for face detection and recognition, electronic equipment, and storage medium

Abstract

The present application provides a face detection and recognition method and device, an electronic device, and a storage medium, wherein the face detection and recognition method includes: using a face detection network to detect faces in a target image to obtain a first face set; using a text detection and recognition network to recognize names in the target image to obtain a first name set; and pairing faces with names based on the position of each face in the first face set and the position of each name in the first name set.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a face detection and recognition method and device, an electronic device, and a storage medium. Background Art

[0002] With the development of face recognition technology, more and more applications use face recognition as identity authentication. Face recognition technology is popular among people for its advantages such as being difficult to be stolen, more secure, contactless, more hygienic, intuitive, and easy to use. It is used in attendance, monitoring, video review, security, and other fields. When training a face recognition network, it is crucial to obtain labeled data containing names corresponding to faces. Generally speaking, labeling requires manual labeling, which is inefficient and requires a lot of manpower. There are often many faces and corresponding names in image materials such as conference videos and news pictures. How to use the faces and corresponding names in these image materials to accurately and efficiently automatically detect and recognize faces in conference videos and news pictures is still a very challenging task. Summary of the invention

[0003] To solve the above technical problems, the embodiments of the present application provide a face detection and recognition method and device, an electronic device, and a storage medium, which can automatically and accurately and efficiently recognize faces from image materials containing names and faces without the need for additional human interaction.

[0004] The present application provides a face detection and recognition method, including:

[0005] Use the face detection network to detect faces in the target image and obtain a first face set;

[0006] Use the text detection and recognition network to identify the names of people in the target image and obtain the first name set;

[0007] Based on the position of each face in the first face set and the position of each name in the first name set, the faces are paired with the names.

[0008] The present application also provides a method for training a face detection and recognition network, including:

[0009] Get multiple target images from the target video;

[0010] Obtaining annotation data for each target image, wherein the annotation data includes at least a name of a person paired with a face in the target image;

[0011] Use multiple target images containing labeled data as image training sets, and use the image training sets to train the face detection and recognition network;

[0012] Wherein, the name of the person paired with the face in the target image is obtained by the face detection and recognition method.

[0013] The present application also provides a face detection and recognition device, including:

[0014] A face set acquisition unit detects faces in a target image using a face detection network to obtain a first face set;

[0015] A name set acquisition unit, which uses a text detection and recognition network to recognize names in a target image and obtain a first name set;

[0016] The pairing unit pairs the faces with the names based on the positions of the faces in the first face set and the positions of the names in the first name set.

[0017] The present application also provides a training device for a face detection and recognition network, including:

[0018] A target image acquisition unit, used to acquire multiple target images from a target video;

[0019] A labeling unit, used to obtain labeling data of each target image, wherein the labeling data at least includes a name of a person paired with a face in the target image; wherein the name of a person paired with a face in the target image is obtained by the face detection and recognition method;

[0020] The training unit is used to use multiple target images containing labeled data as image training sets, and use the image training sets to train the face detection and recognition network.

[0021] The present application also provides an electronic device, including:

[0022] A memory for storing executable instructions;

[0023] The processor is used to implement the face detection and recognition method or the face detection and recognition network training method when executing the executable instructions stored in the memory.

[0024] An embodiment of the present application also provides a storage medium, which stores executable instructions. When the executable instructions are executed by at least one processor, the face detection and recognition method is implemented, or the training method of the face detection and recognition network is implemented.

[0025] The embodiment of the present application provides a face detection and recognition method, wherein a face detection network is used to detect faces in a target image to obtain a first face set; a text detection and recognition network is used to recognize names in the target image to obtain a first name set; and faces are paired with names based on the positions of each face in the first face set and the positions of each name in the first name set.

[0026] In the embodiment of the present application, the face detection network can accurately detect the faces contained in the target image and the positions corresponding to the faces, and the text detection and recognition network can accurately detect the names contained in the target image and the positions corresponding to the names. In this way, based on the positions of the faces in the first face set and the positions of the names in the first name set, the faces in the target image containing the corresponding names can be accurately and efficiently detected and recognized without the need for additional human interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic diagram of the process of face detection and recognition method provided in the embodiment of the present application;

[0028] Figure 2 A flowchart of a method for training a face detection and recognition network provided in an embodiment of the present application;

[0029] Figure 3 A schematic diagram of the structure of a face detection and recognition device provided in an embodiment of the present application;

[0030] Figure 4 A schematic diagram of the structure of a training device for a face detection and recognition network provided in an embodiment of the present application;

[0031] Figure 5 A flowchart of a method for training a face detection and recognition network provided in an application embodiment of the present application;

[0032] Figure 6a A schematic diagram of a seat sign captured from a target image for this application example;

[0033] Figure 6b A schematic diagram of obtaining identification samples from seat cards for this application example;

[0034] Figure 6c A schematic diagram showing scaling the image area corresponding to the text box to the same size for this application example;

[0035] Figure 6d A schematic diagram of extracting texture features of the background of an image area corresponding to a text box for this application example;

[0036] Figure 6e A schematic diagram of calculating the frequency distribution of texture features for the example of this application;

[0037] Figure 6f A schematic diagram of obtaining a set of first person names for this application example;

[0038] Figure 7 A schematic diagram of the structure of a training device for a face detection and recognition network provided in an application embodiment of the present application;

[0039] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0041] It should be noted that in the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the embodiments of the present application, the character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0042] In the description of the embodiments of the present application, the term "corresponding" may indicate a direct or indirect correspondence between two items, or an association relationship between the two items, or a relationship between indication and being indicated, configuration and being configured, and the like.

[0043] To facilitate understanding of the technical solutions of the embodiments of the present application, the relevant technologies of the embodiments of the present application are described below. The following related technologies can be arbitrarily combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.

[0044] Face recognition is a technology widely used in monitoring, video review, security and other fields. For training a face recognition model, the acquisition of labeled data is a crucial step. Generally speaking, such data needs to be manually labeled, which will consume a lot of manpower and time. For example, there are often many faces and corresponding names in images such as news videos and certificate materials. Here are several solutions:

[0045] Solution A: This method first establishes a face sample library, and requires manual labeling of each face sample in the face sample library; then, the face detection algorithm is used to detect the face image area of ​​the photo image to be labeled, and the face fingerprint of the detected face image area is obtained, and the face fingerprint of the face image area is compared with each face fingerprint in the face sample library to identify the similarity between the face image area of ​​the photo image to be labeled and the face in the face sample library, and determine whether they are the same person; wherein, the process of obtaining the face fingerprint is "reducing the face image to a size of N×N, with a total of N×N pixels, and calculating the grayscale average of N×N pixels, comparing the grayscale of each pixel in the face image with the grayscale average, and recording 1 when the comparison result is greater than or equal to, and recording 0 when the comparison result is less than, and forming all the comparison results into an N×N-bit binary code". In the above process of obtaining the face fingerprint, the grayscale value of each pixel is easily affected by factors such as ambient light and human posture, which is easy to cause misidentification. Therefore, this method cannot achieve automatic face detection and recognition.

[0046] Solution B: This method first performs face recognition on the video to obtain a face sequence; then obtains the time when the characters in the video appear in the video; finally, according to the time when the characters appear in the video and the time when the face images involved in the face sequence appear in the video, the face sequence is annotated to obtain an annotated face sequence; wherein, the method for obtaining the time when the characters in the video appear in the video is: text alignment is performed according to the first description information of the video and the second description information of the video to obtain the corresponding relationship between the characters and the time when the characters appear in the video. wherein, the first description information is used to indicate the corresponding relationship between each text and the time when the characters appear in the video; the second description information is used to indicate the corresponding relationship between each text and the characters. as a possible implementation, the first description information is specifically subtitles, and the second description information is specifically a script. the script and subtitles of the video both record the lines, the difference is that the script records the lines spoken by each character, while the subtitles record the time when the lines appear. that is, according to the script, the corresponding relationship between the characters and the lines can be obtained, and according to the subtitles, the corresponding relationship between the time when the characters appear and the lines can be obtained. Then, the subtitles of the video are aligned with the script of the video to obtain the correspondence between the characters and the time when the characters appear in the video. In the above process of obtaining the time when the characters in the video appear in the video, the first description information and the second description information are needed. However, conference videos or news pictures usually do not contain the above description information. Therefore, this method is not suitable for face detection and recognition of image materials such as conference videos.

[0047] Solution C: This method first extracts images and text descriptions of images from news web pages: detects images and their positions in the web page layout; then detects the position of the text paragraph below the image in the web page, uses natural language processing technology to perform named entity recognition on the text description, and identifies a set of names; finally, the candidate database is post-processed using two strategies: using similar avatars from the face database to refine the data in the candidate database and using refinement between the data in the candidate database. This method first requires the existence of text descriptions of images in the news web page, and secondly requires a face database that has been manually annotated in advance. Therefore, this method is not suitable for face detection and recognition of images such as conference videos.

[0048] The above scheme has the following defects:

[0049] For Solution A, firstly, a manually annotated face database is required, which leads to low efficiency of face recognition; secondly, the recognition results are easily affected by factors such as ambient light and human posture, which can easily lead to misrecognition.

[0050] For solution B, not only first description information representing the correspondence between text and face is needed, but also second description information representing the correspondence between text and name is needed, and these description information usually do not exist in the image.

[0051] For Solution C, not only does it require a text description of the image on the news web page, but it also requires a face database that has been manually annotated in advance, and the efficiency of face detection and recognition is low.

[0052] In order to solve at least part of the above-mentioned defects, the following technical solutions of the embodiments of the present application are proposed.

[0053] In order to enable a more detailed understanding of the features and technical contents of the present application, the implementation of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the present application. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0054] The present application provides a method for face detection and recognition. Figure 1 The following is a flow chart of the face detection and recognition method provided in the embodiment of the present application. Figure 1 As shown, the face detection and recognition method comprises the following steps:

[0055] Step 101: Use a face detection network to detect faces in a target image to obtain a first face set;

[0056] Step 102: Use a text detection and recognition network to identify names in the target image to obtain a first name set;

[0057] Step 103: Pairing faces with names based on the positions of the faces in the first face set and the positions of the names in the first name set.

[0058] Explanation of the nouns in the above scheme:

[0059] Face detection network: used to detect and locate the face in the target image, and output the corresponding face frame coordinates, wherein the face frame coordinates at least include the coordinates of a vertex of the face frame and the length and width of the face frame; the face detection network can adopt a target detection network based on RetinaNet.

[0060] The first face set: a set of faces detected from the target image using the face detection network. Each face in the first face set corresponds to its face position information. The face position information contains at least five data: the first data represents the file name, the second and third data represent the coordinates of a vertex of the face frame, the fourth data represents the length of the face frame, and the fifth data represents the width of the face frame (for example, the face position information can be in the form of (file name, upper left corner x1, upper left corner y1, length w1, width h1)), and the face position information can be saved in the format of a box file.

[0061] Text detection and recognition network: used to recognize the name of a person in the target image and output the text box coordinates corresponding to the name, wherein the text box coordinates corresponding to the name at least include the coordinates of a vertex of the text box and the length and width of the text box; the text detection and recognition network can adopt a text recognition network based on a convolutional recurrent neural network (CRNN).

[0062] The first name set: a set of names detected from the target image using a text detection and recognition network. Each name in the first name set corresponds to its name position information. The name position information includes at least five data: the first data represents the file name, the second and third data represent the coordinates of a vertex of the text box corresponding to the name, the fourth data represents the length of the text box corresponding to the name, and the fifth data represents the width of the text box corresponding to the name (for example, the name position information can be in the form of (file name, upper left corner x'1, upper left corner y'1, length w'1, width h'1)), and the name position information can be saved in a box file format.

[0063] Matching faces with names: the correspondence between each face in the first face set and each name in the first name set.

[0064] In the above scheme, different from the news page, as long as the target image contains both the face and the corresponding name information, even if the name information is not a regular text but exists in the form of an image, in one embodiment, the face and the corresponding name can be paired; in actual applications, the source of the target image can be obtained in many ways, which can be a picture taken and transmitted by a video recording device, a picture downloaded and transmitted through the network, a picture transmitted through a local storage device, or a picture extracted frame by frame from the target video.

[0065] Based on this, in one embodiment, before using the face detection network to detect the face in the target image, the method further includes:

[0066] Extract a frame of image from the target video as the target image;

[0067] Using a face detection network to detect whether there is a face in the target image, and using a text detection and recognition network to identify whether there is a name in the target image;

[0068] Accordingly, the method of using a face detection network to detect faces in a target image to obtain a first face set includes:

[0069] If there is a face in the target image, use a face detection network to detect the face in the target image to obtain a first face set face;

[0070] Accordingly, the text detection and recognition network is used to recognize the names of people in the target image to obtain a first name set, including:

[0071] If there are names in the target image, a text detection and recognition network is used to recognize the names in the target image to obtain a first name set.

[0072] In the above scheme, taking the target video selecting a conference video as an example, the conference video usually contains the name information of the participants. For example, there are seat signs in front of the seats corresponding to the participants, and the seat signs record the names, positions and other information of the corresponding participants; however, a certain frame of the target image extracted from the target video may not contain a face or / and a name, and a target image without a face or / and a name does not have the meaning of face detection and recognition.

[0073] Based on this, in one embodiment, the method further includes:

[0074] If there is no human face in the target image and / or there is no human name in the target image, the target image is updated based on the target video.

[0075] When there is no human face or / and no human name in the target picture extracted from the target video, since the video screen has strong continuity, it is very likely that there is no human face or / and no human name in the adjacent frames of the current frame target picture.

[0076] Based on this, in one embodiment, updating the target picture based on the target video includes:

[0077] According to a preset frame interval, the next frame of the current target picture is extracted from the target video as the updated target picture.

[0078] In the above scheme, each frame of the target video can be extracted first, and then the target image can be updated according to the preset frame interval; each frame of the target video can be extracted by using OpenCV or Adobe Premiere software. When using Adobe Premiere software, it is only necessary to import the target video and click the export frame button to select and save each frame or the specified frame of the extracted image; when using OpenCV to extract each frame of the target video, the following steps are included:

[0079] Step a: Install the required library files cv2 and numpy;

[0080] Step b: Import the library files cv2 and numpy, and enable real-time preview of the target video. Since OpenCV provides functions for previewing the target video and reading each frame of the target video, you only need to implement it with the corresponding code. It should be noted that the target video preview here refers to the preview of the local target video. If you need to realize real-time preview, you can also use the camera to capture the target video, and then realize real-time preview by pushing the stream.

[0081] Step c: Extract each frame of the target video during real-time preview.

[0082] In the above step c, the process of extracting the next frame of the current target image from the target video according to the preset frame interval only needs to be implemented with corresponding codes.

[0083] In the above scheme, it is assumed that the first face set contains m faces, represented by M(M1,M2…M m ), the first name set contains n names, expressed as N(N1,N2…N n ), then the first face set and the first name set can form a bipartite graph.

[0084] The definition of a bipartite graph is: given two groups of vertices, there is no edge connecting any two vertices in the group, and there are only edges between the two groups, that is, the points in the first group can only be connected to the points in the second group. The graph constructed in this way is called a bipartite graph.

[0085] Therefore, in the above step 103, the problem of pairing faces with names based on the positions of each face in the first face set and the positions of each name in the first name set can be solved using a bipartite graph matching algorithm.

[0086] Furthermore, in the above scheme, since there is often a specific positional relationship between the face and the corresponding name in the target image, taking the target image extracted from the conference video as an example, the distance between the face frame corresponding to the face and the text frame corresponding to the name is often within a certain threshold range. Therefore, it is also necessary to calculate the relative distance between the face frame and the text frame corresponding to the initially paired face and name. If the relative distance is less than the preset threshold, the initially paired face and name are paired; wherein, the relative distance represents the distance between the face frame and the text frame relative to the length or width of the face frame.

[0087] Based on this, in one embodiment, the pairing of the face and the name based on the position of the face and the position of the name includes:

[0088] Using a bipartite graph matching algorithm to initially pair each face in the first face set with each name in the first name set;

[0089] The relative distance between the face frame and the text frame corresponding to the initially paired face and name is calculated. If the relative distance is less than a preset threshold, the initially paired face and name are paired; wherein the relative distance represents the distance between the face frame and the text frame relative to the length or width of the face frame.

[0090] In the above scheme, the bipartite graph matching algorithm includes the bipartite graph maximum matching algorithm (Hungarian algorithm) and the bipartite graph perfect matching algorithm (KM algorithm), etc. This application further describes the process of initial matching of faces and names using the Hungarian algorithm as an example:

[0091] Calculate the distance between the face frame corresponding to each face in the first face set and the text frame corresponding to each name in the first name set, and the face frame corresponding to each face will obtain multiple distances; wherein, the distance between the face frame corresponding to the face and the text frame corresponding to the name can be expressed by the distance between the center of the face frame and the center of the text frame, and the distance between the center of the face frame and the center of the text frame can be calculated using the corresponding face position information and name position information.

[0092] For example, the face position information of the face frame corresponding to the mth face in the first face set is (file name, upper left corner x 1m , upper left corner y 1m , length w 1m , width h 1m ), then the center coordinates of the face frame corresponding to the mth face are The name position information of the text box corresponding to the nth name in the first name set is (file name, upper left corner x'1, upper left corner y'1, length w'1, width h'1), then the center coordinates of the text box corresponding to the nth name are Then the distance between the face frame corresponding to the mth face in the first face combination and the text frame corresponding to the nth name in the first name set is

[0093] Using the Hungarian algorithm, based on the principle of the closest distance, the minimum distance among the multiple distances is selected during the initial pairing, and the name corresponding to the minimum distance is the name initially paired with the mth face.

[0094] Furthermore, in the above scheme, taking the target image extracted from the conference video as an example, there is a specific positional relationship between the face in the target image and its corresponding name. Therefore, after the face and the name are initially matched using the Hungarian algorithm, it is necessary to calculate the relative distance between the initially matched face frame and the corresponding text frame. The relative distance can be the distance between the face frame and the text frame relative to the length or width of the face frame, or the distance between the face frame and the text frame relative to the diagonal of the face frame. If the relative distance between the face frame and the text frame corresponding to the initially matched face and name is not greater than a preset threshold, the initially matched face and name are paired; the preset threshold can be selected as a distance value three times the length of the diagonal of the face frame.

[0095] In the above scheme, the text of the target image may contain information other than names. For example, in the target image extracted from a conference video, the seat card may contain not only name information but also job information. In this case, the extracted text is (job title) or (name). Therefore, it is necessary to further extract valid name information, that is, the final extracted text is (name).

[0096] Based on this, in one embodiment, the method of using a text detection and recognition network to detect names in a target image to obtain a first name set includes:

[0097] Use a text detection and recognition network to detect text in the target image to obtain a first text set;

[0098] The text detection and recognition network is used to recognize names in the first text set to obtain a first name set.

[0099] In the above scheme, taking the target image extracted from the conference video as an example, there may be multiple seat cards in the target image. When the text detection and recognition network is used to detect the text in the target image, multiple text boxes will also be detected. At this time, since the seat cards are not adjacent, multiple text boxes whose distances are less than a preset threshold are used to form a recognition sample. The multiple text boxes contained in the recognition sample are all the text boxes on a seat card. Furthermore, in order to extract valid name information, that is, the final extracted text is (name), it is also necessary to filter out the text boxes that do not contain names in the recognition sample.

[0100] Based on this, in one embodiment, the step of using a text detection and recognition network to recognize names in the first text set to obtain a first name set includes:

[0101] According to the position information of the text box corresponding to each text in the first text set, a recognition sample is formed by using a plurality of text boxes whose distances to each other are less than a preset threshold;

[0102] For each recognition sample, merging at least part of the text boxes in the recognition sample to obtain a merged box corresponding to the recognition sample;

[0103] A word segmentation filtering tool is used to filter out merged boxes that do not contain names, to obtain merged boxes that contain names, and a first name set is obtained based on the merged boxes that contain names.

[0104] In the above scheme, since the seat plates carrying people's names have the same color, that is, the background texture features of the image area corresponding to the text boxes contained in each recognition sample are similar or identical, the similarity of the texture features can be used to merge adjacent text boxes in the same recognition sample.

[0105] Based on this, in one embodiment, merging at least part of the text boxes in the recognition sample includes:

[0106] Scale the image area corresponding to each text box in the recognition sample to the same size;

[0107] Extract the texture features of the background of the image area corresponding to each text box;

[0108] Calculate the frequency distribution corresponding to each texture feature;

[0109] The similarity of the frequency distribution of texture features corresponding to two adjacent text boxes is calculated. If the similarity is greater than a preset threshold, the two adjacent text boxes are merged.

[0110] In the above scheme, texture feature is a kind of global feature, which is used to describe the surface properties of the scene corresponding to the image or image area. Specifically, texture feature is a visual feature that reflects the homogeneous phenomenon in the image, which reflects the surface structure organization and arrangement properties of the object surface with slow or periodic changes. Texture is different from image features such as grayscale and color. It is expressed by the grayscale distribution of pixels and their surrounding spatial neighborhoods, that is, local texture information.

[0111] In the above scheme, because the seat cards carrying the names have the same color, that is, the texture features of the background of the image area corresponding to the text boxes contained in each recognition sample are similar or the same, the improved LBP operator can be used to extract the texture features of the background of the image area corresponding to each text box, and the frequency distribution of the texture features corresponding to two adjacent text boxes is defined as X(x1, x2…x j )、Y(y1,y2…y j ), then according to the similarity calculation formula, the similarity between X and Y is Among them, j is the dimension of texture features, are the characteristic means of X and Y respectively, σX and σY are the variances of X and Y respectively.

[0112] In the above solution, the preset threshold value of the similarity of the frequency distribution of the texture features corresponding to the two text boxes may be any value greater than 0.85 and less than 1, such as 0.9.

[0113] The embodiment of the present application provides a face detection and recognition method, wherein a face detection network is used to detect faces in a target image to obtain a first face set; a text detection and recognition network is used to recognize names in the target image to obtain a first name set; and faces are paired with names based on the positions of each face in the first face set and the positions of each name in the first name set.

[0114] In the embodiment of the present application, the face detection network can accurately detect the faces contained in the target image and the positions corresponding to the faces, and the text detection and recognition network can accurately detect the names contained in the target image and the positions corresponding to the names. In this way, based on the positions of the faces in the first face set and the positions of the names in the first name set, the faces in the target image containing the corresponding names can be accurately and efficiently detected and recognized without the need for additional human interaction.

[0115] The present application also provides a method for training a face detection and recognition network. Figure 2 As shown, Figure 2 A flowchart of a method for training a face detection and recognition network provided in an embodiment of the present application. The method for training a face detection and recognition network includes:

[0116] Step 201: Acquire multiple target images from a target video;

[0117] Step 202: Obtaining annotation data for each target image, wherein the annotation data at least includes the name of the person paired with the face in the target image;

[0118] Step 203: using the multiple target images containing the labeled data as an image training set, and using the image training set to train a face detection and recognition network;

[0119] Among them, the name of the person paired with the face in the target image is obtained by the face detection and recognition method described in the embodiment of the present application.

[0120] In order to implement the face detection and recognition method described in the embodiment of the present application, the embodiment of the present application also provides a face detection and recognition device 300, such as Figure 3 As shown, Figure 3 The face detection and recognition device 300 provided in the embodiment of the present application is a schematic diagram of the structure of the face detection and recognition device 300. The face detection and recognition device 300 includes:

[0121] A face set acquisition unit 301 is used to detect faces in a target image using a face detection network to obtain a first face set;

[0122] A name set acquisition unit 302 is used to recognize names in a target image using a text detection and recognition network to obtain a first name set;

[0123] The pairing unit 303 is configured to pair faces with names based on the positions of the faces in the first face set and the positions of the names in the first name set.

[0124] Those skilled in the art should understand that Figure 3 The implementation functions of each unit in the face detection and recognition device 300 shown can be understood by referring to the relevant description of the above-mentioned face detection and recognition method. Figure 3 The functions of the various units in the face detection and recognition device 300 shown can be implemented by a program running on a processor, or can be implemented by a specific logic circuit.

[0125] It should be noted that: the face detection and recognition device 300 provided in the above embodiment only uses the division of the above program units as an example when performing face detection and recognition. In actual applications, the above processing can be assigned to different program units as needed, that is, the internal structure of the system is divided into different program units to complete all or part of the processing described above. In addition, the face detection and recognition device 300 provided in the above embodiment and the face detection and recognition method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0126] In order to implement the training method of the face detection and recognition network described in the embodiment of the present application, the embodiment of the present application also provides a training device 400 for the face detection and recognition network, such as Figure 4 As shown, Figure 4 A schematic diagram of the structure of a face detection and recognition network training device 400 provided in an embodiment of the present application. The face detection and recognition network training device 400 comprises:

[0127] The target image acquisition unit 401 is used to acquire multiple target images from the target video;

[0128] A labeling unit 402 is used to obtain labeling data for each target image, wherein the labeling data at least includes a name of a person paired with a face in the target image; wherein the name of a person paired with a face in the target image is obtained by the face detection and recognition method described in the embodiment of the present application;

[0129] The training unit 403 is used to use the multiple target images containing the labeled data as an image training set, and use the image training set to train the face detection and recognition network.

[0130] Those skilled in the art should understand that Figure 4 The implementation functions of each unit in the training device 400 for face detection and recognition network shown can be understood by referring to the relevant description of the training method for face detection and recognition network mentioned above. Figure 4 The functions of the various units in the training device 400 for face detection and recognition network shown can be implemented by a program running on a processor, or by a specific logic circuit.

[0131] It should be noted that: the training device 400 for face detection and recognition network provided in the above embodiment only uses the division of the above program units as an example when training the face detection and recognition network. In actual applications, the above processing can be assigned to different program units as needed, that is, the internal structure of the system is divided into different program units to complete all or part of the processing described above. In addition, the training device 400 for face detection and recognition network provided in the above embodiment and the training method embodiment of face detection and recognition network belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0132] The present application is further described below in conjunction with the application examples:

[0133] Figure 5 A flow chart of a training method for a face detection and recognition network provided in an application embodiment of the present application is shown in FIG. Figure 5As shown, the application embodiment of the present application provides a training method for a face detection and recognition network, which is used to train the face detection and recognition network using conference videos. The training method for a face detection and recognition network provided by the application embodiment of the present application can train the face recognition network without manually annotating the faces in the conference video.

[0134] like Figure 5 As shown, the training method of the face detection and recognition network provided by the application embodiment of the present application includes the following steps:

[0135] Step 501: Select a video segment from the prepared sufficient conference videos, and read a frame of image in the video segment as a target image;

[0136] Step 502: For the target image obtained in step 501, use a face detection network to detect faces in the target image to form a first face set M (M1, M2…M m );

[0137] Step 503: For the target image obtained in step 501, use the text detection and recognition network to detect the names in the target image to form a first name set N (N1, N2...N n ); specifically, it includes the following steps:

[0138] Step 5031: Use the text detection and recognition network to detect the text in the target image to obtain a first text set; according to the position information of the text box corresponding to each text in the first text set, use multiple text boxes whose distances are less than a preset threshold to form a recognition sample T i (t1,t2…t k ,t k+1 …), that is, the multiple text boxes located on the i-th seat sign constitute a recognition sample T i (t1,t2…t k ,t k+1 …), t k and t k+1 Represents two adjacent text boxes. The set of recognition samples on all seat cards in the target image is T(T1, T2…T i …);

[0139] Step 5032: If Then for T i (t1,t2…t k ,t k+1 …) attempt to merge each text box in the same folder, including the following steps:

[0140] Step 50321: T i (t1,t2…t k ,tk+1 …) are scaled to the same size for each text box in the image. k The position order is arranged from left to right according to the position of each text box in the target image;

[0141] Step 50322: Extract each text box t using the improved LBP operator k Texture features of the background of the corresponding image area;

[0142] Step 50323: Calculate each text box t k The frequency distribution corresponding to the texture features of the background of the corresponding image area;

[0143] Step 50324: Calculate the number of adjacent text boxes t k and t k+1 The frequency distribution of the corresponding texture features X(x1,x2…x j ) and Y(y1,y2…y j )

[0144] Step 50325: If the similarity is greater than a preset threshold, such as 0.9, then t k and t k+1 merge;

[0145] Step 50326: Repeat steps 50324 and 50325 until T i (t1,t2…t k ,t k+1 …) the merger is completed, or further merger is impossible;

[0146] Step 5033: Repeat step 5032 until T(T1, T2…T i ...) in all T i The merger is complete;

[0147] Step 5034: Use the word segmentation filter tool to filter out T that does not contain a person's name i , get T containing a person's name i ; Using T containing a person's name i , get the first name set N(N1,N2…N n );

[0148] Step 504: The first face set M (M1, M2…M m ), and the first name set N(N1, N2…N) obtained in step 504 n) Pair the faces with the names based on the positions of the faces in the first face set and the positions of the names in the first name set, and record the pairing information as labeled data;

[0149] Step 505: Steps 501 to 504 until the conference video reading is completed;

[0150] Step 506: Use multiple target pictures containing labeled data as an image training set, and use the image training set to train the face detection and recognition network.

[0151] To facilitate those skilled in the art to more clearly understand the technical solution of the present application, the following will further describe step 503 of the training method of the face detection and recognition network provided by the application embodiment of the present application with examples. The method for obtaining the first name set provided by the example of the present application includes the following steps:

[0152] As Figure 6a shown, Figure 6a is a schematic diagram of the seat card intercepted from the target picture in the example of the present application. The method for obtaining the first name set from the Figure 6a seat card includes the following steps:

[0153] Step 601: As Figure 6b shown, Figure 6b is a schematic diagram of obtaining recognition samples from the seat card in the example of the present application, including: using a text detection and recognition network to detect the text in the target picture to obtain a first text set (Li Yao); since the distance between the text boxes corresponding to "Li" and "Yao" is less than a preset threshold, the text boxes corresponding to "Li" and "Yao" are combined to form a recognition sample (Li Yao);

[0154] Step 602: Merge the text boxes corresponding to "Li" and "Yao", including the following steps:

[0155] Step 6021: As Figure 6c shown, Figure 6c is a schematic diagram of scaling the image regions corresponding to the text boxes to the same size in the example of the present application. Scale the image regions corresponding to "Li" and "Yao" to the same size and arrange them side by side;

[0156] Step 6022: As Figure 6d shown, Figure 6d is a schematic diagram of extracting the texture features of the background of the image regions corresponding to the text boxes in the example of the present application. Extract the texture features of the backgrounds of the image regions corresponding to the text boxes of "Li" and "Yao";

[0157] Step 6023: As Figure 6e shown, Figure 6eA schematic diagram for calculating the frequency distribution of texture features in this application example. Calculate the frequency distribution of the texture features described in step 6022, and determine whether the text boxes corresponding to "Li" and "Yao" can be merged according to the similarity of the frequency distribution.

[0158] Step 6024: In the example of this application, the similarity is 0.9456. Therefore, merge the text boxes corresponding to "Li" and "Yao"; use a word segmentation filtering tool to determine that (Li Yao) is a person's name, and thus obtain a merged box containing a person's name (Li Yao).

[0159] Step 603: As Figure 6f shown, Figure 6f A schematic diagram for obtaining the first set of person's names in this application example. Obtain the first set of person's names (Li Yao).

[0160] To implement the training method of the face detection and recognition network described in the application embodiment of this application, the application embodiment of this application also provides a training device 700 for the face detection and recognition network. Figure 7 A schematic diagram of the structure of the training device 700 for the face detection and recognition network provided by the application embodiment of this application. As Figure 7 shown, the training device 700 for the face detection and recognition network provided by the application embodiment of this application includes:

[0161] A target picture acquisition unit 701, configured to select a video from a sufficient amount of prepared conference videos and read a frame image in the video as a target picture.

[0162] A face set acquisition unit 702, configured to detect faces in the target picture using a face detection network to form a first face set M (M1, M2... M m ).

[0163] A person's name set acquisition unit 703, configured to detect person's names in the target picture using a text detection and recognition network to form a first person's name set N (N1, N2... N n ), including:

[0164] An identification sample acquisition module 7031, configured to detect text in the target picture using a text detection and recognition network to obtain a first text set; according to the position information of the text boxes corresponding to each text in the first text set, use multiple text boxes with distances less than a preset threshold to form an identification sample T i (t1, t2... t k , t k+1 ...);

[0165] An identification sample text box merging module 7032, configured to perform, for each identification sample T i (t1, t2... t k,t k+1 …) try to merge each text box in

[0166] The merge frame word segmentation filtering module 7033 is used to use the word segmentation filtering tool to filter out T words that do not contain human names. i , get T containing a person's name i ; Using T containing a person's name i , get the first name set N(N1,N2…N n );

[0167] A pairing unit 704, configured to pair faces with names based on the positions of the faces in the first face set and the positions of the names in the first name set;

[0168] A labeling unit 705 is used to obtain labeling data of the target image, wherein the labeling data at least includes the name of the person paired with the face in the target image; wherein the labeling data is obtained from the pairing result of the pairing unit 704;

[0169] The training unit 706 is used to use the target image containing the labeled data as an image training set, and use the image training set to train the face detection and recognition network.

[0170] The embodiment of the present application further provides an electronic device 800, including:

[0171] Memory 802, used to store executable instructions;

[0172] The processor 801 is used to implement the face detection and recognition method described in the embodiment of this application, or the training method of the face detection and recognition network described in the embodiment of this application when executing the executable instructions stored in the memory 802.

[0173] In practical application, such as Figure 8 As shown, the various components in the electronic device 800 are coupled together through a bus module 803. It can be understood that the bus module 803 is used to realize the connection and communication between these components. In addition to the data bus, the bus module 803 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 8 Various buses are labeled as bus module 803.

[0174] An embodiment of the present application also provides a storage medium, which stores executable instructions. When the executable instructions are executed by at least one processor 801, the face detection and recognition method described in the embodiment of the present application is implemented, or the training method of the face detection and recognition network described in the embodiment of the present application is implemented.

[0175] In some embodiments, the storage medium can be a magnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface storage, an optical disk, or a compact disc read-only memory (CD-ROM); it can also be various devices including one or any combination of the above memories.

[0176] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0177] As an example, executable instructions may, but do not necessarily, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0178] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0179] It should be noted that: "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0180] The technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0181] In the several embodiments provided in the present application, it should be understood that the disclosed methods and electronic devices can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0182] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0183] In addition, all functional units in the embodiments of the present application may be integrated into a second processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0184] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A face detection and recognition method, characterized in that: include: Use the face detection network to detect faces in the target image and obtain a first face set; Use the text detection and recognition network to identify the names of people in the target image and obtain the first name set; Using a bipartite graph matching algorithm to initially pair each face in the first face set with each name in the first name set; The relative distance between the face frame and the text frame corresponding to the initially paired face and name is calculated. If the relative distance is less than a preset threshold, the initially paired face and name are paired; wherein the relative distance represents the distance between the face frame and the text frame relative to the length or width of the face frame.

2. The method according to claim 1, characterized in that Before using the face detection network to detect the face in the target image, the method further includes: Extract a frame of image from the target video as the target image; Using a face detection network to detect whether there is a face in the target image, and using a text detection and recognition network to identify whether there is a name in the target image; Accordingly, the method of using a face detection network to detect faces in a target image to obtain a first face set includes: If there is a face in the target image, use a face detection network to detect the face in the target image to obtain a first face set; Accordingly, the text detection and recognition network is used to recognize the names of people in the target image to obtain a first name set, including: If there are names in the target image, a text detection and recognition network is used to recognize the names in the target image to obtain a first name set.

3. The method according to claim 2, characterized in that Also includes: If there is no human face in the target image and / or there is no human name in the target image, the target image is updated based on the target video.

4. The method according to claim 3, characterized in that The updating of the target picture based on the target video includes: According to a preset frame interval, the next frame of the current target picture is extracted from the target video as the updated target picture.

5. The method according to any one of claims 1 to 4, characterized in that The method of using the text detection and recognition network to detect names in the target image and obtain a first name set includes: Use a text detection and recognition network to detect text in the target image to obtain a first text set; The text detection and recognition network is used to recognize names in the first text set to obtain a first name set.

6. The method according to claim 5, characterized in that The step of using a text detection and recognition network to identify names in the first text set to obtain a first name set includes: According to the position information of the text box corresponding to each text in the first text set, a recognition sample is formed by using a plurality of text boxes whose distances to each other are less than a preset threshold; For each recognition sample, merging at least part of the text boxes in the recognition sample to obtain a merged box corresponding to the recognition sample; A word segmentation filtering tool is used to filter out merged boxes that do not contain names, to obtain merged boxes that contain names, and a first name set is obtained based on the merged boxes that contain names.

7. The method according to claim 6, characterized in that The merging of at least part of the text boxes in the recognition sample comprises: Scale the image area corresponding to each text box in the recognition sample to the same size; Extract the texture features of the background of the image area corresponding to each text box; Calculate the frequency distribution corresponding to each texture feature; The similarity of the frequency distribution of the texture features corresponding to two adjacent text boxes is calculated. If the similarity is greater than a preset threshold, the two adjacent text boxes are merged.

8. A training method for a face detection and recognition network, characterized in that: include: Get multiple target images from the target video; Obtaining annotation data for each target image, wherein the annotation data includes at least a name of a person paired with a face in the target image; Use multiple target images containing labeled data as image training sets, and use the image training sets to train the face detection and recognition network; The name of a person paired with a face in the target image is obtained by the face detection and recognition method described in any one of claims 1 to 7.

9. A face detection and recognition device, characterized in that: include: A face set acquisition unit detects faces in a target image using a face detection network to obtain a first face set; A name set acquisition unit, which uses a text detection and recognition network to recognize names in a target image and obtain a first name set; a pairing unit, using a bipartite graph matching algorithm to initially pair each face in the first face set with each name in the first name set; The relative distance between the face frame and the text frame corresponding to the initially paired face and name is calculated. If the relative distance is less than a preset threshold, the initially paired face and name are paired; wherein the relative distance represents the distance between the face frame and the text frame relative to the length or width of the face frame.

10. A training device for a face detection and recognition network, characterized in that: include: A target image acquisition unit, used to acquire multiple target images from a target video; A labeling unit, used to obtain labeling data for each target image, wherein the labeling data at least includes a name of a person paired with a face in the target image; wherein the name of a person paired with a face in the target image is obtained by the face detection and recognition method according to any one of claims 1 to 7; The training unit is used to use multiple target images containing labeled data as image training sets, and use the image training sets to train the face detection and recognition network.

11. An electronic device, characterized in that: include: A memory for storing executable instructions; A processor, for implementing the face detection and recognition method described in any one of claims 1 to 7, or the face detection and recognition network training method described in claim 8, when executing the executable instructions stored in the memory.

12. A storage medium, characterized in that: The storage medium stores executable instructions. When the executable instructions are executed by at least one processor, the face detection and recognition method described in any one of claims 1 to 7 is implemented, or the training method of the face detection and recognition network described in claim 8 is implemented.

Images