A design method of an image review and editing system based on face recognition

Abstract

The application relates to a kind of image review and editing system design methods based on face recognition, belonging to computer software field.The application establishes portrait detection model and face recognition model in turn by training, detects and identifies image content by video frame extraction, first obtains face result coordinates by portrait detection, automatically calculates the pixel size of target area, estimates the picture proportion of target portrait, forms picture information component; the face image output by portrait detection is identified by face recognition, the identification result is integrated with picture layout information and returned to the video editing interface, the picture information of current frame is intelligently labeled to form frame track additional information.The application applies face recognition to the design of image review and image editing field, selects appropriate algorithm for the field work, optimizes the framework, adds vector database, and greatly improves the recognition accuracy and efficiency.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer software, and particularly relates to a design method of an image review and editing system based on face recognition. BACKGROUND

[0002] Nowadays, image review and editing has become an important part of work in various fields, especially in the fields of news work, self-media and the like. However, the image editing tools on the market at present are mainly local software, which are not friendly to collaborative office, and mainly have various editing functions, lack of intelligence, and users still need to watch the entire video to obtain image content information and then review or edit. For videos that need to be reviewed and edited, the quantity is large and the content is miscellaneous, which will inevitably cause the problems of large workload and difficult image content positioning, and a large amount of time is wasted on video content searching, resulting in low work efficiency.

[0003] In view of the pain points of the current image review and editing tools, the present application provides a feasible online intelligent image review and editing system implementation scheme by using image detection and face recognition technology, and the recognition accuracy and efficiency are improved by optimization of algorithm and framework. Taking a person as the main body, the face area in the picture and video picture is obtained through portrait detection, the proportion of the person picture is estimated through the area pixel, the target person is identified through face recognition, the identification result, time stamp and picture proportion information are integrated, a video script is formed and displayed to the user, which effectively assists the review and control of the submitted pictures, audios and videos, so that the user can better focus on video review and editing rather than content searching. SUMMARY

[0004] (I) Technical problem to be solved

[0005] The technical problem to be solved by the present application is how to provide a design method of an image review and editing system based on face recognition to solve the problem of low work efficiency of image review and editing.

[0006] (II) Technical scheme

[0007] In order to solve the above technical problem, the present application provides a design method of an image review and editing system based on face recognition, which comprises the following steps:

[0008] S1, uploading video: using object storage technology, uploading the video to a storage service, different users pulling related data according to corresponding permissions, playing and watching the video online, editing, and realizing collaborative office;

[0009] S2, uploading a portrait of a reported person: according to the requirement, whether to upload a portrait of a reported person can be selected, if the portrait of the reported person is selected to be uploaded, step S3 is executed; otherwise, step S4 is executed;

[0010] S3, image analysis: image analysis will call the trained face recognition model, responsible for extracting the uploaded portrait into a multi-dimensional vector feature, stored in the Mlvus vector database, thereby updating the feature library of the model;

[0011] S4, video analysis: video analysis includes two parts of portrait detection and face recognition; first, use the portrait detection model to detect the frame picture of the video, and process the image containing the face; use the face recognition model to recognize the face of the processed face image;

[0012] S5, the result of video analysis is packaged and transmitted to the front end rendering, marked by personnel or content, and two forms of results are generated; the first is the track result in the form of time track; the second is the explicit display of text results;

[0013] S6, artificial review: artificial review of S5 results, record / mark each picture or frame or each shot specific opinion, comprehensive generate the final review result list, related list text file support export.

[0014] Further, the image review and editing system comprises a presentation layer, a transmission layer, a service layer and a data layer;

[0015] The presentation layer is a user-oriented Web end, realizing online image editing in Web mode, facilitating collaborative office and data sharing;

[0016] The transmission layer is responsible for data transmission between the presentation layer and the service layer;

[0017] The service layer is responsible for the realization of all business functions of the system;

[0018] The data layer is responsible for storing user information and historical data.

[0019] Further, the presentation layer is built with the React development framework; the transmission layer uses HTTP network request related technologies for data transmission, while maintaining the RESTful style in interface design; the service layer uses SpringBoot technology, uses SpringAOP to realize the functions of system log, exception handling and permission control, uses Pytorch and Opencv related technologies to realize portrait detection algorithm and face recognition algorithm, uses Flask Python Web framework to deploy the algorithm online, and finally uses ResTemplate to realize the communication between the business server and the algorithm server; the data layer uses Milmvus vector database to store, retrieve and compare the vector features extracted by the model, uses MySQL database to store user personal information and system related data, and uses OOS object storage service to store various cases, activity videos and activity pictures.

[0020] Further, in step S4, the first track result in the form of a time track is displayed on the existing video progress bar, listing time code, main personnel and picture proportion information, and the same person recognition result is aggregated in multiple frames.

[0021] Further, in step S4, the second type is an explicitly displayed text result, including: timestamp, character name and picture proportion information, forming a video script to facilitate quick searching and provide auxiliary reference when manually reviewing.

[0022] Further, the portrait detection model selects the YOLOv5 model, the training set and the verification set use CelebADataset, select the pictures to be trained and make YOLO format labels, and the test data set uses LFW to measure the effect of the model.

[0023] Further, the steps of the portrait detection are as follows:

[0024] S31, first build a portrait detection model for calling;

[0025] S32, frame extraction processing is performed on the video;

[0026] S33, face detection is performed on the extracted pictures to determine whether there are face pictures meeting the requirements, if yes, step S34 is executed, and if no, step S32 is executed;

[0027] S34, when there are face pictures meeting the requirements in the detection result, save and output the face coordinates;

[0028] S35, restore the coordinate values after normalization processing, and calculate the picture proportion through the restored coordinates;

[0029] S36, the detected face target is cropped, and a video timestamp and a picture proportion number are marked and saved for subsequent face recognition.

[0030] Further, the face recognition model selects FaceMobileNet supported by additive angle margin loss Arcface, obtains a portrait data set through data crawling or file system docking, and generates a label text file. The backbone of the face recognition model selects FaceMobileNet based on MobileNet, stacks multiple different convolution blocks, then connects a Flatten block to flatten the input, and then connects a fully connected layer and a 1-dimensional BatchNorm layer. The loss function selects Focal Loss.

[0031] Further, the metric function of the face recognition model implements CosFace and ArcFace, and increases the margin on the classification boundary based on Softmax Loss, and then increases the inter-class distance. Preferably, ArcFace is used first. Since ArcFace has an out-of-bound problem, the face recognition process is optimized to add a judgment logic for whether ArcFace is out of bounds. The out-of-bound of ArcFace is determined by whether the additional angle exceeds 180 degrees. If the out-of-bound occurs, CosFace is used instead of ArcFace.

[0032] Further, the face recognition steps include:

[0033] S41, a trained face recognition model is first called;

[0034] S42, it is judged whether a reported character picture marked is uploaded. If a reported character picture is input, step S43 is executed, otherwise step S44 is executed.

[0035] S43, a face image feature is extracted by using the loaded model, and the feature mean is stored in a Mivus vector database;

[0036] S44, a face image output by a portrait detection model is used as an input of a face recognition model for face recognition. A face image is extracted into a 512-dimensional feature vector through Arcface feature calculation.

[0037] S45, the feature vector obtained in S44 is measured with the reference face feature vector in the Mi lvus vector database, the cosine distance is used to measure the distance of the two faces, the training process is kept corresponding to each other, when the similarity exceeds a certain threshold, then it is determined that the two faces are the same person, if the similarity does not exceed the threshold, the face is not in the Mi lvus vector database, and finally the recognition result is returned to the front-end processing.

[0038] (III) Beneficial Effects

[0039] The present application proposes a kind of image review and editing system design method based on face recognition, compared with prior art, the present application proposes that face recognition is applied to the design of image review, image editing field, and selects suitable algorithm for the field work, optimizes framework, joins vector database, greatly improves recognition accuracy and efficiency. By training, portrait detection model and face recognition model are established in turn. By video frame extraction, the image content is detected and identified, first, the face result coordinates are obtained by portrait detection, the pixel size of target area is automatically calculated, the picture proportion estimation of target portrait is carried out, the picture information component is formed, and the video content position of target picture composition is positioned quickly and conveniently;The face image output by portrait detection is identified by face recognition, and the recognition result is integrated with picture layout information and returned to video editing interface, intelligently labels the picture information of current frame, forms frame track additional information, supports the convergence of multiple frames with same recognition result, so that system user can obtain video content related information without frame-by-frame video viewing. When face feature storage, retrieval, comparison are carried out, Mi lvus vector database is used, real-time updating and high-speed retrieval comparison of vector data are realized, the whole face recognition process is optimized, and the system recognition accuracy and efficiency are improved. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 It is the overall architecture diagram of the present application;

[0041] Figure 2 It is the system flow chart of the present application;

[0042] Figure 3 It is the portrait detection flow chart of the present application;

[0043] Figure 4 It is the face recognition flow chart of the present application. DETAILED DESCRIPTION

[0044] To make the purpose, content and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in combination with the drawings and examples.

[0045] The purpose of the present application is to propose a technology capable of utilizing portrait detection and face recognition to effectively identify image content and apply it to intelligent online image review and editing systems in the news industry and related fields.

[0046] Non-structured data processed by a deep learning model is vectorized, and the analysis and processing of massive non-structured data is converted into approximate search of massive vectors. This causes two problems, one is the low efficiency of analysis, the number of pictures after video frame extraction is large, the target to be analyzed is also large, which naturally causes time-consuming analysis, and the face feature library generated by the model has a large data volume, which is more time-consuming for search and comparison; the second is the management problem of the feature library, face recognition requires massive face feature data, the most common method is text saving, which is not easy to manage, and the characters in news reports are unpredictable, requiring high updateability. To solve the above problems, the face features are stored, searched and compared based on the Milvus vector database, the system framework is optimized, the feature data is updated in real time, and the image analysis efficiency is greatly improved. Since the online image editing technology is mature today, the present application mainly describes the intelligent recognition service part.

[0047] 1. Overall architecture

[0048] Figure 1 The overall architecture of the present application is shown in Figure 1 The system technical architecture can be divided into four layers, the presentation layer, the transmission layer, the service layer and the data layer.

[0049] The presentation layer is mainly for the user's Web end, and the system realizes Web-based online image editing, facilitates collaborative office work, and adopts the React development framework for construction;

[0050] The transmission layer is responsible for data transmission between the presentation layer and the service layer, and the system mainly uses HTTP network request related technology for data transmission, while maintaining the RESTful style in interface design;

[0051] The service layer is responsible for the realization of all business functions of the system, adopts SpringBoot technology, uses Spring AOP to realize system log, exception handling and permission control functions, uses Pytorch and Opencv related technologies to realize portrait detection algorithm and face recognition algorithm, uses Flask Python Web framework to deploy the algorithm online, and finally uses ResTemplate to realize the communication between the business server and the algorithm server.

[0052] The data layer is responsible for storing face vector features and user information, system history and other data. The system uses Milvus vector database to store, retrieve and compare the vector features extracted by the model, uses MySQL database to store user personal information and system related data, and uses OOS object storage service to store various files such as application, activity video and activity picture.

[0053] 2. System use

[0054] The steps of system use are shown in Figure 2

[0055] The present application comprises the following steps:

[0056] S1, uploading video: video uploading uses object storage technology to upload video to storage service. Different users can pull relevant data according to corresponding permissions, and can play, watch and edit video online to realize collaborative office.

[0057] S2, uploading portrait of reported person: it means to avoid recognition failure caused by no specific person during face recognition model training. Whether to upload can be selected according to demand. If uploading is selected, step S3 is executed; otherwise, step S4 is executed.

[0058] S3, image analysis: image analysis will call the trained face recognition model, which is responsible for extracting the uploaded portrait into a multi-dimensional vector feature and storing it in the Milvus vector database, so as to update the feature library of the model.

[0059] S4, video analysis: video analysis includes two parts of portrait detection and face recognition. First, the portrait detection model is used to detect the portrait of the frame picture of the video, and the image containing the face is processed. The specific process is shown in the "portrait detection" step. The face recognition model is used to recognize the face of the processed face image. The specific process is shown in the "face recognition" step.

[0060] S5, the result of video analysis is packaged and transmitted to the front end for rendering, marked by personnel or content, and two forms of results are generated. The first one is the track result in the form of time track, which is displayed on the existing video progress bar, and lists the time code, main personnel and picture proportion information. The same person recognition result will be aggregated in multiple frames. The second one is the explicit display of text result, including: timestamp, person name and picture proportion information, forming a video script, so that artificial review can quickly find and provide auxiliary reference;

[0061] S6, artificial review: artificial review of the result of S5, recording / marking the specific opinions of each picture or each frame or each shot, and comprehensively generating the final review result list. The related list text file supports export. ​

[0062] 3. Portrait detection

[0063] a. Algorithm selection

[0064] To achieve the picture proportion of the reported character, the picture proportion of the detection frame is calculated through the coordinates of the detection frame, and then the picture proportion of the character is estimated. Due to the diversity of image acquisition devices, different standard sizes of image pictures may be caused, resulting in different aspect ratios of pictures after actual video frame extraction. Therefore, after scaling and filling, the size of the black filling area at both ends is different, and if the filling is more, there is information redundancy, which affects the inference speed.

[0065] To solve the above problems, the system selects the YOLOv5 model algorithm for portrait detection, which can realize the acquisition of the coordinates of the four corners of the target detection frame through a small amount of code modification, and then perform subsequent proportion calculation. Compared with the previous versions of the YOLO series, YOLOv5 has added adaptive picture scaling, which reduces the computational complexity during inference and improves the target detection speed.

[0066] b. Dataset

[0067] The training set and the verification set use CelebADataset, select the pictures to be trained and make YOLO format labels. The test dataset uses LFW to measure the effect of the model.

[0068] c. Algorithm network construction

[0069] Since the target detection has requirements for frame rate, the commonly used YOLOv5s model is selected instead of YOLOv5n, YOLOv5m or other models, and considering the speed and accuracy, the model is small, the corresponding inference speed is fast, and the frame rate during target detection is also fast.

[0070] d. Portrait detection process

[0071] The steps of portrait detection are shown in Figure 3 .

[0072] S31, first build a portrait detection algorithm model for calling;

[0073] S32, perform frame extraction processing on the video;

[0074] S33, perform face detection on the extracted pictures, and determine whether there is a face picture meeting the requirements, if yes, execute step S34, if no, execute step S32;

[0075] S34, when there is a face picture meeting the requirements in the detection result, save the output face coordinates;

[0076] S35, restore the normalized coordinate values, and calculate the picture proportion through the restored coordinates;

[0077] S36, the detected face target is cropped, and the video timestamp, picture proportion number is labeled and saved for subsequent face recognition.

[0078] 4. Face recognition

[0079] a. Algorithm selection

[0080] Because the news industry targets a variety of reports, the training data is updated quickly, and the model training is relatively frequent, so the face recognition model selects FaceMobileNet supported by additive angle margin loss Arcface, obtains highly distinguishable features for face recognition, has high training efficiency, and almost no additional computational overhead.

[0081] b. Data preparation

[0082] Obtain the portrait data set through data crawling or archive system docking, and generate a label text file yourself.

[0083] c. Network structure

[0084] The backbone of the model selects FaceMobileNet based on MobileNet, stacks multiple different convolution blocks, then connects a Flatten block to flatten the input, and then connects a fully connected layer and a 1-dimensional BatchNorm layer. The length of the final feature vector is selected as 512, that is, a 512-dimensional vector is used to represent the face.

[0085] The loss function selects Focal Loss, which reduces the contribution of easy samples to the loss, so that the model pays more attention to difficult samples, and the model learns better features.

[0086] The metric function implements CosFace and ArcFace, which are improved based on the original Softmax Loss, increase the margin on the classification boundary, map the features to the angle feature space first, and then increase the inter-class distance. Prefer to use ArcFace, because ArcFace has an out-of-bound problem, so the face recognition process is optimized, and the judgment logic of whether ArcFace is out of bounds (whether the out-of-bound of ArcFace is judged by whether the additional angle exceeds 180 degrees) is added., to determine which metric function to select, if it occurs, use CosFace instead of ArcFace.

[0087] d. Face recognition process

[0088] Because of the diversity of reports, the training data set cannot cover all reporting objects, in order to increase the expandability, provide the function of uploading the picture of the reported person, and update the feature library of the face recognition model in real time.

[0089] The steps of face recognition are as shown in Figure 4

[0090] S41, first call the trained FaceMobileNet face recognition model (a feature extractor trained based on the ArcFace metric learning algorithm);

[0091] S42, judge whether the labeled reportage character picture is uploaded, if the reportage character picture is uploaded, execute step S43, otherwise execute step S44;

[0092] S43, use the loaded model to extract the face image features, and store the feature mean in the feature library (Milvus vector database).

[0093] S44, use the output (the face image obtained by cutting) of the portrait detection model as the input of the face recognition model to perform face recognition, extract the face image into a 512-dimensional feature vector through Arcface feature calculation;

[0094] S45, measure the feature vector obtained in S44 and the reference face feature vector in the Milvus vector database, use the cosine distance to measure the distance between the two faces, keep corresponding to the training process, when the similarity exceeds a certain threshold, it is determined that the two faces are the same person, if the similarity does not exceed the threshold, the face is not in the Milvus vector database, and finally the recognition result is returned to the front end processing.

[0095] Compared with the prior art, the application applies face recognition to the field of image review and image editing, selects appropriate algorithms for the field, optimizes the framework, and adds a vector database, which greatly improves the recognition accuracy and efficiency. The portrait detection model and the face recognition model are established through training. The image content is detected and recognized by video frame extraction. First, the face result coordinates are obtained through portrait detection, the target area pixel size is automatically calculated, the target portrait picture proportion is estimated, the picture information components are formed, and the video content position of the target picture composition is quickly positioned; the face image output by the portrait detection is identified through face recognition, the recognition result is integrated with the picture layout information and returned to the video editing interface, the picture information of the current frame is intelligently labeled, the frame track additional information is formed, the same recognition result is gathered, and the system user can obtain the video content related information without watching the video frame by frame. The Milvus vector database is used for face feature storage, retrieval and comparison, the vector data is updated and high-speed retrieval and comparison in real time, the whole face recognition process is optimized, and the system recognition accuracy and efficiency are improved.

[0096] ​The above merely describes the preferred embodiments of the present application, and it should be pointed out that, for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present application, and these improvements and modifications should also be considered as the protection scope of the present application.

Claims

1. A face recognition-based image review and editing system design method, characterized in that, The method comprises the following steps: S1, uploading a video: using object storage technology, uploading the video to a storage service, different users pulling relevant data according to corresponding permissions, playing and watching the video online, editing, and realizing collaborative office work; S2, uploading a report character portrait: according to the requirement, whether to upload a report character portrait can be selected, if uploading is selected, step S3 is executed; otherwise, step S4 is executed; S3, image analysis: the image analysis will call a trained face recognition model, responsible for extracting the uploaded character portrait into a multi-dimensional vector feature, and storing it in a Milvus vector database, thereby updating the feature library of the model; S4, video analysis: video analysis includes two parts of portrait detection and face recognition; first, a portrait detection model is used to detect the portrait of the frame picture of the video, and the image containing the face detected is processed; a face recognition model is used to recognize the processed face image; S5, the result of video analysis is packaged and transmitted to the front end for rendering, marked by personnel or content, and two forms of results are generated; the first is a track result in the form of a time track; the second is a text result displayed explicitly; S6, artificial review: artificial review of the result of S5, recording / marking the specific opinions of each picture or frame or each shot, and comprehensively generating a final review result list, and a related list text file is supported for export; Among them, The face recognition model selects FaceMobileNet supported by additive angle margin loss Arcface; portrait data set is obtained through data crawling or archive system docking, and a label text file is generated; the backbone of the face recognition model selects FaceMobileNet based on MobileNet, stacks multiple different convolution blocks, then connects a Flatten block to flatten the input, and then connects a fully connected layer and a 1-dimensional BatchNorm layer, and the loss function selects FocalLoss; The metric function of the face recognition model realizes CosFace and ArcFace, increases the margin on the classification boundary based on Softmax Loss, maps the features to the angle feature space first, and then increases the inter-class distance; ArcFace is used, and since ArcFace has an out-of-bound problem, the face recognition process is optimized, the judgment logic of whether ArcFace is out of bound is added, and the out-of-bound of ArcFace is determined by whether the additional angle exceeds 180 degrees. If out-of-bound occurs, CosFace is used instead of ArcFace.

2. The face recognition-based image review and editing system design method of claim 1, wherein The image review and editing system comprises a presentation layer, a transmission layer, a service layer and a data layer; The presentation layer is a Web terminal for users, realizing Web type online image editing, facilitating collaborative office work and data sharing; The transmission layer is responsible for data transmission between the presentation layer and the service layer; The service layer is responsible for the realization of all business functions of the system; The data layer is responsible for storing feature vector data, user information and system historical data.

3. The face recognition based video review and editing system design method of claim 2, wherein, The presentation layer is built using the React development framework; the transmission layer uses HTTP network request related technologies for data transmission, while maintaining the RESTful style in interface design; the service layer uses SpringBoot technology, uses SpringAOP to implement system logging, exception handling, and permission control functions, uses Pytorch and Opencv related technologies to implement portrait detection algorithms and face recognition algorithms, uses the Flask Python web framework to deploy the algorithms online, and finally uses ResTemplate to implement communication between the business server and the algorithm server; the data layer uses Milvus vector database to store, retrieve and compare the vector features extracted by the model, uses MySQL database to store user personal information and system related data, and uses OOS object storage service to store various cases, activity videos and activity pictures.

4. The face recognition based video review and editing system design method of claim 1, wherein, In the step S4, the first kind of track result in the form of time track is displayed on the existing video progress bar, listing time code, main personnel and picture proportion information, and the same person recognition result is aggregated in multiple frames.

5. The face recognition based video review and editing system design method of claim 1, wherein, In the step S4, the second kind of explicit display text result includes: timestamp, person name and picture proportion information, forming a video script, so that manual review can quickly find and provide auxiliary reference.

6. The face recognition based video review and editing system design method of any one of claims 1-5, wherein, The portrait detection model selects YOLOv5 model, the training set and the verification set use CelebA Dataset, select the pictures to be trained and make YOLO format labels, and the test data set uses LFW to measure the effect of the model.

7. The face recognition based video review and editing system design method of claim 6, wherein, The steps of the portrait detection are as follows: S31, first build a portrait detection model for calling; S32, frame processing is performed on the video; S33, the extracted pictures are subjected to face detection to determine whether there are face pictures meeting the requirements, if yes, step S34 is executed, and if no, step S32 is executed; S34, when there are face pictures meeting the requirements in the detection result, the output face coordinates are saved; S35, the normalized coordinate values are restored, and the picture proportion is calculated through the restored coordinates; S36, the face target detected is cropped, and the video timestamp and picture proportion number are labeled and saved for subsequent face recognition.

8. The face recognition based video review and editing system design method of claim 7, wherein, The steps of the face recognition include: S41, first call the trained face recognition model; S42, determine whether the reported character picture labeled is uploaded, if yes, step S43 is executed, and if no, step S44 is executed; S43, the loaded model is used to extract face image features, and the feature mean is stored in the Milvus vector database; S44, the face image output by the portrait detection model is used as the input of the face recognition model for face recognition, and the face image is extracted into a 512-dimensional feature vector through Arcface feature calculation; S45, the feature vector obtained in S44 is measured with the reference face feature vector in the Milvus vector database, the cosine distance is used to measure the distance between the two faces, and the training process is kept corresponding to each other, when the similarity exceeds a certain threshold, it is determined that the two faces are the same person, if the similarity does not exceed the threshold, the face is not in the Milvus vector database, and finally the recognition result is returned to the front end processing.

Images