Method, device and computer-readable storage medium for determining facial features
By segmenting the video and performing image alignment, rearrangement and feature extraction, the problem of long-term facial feature extraction time is solved, and fast and efficient facial feature extraction and monitoring and tracking are achieved.
Patent Information
- Application Number
- CN202110403314.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-04-15
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-04-15
AI Technical Summary
In the prior art, the extraction time of face features is long, resulting in low monitoring and tracking efficiency.
By segmenting the target video into multiple segmented videos, and performing image alignment, rearrangement and feature extraction for each segmented video based on the face detection model and feature extraction model, rapid batch processing of face features is achieved.
The extraction time of face features is shortened, the efficiency of monitoring and tracking is improved, and the necessity of extracting face features frame by frame from the entire video.
Smart Images

Figure CN113158867B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a method and device for determining facial features and a computer-readable storage medium. Background Art
[0002] In some places that require a safe environment, it is necessary to monitor and track the people entering and leaving, such as banks, shopping malls, and residential areas. The monitoring and tracking of the crowd is achieved by extracting facial features from the collected images.
[0003] In the prior art, facial features are extracted through collected videos, but the videos include multiple images, which makes the extraction of facial features take a long time. Summary of the invention
[0004] The main purpose of the present invention is to provide a method, device and computer-readable storage medium for determining facial features, aiming to solve the problem of long time required for extracting facial features.
[0005] To achieve the above object, the present invention provides a method for determining facial features, the method comprising the following steps:
[0006] Segmenting the target video to obtain a plurality of segmented videos, and determining a first processing parameter corresponding to a first image in the segmented videos according to a face detection model;
[0007] Aligning and rearranging the first images of the segmented video according to the first processing parameter to obtain second images, and using the face detection model to detect the second images to obtain third images and first screening parameters corresponding to the third images;
[0008] Determine a target image according to the third image corresponding to the first screening parameter that meets the first preset condition, and determine a second processing parameter of the target image according to the feature extraction model, so as to align and rearrange each of the target images according to the second processing parameter to obtain each fourth image;
[0009] The feature extraction model is used to extract features from each of the fourth images to obtain facial features, and the facial features are associated with the segmented videos and saved.
[0010] In one embodiment, the step of determining the target image according to the third image corresponding to the first screening parameter that meets the first preset condition comprises:
[0011] Determine the third image corresponding to the first screening parameter that meets the first preset condition as the fifth image, and determine the third processing parameter of the fifth image according to the feature prediction model;
[0012] Aligning and rearranging the fifth images according to the third processing parameter to obtain sixth images;
[0013] Using the feature prediction model to perform feature prediction on each of the sixth images to obtain a seventh image and a second screening parameter of the seventh image;
[0014] A target image is determined according to a seventh image corresponding to the second screening parameter that meets a second preset condition.
[0015] In one embodiment, the step of determining the target image according to the seventh image corresponding to the second screening parameter that meets the second preset condition comprises:
[0016] Determine the seventh image corresponding to the second screening parameter that meets the second preset condition as the eighth image, and determine the fourth processing parameter of the eighth image according to the feature optimization model;
[0017] Aligning and rearranging the eighth images according to the fourth processing parameter to obtain ninth images;
[0018] Using the feature optimization model to perform feature optimization on each of the ninth images to obtain a tenth image and a third screening parameter of the tenth image;
[0019] A tenth image corresponding to the third screening parameter that meets the third preset condition is determined as a target image.
[0020] In one embodiment, the processing parameters are determined according to the number of image channels, the number of image rearrangements, and the image scaling ratio, and the processing parameters include a first processing parameter, the second processing parameter, the third processing parameter, and the fourth processing parameter.
[0021] In one embodiment, the step of segmenting the target video to obtain a plurality of segmented videos includes:
[0022] Determine the number of segments of the target video, and segment the target video according to the number of segments to obtain segmented videos to be determined;
[0023] Decoding each of the to-be-determined segmented videos to obtain a first time point corresponding to the completion of decoding of the to-be-determined segmented videos;
[0024] Obtaining a second time point corresponding to segmentation of each of the to-be-determined segmented videos, and associating and saving the to-be-determined segmented videos, the first time point corresponding to the to-be-determined segmented videos, and the second time period to obtain segmented videos;
[0025] The step of associating and saving the facial features with the segmented videos comprises:
[0026] The facial feature is associated with the first time point and the second time point of the segmented video corresponding to the facial feature and saved to complete the associated saving of the facial feature and the segmented video.
[0027] In one embodiment, each step in the facial feature determination process is executed by a module corresponding to the step, and the module executes one or more identical steps in parallel.
[0028] In one embodiment, the number of identical steps executed by the module is determined based on the ratio between the target duration and the duration required for the module to execute the steps, and the target duration is determined based on the least common multiple of the durations required for each module to execute the steps.
[0029] In one embodiment, the facial feature determination device determines the facial features of each of the segmented videos through multiple threads, and the number of the threads is less than the number of the segmented videos; when the thread completes the facial feature determination of the current segmented video, the facial feature determination device assigns the target segmented video to the thread, so as to determine the facial features of the target segmented video through the thread, wherein the target segmented video is a segmented video for which facial feature determination has not been performed, or the target video is a video segment split from a segmented video for which facial feature determination is being performed, and the images in the split video segments are not processed.
[0030] To achieve the above-mentioned purpose, the present invention also provides a device for determining facial features, which includes a memory, a processor, and a determination program stored in the memory and executable on the processor, and when the determination program is executed by the processor, the method for determining facial features as described above is implemented.
[0031] To achieve the above-mentioned purpose, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a determination program, and when the determination program is executed by a processor, the method for determining facial features as described above is implemented.
[0032] The present invention provides a method, device and computer-readable storage medium for determining facial features. The device for determining facial features segments a target video to obtain multiple segmented videos, then determines a first processing parameter corresponding to a first image in the segmented video based on a face detection model, and aligns and rearranges each first image of the segmented video according to the first processing parameter to obtain a second image, then uses a face detection model to detect each second image to obtain each third image and screening parameters of the third image, then determines a target image according to the third image corresponding to the screening parameters that meet preset conditions, and determines the second processing parameter of the target image based on a feature extraction model, and aligns and rearranges each target image to obtain a fourth image, finally uses a feature extraction model to extract features from each fourth image to obtain facial features, and then associates the facial features with the segmented videos for storage. The present invention first divides a longer video into multiple segmented videos, so that the device can extract facial features from multiple segmented videos at the same time, without extracting facial features from the entire video, thereby shortening the facial feature extraction time; further, when extracting facial features from segmented videos, the images of the segmented videos are aligned and rearranged, so that the device can quickly batch process the images to facilitate the model to extract facial features in the images, thereby shortening the facial feature extraction time. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 A schematic diagram of the hardware structure of a device for determining facial features according to an embodiment of the present invention;
[0034] Figure 2 A schematic diagram of a flow chart of an embodiment of a method for determining facial features of the present invention;
[0035] Figure 3 A schematic diagram of a flow chart of another embodiment of a method for determining facial features of the present invention;
[0036] Figure 4 The figure is a flow chart of another embodiment of the method for determining facial features of the present invention.
[0037] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0038] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0039] The main solution of the embodiment of the present invention is: segmenting the target video to obtain multiple segmented videos, and determining the first processing parameter corresponding to the first image in the segmented video according to the face detection model; aligning and rearranging the first images of the segmented video according to the first processing parameter to obtain the second image, and using the face detection model to detect the second images to obtain the third images and the first screening parameters corresponding to the third images; determining the target image according to the third image corresponding to the first screening parameter that meets the first preset condition, and determining the second processing parameter of the target image according to the feature extraction model, so as to align and rearrange the target images according to the second processing parameter to obtain the fourth images; using the feature extraction model to extract features of each fourth image to obtain facial features, and the facial features are associated with the segmented video and saved.
[0040] The present invention first divides a longer video into multiple segmented videos, so that the device can extract facial features from multiple segmented videos at the same time, without extracting facial features from the entire video, thereby shortening the facial feature extraction time; further, when extracting facial features from segmented videos, the images of the segmented videos are aligned and rearranged, so that the device can quickly batch process the images to facilitate the model to extract facial features in the images, thereby shortening the facial feature extraction time.
[0041] As an implementation scheme, the device for determining facial features can be as follows: Figure 1 shown.
[0042] The embodiment of the present invention relates to a device for determining facial features, which includes: a processor 1001, such as a CPU, a memory 1002, and a communication bus 1003. The communication bus 1003 is configured to achieve connection and communication between these components.
[0043] The memory 1002 may be a high-speed RAM memory (Random Access Memory) or a stable memory (non-volatile memory), such as a disk memory. Figure 1 As shown, the memory 1002 as a computer storage medium may include a determination program; and the processor 1001 may be configured to call the determination program in the memory 1002 and perform the following operations:
[0044] Segmenting the target video to obtain a plurality of segmented videos, and determining a first processing parameter corresponding to a first image in the segmented videos according to a face detection model;
[0045] Aligning and rearranging the first images of the segmented video according to the first processing parameter to obtain second images, and using the face detection model to detect the second images to obtain third images and first screening parameters corresponding to the third images;
[0046] Determine a target image according to the third image corresponding to the first screening parameter that meets the first preset condition, and determine a second processing parameter of the target image according to the feature extraction model, so as to align and rearrange each of the target images according to the second processing parameter to obtain each fourth image;
[0047] The feature extraction model is used to extract features from each of the fourth images to obtain facial features, and the facial features are associated with the segmented videos and saved.
[0048] In one embodiment, the processor 1001 may be configured to call a determination program stored in the memory 1002 and perform the following operations:
[0049] Determine the third image corresponding to the first screening parameter that meets the first preset condition as the fifth image, and determine the third processing parameter of the fifth image according to the feature prediction model;
[0050] Aligning and rearranging the fifth images according to the third processing parameter to obtain sixth images;
[0051] Using the feature prediction model to perform feature prediction on each of the sixth images to obtain a seventh image and a second screening parameter of the seventh image;
[0052] A target image is determined according to a seventh image corresponding to the second screening parameter that meets a second preset condition.
[0053] In one embodiment, the processor 1001 may be configured to call a determination program stored in the memory 1002 and perform the following operations:
[0054] Determine the seventh image corresponding to the second screening parameter that meets the second preset condition as the eighth image, and determine the fourth processing parameter of the eighth image according to the feature optimization model;
[0055] Aligning and rearranging the eighth images according to the fourth processing parameter to obtain ninth images;
[0056] Using the feature optimization model to perform feature optimization on each of the ninth images to obtain a tenth image and a third screening parameter of the tenth image;
[0057] A tenth image corresponding to the third screening parameter that meets the third preset condition is determined as a target image.
[0058] In one embodiment, the processing parameters are determined according to the number of image channels, the number of image rearrangements, and the image scaling ratio, and the processing parameters include a first processing parameter, the second processing parameter, the third processing parameter, and the fourth processing parameter.
[0059] In one embodiment, the processor 1001 may be configured to call a determination program stored in the memory 1002 and perform the following operations:
[0060] Determine the number of segments of the target video, and segment the target video according to the number of segments to obtain segmented videos to be determined;
[0061] Decoding each of the to-be-determined segmented videos to obtain a first time point corresponding to the completion of decoding of the to-be-determined segmented videos;
[0062] Obtaining a second time point corresponding to segmentation of each of the to-be-determined segmented videos, and associating and saving the to-be-determined segmented videos, the first time point corresponding to the to-be-determined segmented videos, and the second time period to obtain segmented videos;
[0063] The step of associating and saving the facial features with the segmented videos comprises:
[0064] The facial feature is associated with the first time point and the second time point of the segmented video corresponding to the facial feature and saved to complete the associated saving of the facial feature and the segmented video.
[0065] In one embodiment, the processor 1001 may be configured to call a determination program stored in the memory 1002 and perform the following operations:
[0066] Each step in the facial feature determination process is executed by a module corresponding to the step, and the module executes one or more identical steps in parallel.
[0067] In one embodiment, the processor 1001 may be configured to call a determination program stored in the memory 1002 and perform the following operations:
[0068] The number of identical steps executed by the module is determined based on the ratio between the target duration and the duration required for the module to execute the steps, and the target duration is determined based on the least common multiple of the durations required for each module to execute the steps.
[0069] In one embodiment, the processor 1001 may be configured to call a determination program stored in the memory 1002 and perform the following operations:
[0070] The facial feature determination device determines the facial features of each of the segmented videos through multiple threads, and the number of the threads is less than the number of the segmented videos; when the thread completes the facial feature determination of the current segmented video, the facial feature determination device allocates the target segmented video to the thread, so as to determine the facial features of the target segmented video through the thread, wherein the target segmented video is a segmented video for which facial feature determination has not been performed, or the target video is a video segment split from a segmented video for which facial feature determination is being performed, and the images in the split video segments are not processed.
[0071] According to the above scheme, the facial feature determination device of this embodiment segments the target video to obtain multiple segmented videos, and then determines the first processing parameter corresponding to the first image in the segmented video based on the face detection model, and aligns and rearranges the first images of the segmented video according to the first processing parameter to obtain the second image, and then uses the face detection model to detect the second images to obtain the third images and the screening parameters of the third images, and then determines the target image according to the third image corresponding to the screening parameters that meet the preset conditions, and determines the second processing parameter of the target image based on the feature extraction model, and aligns and rearranges the target images to obtain the fourth image, and finally uses the feature extraction model to extract features from the fourth images to obtain facial features, and then associates the facial features with the segmented videos and stores them. The present invention first divides a longer video into multiple segmented videos, so that the device can extract facial features from multiple segmented videos at the same time, without extracting facial features from the entire video, thereby shortening the facial feature extraction time; further, when extracting facial features from segmented videos, the images of the segmented videos are aligned and rearranged, so that the device can quickly batch process the images to facilitate the model to extract facial features in the images, thereby shortening the facial feature extraction time.
[0072] Based on the above hardware architecture, an embodiment of the method for determining facial features of the present invention is proposed.
[0073] Reference Figure 2 , Figure 2 In one embodiment of the present invention, the method for determining facial features includes the following steps:
[0074] Step S10, obtain the target video, and segment the target video to obtain multiple segmented videos, and determine the first processing parameter corresponding to the first image in the segmented video according to the face detection model; in this embodiment, the execution subject is a device for determining facial features. For the sake of ease of description, the device is used below to refer to the device for determining facial features.
[0075] The device first obtains the video that needs to be used for facial feature extraction, which is the target video. In order to quickly analyze the video to obtain facial features, a sufficiently fast data input speed is required, that is, high-speed decoding of the data in advance is required. Therefore, the target video is segmented, and then the segmented video is decoded, so that the video can be read in segments, and the overall data input speed is doubled. The device can segment the target video according to the fixed duration of the segmented video. For example, the fixed duration of the segmented video is 2 minutes, then the 9-minute target video is divided into 5 segmented videos. In addition, the device can divide the video into a fixed number of segmented videos. For example, if the fixed number is 5, the target video is divided into 5 segmented videos.
[0076] The device is provided with a face detection model for identifying the facial features in the image and marking the facial features in the image. After obtaining the segmented video, the face model sequentially identifies the facial features of the first image in each segmented video. Before identification, the device needs to align and rearrange each first image. Specifically, the device obtains the first processing parameter of the first image, and the first processing parameter is used to align and rearrange each first image. The first processing parameter includes the number of image channels of the first image, the ratio of the first image to be scaled, and the number of aligned and rearranged first images. The scaling ratio of the first image and the number of aligned and rearranged first images are determined according to the face detection model, that is, the face detection model has size requirements for image recognition and the number of aligned target images. The first processing parameter can be represented as (W, H, C, N), where (W, H,) represents the image size after the first image is scaled, C is the number of aligned first images, and N is the number of images processed simultaneously by the face recognition model; for example, the first processing parameter (W, H, C, N) = (416, 416, 3, 2).
[0077] Step S20, aligning and rearranging the first images of the segmented video according to the first processing parameter to obtain second images, and using the face detection model to detect the second images to obtain third images and first screening parameters corresponding to the third images;
[0078] After determining the first processing parameter corresponding to the first image, the device aligns and rearranges the first images in the segmented video according to the first processing parameter to obtain the second image. The alignment and rearrangement of the first images can facilitate batch processing by the GPU (Graphics Processing Unit) in the device.
[0079] After obtaining the second image, the device uses a face detection model to perform feature detection on each second image to obtain each third image, and the first screening parameter corresponding to the third image. The third image marks the location of the facial features, for example, the face is framed in the form of a square. Not every image in the segmented video has a face, and not every image can clearly extract facial features. Therefore, it is necessary to screen out images with faces, and the first screening parameter can be used to screen out third images containing faces. The first screening parameter can be the confidence of the third image.
[0080] Step S30, determining a target image according to the third image corresponding to the first screening parameter that meets the first preset condition, and determining a second processing parameter of the target image according to the feature extraction model, so as to align and rearrange each of the target images according to the second processing parameter to obtain each fourth image;
[0081] The device is provided with a first preset condition. After obtaining each third image, the device determines whether the first screening parameter of the third image satisfies the first preset condition. The first preset condition may be that the confidence is greater than a first preset threshold. Therefore, the device selects the third image corresponding to the confidence greater than the first preset threshold as the target image, and the confidence greater than the first preset threshold is the first screening parameter that satisfies the first preset condition.
[0082] The device is also provided with a feature extraction model, which is used to extract the facial features in the target image, that is, to extract the facial features within the frame of the target image.
[0083] Before facial feature extraction, each target image needs to be aligned and rearranged, that is, the second processing parameters of the target image are obtained according to the feature extraction model. The second processing parameters include the number of image channels of the target image, the ratio at which the target image needs to be scaled, and the number of aligned and rearranged target images. The scaling ratio of the target image and the number of aligned and rearranged target images are determined according to the feature extraction model, that is, the feature extraction model has size requirements for image recognition and requirements for the number of aligned target images. The second processing parameters can also be represented as (W, H, C, N), for example, the second processing parameters (W, H, C, N) = (112, 112, 3, 4).
[0084] After determining the second processing parameter corresponding to the target image, the device aligns and rearranges each target image according to the second processing parameter to obtain a fourth image. The alignment and rearrangement of the fourth image can facilitate the GPU (Graphics Processing Unit) in the device to perform batch processing. It should be noted that the blank areas of the scaled image are filled with 0, and 0 is the pixel value.
[0085] Step S40: extract features from each of the fourth images using the feature extraction model to obtain facial features, and associate the facial features with the segmented videos for storage.
[0086] After obtaining the fourth image, the device uses a feature extraction model to extract features from each fourth image, thereby obtaining facial features, which can exist in the form of feature vectors. The device can process the facial features of each segmented video in turn, or extract the facial features of each segmented video at the same time, and then associate and save the facial features with the segmented videos corresponding to the facial features. In addition, if the device needs to monitor and track immediately, the human features are tracked based on the acquired facial features. It should be noted that the target video can be an offline video or a real-time video. When the device extracts facial features from real-time video, it will capture a smaller number of images for facial feature extraction, for example, it will capture the currently received image and the last received image for facial feature extraction, or even only capture the currently received image for facial feature extraction. Therefore, the efficiency of the device in extracting facial features from offline videos is higher than the efficiency of extracting facial features from real-time videos. Therefore, when the device receives real-time video and there is no time limit for monitoring and tracking, it can first store the real-time received image to convert it into an offline video, and then extract features from the offline video to improve the efficiency of facial feature extraction.
[0087] In the technical solution provided in this embodiment, the facial feature determination device segments the target video to obtain multiple segmented videos, and then determines the first processing parameter corresponding to the first image in the segmented video based on the face detection model, and aligns and rearranges the first images of the segmented video according to the first processing parameter to obtain the second image, and then uses the face detection model to detect the second images to obtain the third images and the screening parameters of the third images, and then determines the target image according to the third image corresponding to the screening parameters that meet the preset conditions, and determines the second processing parameter of the target image based on the feature extraction model, and aligns and rearranges the target images to obtain the fourth image, and finally uses the feature extraction model to extract features from the fourth images to obtain facial features, and then associates the facial features with the segmented videos and stores them. The present invention first divides a longer video into multiple segmented videos, so that the device can extract facial features from multiple segmented videos at the same time, without extracting facial features from the entire video, thereby shortening the facial feature extraction time; further, when extracting facial features from segmented videos, the images of the segmented videos are aligned and rearranged, so that the device can quickly batch process the images to facilitate the model to extract facial features in the images, thereby shortening the facial feature extraction time.
[0088] Reference Figure 3 , Figure 3 In another embodiment of the method for determining facial features of the present invention, based on the above embodiment, in step S30, "determining the target image according to the third image corresponding to the first screening parameter that meets the first preset condition" includes:
[0089] Step S31, determining the third image corresponding to the first screening parameter that meets the first preset condition as the fifth image, and determining the third processing parameter of the fifth image according to the feature prediction model;
[0090] Step S32, aligning and rearranging the fifth images according to the third processing parameter to obtain sixth images;
[0091] Step S33, using the feature prediction model to perform feature prediction on each of the sixth images to obtain a seventh image and a second screening parameter of the seventh image;
[0092] Step S34, determining a target image according to the seventh image corresponding to the second screening parameter that meets the second preset condition.
[0093] The range of the third image obtained by the face detection model after detecting the second image is relatively large, so that when the feature extraction model extracts the face features of the third image as the target image, it takes a long time, and the extracted face features are not accurate. In this regard, a feature prediction model is set in the device, that is, the box in the third image is further reduced through the feature preset model, the extraction time of the face features is reduced, and the extracted face features are more accurate.
[0094] In this regard, the device first determines the third image corresponding to the first screening parameter that meets the first preset condition as the fifth image, and then determines the third processing parameter of the fifth image according to the feature prediction model. The third processing parameter includes the number of image channels of the fifth image, the ratio at which the fifth image needs to be scaled, and the number of aligned and rearranged fifth images. The scaling ratio of the fifth image and the number of aligned and rearranged fifth images are determined according to the feature prediction model, that is, the feature prediction model has size requirements and number requirements for aligned fifth images for image recognition. The third processing parameter can also be represented as (W, H, C, N), for example, the third processing parameter (W, H, C, N) = (48, 48, 3, any number).
[0095] The device aligns and rearranges each fifth image based on the third processing parameter to obtain a sixth image, and then uses the feature prediction model to predict each sixth image to obtain a seventh image and a second screening parameter of the seventh image. The second screening parameter can be the confidence of the seventh image. The seventh image is still an image marked with a box, and the size of the box in the seventh box is smaller than the size of the box in the third image.
[0096] The device is provided with a second preset condition. After obtaining each seventh image, the device determines whether the second screening parameter of the seventh image satisfies the second preset condition. The second preset condition may be that the confidence is greater than the second preset threshold. Therefore, the device selects the seventh image corresponding to the confidence greater than the second preset threshold as the target image, and the confidence greater than the second preset threshold is the second screening parameter that satisfies the second preset condition.
[0097] In the technical solution provided in this embodiment, the device determines the third image corresponding to the first screening parameter that meets the first preset condition as the fifth image, and determines the third processing parameter of the fifth image according to the feature prediction model, and then aligns and rearranges the fifth images according to the third processing parameter to obtain the sixth image, and then uses the feature prediction model to predict the sixth image to obtain the seventh image and the second screening parameter of the seventh image, and finally uses the seventh image corresponding to the second screening parameter that meets the second preset condition as the target image, so that the target image can be quickly extracted with facial features by the feature extraction model, and the extracted facial features are relatively accurate.
[0098] Reference Figure 4 , Figure 4 Another embodiment of the method for extracting facial features of the present invention is based on Figure 3 In the embodiment shown, step S34 includes:
[0099] Step S341, determining the seventh image corresponding to the second screening parameter that meets the second preset condition as the eighth image, and determining the fourth processing parameter of the eighth image according to the feature optimization model;
[0100] Step S342, aligning and rearranging the eighth images according to the fourth processing parameter to obtain ninth images;
[0101] Step S343, using the feature optimization model to perform feature optimization on each of the ninth images to obtain a tenth image and a third screening parameter of the tenth image;
[0102] Step S344: determine the tenth image corresponding to the third screening parameter that meets the third preset condition as the target image.
[0103] The box marked in the seventh image is composed of feature points. In order to further improve the accuracy of facial feature extraction, a feature optimization model can be set in the device, that is, the feature points of the box in the seventh image are optimized by the feature optimization model. Optimizing feature points means removing feature points outside the straight line (the straight line is the straight line fitted by the device to the feature points in the seventh image).
[0104] In this regard, the device first determines the seventh image corresponding to the second screening parameter that meets the second preset condition as the eighth image, and then determines the fourth processing parameter of the eighth image according to the feature prediction model. The fourth processing parameter includes the number of image channels of the eighth image, the ratio at which the eighth image needs to be scaled, and the number of aligned and rearranged eighth images. The scaling ratio of the eighth image and the number of aligned and rearranged eighth images are determined according to the feature optimization model, that is, the feature optimization model has size requirements and number requirements for aligned eighth images for image recognition. The fourth processing parameter can also be represented as (W, H, C, N), for example, the fourth processing parameter (W, H, C, N) = (24, 24, 15, any number).
[0105] The device aligns and rearranges each of the eighth images based on the fourth processing parameter to obtain a ninth image, and then uses the feature optimization model to optimize the features of each of the seventh images to obtain a tenth image and a third screening parameter of the tenth image. The third screening parameter can be the confidence of the tenth image. The tenth image is still an image marked with a box, and the size of the box in the tenth box is smaller than the size of the box in the seventh image.
[0106] The device is provided with a third preset condition. After obtaining each tenth image, the device determines whether the third screening parameter of the tenth image satisfies the third preset condition. The third preset condition may be that the confidence is greater than the third preset threshold. Therefore, the device selects the tenth image corresponding to the confidence greater than the third preset threshold as the target image, and the confidence greater than the third preset threshold is the third screening parameter that satisfies the third preset condition.
[0107] It can be seen from the above embodiments that the processing parameters are determined according to the number of image channels, the number of image rearrangements and the image scaling ratio, and the processing parameters include a first processing parameter, a second processing parameter, a third processing parameter and a fourth processing parameter.
[0108] In the technical solution provided in this embodiment, the device determines the seventh image corresponding to the second screening parameter that meets the second preset condition as the eighth image, and determines the fourth processing parameter of the eighth image according to the feature optimization model, and then aligns and rearranges the eighth images according to the fourth processing parameter to obtain the ninth image, and then uses the feature optimization model to predict the ninth image to obtain the tenth image and the third screening parameter of the tenth image, and finally uses the tenth image corresponding to the third screening parameter that meets the third preset condition as the target image, so that the extracted facial features are more accurate.
[0109] In one embodiment, the step of segmenting the target video to obtain a plurality of segmented videos includes:
[0110] A. determining the number of segments of the target video, and segmenting the target video according to the number of segments to obtain segmented videos to be determined;
[0111] B. Decoding each of the to-be-determined segmented videos to obtain a first time point corresponding to the completion of decoding of the to-be-determined segmented videos;
[0112] C. Obtaining the second time point corresponding to the segmentation of each of the to-be-determined segmented videos, and associating and saving the to-be-determined segmented videos, the first time point corresponding to the to-be-determined segmented videos, and the second time period to obtain the segmented videos;
[0113] In this embodiment, the device is provided with a number of segments n. After obtaining the target video, the device segments the target video based on the number of segments to obtain the segmented video to be determined. The segmented video to be determined is an unmarked segmented video. The device obtains the starting time point for segmenting each segmented video to be determined, and the starting time point is the second time point. For example, if the duration of the target video is T, the second time point t1 of the first segmented video to be determined is 0, the second time point t2 of the second segmented video to be determined is t / n, ... the second time point tn of the nth segmented video to be determined is t*((n-1) / n).
[0114] After obtaining each segmented video to be determined, the device decodes each segmented video to be determined at the same time, and records the time when the decoding of the segmented video to be determined is completed as the first time point, and the device jumps to read the segmented video to be determined and decodes it according to the second time points t1, t2, ... tn respectively until the decoding is successful. At this time, the decoding completion time points of each segmented video to be determined are s1, s2, ... sn. Therefore, the first time point and the second time point corresponding to the first segmented video to be determined are (t1, s1), the first time point and the second time point corresponding to the second segmented video to be determined are (t2, s2), ... The first time point and the second time point corresponding to the nth segmented video to be determined are (tn, sn).
[0115] The device associates and saves the segmented video to be determined, the first time point and the second time point corresponding to the segmented video to be determined, thereby obtaining the segmented video. The marked first time point and the second time point can be understood as the mark of the segmented video. Therefore, after obtaining the facial features, the facial features are associated with the first time point and the second time point of the segmented video corresponding to the facial features, which can be regarded as the associated storage of the facial features and the segmented video, that is, the step of associating and saving the facial features with the segmented video includes:
[0116] The facial feature is associated with the first time point and the second time point of the segmented video corresponding to the facial feature and saved to complete the associated saving of the facial feature and the segmented video.
[0117] In this embodiment, the segmented videos are marked based on the segmentation time points and decoding time points of the segmented videos, so that the determined facial features are associated and saved with the segmented videos.
[0118] In one embodiment, each step in the facial feature determination process is executed by a module corresponding to the step, and the module executes one or more identical steps in parallel. Specifically, there are fourteen steps in the facial feature extraction process. They are:
[0119] 1. Video segmentation;
[0120] 2. Alignment and rearrangement of the initial image;
[0121] 3. The face detection model detects the aligned and rearranged image to obtain image A;
[0122] 4. Perform result screening on image A (result screening is screening parameters that meet preset conditions) to obtain image B;
[0123] 5. Align and rearrange the B image;
[0124] 6. Use the feature prediction model to align the rearranged B image and predict the C image;
[0125] 7. Filter the results of image C to obtain image D;
[0126] 8. Align and rearrange the D image;
[0127] 9. Use the feature optimization model to optimize the features of the aligned and rearranged D image to obtain the E image;
[0128] 10. Filter the results of image E to obtain image F;
[0129] 11. Align and rearrange the F image;
[0130] 12. Use the feature extraction model to extract features from the aligned and rearranged F image to obtain facial features;
[0131] 13. Use facial features to track features and get results;
[0132] 14. Analyze the results and save them.
[0133] The above fourteen steps are performed by different modules in the device, and since there are multiple images in the segmented video, the modules can perform one or more identical steps in parallel. For example, the modules can align and rearrange three groups of F images at the same time.
[0134] Further, the number of the same steps executed by the module is determined according to the ratio between the target duration and the duration required for the module to execute the steps, and the target duration is determined according to the least common multiple corresponding to the duration required for each module to execute the steps. For example, the device executes four steps 1, 2, 3, and 4, and the duration for step 1 to complete execution is 20s, the duration for step 2 to complete execution is 30s, the duration for step 3 to complete execution is 10s, and the duration for step 4 to complete execution is 40s. Then the least common multiple of 20s, 30s, 10s, and 40s is 120s. Then the number of steps 1 that can be executed simultaneously by module 1 = 120s / 20s = 6, the number of steps 1 that can be executed simultaneously by module 2 = 120s / 30s = 4, the number of steps 3 that can be executed simultaneously by module 3 = 120s / 10s = 12, and the number of steps 4 that can be executed simultaneously by module 4 = 120s / 40s = 3. Similarly, the number of steps executed by other modules can be calculated according to the above example.
[0135] By adopting this method, the CPU and GPU in the device can balance the load and optimize the extraction speed of facial features.
[0136] In one embodiment, since the speed of extracting human features from a segmented video is related to the content of the assigned video, the time for feature extraction estimated based on the file length of the segmented video may not be optimal. This embodiment proposes a method for dynamically splitting segmented videos. During the facial feature extraction process of a segmented video, if a certain analysis instance completes the extraction of facial features from the segmented video earlier than other analysis instances, the device will call the dynamic splitting method to split a new segmented video and assign it to the analysis instance. The idle instance will start the analysis again until the entire video analysis is completed.
[0137] Specifically, the device determines the facial features of each segmented video through multiple threads (one thread can be regarded as an analysis instance), and the number of threads is less than or equal to the number of segmented videos, so that there are redundant segmented videos that are not assigned to threads for facial feature extraction. When the thread completes the facial feature determination of the current segmented video, the device assigns the target segmented video to the thread that completes the human body feature extraction, so as to determine the facial features of the target segmented video through the thread, and the target segmented video is a segmented video that has not undergone facial feature determination; or, the target video is a video segment split from a segmented video that is undergoing facial feature determination, and the image in the split video segment is not processed.
[0138] The following description is given by taking the target segmented video as the video segments split from the segmented video of the facial features being detected.
[0139] The device first finds an idle analysis instance; the device then checks the progress of facial feature extraction of the segmented video, and then finds the longest unprocessed video segment from the segmented video, inserts a video reading start point in the middle of the unprocessed video segment, and modifies the end time point of the original segmented video. The time point from the video reading start point to the end time point of the video segment is the target segmented video. The disassembled video segment is assigned to the analysis instance for human feature extraction. The thread facial features can refer to steps S10-S40, which will not be repeated here. For example, the segmented video is 10 minutes and 10 seconds-20 minutes and 10 seconds, and the segmented video has been processed to 15 minutes and 30 seconds, then the video reading point is found in 15 minutes and 31 seconds-20 minutes and 10 seconds. If 16 minutes and 10 seconds is selected, the end time point of the original segmented video 20 minutes and 10 seconds is changed to 16 minutes and 9 seconds, and the target video segment is 16 minutes and 10 seconds-20 minutes and 10 seconds.
[0140] The present invention also provides a device for determining facial features, which includes a memory, a processor, and a determination program stored in the memory and executable on the processor. When the determination program is executed by the processor, the method for determining facial features as described in the above embodiment is implemented.
[0141] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a determination program, and when the determination program is executed by a processor, the method for determining facial features as described in the above embodiment is implemented.
[0142] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0143] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0144] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is an optional implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, including a number of instructions for a terminal device (which can be a mobile phone, computer, server, television, or network device, etc.) to execute the methods described in each embodiment of the present invention.
[0145] The above are only optional embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A method for determining facial features, characterized in that: The method for determining facial features comprises the following steps: Segmenting the target video to obtain a plurality of segmented videos, and determining a first processing parameter corresponding to a first image in the segmented videos according to a face detection model; Aligning and rearranging the first images of the segmented video according to the first processing parameter to obtain second images, and using the face detection model to detect the second images to obtain third images and first screening parameters corresponding to the third images, wherein the first screening parameters corresponding to the third images are confidence levels of the third images; Determine a target image according to the third image corresponding to the first screening parameter that meets the first preset condition, and determine a second processing parameter of the target image according to the feature extraction model, so as to align and rearrange each of the target images according to the second processing parameter to obtain each fourth image; The feature extraction model is used to extract features from each of the fourth images to obtain facial features, and the facial features are associated with the segmented videos and saved.
2. The method for determining facial features according to claim 1, wherein: The step of determining the target image according to the third image corresponding to the first screening parameter that meets the first preset condition comprises: Determine the third image corresponding to the first screening parameter that meets the first preset condition as the fifth image, and determine the third processing parameter of the fifth image according to the feature prediction model; Aligning and rearranging the fifth images according to the third processing parameter to obtain sixth images; Using the feature prediction model to perform feature prediction on each of the sixth images to obtain a seventh image and a second screening parameter of the seventh image, wherein the second screening parameter of the seventh image is a confidence level of the seventh image; A target image is determined according to a seventh image corresponding to the second screening parameter that meets a second preset condition.
3. The method for determining facial features according to claim 2, wherein: The step of determining the target image according to the seventh image corresponding to the second screening parameter that meets the second preset condition comprises: Determine the seventh image corresponding to the second screening parameter that meets the second preset condition as the eighth image, and determine the fourth processing parameter of the eighth image according to the feature optimization model; Aligning and rearranging the eighth images according to the fourth processing parameter to obtain ninth images; Using the feature optimization model to perform feature optimization on each of the ninth images to obtain a tenth image and a third screening parameter of the tenth image, wherein the third screening parameter of the tenth image is the confidence of the tenth image; A tenth image corresponding to the third screening parameter that meets the third preset condition is determined as a target image.
4. The method for determining facial features according to claim 3, wherein: The processing parameters are determined according to the number of image channels, the number of image rearrangements, and the image scaling ratio, and the processing parameters include a first processing parameter, the second processing parameter, the third processing parameter, and the fourth processing parameter.
5. The method for determining facial features according to any one of claims 1 to 4, characterized in that: The step of segmenting the target video to obtain a plurality of segmented videos comprises: Determine the number of segments of the target video, and segment the target video according to the number of segments to obtain segmented videos to be determined; Decoding each of the to-be-determined segmented videos to obtain a first time point corresponding to the completion of decoding of the to-be-determined segmented videos; Obtaining a second time point corresponding to segmentation of each of the to-be-determined segmented videos, and associating and saving the to-be-determined segmented videos, the first time point corresponding to the to-be-determined segmented videos, and the second time period to obtain segmented videos; The step of associating and saving the facial features with the segmented videos comprises: The facial feature is associated with the first time point and the second time point of the segmented video corresponding to the facial feature and saved to complete the associated saving of the facial feature and the segmented video.
6. The method for determining facial features according to any one of claims 1 to 4, characterized in that: Each step in the facial feature determination process is executed by a module corresponding to the step, and the module executes one or more identical steps in parallel.
7. The method for determining facial features according to claim 6, wherein: The number of identical steps executed by the module is determined based on the ratio between the target duration and the duration required for the module to execute the steps, and the target duration is determined based on the least common multiple of the durations required for each module to execute the steps.
8. The method for determining facial features according to any one of claims 1 to 4, characterized in that: The facial feature determination device determines the facial features of each of the segmented videos through multiple threads, and the number of the threads is less than the number of the segmented videos; when the thread completes the facial feature determination of the current segmented video, the facial feature determination device allocates the target segmented video to the thread, so as to determine the facial features of the target segmented video through the thread, wherein the target segmented video is a segmented video for which facial feature determination has not been performed, or the target video is a video segment split from a segmented video for which facial feature determination is being performed, and the images in the split video segments are not processed.
9. A device for determining facial features, characterized in that: The facial feature determination device includes a memory, a processor, and a determination program stored in the memory and executable on the processor. When the determination program is executed by the processor, the facial feature determination method according to any one of claims 1 to 8 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a determination program, and when the determination program is executed by a processor, the method for determining facial features according to any one of claims 1 to 8 is implemented.
