Video face segmentation method, device, computer equipment and storage medium

Through the image processing model of four-layer feature branching and attention mechanism, the problem of low accuracy of video face segmentation is solved, the quality of image edge information segmentation and feature fusion are improved, and more efficient video face segmentation is achieved.

CN114283475BActive Publication Date: 2024-11-12SHENZHEN WONDERSHARE SOFTWARE CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202111597923.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-24
Publication Date
2024-11-12
Estimated Expiration
2041-12-24

AI Technical Summary

Technical Problem

The existing video face segmentation method is based on the full convolutional network FCN, which cannot quickly perform real-time inference and fail to effectively consider global context information, resulting in low accuracy of video face segmentation.

Method used

The image processing model of four-layer feature branches is adopted, and through filtering processing, differentiation processing, image resolution reduction and attention mechanism, the image edge detail information and feature fusion is enhanced, and the image segmentation quality is improved.

Benefits of technology

It improves the accuracy and efficiency of video face segmentation, enhances the quality of image edge information segmentation, enhances image fusion characteristics, and improves the accuracy of video face segmentation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114283475B_ABST
    Figure CN114283475B_ABST
Patent Text Reader

Abstract

The present application relates to the field of video image processing technology, and discloses a video face segmentation method, device, computer equipment and storage medium, wherein the method pre-processes the acquired sample image, and inputs the obtained true value image into the image processing model; filters the true value image according to a preset multiple, and superimposes the obtained filtering results, and performs differentiation processing and image resolution reduction processing on the obtained first image feature and the true value image to obtain a third image feature and a fourth image feature; based on the attention mechanism, the attention matrix and feature fusion of the third image feature and the fourth image feature are calculated to obtain an enhanced image, and then a trained image processing model is obtained based on the loss value; the acquired video to be processed is input into the trained image processing model for image segmentation processing to obtain a target image. The present application is beneficial to the accuracy of video face segmentation by enhancing the edge details of the image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of video image processing, and in particular to a video face segmentation method, device, computer equipment and storage medium. Background Art

[0002] Semantic segmentation is an important core technology of many visual understanding systems, and plays an irreplaceable role in fields including medical image analysis, vehicle autonomous driving, video surveillance, and virtual reality enhancement. Earlier segmentation technology was mainly based on traditional image algorithms, and derived methods such as threshold method, histogram statistics method, pixel region growth method, k-means clustering, and Markov random field. In recent years, with the rapid development of deep learning technology, more algorithms with superior performance and better accuracy have been proposed. In the final analysis, semantic segmentation can actually be considered as a semantic classification problem for each pixel in an image, and video face segmentation is an important branch of semantic segmentation.

[0003] Existing video face segmentation methods are implemented through fully convolutional networks (FCNs), which only contain convolutional modules. For the first time, deep networks are trained in an end-to-end manner on images of variable sizes to output segmented images of the same resolution. However, although the traditional fully convolutional network (FCN) model is universal and effective, it also has certain limitations. It cannot perform real-time reasoning quickly and cannot effectively consider global context information, resulting in low accuracy of video face segmentation. There is an urgent need for a method that can improve the accuracy of video face segmentation. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a video face segmentation method, apparatus, computer equipment and storage medium to improve the accuracy of video face segmentation.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a video face segmentation method, comprising:

[0006] Acquire a sample image, preprocess the sample image to obtain a true value image, and input the true value image into an image processing model, wherein the image processing model includes four layers of feature branches;

[0007] In the first layer feature branch, the true value image is filtered according to a preset multiple to obtain a filtering result, and the filtering results are superimposed to obtain a first image feature;

[0008] In the second-layer feature branch, the first image feature and the true value image are differentiated to obtain a second image feature;

[0009] The second image feature is processed by image resolution reduction of the third layer feature branch and the fourth layer feature branch respectively, to obtain a third image feature and a fourth image feature;

[0010] Based on the attention mechanism, an attention matrix of the third image feature and the fourth image feature is calculated, and feature fusion is performed based on the attention matrix to obtain an enhanced image feature;

[0011] Calculating the loss value between the enhanced image feature and the true value image, and if the loss value reaches a preset threshold, obtaining a trained image processing model;

[0012] The video to be processed is obtained, and frame-by-frame extraction is performed on the video to be processed to obtain an initial image, and the initial image is input into the trained image processing model for image segmentation processing to obtain a target image.

[0013] Furthermore, the acquiring of sample images, preprocessing of the sample images to obtain true value images, and inputting the true value images into the image processing model includes:

[0014] Acquire the sample image, and perform normalization processing on the sample image to obtain a normalized image;

[0015] The normalized image is resized according to a preset image size to obtain a true value image, and the true value image is input into the image processing model.

[0016] Furthermore, in the first-layer feature branch, the true value image is filtered according to a preset multiple to obtain a filtering result, and the filtering results are superimposed to obtain a first image feature, including:

[0017] In the first-layer feature branch, image features are obtained by performing label preprocessing on the true value image;

[0018] Using Laplace transform, filtering the true value image according to the preset multiple to obtain a filtering result, and superimposing the filtering results to obtain edge detail pyramid features;

[0019] Calculating a loss value between the edge detail pyramid feature and the true value image to obtain a first loss value;

[0020] If the first loss value does not reach the first preset threshold, the corresponding parameters of the first-layer feature branch are updated to re-perform image processing of the first-layer feature branch. If the first loss value reaches the first preset threshold, the edge detail pyramid feature is used as the first image feature.

[0021] Furthermore, the method of using Laplace transform to filter the true value image according to the preset multiple to obtain a filtering result, and superimposing the filtering results to obtain edge detail pyramid features includes:

[0022] Acquire multiple different preset magnifications, and use Laplace transform to perform convolution processing on the true value image according to the preset magnifications respectively, so as to obtain edge detail features corresponding to the multiple preset magnifications;

[0023] By adopting an interpolation calculation method, edge detail features corresponding to a plurality of the preset multiples are superimposed and calculated according to preset weights at the same image resolution to obtain the edge detail pyramid feature.

[0024] Furthermore, in the second-layer feature branch, performing differentiation processing on the first image feature and the true value image to obtain a second image feature includes:

[0025] In the second-layer feature branch, performing image resolution reduction processing on the first image feature to obtain a resolution processed feature;

[0026] Calculating a loss value between the resolution reduction processing feature and the true value image to obtain a second loss value;

[0027] If the second loss value does not reach the second preset threshold, the corresponding parameters of the first-layer feature branch and the second-layer feature branch are updated, and the true value image is reprocessed through the first-layer feature branch and the second-layer feature branch until the second loss value reaches the second preset threshold, and the resolution processing feature is used as the second image feature.

[0028] Furthermore, the step of subjecting the second image feature to image resolution reduction processing of the third layer feature branch and the fourth layer feature branch respectively to obtain the third image feature and the fourth image feature includes:

[0029] The second image feature is processed by image resolution reduction of the third layer feature branch and the fourth layer feature branch respectively, and the third image feature and the fourth initial image feature are obtained;

[0030] Calculating a loss value between the fourth initial image feature and the true value image to obtain a third loss value;

[0031] If the third loss value does not reach the third preset threshold, the corresponding parameters of the image processing model are updated, and the true value image is re-input into the image processing model for image processing until the third loss value reaches the third preset threshold, and the fourth initial image feature is used as the fourth image feature.

[0032] Furthermore, the attention matrix of the third image feature and the fourth image feature is calculated based on the attention mechanism, and feature fusion is performed based on the attention matrix to obtain enhanced image features, including:

[0033] Based on the attention mechanism, calculating the attention matrix of the third image feature and the fourth image feature;

[0034] Performing feature multiplication on the attention matrix and the fourth image feature to obtain a feature multiplication result;

[0035] The feature multiplication result and the third image feature are convolved to obtain the enhanced image feature.

[0036] In order to solve the above technical problems, the embodiment of the present application provides a video face segmentation device, comprising:

[0037] A sample image acquisition module, used to acquire a sample image, preprocess the sample image to obtain a true value image, and input the true value image into an image processing model, wherein the image processing model includes four layers of feature branches;

[0038] A first image feature generation module is used to filter the true value image according to a preset multiple in the first layer feature branch to obtain a filtering result, and superimpose the filtering results to obtain a first image feature;

[0039] A second image feature generating module, configured to perform a differentiation process on the first image feature and the true value image in a second layer feature branch to obtain a second image feature;

[0040] A third image feature generating module, configured to subject the second image feature to image resolution reduction processing through a third-layer feature branch and a fourth-layer feature branch, respectively, to obtain a third image feature and a fourth image feature;

[0041] An enhanced image feature generation module, used to calculate the attention matrix of the third image feature and the fourth image feature based on the attention mechanism, and perform feature fusion based on the attention matrix to obtain the enhanced image feature;

[0042] An image processing model completion module is used to calculate the loss value between the enhanced image features and the true value image, and if the loss value reaches a preset threshold, a trained image processing model is obtained;

[0043] The target image generation module is used to obtain the video to be processed, and extract the video to be processed frame by frame to obtain an initial image, and input the initial image into the trained image processing model for image segmentation processing to obtain a target image.

[0044] To solve the above technical problems, a technical solution adopted by the present invention is: providing a computer device, including one or more processors; a memory for storing one or more programs, so that the one or more processors implement any one of the above-mentioned video face segmentation methods.

[0045] In order to solve the above technical problems, a technical solution adopted by the present invention is: a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, any one of the above-mentioned video face segmentation methods is implemented.

[0046] The embodiment of the present invention provides a video face segmentation method, device, computer equipment and storage medium. The method includes: obtaining a sample image, preprocessing the sample image to obtain a true value image, and inputting the true value image into an image processing model; in the first layer feature branch, filtering the true value image according to a preset multiple to obtain a filtering result, and superimposing the filtering results to obtain a first image feature; in the second layer feature branch, performing a differential processing on the first image feature and the true value image to obtain a second image feature; subjecting the second image feature to image resolution reduction processing of the third layer feature branch and the fourth layer feature branch respectively to obtain a third image feature and a fourth image feature; based on the attention mechanism, calculating the attention matrix of the third image feature and the fourth image feature, and performing feature fusion based on the attention matrix to obtain an enhanced image feature; calculating the loss value of the enhanced image feature and the true value image, and if the loss value reaches a preset threshold, obtaining a trained image processing model; obtaining a video to be processed, extracting the video to be processed frame by frame to obtain an initial image, and inputting the initial image into the trained image processing model for image segmentation processing to obtain a target image. The embodiment of the present application performs filtering processing on the true value image to enhance the image edge detail information and improve the image edge information segmentation quality. At the same time, the embodiment of the present application also introduces an attention mechanism to enhance the image fusion feature, which is beneficial to the accuracy of video face segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 This is a flowchart of a sub-process in the video face segmentation method provided in an embodiment of the present application;

[0049] Figure 2This is another implementation flow chart of a sub-process in the video face segmentation method provided in an embodiment of the present application;

[0050] Figure 3 This is another implementation flow chart of a sub-process in the video face segmentation method provided in an embodiment of the present application;

[0051] Figure 4 This is another implementation flow chart of a sub-process in the video face segmentation method provided in an embodiment of the present application;

[0052] Figure 5 This is another implementation flow chart of a sub-process in the video face segmentation method provided in an embodiment of the present application;

[0053] Figure 6 This is another implementation flow chart of a sub-process in the video face segmentation method provided in an embodiment of the present application;

[0054] Figure 7 This is another implementation flow chart of a sub-process in the video face segmentation method provided in an embodiment of the present application;

[0055] Figure 8 is a schematic diagram of a video face segmentation device provided in an embodiment of the present application;

[0056] Fig. 9 It is a schematic diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0058] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0059] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0060] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0061] It should be noted that the video face segmentation method provided in the embodiment of the present application is generally executed by a server, and accordingly, the video face segmentation device is generally configured in the server.

[0062] See also Figure 1 , Figure 1 A specific implementation of the video face segmentation method is shown.

[0063] It should be noted that if there are substantially the same results, the method of the present invention is not limited to Figure 1 The process sequence shown is limited to the following steps:

[0064] S1: Obtain a sample image, preprocess the sample image to obtain a true value image, and input the true value image into the image processing model.

[0065] Specifically, a sample image is obtained to train an image processing model, thereby obtaining a model for video face segmentation. The image processing model is EfficientNet, which is a multi-dimensional hybrid model scaling method. The image processing model includes four layers of feature branches, which are feature branches of four different resolutions derived from four size reduction layers in the image processing model.

[0066] See also Figure 2 , Figure 2 A specific implementation of step S1 is shown, which is described in detail as follows:

[0067] S11: Acquire a sample image, and perform normalization processing on the sample image to obtain a normalized image.

[0068] S12: resizing the normalized image according to a preset image size to obtain a true value image, and input the true value image into the image processing model.

[0069] Specifically, the preprocessing of the sample image mainly includes normalization and resizing. The normalization in the embodiment of the present application refers to image normalization, which refers to the process of performing a series of standard processing transformations on the image to transform it into a fixed standard form. The standard image is called a normalized image. After some processing or attack, the original image can obtain a variety of copy images, which can obtain standard images of the same form after image normalization processing with the same parameters. The resizing is to adjust the sample image to an image of the same size to facilitate subsequent training of the model. It should be noted that the preset image size is set according to the actual situation and is not limited here.

[0070] In this embodiment, a sample image is acquired and normalized to obtain a normalized image, and then the normalized image is resized according to a preset image size to obtain a true image, and the true image is input into an image processing model to achieve acquisition and preprocessing of the sample image, so that the image input to the model remains uniform, which is convenient for subsequent model training.

[0071] S2: In the first-layer feature branch, the true value image is filtered according to a preset multiple to obtain a filtering result, and the filtering results are superimposed to obtain a first image feature.

[0072] See also Figure 3 , Figure 3 A specific implementation of step S2 is shown, which is described in detail as follows:

[0073] S21: In the first-layer feature branch, image features are obtained by performing label preprocessing on the true value image.

[0074] S22: Using Laplace transform, filter the true value image according to a preset multiple to obtain a filtering result, and superimpose the filtering results to obtain edge detail pyramid features.

[0075] S23: Calculate the loss value between the edge detail pyramid feature and the true value image to obtain a first loss value.

[0076] S24: If the first loss value does not reach the first preset threshold, the corresponding parameters of the first layer feature branch are updated to re-process the image of the first layer feature branch. If the first loss value reaches the first preset threshold, the edge detail pyramid feature is used as the first image feature.

[0077] Specifically, label preprocessing is to distinguish the person and background in the true value image by labeling, marking the person in the true value image as 1, and marking the background as 0. Through the Laplace transform method, the true value image is filtered according to the preset multiple to obtain the filtering result, wherein the preset multiple is set according to the actual situation, and is not limited here. In a specific embodiment, the preset multiples are 1 times, 2 times, and 4 times. Then the filtering result is interpolated to the same resolution, and superimposed to obtain the edge detail pyramid feature. In a specific embodiment, the image feature is subjected to three times of 2 times, 4 times, and 8 times of downsampling filtering using a Laplace filter, and the results of the three times of downsampling filtering are interpolated to the same resolution, and superimposed together to construct an edge detail information pyramid, and then the loss value of the edge detail pyramid feature and the true value image is calculated to obtain the first loss value; if the first loss value does not reach the first preset threshold, the corresponding parameters of the first layer feature branch are updated to re-perform the image processing of the first layer feature branch, and if the first loss value reaches the first preset threshold, the edge detail pyramid feature is used as the first image feature. It should be noted that the first preset threshold is set according to actual conditions and is not limited here. In a specific embodiment, the first preset threshold is 0.3.

[0078] In this embodiment, in the first-layer feature branch, the true value image is preprocessed with labels to obtain image features, and the true value image is filtered according to a preset multiple using Laplace transform to obtain a filtering result, and the filtering results are superimposed to obtain an edge detail pyramid feature, a loss value is calculated, and the model parameters are updated based on the loss value. The corresponding first image feature is obtained to enhance the image edge detail feature and construct an edge detail pyramid feature, which is beneficial to the subsequent segmentation of facial images and improves the accuracy of video face segmentation.

[0079] See also Figure 4 , Figure 4 A specific implementation of step S22 is shown, which is described in detail as follows:

[0080] S221: Acquire multiple different preset magnifications, and use Laplace transform to perform convolution processing on the true value image according to the preset magnifications to obtain edge detail features corresponding to the multiple preset magnifications.

[0081] S222: Using an interpolation calculation method, edge detail features corresponding to a plurality of preset multiples are superimposed and calculated according to preset weights at the same image resolution to obtain edge detail pyramid features.

[0082] In a specific embodiment, the image features are convolved with 1, 2, and 4 steps respectively by Laplace transform to obtain edge detail features of 1, 2, and 4 times downsampling, and the three layers of features are interpolated to a uniform size for feature superposition. The three layers of feature elements (x1, x2, x3) at the same position are superimposed and calculated according to the preset weights of 0.6\0.3\0.1. The calculation method is (0.6*x1+0.3*x2+0.1*x3) to obtain the final edge detail pyramid feature. Furthermore, considering that the number of categories of portrait segmentation is only foreground and background, the edge category information is very small and extremely unbalanced with the background category, so the edge detail information of the true value image is supervised by combining binary cross entropy with the calculation detail loss Dice loss, so that the shallow features of the network have a stronger ability to perceive the edge of the portrait. At the same time, this process only exists in model training, and the edge detail information supervision part will not be run in the inference stage of the model, which does not affect the speed of model use at all.

[0083] S3: In the second-layer feature branch, the first image feature is differentiated from the true value image to obtain the second image feature.

[0084] See also Figure 5 , Figure 5 A specific implementation of step S3 is shown, which is described in detail as follows:

[0085] S31: In the second-layer feature branch, image resolution reduction processing is performed on the first image feature to obtain a resolution processed feature.

[0086] S32: Calculate the loss value of the resolution reduction processing feature and the true value image to obtain a second loss value.

[0087] S32: If the second loss value does not reach the second preset threshold, the corresponding parameters of the first-layer feature branch and the corresponding parameters of the second-layer feature branch are updated, and the true value image is reprocessed through the first-layer feature branch and the second-layer feature branch until the second loss value does not reach the second preset threshold, and the resolution processing feature is used as the second image feature.

[0088] Specifically, in the above steps, the edge detail features corresponding to multiple preset multiples are superimposed and calculated according to the preset weights to the same image resolution by interpolation calculation, thereby realizing the image resolution reduction of the image features. In a specific embodiment, if the image resolution of the true value image is 256, after the first layer of feature branches, the image resolution of the first image feature is 128, and the image resolution of the second image feature is 64. Then calculate the loss value of the resolution reduction processing feature and the true value image to obtain the second loss value; if the second loss value does not reach the second preset threshold, update the corresponding parameters of the first layer feature branch and the corresponding parameters of the second layer feature branch, and reprocess the true value image through the first layer feature branch and the second layer feature branch until the second loss value does not reach the second preset threshold, and use the resolution processing feature as the second image feature. It should be noted that the second preset threshold is set according to the actual situation and is not limited here. In a specific embodiment, the second preset threshold is 0.2. Among them, the differential processing includes image resolution reduction processing, and resolution reduction processing features and true value image loss calculation processing.

[0089] In this embodiment, in the second-layer feature branch, the first image feature and the true image are differentiated to obtain the second image feature, and then the loss value of the resolution reduction processing feature and the true image is calculated to obtain the second loss value. Based on the second loss value, the model parameters are updated and the second image feature is obtained, which is conducive to further processing the image features and improving the accuracy of video face segmentation.

[0090] S4: The second image feature is subjected to image resolution reduction processing by the third-layer feature branch and the fourth-layer feature branch respectively to obtain a third image feature and a fourth image feature.

[0091] See also Figure 6 , Figure 6 A specific implementation of step S4 is shown, which is described in detail as follows:

[0092] S41: The second image feature is subjected to image resolution reduction processing of the third-layer feature branch and the fourth-layer feature branch respectively to obtain a third image feature and a fourth initial image feature.

[0093] S42: Calculate the loss value between the fourth initial image feature and the true value image to obtain a third loss value.

[0094] S43: If the third loss value does not reach the third preset threshold, update the corresponding parameters of the image processing model, and re-input the true value image into the image processing model for image processing until the third loss value reaches the third preset threshold, and use the fourth initial image feature as the fourth image feature.

[0095] Specifically, the embodiment of the present application further performs image resolution reduction processing on the image features. In a specific embodiment, if the image resolution of the true value image is 256, after the first layer of feature branching, the image resolution of the first image feature is 128, the image resolution of the second image feature is 64, the image resolution of the third image feature is 32, and the image resolution of the fourth image feature is 16. At the same time, the loss value of the fourth initial image feature and the true value image is calculated to obtain a third loss value. If the third loss value does not reach the third preset threshold, the corresponding parameters of the image processing model are updated, and the true value image is re-input into the image processing model for image processing until the third loss value reaches the third preset threshold, and the fourth initial image feature is used as the fourth image feature. It should be noted that the third preset threshold is set according to the actual situation and is not limited here. In a specific embodiment, the third preset threshold is 0.1.

[0096] S5: Based on the attention mechanism, the attention matrix of the third image feature and the fourth image feature is calculated, and feature fusion is performed based on the attention matrix to obtain enhanced image features.

[0097] See also Figure 7 , Figure 7 A specific implementation of step S5 is shown, which is described in detail as follows:

[0098] S51: Based on the attention mechanism, calculate the attention matrix of the third image feature and the fourth image feature.

[0099] S52: Perform feature multiplication on the attention matrix and the fourth image feature to obtain a feature multiplication result.

[0100] S53: performing convolution processing on the feature multiplication result and the third image feature to obtain an enhanced image feature.

[0101] Specifically, by introducing the attention model, the focus areas of the third image feature and the fourth image feature are identified, and a corresponding attention matrix is ​​constructed, and then the attention matrix is ​​feature-multiplied with the fourth image feature to obtain a feature multiplication result, and then the feature multiplication result and the third image feature are convolved to obtain an enhanced image feature. In this embodiment, the attention matrix of the third image feature and the fourth image feature is calculated to focus on the focus areas in the image features, which is beneficial to enhancing the perception of image features, thereby facilitating improving the accuracy of video face segmentation.

[0102] S6: Calculate the loss value between the enhanced image features and the true value image. If the loss value reaches a preset threshold, a trained image processing model is obtained.

[0103] Specifically, the enhanced image features obtained in the above steps are features extracted by four layers of feature branches. They are compared with the original true value image to determine whether they are close to the true value image. Therefore, the loss value between the enhanced image features and the true value image is calculated. The smaller the loss value, the closer the enhanced image features are to the true value image. If the loss value reaches the preset threshold, a trained image processing model is obtained. It should be noted that the preset threshold is set according to the actual situation and is not limited here. In a specific embodiment, the preset threshold is 0.05.

[0104] S7: Obtain the video to be processed, extract the video to be processed frame by frame, obtain an initial image, and input the initial image into the trained image processing model for image segmentation processing to obtain a target image.

[0105] Specifically, when it is necessary to segment the face in a video, the video to be processed is obtained, and an image is extracted from the video to be processed by frame-by-frame extraction to obtain an initial image, and the initial image is input into a trained image processing model for image segmentation processing to obtain a target image.

[0106] In this embodiment, a sample image is obtained, and the sample image is preprocessed to obtain a true image, and the true image is input into the image processing model; in the first-layer feature branch, the true image is filtered according to a preset multiple to obtain a filtering result, and the filtering results are superimposed to obtain a first image feature; in the second-layer feature branch, the first image feature and the true image are differentiated to obtain a second image feature; the second image feature is subjected to image resolution reduction processing by the third-layer feature branch and the fourth-layer feature branch respectively to obtain a third image feature and a fourth image feature; based on the attention mechanism, the attention matrix of the third image feature and the fourth image feature is calculated, and feature fusion is performed based on the attention matrix to obtain an enhanced image feature; the loss value of the enhanced image feature and the true image is calculated, and if the loss value reaches a preset threshold, a trained image processing model is obtained; the video to be processed is obtained, and the video to be processed is extracted frame by frame to obtain an initial image, and the initial image is input into the trained image processing model for image segmentation processing to obtain a target image. The embodiment of the present application performs filtering processing on the true value image to enhance the image edge detail information and improve the image edge information segmentation quality. At the same time, the embodiment of the present application also introduces an attention mechanism to enhance the image fusion feature, which is beneficial to the accuracy of video face segmentation.

[0107] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0108] Please refer to Figure 8 , as a response to the above Figure 1 The present application provides an embodiment of a video face segmentation device, and the device embodiment is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0109] like Figure 8 As shown, the video face segmentation device of this embodiment includes: a sample image acquisition module 81, a first image feature generation module 82, a second image feature generation module 83, a third image feature generation module 84, an enhanced image feature generation module 85, an image processing model completion module 86 and a target image generation module 87, wherein:

[0110] The sample image acquisition module 81 is used to acquire a sample image, pre-process the sample image, obtain a true value image, and input the true value image into the image processing model, wherein the image processing model includes four layers of feature branches;

[0111] A first image feature generating module 82 is used to filter the true value image according to a preset multiple in the first layer feature branch to obtain a filtering result, and to superimpose the filtering results to obtain a first image feature;

[0112] A second image feature generating module 83 is used to perform a differentiation process on the first image feature and the true value image in the second layer feature branch to obtain a second image feature;

[0113] A third image feature generating module 84 is used to process the second image feature through the image resolution reduction process of the third layer feature branch and the fourth layer feature branch respectively, so as to obtain a third image feature and a fourth image feature;

[0114] An enhanced image feature generation module 85 is used to calculate the attention matrix of the third image feature and the fourth image feature based on the attention mechanism, and perform feature fusion based on the attention matrix to obtain the enhanced image feature;

[0115] The image processing model completion module 86 is used to calculate the loss value between the enhanced image features and the true value image. If the loss value reaches a preset threshold, a trained image processing model is obtained;

[0116] The target image generation module 87 is used to obtain the video to be processed, extract the video to be processed frame by frame, obtain the initial image, and input the initial image into the trained image processing model for image segmentation processing to obtain the target image.

[0117] Furthermore, the sample image acquisition module 81 includes:

[0118] A normalization processing unit, used for acquiring a sample image and performing normalization processing on the sample image to obtain a normalized image;

[0119] The size adjustment unit is used to adjust the size of the normalized image according to a preset image size to obtain a true value image, and input the true value image into the image processing model.

[0120] Furthermore, the first image feature generating module 82 includes:

[0121] An image feature acquisition unit, used to obtain image features by performing label preprocessing on the true value image in the first layer feature branch;

[0122] A filtering result generating unit is used to filter the true value image according to a preset multiple by adopting Laplace transform to obtain a filtering result, and to superimpose the filtering results to obtain edge detail pyramid features;

[0123] A first loss value generating unit, used to calculate the loss value of the edge detail pyramid feature and the true value image to obtain a first loss value;

[0124] The first parameter updating unit is used to update the corresponding parameters of the first-layer feature branch if the first loss value does not reach the first preset threshold value, so as to re-process the image of the first-layer feature branch. If the first loss value reaches the first preset threshold value, the edge detail pyramid feature is used as the first image feature.

[0125] Furthermore, the filtering result generating unit includes:

[0126] The convolution processing subunit is used to obtain a plurality of different preset multiples, and to perform convolution processing on the true value image according to the preset multiples by using Laplace transform to obtain edge detail features corresponding to the plurality of preset multiples;

[0127] The superposition calculation subunit is used to superimpose edge detail features corresponding to multiple preset multiples according to preset weights at the same image resolution by using an interpolation calculation method to obtain edge detail pyramid features.

[0128] Furthermore, the second image feature generating module 83 includes:

[0129] A first resolution processing unit is used to perform image resolution reduction processing on the first image feature in the second layer feature branch to obtain a resolution processing feature;

[0130] A second loss value generating unit, used to calculate the loss value of the resolution reduction processing feature and the true value image to obtain a second loss value;

[0131] The second parameter updating unit is used to update the corresponding parameters of the first-layer feature branch and the corresponding parameters of the second-layer feature branch if the second loss value does not reach the second preset threshold, and re-process the true value image through the first-layer feature branch and the second-layer feature branch until the second loss value reaches the second preset threshold, and use the resolution processing feature as the second image feature.

[0132] Furthermore, the third image feature generating module 84 includes:

[0133] A second resolution processing unit is used to process the second image feature through the image resolution reduction of the third layer feature branch and the fourth layer feature branch respectively, so as to obtain a third image feature and a fourth initial image feature;

[0134] A third loss value generating unit, used to calculate the loss value of the fourth initial image feature and the true value image to obtain a third loss value;

[0135] The third parameter updating unit is used to update the corresponding parameters of the image processing model if the third loss value does not reach the third preset threshold, and re-input the true value image into the image processing model for image processing until the third loss value reaches the third preset threshold, and use the fourth initial image feature as the fourth image feature.

[0136] Furthermore, the enhanced image feature generation module 85 includes:

[0137] an attention matrix generating unit, used for calculating the attention matrix of the third image feature and the fourth image feature based on the attention mechanism;

[0138] A feature multiplication unit, used for performing feature multiplication on the attention matrix and the fourth image feature to obtain a feature multiplication result;

[0139] The image convolution processing unit is used to perform convolution processing on the feature multiplication result and the third image feature to obtain enhanced image features.

[0140] To solve the above technical problems, the present application also provides a computer device. Fig. 9 , Fig. 9 This is a basic structural block diagram of the computer device in this embodiment.

[0141] The computer device 9 includes a memory 91, a processor 92, and a network interface 93 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 9 having three components: a memory 91, a processor 92, and a network interface 93, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0142] Computer devices can be computing devices such as desktop computers, notebooks, PDAs, and cloud servers. Computer devices can interact with users through keyboards, mice, remote controls, touch pads, or voice control devices.

[0143] The memory 91 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 91 can be an internal storage unit of the computer device 9, such as a hard disk or memory of the computer device 9. In other embodiments, the memory 91 can also be an external storage device of the computer device 9, such as a plug-in hard disk equipped on the computer device 9, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 91 can also include both the internal storage unit of the computer device 9 and its external storage device. In this embodiment, the memory 91 is generally used to store the operating system and various application software installed on the computer device 9, such as the program code of the video face segmentation method, etc. In addition, the memory 91 can also be used to temporarily store various types of data that have been output or are to be output.

[0144] In some embodiments, the processor 92 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 92 is generally used to control the overall operation of the computer device 9. In this embodiment, the processor 92 is used to run the program code stored in the memory 91 or process data, such as running the program code of the above-mentioned video face segmentation method to implement various embodiments of the video face segmentation method.

[0145] The network interface 93 may include a wireless network interface or a wired network interface, and the network interface 93 is generally used to establish a communication connection between the computer device 9 and other electronic devices.

[0146] The present application also provides another implementation, namely, providing a computer-readable storage medium, which stores a computer program. The computer program can be executed by at least one processor to enable the at least one processor to perform the steps of a video face segmentation method as described above.

[0147] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0148] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0149] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A video face segmentation method, It is characterized in that include: Acquire a sample image, preprocess the sample image to obtain a true value image, and input the true value image into an image processing model, wherein the image processing model includes four layers of feature branches; In the first layer feature branch, the true value image is filtered according to a preset multiple to obtain a filtering result, and the filtering results are superimposed to obtain a first image feature; In the second-layer feature branch, the first image feature and the true value image are differentiated to obtain a second image feature; The second image feature is processed by image resolution reduction of the third layer feature branch and the fourth layer feature branch respectively, to obtain a third image feature and a fourth image feature; Based on the attention mechanism, an attention matrix of the third image feature and the fourth image feature is calculated, and feature fusion is performed based on the attention matrix to obtain an enhanced image feature; Calculating the loss value between the enhanced image feature and the true value image, and if the loss value reaches a preset threshold, obtaining a trained image processing model; The video to be processed is obtained, and frame-by-frame extraction is performed on the video to be processed to obtain an initial image, and the initial image is input into the trained image processing model for image segmentation processing to obtain a target image.

2. The video face segmentation method according to claim 1, It is characterized in that The step of obtaining a sample image, preprocessing the sample image to obtain a true value image, and inputting the true value image into an image processing model includes: Acquire the sample image, and perform normalization processing on the sample image to obtain a normalized image; The normalized image is resized according to a preset image size to obtain a true value image, and the true value image is input into the image processing model.

3. The video face segmentation method according to claim 1, It is characterized in that In the first layer feature branch, the true value image is filtered according to a preset multiple to obtain a filtering result, and the filtering results are superimposed to obtain a first image feature, including: In the first-layer feature branch, image features are obtained by performing label preprocessing on the true value image; Using Laplace transform, filtering the true value image according to the preset multiple to obtain a filtering result, and superimposing the filtering results to obtain edge detail pyramid features; Calculating a loss value between the edge detail pyramid feature and the true value image to obtain a first loss value; If the first loss value does not reach the first preset threshold, the corresponding parameters of the first-layer feature branch are updated to re-perform image processing of the first-layer feature branch. If the first loss value reaches the first preset threshold, the edge detail pyramid feature is used as the first image feature.

4. The video face segmentation method according to claim 3, It is characterized in that The method of using Laplace transform to filter the true value image according to the preset multiple to obtain a filtering result, and superimposing the filtering results to obtain edge detail pyramid features includes: Acquire multiple different preset magnifications, and use Laplace transform to perform convolution processing on the true value image according to the preset magnifications respectively, so as to obtain edge detail features corresponding to the multiple preset magnifications; By adopting an interpolation calculation method, edge detail features corresponding to a plurality of the preset multiples are superimposed and calculated according to preset weights at the same image resolution to obtain the edge detail pyramid feature.

5. The video face segmentation method according to claim 1, It is characterized in that In the second-layer feature branch, performing differentiation processing on the first image feature and the true value image to obtain a second image feature includes: In the second-layer feature branch, performing image resolution reduction processing on the first image feature to obtain a resolution processed feature; Calculating a loss value between the resolution reduction processing feature and the true value image to obtain a second loss value; If the second loss value does not reach the second preset threshold, the corresponding parameters of the first-layer feature branch and the second-layer feature branch are updated, and the true value image is reprocessed through the first-layer feature branch and the second-layer feature branch until the second loss value reaches the second preset threshold, and the resolution processing feature is used as the second image feature.

6. The video face segmentation method according to claim 1, It is characterized in that The step of subjecting the second image feature to image resolution reduction processing of the third layer feature branch and the fourth layer feature branch respectively to obtain the third image feature and the fourth image feature comprises: The second image feature is processed by image resolution reduction of the third layer feature branch and the fourth layer feature branch respectively, so as to obtain a third image feature and a fourth initial image feature; Calculating a loss value between the fourth initial image feature and the true value image to obtain a third loss value; If the third loss value does not reach the third preset threshold, the corresponding parameters of the image processing model are updated, and the true value image is re-input into the image processing model for image processing until the third loss value reaches the third preset threshold, and the fourth initial image feature is used as the fourth image feature.

7. The video face segmentation method according to any one of claims 1 to 6, It is characterized in that The method of calculating the attention matrix of the third image feature and the fourth image feature based on the attention mechanism, and performing feature fusion based on the attention matrix to obtain enhanced image features includes: Based on the attention mechanism, calculating the attention matrix of the third image feature and the fourth image feature; Performing feature multiplication on the attention matrix and the fourth image feature to obtain a feature multiplication result; The feature multiplication result and the third image feature are convolved to obtain the enhanced image feature.

8. A video face segmentation device, It is characterized in that include: A sample image acquisition module, used to acquire a sample image, preprocess the sample image to obtain a true value image, and input the true value image into an image processing model, wherein the image processing model includes four layers of feature branches; A first image feature generation module is used to filter the true value image according to a preset multiple in the first layer feature branch to obtain a filtering result, and superimpose the filtering results to obtain a first image feature; A second image feature generating module, configured to perform a differentiation process on the first image feature and the true value image in a second layer feature branch to obtain a second image feature; A third image feature generating module, configured to subject the second image feature to image resolution reduction processing through a third-layer feature branch and a fourth-layer feature branch, respectively, to obtain a third image feature and a fourth image feature; An enhanced image feature generation module, used to calculate the attention matrix of the third image feature and the fourth image feature based on the attention mechanism, and perform feature fusion based on the attention matrix to obtain the enhanced image feature; An image processing model completion module is used to calculate the loss value between the enhanced image features and the true value image, and if the loss value reaches a preset threshold, a trained image processing model is obtained; The target image generation module is used to obtain the video to be processed, and extract the video to be processed frame by frame to obtain an initial image, and input the initial image into the trained image processing model to perform image segmentation processing to obtain a target image.

9. A computer device, It is characterized in that The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the video face segmentation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the video face segmentation method according to any one of claims 1 to 7 is implemented.

Citation Information

Cited By

  • Video splitting method and system

    CN121078265A