Special effect processing method and device, equipment, chip, storage medium and product
By performing object detection on video frames in a live broadcast application, it determines whether to perform feature point detection, which solves the high power consumption problem caused by frequent feature point detection, and ensures that face special effects can be quickly obtained when faces are detected.
Patent Information
- Application Number
- CN202411962443.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-27
- Publication Date
- 2025-04-11
AI Technical Summary
In existing live broadcast applications, frequent use of face feature point detection (FFD) algorithm results in high power consumption of the device, and when entering the live broadcast interface at non-FFD moments, face special effects cannot be obtained in the first few video frames.
By performing object detection on the current video frame, it is determined whether the target is included, and then whether the feature point detection for the target is performed. This method uses the result of object detection to control the frequency of feature point detection and reduces equipment power consumption.
By reducing unnecessary feature point detection, the power consumption of the device is reduced, while ensuring that face special effects can be quickly obtained when faces are detected, solving the problem that face special effects cannot be obtained in the first few video frames.
Smart Images
Figure CN119815107A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to, but is not limited to, the field of computer technology, and in particular to a special effects processing method, device, equipment, chip, storage medium and product. Background Art
[0002] After receiving the image (video frame) sent by the camera, the existing live broadcast application (APP) uses the self-developed Face Feature Point Detection (FFD) algorithm to perform special effects such as beautification, makeup, and face slimming. In this process, the FFD algorithm is usually executed once every several frames.
[0003] Due to the complexity of the FFD algorithm and the high power consumption required, frequent use of the FFD algorithm will result in high power consumption of the device. Summary of the invention
[0004] The present application at least provides a special effects processing method, device, equipment, chip, storage medium and product.
[0005] The technical solution of this application is implemented as follows:
[0006] In a first aspect, the present application provides a special effects processing method, the method comprising: obtaining a current video frame; determining whether the current video frame contains a target based on a result of target detection performed on the current video frame; determining whether to perform feature point detection for the target on the current video frame based on whether the current video frame contains the target; wherein the feature point information obtained by performing feature point detection for the target on the current video frame is used to perform special effects processing on the target in the current video frame.
[0007] In a second aspect, an embodiment of the present application provides a special effects processing device, which includes: an acquisition unit, used to acquire a current video frame; a first determination unit, used to determine whether the current video frame contains a target based on a result of performing target detection on the current video frame; a second determination unit, used to determine whether to perform feature point detection for the target on the current video frame based on whether the current video frame contains the target; wherein the feature point information obtained by performing feature point detection for the target on the current video frame is used to perform special effects processing on the target in the current video frame.
[0008] In a third aspect, an embodiment of the present application provides a special effects processing device, which includes a memory and a processor; wherein the memory is used to store computer-executable instructions; the processor is connected to the memory and is used to implement the method described in the first aspect by executing the computer-executable instructions.
[0009] In a fourth aspect, an embodiment of the present application provides a chip, which includes: a processor, configured to call and run a computer program from a memory, so that a device equipped with the chip executes the method described in the first aspect.
[0010] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by at least one processor, it implements the method described in the first aspect.
[0011] In a sixth aspect, an embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the method described in the first aspect.
[0012] In the embodiment of the present application, it is possible to determine whether the current video frame contains a target based on the result of performing target detection on the current video frame, and then determine whether to perform feature point detection on the target on the current video frame based on whether the current video frame contains the target. The feature point information obtained by performing feature point detection on the current video frame is used to perform special effects processing on the target in the current video frame.
[0013] According to the method of the embodiment of the present application, since the target detection scheme is adopted to determine whether the current video frame contains the target, and the complexity and power consumption required to perform target detection are relatively low, therefore, judging whether to trigger feature point detection for the target on the current video frame based on whether the current video frame contains the target is beneficial to reducing device power consumption.
[0014] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.
[0016] Figure 1 This is a flowchart diagram of a commonly used live broadcast APP performing special effects processing on faces;
[0017] Figure 2 A flowchart of a special effect processing method provided in an embodiment of the present application;
[0018] Figure 3 A possible implementation flow diagram of the special effect processing method provided in the embodiment of the present application;
[0019] Figure 4 A schematic diagram of the structure of a special effect processing device provided in an embodiment of the present application;
[0020] Figure 5 A hardware entity schematic diagram of a special effects processing device in an embodiment of the present application. DETAILED DESCRIPTION
[0021] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.
[0022] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meaning as those commonly understood by those skilled in the art of the present application. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.
[0023] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. It should also be noted that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first\second\third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.
[0024] It should be understood that the term "and / or" in the embodiments of the present application is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0025] After receiving the image (video frame) sent by the camera, the existing live broadcast application (APP) uses the self-developed facial feature detection (FFD) algorithm to perform special effects such as beautification, makeup, and face slimming. In this process, the FFD algorithm is usually executed once every several frames, and the facial feature points are obtained through the tracking algorithm in the intermediate frames.
[0026] For ease of understanding, the following Figure 1 This article introduces a common live broadcast APP process for special effects processing of faces. Among them, the P1 module and the P2 module represent the functional modules of the system camera processing.
[0027] like Figure 1 As shown in the figure, the process of special effects processing of faces by common live broadcast APPs may include:
[0028] S101, determine whether Frame_Num%N is equal to 0.
[0029] Wherein, Frame_Num represents the frame number of the current video frame (referred to as the current frame); "%" is the modulo operator; N is a positive integer, and it is assumed here that FDD is performed once every N frames.
[0030] By determining whether Frame_Num%N is equal to 0, video frames with frame numbers that are integer multiples of N can be screened out. For example, when N=10, video frames with frame numbers of 0, 10, 20, ... can be screened out.
[0031] S102, execute FFD.
[0032] If the determination result of S101 is "yes", FFD may be performed on the current frame.
[0033] Taking N=10 as an example, if the frame sequence number of the current frame is 0, 10, 20, ..., then FFD can be performed on the current frame.
[0034] S103, executing the tracking algorithm.
[0035] If the determination result of S101 is "No", the tracking algorithm may be executed on the current frame.
[0036] Taking N=10 as an example, if the frame number of the current frame is not 0, 10, 20, ..., the tracking algorithm can be executed on the current frame.
[0037] S104, determining whether there are facial feature points.
[0038] In this step, it can be determined whether the current frame has facial feature points based on the result of executing the FFD in S102 or the result of executing the tracking algorithm in S103.
[0039] S105, performing special effects processing on the face.
[0040] If the judgment result of S104 is "yes", that is, the current frame has facial feature points, special effects processing (such as beautification, makeup, face slimming, etc.) can be performed on the face in the current frame according to the facial feature points.
[0041] S106, display and encoding.
[0042] In one possible situation, the judgment result of S104 is "yes". In this case, special effects processing needs to be performed on the face in the current frame, so the video frame processed with special effects can be displayed and encoded.
[0043] In another possible situation, the judgment result of S104 is "No". In this case, the face special effects processing cannot be performed on the current frame, so the video frame output by the P2 module can be directly displayed and encoded.
[0044] according to Figure 1 As can be seen from the flow, the existing solutions for special effects processing of faces usually execute the FFD algorithm once every multiple frames. However, due to the complexity of the FFD algorithm and the high power consumption required, frequent use of the FFD algorithm will result in high power consumption of the device.
[0045] In view of this, the embodiments of the present application provide a special effect processing method, device, equipment, chip, storage medium and product, which can be applied to scenarios such as live broadcast and video recording. In this method, it can be determined whether the current video frame contains a target based on the result of performing target detection on the current video frame, and then it can be determined whether to perform feature point detection for the target on the current video frame based on whether the current video frame contains the target. The feature point information obtained by performing feature point detection for the target on the current video frame is used to perform special effect processing on the target in the current video frame.
[0046] That is to say, whether to perform feature point detection for the target on the current video frame can be determined based on whether the current video frame contains the target. Since the target detection scheme is used to determine whether the current video frame contains the target, and the complexity and power consumption required for performing target detection are relatively low, therefore, determining whether to trigger the feature point detection for the target on the current video frame based on whether the current video frame contains the target is conducive to reducing the power consumption of the device.
[0047] It should be noted that the embodiment of the present application does not limit the type of target. For example, the target in the embodiment of the present application can be a person (such as a face), an animal (such as a cat or a dog), a plant (such as a flower or grass), food, etc.
[0048] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.
[0049] The present application embodiment provides a special effects processing method. Figure 2 As shown, the method may include:
[0050] S201, obtaining the current video frame.
[0051] Exemplarily, a video frame currently generated by a system camera of the device may be obtained as the current video frame.
[0052] In some scenarios, the "video frame" in the embodiments of the present application may also be referred to as a "frame". As an example, a video frame may be a video frame in a live broadcast process (such as a video frame generated by a live broadcast APP during a live broadcast), or may be a video frame in a video recording process (such as a video frame generated by a short video recording APP during a video recording process).
[0053] S202: Determine whether the current video frame contains a target according to a result of performing target detection on the current video frame.
[0054] Exemplarily, target detection may be performed on the acquired current video frame, so as to determine / judge whether the current video frame contains a target based on the result of the target detection.
[0055] The result of target detection may include, but is not limited to, at least one of the following: target angle, confidence in target detection (ie, the probability of determining it as a target), target feature points, size coordinates, etc.
[0056] For example, when the target is a human face, face detection (FD) may be performed on the acquired current video frame, so as to determine / judge whether the current video frame contains a human face based on the result of the face detection.
[0057] The result of face detection may include, but is not limited to, at least one of the following: face angle, confidence of face detection (ie, the probability of determining it to be a face), face feature points, size coordinates, etc.
[0058] It is understandable that, compared with the target feature points obtained by performing feature point detection on the target, the number of target feature points obtained by performing target detection is relatively small and the precision is relatively low. Therefore, compared with performing feature point detection on the target, the complexity and power consumption required for performing target detection are relatively low. For example, compared with facial feature point detection, the complexity and power consumption required for performing face detection are relatively low.
[0059] It should be noted that the embodiments of the present application do not limit the target detection algorithm (such as face detection algorithm) used to perform target detection (such as face detection).
[0060] S203, determining whether to perform feature point detection on the current video frame according to whether the current video frame contains the target; wherein the feature point information obtained by performing feature point detection on the current video frame is used to perform special effects processing on the target in the current video frame.
[0061] In this step, whether to perform feature point detection for the target on the current video frame may be determined according to whether the current video frame contains the target.
[0062] For example, when the target is a human face, it can be determined whether to perform feature point detection for the face (ie, facial feature point detection) on the current video frame according to whether the current video frame contains a human face.
[0063] That is to say, whether to perform feature point detection for the target on the current video frame can be determined based on whether the current video frame contains the target. Since the target detection scheme is used to determine whether the current video frame contains the target, and the complexity and power consumption required for performing target detection are relatively low, therefore, determining whether to trigger the feature point detection for the target on the current video frame based on whether the current video frame contains the target is conducive to reducing the power consumption of the device.
[0064] In this embodiment, feature point information obtained by performing feature point detection on the current video frame (i.e., feature point information of the target in the current video frame) can be used to perform special effects processing on the target in the current video frame. For example, when the target is a face, facial feature point information obtained by performing facial feature point detection on the current video frame can be used to perform special effects processing on the face in the current video frame.
[0065] That is, after performing feature point detection on the current video frame, if the feature point information of the target is obtained, the target in the current video frame can be processed with special effects according to the feature point information. For example, if the target is a face, after performing facial feature point detection on the current video frame, if the facial feature point information is obtained, the face in the current video frame can be processed with special effects according to the facial feature point information.
[0066] Taking a human face as an example, the special effects processing may include but is not limited to at least one of the following: beautification, makeup, and face slimming.
[0067] In some embodiments, the steps of "performing feature point detection on the target" and "performing special effects processing on the target" can be executed by an APP in the device.
[0068] It should be noted that the embodiment of the present application does not limit the feature point detection algorithm used to perform feature point detection (such as facial feature point detection).
[0069] In some embodiments, whether to perform feature point detection for the target on the current video frame is determined based on whether the current video frame contains the target, including: when the current video frame is the first video frame, if the current video frame contains the target, determining to perform feature point detection for the target on the current video frame.
[0070] That is to say, when the current video frame is the first video frame, or when the current video frame is the first frame, if it is known from the target detection result that the current video frame contains a target, then feature point detection for the target can be performed on the current video frame, so that special effects processing can be performed on the target in the current video frame based on the detected feature point information. If it is known from the target detection result that the current video frame does not contain a target, there is no need to perform feature point detection for the target on the current video frame, which is conducive to saving power consumption.
[0071] Taking the target as a face as an example, when the current video frame is the first video frame, or in other words, when the current video frame is the first frame, if it is known from the face detection result that the current video frame contains a face, then facial feature point detection can be performed on the current video frame, so that special effects processing can be performed on the face in the current video frame based on the detected facial feature point information. If it is known from the face detection result that the current video frame does not contain a face, there is no need to perform facial feature point detection on the current video frame, which is conducive to saving power consumption.
[0072] according to Figure 1 As can be seen from the flow shown, the existing solution performs facial feature point detection once every multiple frames, so when the host enters the live broadcast interface at a non-face feature point detection time, the first few video frames do not perform facial feature point detection, so the first few video frames cannot obtain face special effects. With the solution of this embodiment, for the first video frame, if it contains a face, facial feature point detection can be performed. Therefore, no matter when the host enters the live broadcast interface, facial feature point detection can be performed as long as a face is detected, so that facial feature point information can be quickly obtained, and then facial special effects can be quickly obtained.
[0073] In some embodiments, determining whether to perform feature point detection for the target on the current video frame according to whether the current video frame contains the target includes the following steps 11) and 12):
[0074] 11) In the case that the current video frame is not the first video frame, if the current video frame contains the target, it is determined whether a previous video frame of the current video frame has feature point information of the target.
[0075] Taking the target as a human face as an example, when the current video frame is not the first video frame, if the current video frame contains a human face, it can be determined whether a video frame before the current video frame has facial feature point information.
[0076] For example, if a feature point detection or tracking algorithm for a target has been performed on a previous video frame (ie, the previous frame), feature point information of the target in the previous video frame may be obtained. Furthermore, the feature point information may be saved / cached for subsequent use.
[0077] In step 11), when the current video frame is not the first video frame, if the current video frame contains a target, it can be determined whether the previous video frame has feature point information of the target, that is, it can be determined whether the feature point information of the target in the previous video frame is currently saved / cached.
[0078] 12) Determine whether to perform feature point detection for the target on the current video frame based on whether the previous video frame has feature point information of the target.
[0079] For example, when the target is a face, whether to perform facial feature point detection on the current video frame can be determined based on whether the previous video frame has facial feature point information.
[0080] According to the method of this embodiment, in the case where the current video frame is not the first video frame, if it is known from the target detection result that the current video frame contains the target, then it can be determined whether to perform feature point detection for the target on the current video frame in combination with whether the previous video frame has feature point information of the target. In this way, it is helpful to avoid performing feature point detection when it is not necessary to perform feature point detection, thereby avoiding unnecessary power consumption. For example, if the previous video frame already has feature point information of the target, it can be considered to perform a tracking algorithm on the current video frame to track the feature points of the target in the previous video frame, without performing feature point detection for the target on the current video frame. Since the power consumption required by the tracking algorithm is relatively low, it is helpful to reduce the power consumption of the device.
[0081] In some embodiments, when the current video frame is not the first video frame, if it is known from the target detection result that the current video frame does not contain the target, there is no need to perform feature point detection for the target on the current video frame, which is beneficial for saving power consumption.
[0082] In some embodiments, whether to perform feature point detection for the target on the current video frame is determined based on whether the previous video frame has feature point information of the target, including: if the previous video frame has no feature point information of the target, determining to perform feature point detection for the target on the current video frame.
[0083] It can be understood that, when the current video frame is not the first video frame, if the previous video frame does not have the feature point information of the target, or in other words, the feature point information of the target in the previous video frame is not currently saved / cached, it means that the feature point information of the target in the video is lost. Therefore, in this case, it is necessary to perform feature point detection for the target on the current video frame to regain the feature point information of the target.
[0084] According to the method of this embodiment, when the feature point information of the target is lost, the feature point information of the target can be regained by performing feature point detection on the current video frame, thereby avoiding the loss of target special effects due to the lack of target feature point information.
[0085] In some embodiments, determining to perform feature point detection on a target on a current video frame includes: determining whether the target in the current video frame satisfies conditions for performing special effects processing based on a result of performing target detection on the current video frame; if the target in the current video frame satisfies conditions for performing special effects processing, then determining to perform feature point detection on the target on the current video frame.
[0086] That is, before performing target feature point detection on the current video frame, it is possible to determine / judge whether the target in the current video frame meets the conditions for special effect processing based on the results of target detection on the current video frame (such as target angle, confidence of target detection, target feature points, size coordinates, etc.). Further, if the target in the current video frame meets the conditions for special effect processing, feature point detection for the target can be performed on the current video frame; if the target in the current video frame does not meet the conditions for special effect processing, there is no need to perform feature point detection for the target on the current video frame.
[0087] It is understandable that when the target in the current video frame does not meet the conditions for special effects processing, even if the feature point information of the target is obtained, special effects processing cannot be performed on the target. Therefore, in this case, there is no need to perform feature point detection for the target in the current video frame, thereby further reducing device power consumption.
[0088] For example, if the angle of the target in the current video frame is not convenient for special effects processing, or the confidence of target detection is low (that is, the probability of determining it as a target is low), then it can be considered that the target in the current video frame does not meet the conditions for special effects processing. In this case, there is no need to perform special effects processing on the target in the current video frame, so there is no need to perform feature point detection for the target in the current video frame, thereby further reducing device power consumption.
[0089] In some embodiments, the step of "determining whether the target in the current video frame meets the conditions for performing special effects processing" can be performed on the system camera side of the device, or can also be performed by an APP in the device.
[0090] In some embodiments, whether to perform feature point detection for the target on the current video frame is determined based on whether the previous video frame has feature point information of the target, including: in the case where the previous video frame has feature point information of the target, if the current video frame is the Nth video frame after the last feature point detection for the target was performed, then determining to perform feature point detection for the target on the current video frame; wherein N is a positive integer.
[0091] For example, assuming that the last feature point detection for the target occurred on the 5th video frame, then if the previous video frame of the current video frame has feature point information of the target, if the current video frame is the 5+Nth video frame, feature point detection for the target can be performed on the current video frame.
[0092] According to the method of this embodiment, when the feature point information of the target in the video is not lost, feature point detection for the target can be performed once every N frames to ensure that accurate target feature point information can be obtained in real time.
[0093] In some embodiments, the method may further include: in a case where the previous video frame has feature point information of the target, if the current video frame is not the Nth video frame after the last feature point detection for the target is performed, then determining to execute a tracking algorithm on the current video frame; wherein the tracking algorithm is used to obtain the feature point information of the target in the current video frame by tracking the feature points of the target in the previous video frame; the feature point information of the target in the current video frame is used to perform special effects processing on the target in the current video frame.
[0094] For example, assuming that the last feature point detection for the target occurred in the 5th video frame, then if the previous video frame of the current video frame has the feature point information of the target, if the current video frame is not the 5th+Nth video frame, the tracking algorithm can be executed on the current video frame.
[0095] Taking N=10 as an example, assuming that the last feature point detection for the target occurred in the 5th frame of the video, then, if the previous video frame of the current video frame has the feature point information of the target, if the current video frame is not the 15th frame of the video (for example, the current video frame is a video frame between the 6th frame and the 14th frame), the tracking algorithm can be executed on the current video frame.
[0096] As an example, when executing the tracking algorithm on the kth video frame, the feature points of the target in the k-1th video frame can be tracked based on the feature point information of the target in the k-1th video frame, thereby obtaining the feature point information of the target in the kth video frame. Furthermore, special effects processing can be performed on the target in the kth video frame based on the feature point information of the target in the kth video frame.
[0097] According to the method of this embodiment, if the feature point information of the target in the video is not lost, if the current video frame is not the Nth video frame after the last feature point detection for the target, the tracking algorithm can be executed on the current video frame. Since the power consumption required by the tracking algorithm is relatively low, it is beneficial to reduce the power consumption of the device.
[0098] In some embodiments, the method may further include: when it is determined that the current video frame contains the target, loading an algorithm component required for special effect processing.
[0099] For example, if the current video frame is determined to contain an object based on the result of object detection performed on the current video frame, the algorithm components required for special effect processing can be loaded. In this way, after obtaining the feature point information of the object in the current video frame, the loaded algorithm components can be used to perform special effect processing on the object in the current video frame.
[0100] According to the method of this embodiment, the algorithm components required for special effects processing can be loaded when it is determined that the current video frame contains the target, without having to wait until the feature point detection results of the target are obtained before deciding whether to load the algorithm components required for special effects processing, thereby improving the efficiency of special effects processing on the target.
[0101] In some embodiments, the step of "performing target detection on the current video frame" may be performed on the system camera side of the device (e.g., performed by the platform hardware on the system camera side). Taking the target as a face, for example, face detection on the system camera side is generally enabled by default, so the face detection result on the system camera side may be reused without increasing the power consumption of the device to determine whether to trigger the detection of facial feature points on the current video frame.
[0102] In some embodiments, the step of "determining whether to perform feature point detection for the target on the current video frame" can be executed on the system camera side of the device, and the system camera side can send the judgment result of "whether to perform feature point detection for the target on the current video frame" to the APP; alternatively, the step of "determining whether to perform feature point detection for the target on the current video frame" can also be executed by the APP in the device.
[0103] To facilitate understanding of the embodiments of the present application, a possible implementation process of the special effects processing method provided in the embodiments of the present application is introduced below, taking a human face as an example.
[0104] Figure 3 A possible implementation flow diagram of the special effect processing method provided in the embodiment of the present application is shown, and the implementation flow can be applied to a live broadcast scene, for example. Among them, the P1 module and the P2 module represent the functional modules of the system camera processing.
[0105] like Figure 3 As shown, the implementation process may include:
[0106] S301, using platform hardware to execute FD.
[0107] In this step, the platform hardware can be used to perform face detection (FD) on the video frames output by the P1 module. By performing FD, the following FD detection results can be obtained: face angle, confidence (that is, the probability of determining it to be a face), face feature points, size coordinates, etc.
[0108] In this embodiment, the default platform hardware executes FD for each video frame.
[0109] As a possible implementation manner, the video frames output by the P1 module are video frames with relatively low resolution.
[0110] It should be noted that, in some embodiments, “using platform hardware to execute FD on the video frames output by the P1 module” can also be replaced by “using platform hardware to execute FD on the video frames output by the P2 module”.
[0111] S302, executing boundary condition judgment.
[0112] In this step, the boundary condition judgment of the face information can be added on the system camera side, that is, whether the boundary condition is met according to the result of executing FD by the platform hardware.
[0113] Exemplarily, boundary condition judgment (judging whether boundary conditions are met) refers to judging whether the face in the current frame meets the conditions for special effects processing (for example, whether makeup conditions are met). As a possible implementation, it can be judged whether the face in the current frame meets the conditions for special effects processing based on the confidence or face angle obtained in S301.
[0114] Furthermore, the judgment result of S302, the FD detection result obtained in S301, and the video frame output by the P2 module may be sent to the live broadcast APP together.
[0115] In some embodiments, S302 may be omitted, that is, S302 may not be executed.
[0116] S303, determining whether the current frame is the first frame of the live broadcast.
[0117] In this step, it can be determined whether the current frame is the first live frame. If the determination result is "yes", that is, the current frame is the first live frame, then S304 is executed; if the determination result is "no", that is, the current frame is not the first live frame, then S305 is executed.
[0118] S304, determining whether there is a face in the FD detection result.
[0119] If the judgment result of S303 is "the current frame is the first frame of live broadcast", it can be determined whether there is a face in the current frame according to the detection result of FD executed by the platform hardware. If the judgment result is "yes", that is, there is a face in the current frame, S309 is executed; if the judgment result is "no", that is, there is no face in the current frame, the video frame output by the P2 module is directly displayed and encoded.
[0120] S305, determining whether there is a face in the FD detection result.
[0121] If the judgment result of S303 is "the current frame is not the first live frame", it can be determined whether there is a face in the current frame according to the detection result of the FD executed by the platform hardware. If the judgment result is "yes", that is, there is a face in the current frame, then S306 is executed; if the judgment result is "no", that is, there is no face in the current frame, then the video frame output by the P2 module is directly displayed and encoded.
[0122] S306, determining whether there are facial feature points in the previous frame.
[0123] For example, it can be determined whether the previous frame has facial feature points according to the determination result obtained when the previous frame executes S311. For example, if facial feature point information is obtained after executing FFD or tracking algorithm on the previous frame, it can be considered that the previous frame has facial feature points; otherwise, it can be considered that the previous frame has no facial feature points.
[0124] If the judgment result of S306 is "yes", that is, there are facial feature points in the previous frame, then S308 is executed; if the judgment result is "no", that is, there are no facial feature points in the previous frame, then S307 is executed.
[0125] S307, boundary condition determination.
[0126] If the judgment result of S306 is "No", that is, there is no facial feature point in the previous frame, then a boundary condition judgment may be performed, that is, whether the face in the current frame satisfies the conditions for special effect processing.
[0127] In some embodiments, if 302 has not been performed before, in this step, it can be determined whether the face in the current frame meets the conditions for special effects processing based on the FD detection result (such as face angle, confidence, etc.) obtained in S301.
[0128] In some embodiments, if S302 has been executed previously, then in this step, the boundary condition judgment result obtained in S302 can be directly used without having to judge again based on the FD detection result (such as face angle, confidence, etc.) whether the face in the current frame meets the conditions for special effects processing.
[0129] If the judgment result of S307 is "yes", that is, the face in the current frame meets the conditions for special effects processing, then S309 is executed; if the judgment result is "no", that is, the face in the current frame does not meet the conditions for special effects processing, then the video frame output by the P2 module is directly displayed and encoded.
[0130] S308, determining whether it is the Nth frame after executing FFD.
[0131] If the judgment result of S306 is "yes", that is, there are facial feature points in the previous frame, it can be determined whether the current frame is the Nth frame after the last FFD execution. If the judgment result is "yes", that is, the current frame is the Nth frame after the last FFD execution, S309 is executed; if the judgment result is "no", that is, the current frame is not the Nth frame after the last FFD execution, S310 is executed.
[0132] Taking N=10 as an example, if the video frame for which FFD was last executed is the 4th frame, then in this step, it can be determined whether the current frame is the 10th frame after the 4th frame, that is, whether the current frame is the 14th frame. If the current frame is the 14th frame, S309 is executed; if the current frame is not the 14th frame, S310 is executed.
[0133] S309, execute FFD.
[0134] If the judgment result of S304 is "yes", or the judgment result of S307 is "yes", or the judgment result of S308 is "yes", this step can be executed to perform FFD on the current frame.
[0135] S310, executing a tracking algorithm.
[0136] If the judgment result of S308 is "No", that is, the current frame is not the Nth frame after the last execution of FFD, then the tracking algorithm can be executed to track the facial feature points in the previous frame.
[0137] Taking N=10 as an example, assume that the video frame for which FFD was last performed is frame 4, and the current frame is frame 7. Since frame 7 is not the 10th frame after frame 4 (that is, not the 14th frame), the tracking algorithm can be executed on the current frame (frame 7) to track the facial feature points in the previous frame (frame 6).
[0138] S311, determine whether there are facial feature points.
[0139] In this step, it can be determined whether the current frame has facial feature points according to the result of executing FFD in S309 or the result of executing the tracking algorithm in S310. If the result of the determination is "yes", that is, the current frame has facial feature points, then S312 is executed; if the result of the determination is "no", that is, the current frame does not have facial feature points, then S313 is executed.
[0140] S312, performing special effects processing on the face.
[0141] If the judgment result of S311 is "yes", that is, the current frame has facial feature points, special effects processing (such as beautification, makeup, face slimming, etc.) can be performed on the face in the current frame according to the facial feature points.
[0142] S313, display and encoding.
[0143] In some embodiments, if S312 is executed, that is, special effects processing is performed on the face in the current frame, the video frame processed with special effects can be displayed and encoded.
[0144] In some embodiments, if S312 is not executed, the video frames output by the P2 module may be directly displayed and encoded.
[0145] According to the above implementation process, when the target is a human face, the special effects processing method provided by the embodiment of the present application may include the following technical means: giving one channel of data output by the P1 module of the system camera to the platform hardware to execute FD, and making a boundary condition judgment on the FD detection result; giving the boundary condition judgment result, the FD detection result (including information such as face angle, confidence, feature points, size coordinates, etc.) together with the result output by the P2 module to the live broadcast APP; the live broadcast APP makes a logical judgment on the information input by the system camera and the detection result of the previous frame of the APP (such as whether there is a face feature point in the previous frame) to determine the processing method for the current frame.
[0146] This method reuses the FD detection results of the platform hardware without increasing the power consumption of the platform, reduces the power consumption of the live broadcast APP, reduces the frequency of losing facial special effects (such as lost makeup), and improves the performance of facial special effects processing. Facial special effects processing can be performed when the first frame of valid face is detected.
[0147] It should be noted that the method of the embodiment of the present application can also be extended to make logical judgments on the system camera side. For example, the system camera can determine whether to perform special effects processing and whether to execute FFD / tracking algorithms, thereby outputting the judgment results of whether to perform special effects processing and whether to execute FFD / tracking algorithms.
[0148] It should also be noted that the method of the embodiment of the present application is not limited to live broadcast scenarios. For example, in application scenarios such as three-party short video recording, the method of the embodiment of the present application can also be applied to improve the performance of face special effects processing and reduce power consumption.
[0149] The above technical means solve the following problems in the prior art:
[0150] 1) In relatively static live broadcast scenarios, frequent use of the FFD algorithm results in high power consumption.
[0151] 2) The FFD algorithm is executed once every several frames, resulting in the inability to obtain facial effects for the first few frames of data when the anchor enters the live broadcast interface at a non-FFD moment.
[0152] 3) Since the tracking algorithm is not as accurate as the FFD algorithm, it may not be able to track facial feature points in motion scenes, resulting in loss of facial effects (such as makeup loss).
[0153] 4) Existing solutions usually dynamically control the number of intermediate frames (i.e., the number of frames for continuously executing the tracking algorithm) according to the state of detected face information. For example, when face information cannot be detected for multiple times, the number of frames for continuously executing the tracking algorithm will increase. In this way, if the FFD algorithm does not detect facial feature points in the previous execution, the intermediate frames for executing the tracking algorithm cannot be processed for facial special effects, resulting in a slow speed for obtaining facial special effects, and the time for obtaining facial special effects is unstable, sometimes fast and sometimes slow.
[0154] 5) The existing solution needs to wait until the FFD result of the live broadcast APP is obtained before starting to load special effect algorithm components such as beauty, makeup, and face slimming, resulting in a slow speed in obtaining facial special effects.
[0155] Through the above technical means, the following technical effects can be achieved:
[0156] 1) The scheduling mechanism of the FFD algorithm and tracking algorithm can be accurately controlled to prevent the loss of facial effects in moving scenes.
[0157] 2) The FFD activation mechanism has been optimized so that face effects processing can be advanced to the first valid face frame. That is, face effects processing can be performed when the first valid face frame is detected.
[0158] 3) The live broadcast APP can decide whether to start the special effects algorithm (including FFD, tracking algorithm, special effects processing, etc.) based on the FD detection results of each frame, which can greatly reduce power consumption.
[0159] 4) You can load special effects algorithm components such as beauty, makeup, and face slimming in advance without waiting for the FFD detection results of the APP, thereby improving algorithm performance and increasing the speed of obtaining facial special effects.
[0160] The preferred embodiments of the present application are described in detail above in conjunction with the accompanying drawings. However, the present application is not limited to the specific details in the above embodiments. Within the technical concept of the present application, the technical solution of the present application can be subjected to a variety of simple modifications, and these simple modifications all belong to the protection scope of the present application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present application will not further explain various possible combinations. For another example, the various different embodiments of the present application can also be arbitrarily combined, as long as they do not violate the idea of the present application, they should also be regarded as the contents disclosed in the present application. For another example, under the premise of no conflict, the various embodiments and / or the technical features in the various embodiments described in the present application can be arbitrarily combined with the prior art, and the technical solution obtained after the combination should also fall within the protection scope of the present application.
[0161] Based on the above embodiments, the embodiments of the present application provide a corresponding special effect processing device. The device includes the modules included, and the submodules included in each module, which can be implemented by a processor in a computer device with information processing capabilities; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.
[0162] The present application embodiment provides a special effects processing device. Figure 4 As shown, the special effect processing device 400 may include:
[0163] An acquisition unit 401 is used to acquire a current video frame; a first determination unit 402 is used to determine whether the current video frame contains the target based on a result of target detection performed on the current video frame; a second determination unit 403 is used to determine whether to perform feature point detection for the target on the current video frame based on whether the current video frame contains the target; wherein the feature point information obtained by performing feature point detection for the target on the current video frame is used to perform special effects processing on the target in the current video frame.
[0164] In some embodiments, the second determination unit 403 is specifically used for: when the current video frame is the first video frame, if the current video frame includes the target, determining to perform feature point detection for the target on the current video frame.
[0165] In some embodiments, the second determination unit 403 includes: a first determination subunit, used to determine whether a previous video frame of the current video frame has feature point information of the target if the current video frame contains the target, when the current video frame is not the first video frame; and a second determination subunit, used to determine whether to perform feature point detection for the target on the current video frame based on whether the previous video frame has feature point information of the target.
[0166] In some embodiments, the second determining subunit is specifically used to: determine to perform feature point detection for the target on the current video frame when the previous video frame does not have feature point information of the target.
[0167] In some embodiments, the second determination unit 403 is specifically used to: determine whether the target in the current video frame meets the conditions for performing special effects processing based on the result of performing the target detection on the current video frame; if the target in the current video frame meets the conditions for performing special effects processing, determine to perform feature point detection on the current video frame.
[0168] In some embodiments, the second determination subunit is specifically used for: when the previous video frame has feature point information of the target, if the current video frame is the Nth video frame after the last feature point detection for the target was performed, then determining to perform feature point detection for the target on the current video frame; wherein N is a positive integer.
[0169] In some embodiments, the second determination subunit is also used for: when the previous video frame has feature point information of the target, if the current video frame is not the Nth video frame after the last feature point detection for the target, then determining to execute a tracking algorithm on the current video frame; wherein the tracking algorithm is used to obtain feature point information of the target in the current video frame by tracking the feature points of the target in the previous video frame; the feature point information of the target in the current video frame is used to perform special effects processing on the target in the current video frame.
[0170] In some embodiments, the special effect processing device 400 further includes: a loading unit, configured to load an algorithm component required for the special effect processing when it is determined that the current video frame includes the target.
[0171] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules / units included in the device provided in the embodiment of the present application can be used to execute the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.
[0172] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific hardware, software or firmware, or any combination of hardware, software, and firmware.
[0173] An embodiment of the present application also provides a special effects processing device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.
[0174] The embodiment of the present application further provides a chip, which includes: a processor, which is used to call and run a computer program from a memory, so that a device equipped with the chip executes some or all of the steps in the above method.
[0175] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, some or all of the steps in the above method are implemented. The computer-readable storage medium can be transient or non-transient.
[0176] An embodiment of the present application also provides a computer program, including a computer-readable code. When the computer-readable code runs in a device, a processor in the device executes part or all of the steps in the above method.
[0177] The present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.
[0178] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, chip, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, chip, storage medium, computer program and computer program product of this application, please refer to the description of the method embodiment of this application for understanding.
[0179] The present application embodiment provides a special effects processing device. Figure 5 As shown, the special effect processing device 500 (hereinafter referred to as the device 500) includes a processor 510, and the processor 510 can call and run a computer program from a memory to implement the method in the embodiment of the present application.
[0180] In some embodiments, Figure 5 As shown, the device 500 may further include a memory 520. The processor 510 may call and run a computer program from the memory 520 to implement the method in the embodiment of the present application. The memory 520 may be a separate device independent of the processor 510, or may be integrated in the processor 510.
[0181] In some embodiments, Figure 5 As shown, the device 500 may also include a transceiver 530, and the processor 510 may control the transceiver 530 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 530 may include a transmitter and a receiver. The transceiver 530 may further include an antenna, and the number of antennas may be one or more.
[0182] It should be understood that "one embodiment", "an embodiment" or "some embodiments" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment", "in an embodiment" or "in some embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial number of each step / process mentioned above does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The serial numbers of the embodiments of the present application mentioned above are for description only and do not represent the advantages and disadvantages of the embodiments.
[0183] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0184] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units or modules is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0185] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0186] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0187] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.
[0188] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.
[0189] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.
Claims
1. A special effect processing method, characterized in that: The method comprises: Get the current video frame; Determining whether the current video frame contains the target according to a result of performing target detection on the current video frame; Determine whether to perform feature point detection on the target on the current video frame based on whether the current video frame contains the target; wherein the feature point information obtained by performing feature point detection on the current video frame is used to perform special effects processing on the target in the current video frame.
2. The method according to claim 1, characterized in that The determining whether to perform feature point detection for the target on the current video frame according to whether the current video frame contains the target includes: In the case that the current video frame is the first video frame, if the current video frame includes the target, it is determined to perform feature point detection for the target on the current video frame.
3. The method according to claim 1, characterized in that The determining whether to perform feature point detection for the target on the current video frame according to whether the current video frame contains the target includes: In the case where the current video frame is not the first video frame, if the current video frame includes the target, determining whether a previous video frame of the current video frame has feature point information of the target; Whether to perform feature point detection for the target on the current video frame is determined according to whether the previous video frame has feature point information of the target.
4. The method according to claim 3, characterized in that The determining whether to perform feature point detection for the target on the current video frame according to whether the previous video frame has feature point information of the target comprises: In a case where the previous video frame does not have feature point information of the target, it is determined to perform feature point detection for the target on the current video frame.
5. The method according to claim 2 or 4, characterized in that: The determining to perform feature point detection on the target on the current video frame includes: Determining, according to a result of performing the target detection on the current video frame, whether the target in the current video frame meets a condition for performing special effect processing; If the target in the current video frame meets the condition for performing special effect processing, it is determined to perform feature point detection on the target on the current video frame.
6. The method according to claim 3 or 4, characterized in that: The determining whether to perform feature point detection for the target on the current video frame according to whether the previous video frame has feature point information of the target comprises: In the case that the previous video frame has feature point information of the target, if the current video frame is the Nth video frame after the last feature point detection for the target is performed, it is determined to perform feature point detection for the target on the current video frame; wherein N is a positive integer.
7. The method according to claim 6, characterized in that The method further comprises: In the case where the previous video frame has feature point information of the target, if the current video frame is not the Nth video frame after the feature point detection for the target was last performed, determining to perform a tracking algorithm on the current video frame; Among them, the tracking algorithm is used to obtain feature point information of the target in the current video frame by tracking the feature points of the target in the previous video frame; the feature point information of the target in the current video frame is used to perform special effects processing on the target in the current video frame.
8. The method according to any one of claims 1 to 4, characterized in that The method further comprises: When it is determined that the current video frame contains the target, an algorithm component required for the special effect processing is loaded.
9. A special effects processing device, characterized in that: The device comprises: An acquisition unit, used for acquiring a current video frame; A first determining unit, configured to determine whether the current video frame contains the target according to a result of performing target detection on the current video frame; The second determination unit is used to determine whether to perform feature point detection on the target on the current video frame according to whether the current video frame contains the target; wherein the feature point information obtained by performing feature point detection on the current video frame is used to perform special effects processing on the target in the current video frame.
10. A special effects processing device, characterized in that: The device comprises: A memory for storing computer executable instructions; A processor, connected to the memory, configured to implement the method according to any one of claims 1 to 8 by executing the computer executable instructions.
11. A chip, characterized in that: The chip comprises: A processor, configured to call and run a computer program from a memory, so that a device equipped with the chip executes the method according to any one of claims 1 to 8.
12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by at least one processor, the method according to any one of claims 1 to 8 is implemented.
13. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the method according to any one of claims 1 to 8 is implemented.