Video processing method, device, electronic device and storage medium
By mirroring the live video screen on the anchor side, removing and processing sticker special effects elements, the mirrored video streaming data and additional special effects data are pushed to the audience side, which solves the garbled problem when using mirror special effects and stickers, and improves the video The accuracy of the presentation.
Patent Information
- Application Number
- CN202111017357.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-08-31
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-08-31
AI Technical Summary
When mirroring special effects are used in combination with stickers, garbled information that is difficult to understand by the audience is likely to appear in the video content after the mirror is flipped, resulting in the problem of errors in display elements, which is not effectively solved by the existing technology.
When mirroring the live video screen on the anchor side, remove the target sticker special effect elements and perform mirroring processing to generate mirror candidate screens, and push the mirror video stream data and additional special effect data of the target sticker special effect elements to the audience side to avoid direct mirroring Dealing with presentation errors.
It realizes the normal rendering and display of target sticker special effects elements in the mirrored video screen, improving the accuracy of content display, and avoiding error problems caused by direct mirroring processing of mirror-sensitive sticker special effects elements.
Smart Images

Figure CN113793410B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the computer field, and in particular to a video processing method, device, electronic device and storage medium. Background Art
[0002] Nowadays, many anchors often use some special effects to increase the traffic of the current content during the video. Common special effects include: mirror and sticker. Among them, the mirror effect here is to flip the video screen in the video stream left and right.
[0003] When the mirror effect is used in combination with stickers, usually when the original video captured by the anchor's camera is mirrored and flipped, the stickers attached to the video will also be mirrored and flipped at the same time, but many stickers contain elements that are sensitive to left-right flipping (such as Chinese and English characters, numbers, etc.). In this way, the content presented in the mirrored flipped video will be difficult for the audience to understand and receive, which seriously affects the presentation effect of the video.
[0004] That is to say, when the video processing method provided by the related technology is adopted, garbled information that is difficult to understand will appear in the displayed content, resulting in the problem of display element errors.
[0005] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention
[0006] The present disclosure provides a video processing method, device, electronic device and storage medium to at least solve the problem that the video processing method provided by the related art will display garbled information that is difficult to understand, resulting in display element errors. The technical solution of the present disclosure is as follows:
[0007] According to a first aspect of an embodiment of the present disclosure, a video processing method is provided, comprising: displaying a live video screen on an anchor end, wherein the live video screen includes a scene screen input by a camera of the anchor end, and a sticker special effect element superimposed on the scene screen; in response to a mirror processing instruction for the live video screen, mirror processing is performed on a candidate screen extracted from the live video screen to obtain a mirror candidate screen, wherein the candidate screen is a video screen obtained by removing a target sticker special effect element from the live video screen, and a difference between the mirror sticker special effect element obtained after the target sticker special effect element is mirrored and the target sticker special effect element is greater than a target threshold; and the mirror video stream data corresponding to the mirror candidate screen and the additional special effect data corresponding to the target sticker special effect element are pushed to an audience end, so that the audience end displays a screen obtained by integrating the mirror candidate screen with the target sticker special effect element.
[0008] As an optional embodiment, before mirroring the candidate picture extracted from the above-mentioned live video picture to obtain the mirrored candidate picture, it also includes: when the above-mentioned sticker special effect element superimposed on the above-mentioned scene picture includes the above-mentioned target sticker special effect element and the basic sticker special effect element, merging the above-mentioned scene picture with the above-mentioned basic sticker special effect element to obtain the above-mentioned candidate picture, wherein the difference between the mirrored sticker special effect element obtained after the above-mentioned basic sticker special effect element is mirrored and the above-mentioned basic sticker special effect element is less than or equal to the above-mentioned target threshold; when the above-mentioned sticker special effect element superimposed on the above-mentioned scene picture is the above-mentioned target sticker special effect element, the above-mentioned scene picture input by the camera of the above-mentioned anchor end is used as the above-mentioned candidate picture.
[0009] As an optional embodiment, after mirroring the candidate picture extracted from the above-mentioned live video picture to obtain the mirror candidate picture, it also includes: determining the facial key point position associated with the above-mentioned target sticker special effect element from the face area presented in the above-mentioned live video picture; performing feature extraction on the above-mentioned facial key point position to obtain facial feature information; determining the facial picture area corresponding to the above-mentioned facial feature information in the above-mentioned mirror candidate picture; and generating the above-mentioned mirror video stream data based on the above-mentioned facial picture area.
[0010] As an optional embodiment, after extracting features from the above-mentioned facial key point positions to obtain facial feature information, it also includes: storing the coordinate information of the pixel points corresponding to each of the above-mentioned facial key point positions in each data bit corresponding to the integer type variable.
[0011] As an optional embodiment, the above-mentioned storing the coordinate information of the pixel points corresponding to each of the above-mentioned facial key point positions in the respective data bits corresponding to the integer type variable includes: storing the horizontal coordinate information of the above-mentioned pixel points in the first sequence data bits of the above-mentioned integer type variable, and storing the vertical coordinate information of the above-mentioned pixel points in the second sequence data bits of the above-mentioned integer type variable.
[0012] According to a second aspect of an embodiment of the present disclosure, a video processing method is provided, including: receiving mirror video stream data and additional special effect data corresponding to a target sticker special effect element sent by a host end, wherein the difference between a mirror sticker special effect element obtained after the target sticker special effect element is mirrored and the target sticker special effect element is greater than a target threshold; parsing a mirror candidate picture from the mirror video stream data, wherein the mirror candidate picture is a picture obtained after mirroring the candidate picture, and the candidate picture is a video picture obtained after removing the target sticker special effect element from a live video picture; rendering the target sticker special effect element onto the mirror candidate picture according to the additional special effect data to obtain a mirror video picture.
[0013] As an optional embodiment, the above-mentioned rendering of the target sticker special effect element onto the above-mentioned mirror candidate screen according to the above-mentioned additional special effect data to obtain the mirror video screen includes: obtaining the first display area corresponding to the above-mentioned target sticker special effect element from the above-mentioned additional special effect data, wherein the above-mentioned first display area is the display area of the above-mentioned target sticker special effect element in the video screen displayed on the above-mentioned anchor end; determining the second display area corresponding to the above-mentioned first display area on the audience end after the mirror processing; when displaying the above-mentioned mirror candidate screen, rendering the above-mentioned target sticker special effect element in the above-mentioned second display area to display the above-mentioned mirror video screen.
[0014] As an optional embodiment, when displaying the mirror video screen, it includes: when receiving the mirror video stream data according to a preset transmission frame rate, performing linear interpolation processing between any two frames of the mirror video screen displayed.
[0015] According to a third aspect of an embodiment of the present disclosure, a video processing device is provided, including: a display unit, configured to display a live video screen on an anchor end, wherein the live video screen includes a scene screen input by a camera of the anchor end, and a sticker special effect element superimposed on the scene screen; a mirror processing unit, configured to perform mirror processing on a candidate screen extracted from the live video screen in response to a mirror processing instruction for the live video screen, to obtain a mirror candidate screen, wherein the candidate screen is a video screen obtained by removing a target sticker special effect element from the live video screen, and a difference between the mirror sticker special effect element obtained after the target sticker special effect element is mirrored and the target sticker special effect element is greater than a target threshold; a push unit, configured to push the mirror video stream data corresponding to the mirror candidate screen and the additional special effect data corresponding to the target sticker special effect element to the audience end, so that the audience end displays a screen obtained by integrating the mirror candidate screen and the target sticker special effect element.
[0016] As an optional embodiment, it also includes: a first determination unit, configured to perform mirroring processing on the candidate picture extracted from the above-mentioned live video picture to obtain the mirrored candidate picture, and when the above-mentioned sticker special effect element superimposed on the above-mentioned scene picture includes the above-mentioned target sticker special effect element and the basic sticker special effect element, merge the above-mentioned scene picture with the above-mentioned basic sticker special effect element to obtain the above-mentioned candidate picture, wherein the difference between the mirrored sticker special effect element obtained after the above-mentioned basic sticker special effect element is mirrored and the above-mentioned basic sticker special effect element is less than or equal to the above-mentioned target threshold; a second determination unit, configured to use the above-mentioned scene picture input by the camera of the above-mentioned anchor end as the above-mentioned candidate picture when the above-mentioned sticker special effect element superimposed on the above-mentioned scene picture is the above-mentioned target sticker special effect element.
[0017] As an optional embodiment, it also includes: a third determination unit, which is configured to perform mirror processing on the candidate picture extracted from the above-mentioned live video picture to obtain the mirror candidate picture, and then determine the facial key point position associated with the above-mentioned target sticker special effect element from the face area presented in the above-mentioned live video picture; an extraction unit, which is configured to perform feature extraction on the above-mentioned facial key point position to obtain facial feature information; a fourth determination unit, which is configured to determine the facial picture area corresponding to the above-mentioned facial feature information in the above-mentioned mirror candidate picture; and a generation unit, which is configured to generate the above-mentioned mirror video stream data based on the above-mentioned facial picture area.
[0018] As an optional embodiment, it also includes: a storage unit, which is configured to store the coordinate information of the pixel points corresponding to each of the above-mentioned facial key point positions in each data bit corresponding to the integer type variable after extracting the features of the above-mentioned facial key point positions to obtain the facial feature information.
[0019] As an optional embodiment, the above-mentioned storage unit includes: a storage module, configured to store the horizontal coordinate information of the above-mentioned pixel point in the first sequence data bit of the above-mentioned integer type variable, and store the vertical coordinate information of the above-mentioned pixel point in the second sequence data bit of the above-mentioned integer type variable.
[0020] According to a fourth aspect of an embodiment of the present disclosure, a video processing device is provided, including: a receiving unit, configured to receive mirror video stream data and additional special effect data corresponding to a target sticker special effect element sent by a host end, wherein the difference between a mirror sticker special effect element obtained after the target sticker special effect element is mirrored and the target sticker special effect element is greater than a target threshold; a parsing unit, configured to parse a mirror candidate picture from the mirror video stream data, wherein the mirror candidate picture is a picture obtained after mirroring the candidate picture, and the candidate picture is a video picture obtained after removing the target sticker special effect element from a live video picture; a rendering unit, configured to render the target sticker special effect element onto the mirror candidate picture according to the additional special effect data to obtain a mirror video picture.
[0021] As an optional embodiment, the rendering unit includes: an acquisition module, configured to acquire the first display area corresponding to the target sticker special effect element from the additional special effect data, wherein the first display area is the display area of the target sticker special effect element in the video screen displayed on the host end; a determination module, configured to determine the second display area corresponding to the first display area on the audience end after the first display area is mirrored; a rendering module, configured to render the target sticker special effect element in the second display area when displaying the mirror candidate screen to display the mirror video screen.
[0022] As an optional embodiment, the rendering unit further includes: a processing module configured to perform linear interpolation processing between any two displayed frames of the mirror video images when receiving the mirror video stream data at a preset transmission frame rate.
[0023] According to a fifth aspect of an embodiment of the present disclosure, there is also provided an electronic device, comprising a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the video processing method.
[0024] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor in an electronic device, the electronic device is enabled to perform the above-mentioned video processing method.
[0025] According to a seventh aspect of an embodiment of the present disclosure, there is provided a computer program product, comprising a computer program, wherein the computer program implements the video processing method when executed by a processor.
[0026] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:
[0027] In an embodiment of the present invention, a live video screen is displayed on the host side, wherein the live video screen includes a scene screen input by the camera of the host side, and a sticker special effect element superimposed on the scene screen; in response to a mirror processing instruction for the live video screen, a candidate screen extracted from the live video screen is mirrored to obtain a mirror candidate screen, wherein the candidate screen is a video screen after the target sticker special effect element is removed, and the target sticker special effect element is a sticker special effect element with a large difference before and after the mirror. The mirror video stream data corresponding to the mirror candidate screen and the additional special effect data corresponding to the target sticker special effect element are pushed to the audience side, so that the audience side displays the screen obtained by integrating the mirror candidate screen and the target sticker special effect element. Thereby, the target sticker special effect element is rendered and displayed normally in the mirror video screen, and the target sticker special effect element that is sensitive to the mirror is directly mirrored, which leads to the problem of incorrect display of the sticker special effect element. Thereby, the accuracy of content display is improved.
[0028] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0030] Figure 1 is a schematic diagram of an application environment of an optional video processing method according to an embodiment of the present disclosure;
[0031] Figure 2 is a flowchart of an optional video processing method according to an embodiment of the present disclosure;
[0032] Figure 3 is a schematic diagram of an optional video processing method according to an embodiment of the present disclosure;
[0033] Figure 4 is a schematic diagram of another optional video processing method according to an embodiment of the present disclosure;
[0034] Figure 5 is a schematic diagram of another optional video processing method according to an embodiment of the present disclosure;
[0035] Figure 6 is a schematic diagram of another optional video processing method according to an embodiment of the present disclosure;
[0036] Figure 7 is a schematic diagram of another optional video processing method according to an embodiment of the present disclosure;
[0037] Figure 8 is a flowchart of another optional video processing method according to an embodiment of the present disclosure;
[0038] Fig. 9 is a structural schematic diagram of an optional video processing device according to an embodiment of the present disclosure;
[0039] Fig.10 It is a structural schematic diagram of another optional video processing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0040] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.
[0041] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0042] According to one aspect of an embodiment of the present invention, a video processing method is provided. Optionally, the video processing method can be but is not limited to being applied to Figure 1In the video processing system in the application environment shown, the system includes: a terminal device 102, a network 104 and a terminal device 106. Among them, the terminal device 102 is a hardware device used by the anchor end, and runs a client logged in using the anchor account. After collecting the desired scene picture, it can be shared with other audience accounts that follow the anchor account through the platform. The terminal device 102 includes a human-computer interaction screen 1022, a processor 1024 and a processor 1026. The human-computer interaction screen 1022 is used to display the above video picture and is also used to detect touch operations. The processor 1024 is used to generate a corresponding human-computer interaction instruction according to the detected touch operation, so as to perform background operations according to the human-computer interaction instruction. The memory 1026 is used to store the above video picture. The terminal device 106 is a hardware device used by the audience end, and runs a playback client logged in using the audience account, which is used to play the content shared by the anchor end. The terminal device 108 includes a human-computer interaction screen 1062, a processor 1064 and a processor 1066. The human-computer interaction screen 1062 is used to display the above mirror video picture. The processor 1024 is used to render the target sticker special effect element on the parsed mirror candidate picture to generate the above-mentioned mirror video picture. The memory 1026 is used to store the above-mentioned mirror video picture.
[0043] The implementation is specifically carried out by the following steps: In step S102, the terminal device 102 displays a live video screen, such as Figure 1 As shown, the live video screen displays a target object with a baseball cap facing the right, and a combination of a sun sticker and a text sticker "You are awesome!" The terminal device 102 responds to the touch operation of mirroring the live video screen displayed on the human-computer interaction screen 1022, and mirrors the candidate screen extracted from the live video screen to obtain a mirrored candidate screen, as shown in step S104.
[0044] When the mirror video stream data obtained based on the mirror candidate screen and the additional special effect data corresponding to the target sticker special effect element (such as the above-mentioned text sticker) are obtained, step S106 is executed to push the mirror video stream data obtained based on the mirror candidate screen and the additional special effect data corresponding to the target sticker special effect element to the terminal device 106 through the network 104. Steps S108-S110 are executed in the terminal device 106 to parse the mirror candidate screen from the above-mentioned mirror video stream data, and render the target sticker special effect element onto the above-mentioned mirror candidate screen according to the additional special effect data to obtain the mirror video screen. Figure 1As shown in the mirrored video screen, a target object with a baseball cap facing left is displayed, and a sun sticker and a text sticker "You are awesome!" are displayed in combination. The sun sticker is mapped from the left side of the screen to the right side of the screen due to the mirroring process, while the text sticker, which is a target sticker special effect element, is rendered locally in the terminal device 106 and will not be mirrored.
[0045] It should be noted that in this embodiment, a live video screen is displayed on the host side, wherein the live video screen includes a scene screen input by the camera of the host side, and a sticker special effect element superimposed on the scene screen; in response to the mirror processing instruction of the live video screen, the candidate screen extracted from the live video screen is mirrored to obtain a mirror candidate screen, where the candidate screen is a video screen after the target sticker special effect element is removed, and the target sticker special effect element is a sticker special effect element with a large difference before and after the mirror. The mirror video stream data corresponding to the mirror candidate screen and the additional special effect data corresponding to the target sticker special effect element are pushed to the audience side, so that the audience side displays the screen obtained by integrating the mirror candidate screen and the target sticker special effect element. Thereby, the target sticker special effect element is rendered and displayed normally in the mirror video screen, avoiding the direct mirroring of the target sticker special effect element that is sensitive to the mirror, resulting in the error of the special effect element display. Thereby achieving the effect of improving the accuracy of content display.
[0046] Optionally, in this embodiment, the terminal device may be a terminal device configured with a target client, which may include but is not limited to at least one of the following: a mobile phone (such as an Android phone, an iOS phone, etc.), a laptop, a tablet computer, a PDA, a MID (Mobile Internet Devices), a PAD, a desktop computer, a smart TV, etc. The target client may be a video client, an instant messaging client, a browser client, an education client, or other online-supported clients. The network may include but is not limited to: a wired network, a wireless network, wherein the wired network includes: a local area network, a metropolitan area network, and a wide area network, and the wireless network includes: Bluetooth, WIFI, and other networks that implement wireless communication. The server may be a single server, or a server cluster consisting of multiple servers, or a cloud server. The above is only an example, and this embodiment does not impose any limitation on this.
[0047] Optionally, as an optional implementation, as Figure 2 As shown, the above video processing method includes:
[0048] S202, displaying a live video screen on the host end, wherein the live video screen includes a scene screen input by a camera of the host end and sticker special effect elements superimposed on the scene screen;
[0049] S204, in response to the mirror processing instruction for the live video screen, mirror processing is performed on the candidate screen extracted from the live video screen to obtain a mirror candidate screen, wherein the candidate screen is a video screen obtained by removing the target sticker special effect element from the live video screen, and the difference between the mirror sticker special effect element obtained after the target sticker special effect element is mirrored and the target sticker special effect element is greater than the target threshold;
[0050] S206, pushing the mirror video stream data corresponding to the mirror candidate picture and the additional special effect data corresponding to the target sticker special effect element to the viewer end, so that the viewer end displays the picture obtained by integrating the mirror candidate picture and the target sticker special effect element.
[0051] Optionally, in this embodiment, the above video processing method can be widely used in scenes such as mobile phone front camera, short video shooting, video, etc., but not limited to. The video scenes here can include but are not limited to: online education, online food broadcasting, online shopping, etc. In addition, in this embodiment, the timing of the above-mentioned mirror processing of the collected scene images can include but are not limited to: camera collection, anchor presentation, push streaming, etc. The execution subject of the video processing method here can be Figure 1 The terminal device 102 shown is used as the host terminal in the process. The above is an example and is not limited in this embodiment.
[0052] Optionally, in this embodiment, the target sticker special effect element may be, but is not limited to, a sticker special effect element that is more sensitive to mirror processing, such as text special effects, such as Chinese and English characters, numbers and other elements. Among them, the text special effects may be, but are not limited to, sticker special effects, which refers to drawing additional pictures or videos at specific positions on the screen. Therefore, the additional special effect data corresponding to the target sticker special effect element here will be pushed to the audience end for playback as separate business data and mirror video stream data.
[0053] In addition, in this embodiment, it is also possible but not limited to combine artificial intelligence (AI) technology to identify the facial feature information corresponding to the position of the key points of the face in the face area of the live video screen, and push the mirror video stream data generated by the face screen area determined based on the above facial feature information and the additional special effect data corresponding to the target sticker special effect element to the audience. This makes it easier for the audience to add the target sticker special effect element to the corresponding position in the face area more accurately when rendering it locally, and then in the dynamic video effect, make it compatible with various facial postures (such as frontal face posture, side face posture, etc.), and achieve the display effect of face stickers.
[0054] Optionally, in this embodiment, the sticker special effect elements configured in the above video screen may also include basic sticker special effect elements. Here, the basic sticker special effect elements are elements that are insensitive to mirror processing, such as sticker special effect elements that are symmetrical in themselves, for example, Figure 1 Sun sticker shown.
[0055] Optionally, in this embodiment, when the anchor pushes the mirrored video stream data and the additional special effect data corresponding to the target sticker special effect element to the audience, the transmission rate can be reduced based on the network conditions, such as reduced to 5 to 10 frames per second. In this way, the amount of data transmitted can be reduced from the original 101 points / frame*8bit / point*15 frames / second=12000bit / second to 5 points / frame*4bit / point*10 frames / second=200bit / second. This achieves nearly 60 times of traffic optimization.
[0056] Specific combination Figure 3 The example shown is used to illustrate: Assume that the host's camera inputs a scene image, and adds basic special effects (basic sticker special effects elements) and additional special effects (target sticker special effects elements). Therefore, the host's screen will present a video image including the above scene image, basic special effects and additional special effects. The display effect here can be as follows: Figure 4 The corresponding screen content of the anchor side is shown on the left.
[0057] In response to the mirror processing instruction for the live video screen, the candidate screen including the scene screen and the basic special effects is mirrored to obtain the mirror candidate screen. Then, the mirror video stream data based on the mirror candidate screen and the special effect data of the above-mentioned additional special effects are pushed to the viewer through the network as push content. The viewer performs the process of rendering the additional special effects onto the parsed mirror candidate screen locally, so that the screen of the viewer presents the mirror video screen including the above-mentioned mirror candidate screen and the additional special effects. The display effect here can be as follows: Figure 4 The corresponding screen content of the audience end is shown on the right.
[0058] It should be noted that the timing of mirroring the candidate images can be combined with Figure 5 The following are provided for illustration and may include:
[0059] For the host end: after the camera inputs the scene picture, the scene picture is first mirrored, and after the basic sticker special effect elements are obtained and before they are presented on the host end, the basic sticker special effect elements are mirrored again; or, after the basic sticker special effect elements are obtained and before they are presented on the host end, the scene picture and the basic sticker special effect elements are mirrored once.
[0060] For the audience: after the camera inputs the scene image, the scene image is first mirrored. After the basic sticker special effect elements are obtained and before the network is pushed, the basic sticker special effect elements are mirrored again; or, before presenting it to the audience, the scene image and the basic sticker special effect elements are mirrored once.
[0061] That is to say, after adopting the method provided in this embodiment, since the anchor end transmits the mirror candidate picture and the target sticker special effect element separately to the audience end, the audience end can complete the rendering of the target sticker special effect element locally, instead of uniformly mirroring all elements in the video picture, thereby avoiding the garbled information that appears after the target sticker special effect element is mirrored, thereby improving the video processing effect.
[0062] Through the embodiments provided by the present application, in response to the mirror processing instruction for the live video screen, the candidate screen extracted from the live video screen is mirrored to obtain a mirror candidate screen, wherein the candidate screen eliminates the target sticker special effect element in the live video screen, where the target sticker special effect element is a mirror-sensitive element, and the difference between the mirror sticker special effect element obtained after the mirror processing and the target sticker special effect element will be greater than the target threshold. The mirror video stream data obtained based on the mirror candidate screen and the additional special effect data corresponding to the target sticker special effect element are pushed to the audience end, so that the audience end displays the mirror video screen corresponding to the live video screen. Thereby, the target sticker special effect element is rendered and displayed normally in the mirror video screen, and the mirror processing of the target sticker special effect element that is sensitive to the mirror is avoided, resulting in the display error of the special effect element, thereby achieving the effect of improving the accuracy of content display.
[0063] As an optional solution, before performing mirror processing on the candidate picture extracted from the live video picture to obtain the mirror candidate picture, the method further includes:
[0064] 1) When the sticker special effect element superimposed on the scene screen includes a target sticker special effect element and a basic sticker special effect element, the scene screen and the basic sticker special effect element are merged to obtain a candidate screen, wherein the difference between the mirror sticker special effect element obtained by mirroring the basic sticker special effect element and the basic sticker special effect element is less than or equal to the target threshold;
[0065] 2) When the sticker special effect element superimposed on the scene screen is the target sticker special effect element, the scene screen input by the camera of the anchor end is used as the candidate screen.
[0066] It should be noted that, in the present embodiment, the above-mentioned video screen may include the scene screen captured by the camera, and the sticker special effect elements configured in the anchor end. Among them, the sticker special effect elements here may be, but are not limited to, elements that are additionally superimposed on the above-mentioned scene screen, and different sticker special effect elements may be, but are not limited to, drawn in layers on the scene screen. For example, the above-mentioned basic sticker special effect elements and target sticker special effect elements may be, but are not limited to, drawn in different layers. The above-mentioned video screen may include basic sticker special effect elements, or may not include basic sticker special effect elements. The basic sticker special effect elements here are sticker special effect elements that are insensitive to mirroring, and the difference in changes after being mirrored is small, which does not affect the user's understanding and viewing.
[0067] For example, assuming that the host side is still the host side and the audience side is still the audience side, the video screen displayed by the host side includes: the scene screen input by the camera, basic special effects, and additional special effects (i.e., target sticker special effect elements). Then, the scene screen input by the camera can be merged with the basic special effects to obtain a candidate screen, and then multiple candidate screens can be compressed to obtain a video stream, which is pushed to the remote audience side through the network. Among them, the above-mentioned additional special effects are elements that are sensitive to mirror processing. In order to avoid being mirrored, they will be integrated into the network push stream as a separate part of the push content.
[0068] Through the embodiments provided by the present application, when there are basic sticker special effects elements in the video screen, the scene screen captured by the camera is merged with the basic sticker special effects elements to obtain a candidate screen; and when the video screen does not include basic sticker special effects elements, the scene screen captured by the camera is used as the candidate screen. By mirroring the candidate screen, it is avoided that the target sticker special effects elements that are sensitive to mirroring are directly mirrored, resulting in the display error of the special effects elements. This achieves the effect of improving the accuracy of content display.
[0069] As an optional solution, after the candidate picture extracted from the live video picture is subjected to mirror processing to obtain the mirror candidate picture, the method further includes:
[0070] S1, determining the position of the key points of the face associated with the target sticker special effect element from the face area presented in the live video screen;
[0071] S2, extracting features of key points of the face to obtain facial feature information;
[0072] S3, determining a face picture area corresponding to the face feature information in the mirror candidate picture;
[0073] S4, generating mirror video stream data based on the face picture area.
[0074] It should be noted that although the garbled code problem caused by mirror flipping can be solved by rendering the target sticker special effect element separately, some target sticker special effect elements may depend on the face area of the target object presented in the video screen (that is, there is an association relationship). For example, if the target sticker special effect element is a face sticker, as the face posture (such as front face, side face, etc.) changes, the display of the face area will also change accordingly. The display position of the above target sticker special effect element will also change with the change of the face area. The type needs to rely on the processing module to determine where to render in the face area.
[0075] Optionally, in this embodiment, the facial key point position associated with the target sticker special effect element can be determined from the face area presented in the live video screen, but is not limited to, and then the face screen area determined based on the facial feature information extracted from the facial key point position is used to generate the mirror video stream data. Among them, the facial key point position associated with the target sticker special effect element can be, but is not limited to, used to determine the display position of the target sticker special effect element with a face-following effect.
[0076] It should be noted that the face area often contains multiple face key points, but the main purpose of this embodiment is to realize the push display of the target sticker special effect element. Therefore, for this scenario, during the transmission process, it is possible but not limited to transmitting the face picture area determined according to the position of the face key points associated with the target sticker special effect element, without transmitting the entire content in the face area, thereby achieving the effect of reducing the transmission amount.
[0077] The mainstream business algorithms in the related technology output about 100 facial key points, and each facial key point can be represented by, but not limited to, two floating-point numbers to represent its X-axis and Y-axis coordinates respectively.
[0078] For example, Figure 6 As shown, the anchor end (i.e., the anchor end) can be but not limited to using five facial key points, namely, the sides of the face (A, B), the chin (E), the center of the eyebrows (B), and the tip of the nose (D) to locate the target sticker special effect element with a face-following effect. In other words, the relative position relationship between the target sticker special effect element and the above facial key points is obtained, so that the audience end (i.e., the audience end) can accurately locate the current facial posture based on the above five facial key points, as well as the corresponding display position of the target sticker special effect element under the facial posture. For example, the head width (AC) is calculated by combining the left and right foreheads, the head height (BE) is calculated by the center of the eyebrows (B) and the lower chin (E), and the distance between the tip of the nose (D) and the sides of the face (A, B) is calculated to calculate the orientation of the face and other information (BD:DE and AB:BC).
[0079] In addition to the above-mentioned positioning mode using five facial key points, more or fewer key points can be transmitted according to business needs to produce more detailed special effects, such as transmitting all facial key points to perform facial liquefaction beautification and makeup processing.
[0080] For example, Figure 7 As shown, when the host side pushes the network stream to the audience side, when it is determined that the target sticker special effect element (i.e., additional special effect) is configured, the facial key point position associated with the target sticker special effect element can be determined from the face area presented in the live video screen, and the face screen area can be determined based on the facial feature information extracted from the facial key point position, and then the mirror video stream data generated based on the face screen area (such as the accompanying facial AI data shown in the figure) is pushed and sent to the audience side.
[0081] Through the embodiments provided by the present application, when the target sticker special effect element is associated with the face area presented in the live video screen, the face screen area of the face key point position associated with the target sticker special effect element determined from the face area and the target sticker special effect element are compressed together to generate mirror video stream data. This achieves the purpose of reducing the amount of live transmission data and improving transmission efficiency.
[0082] As an optional solution, after extracting the features of the key points of the face and obtaining the face feature information, the following is also included:
[0083] S1, store the coordinate information of the pixel points corresponding to each facial key point position into the data bits corresponding to the integer type variable respectively.
[0084] It should be noted that when the above key feature points are accurate to the pixel level, considering that the screen of the mobile terminal has a pixel ratio of 2000*3000, in this embodiment, a single integer type variable int can be used but is not limited to represent the coordinates of a pixel point.
[0085] Through the embodiments provided by the present application, by using integer type variables to store the coordinate information of the target sticker special effect element, not only the position deviation is small, but also the storage space is small, which further reduces the amount of data processing.
[0086] As an optional solution, the coordinate information of the pixel points corresponding to the position of each facial key point is stored in the data bits corresponding to the integer type variable, including: storing the horizontal coordinate information of the pixel points in the first sequence data bits of the integer type variable, and storing the vertical coordinate information of the pixel points in the second sequence data bits of the integer type variable.
[0087] For example, a value within 100 million is used to represent the coordinate information of a key feature point, where the tens of millions to tens of thousands are used to represent the X-axis pixel coordinates, and the thousands to ones are used to represent the Y-axis pixel coordinates. Each coordinate component has a total of 10,000 subdivisions to cover the pixel positions of most mobile terminals and avoid large position deviations.
[0088] Through the embodiments provided by the present application, the horizontal coordinate information and the vertical coordinate information of the target sticker special effect element are respectively stored in different sequence data bits of the integer type variable, so that the transmission is achieved by using one variable and the storage occupies a small space, further reducing the data processing amount.
[0089] According to another aspect of an embodiment of the present invention, a video processing method is provided. Optionally, Figure 8 As shown, the above video processing method includes:
[0090] S802, receiving the mirrored video stream data and the additional special effect data corresponding to the target sticker special effect element sent by the anchor end, wherein the difference between the mirrored sticker special effect element obtained after the target sticker special effect element is mirrored and the target sticker special effect element is greater than the target threshold;
[0091] S804, parsing the mirrored video stream data to obtain a mirrored candidate picture, wherein the mirrored candidate picture is a picture obtained by mirroring the candidate picture, and the candidate picture is a video picture obtained by removing the target sticker special effect element from the live video picture;
[0092] S806, rendering the target sticker special effect element onto the mirror candidate picture according to the additional special effect data to obtain a mirror video picture.
[0093] Optionally, in this embodiment, the above video processing method can be widely used in scenes such as mobile phone front camera, short video shooting, video, etc., but not limited to. The video scenes here can include but are not limited to: online education, online food broadcasting, online shopping, etc. In addition, in this embodiment, the timing of the above-mentioned mirror processing of the collected scene images can include but are not limited to: camera collection, anchor presentation, push streaming, etc. The execution subject of the video processing method here can be Figure 1 The terminal device 106 shown is used as the viewer terminal in the process. The above is an example and is not limited in this embodiment.
[0094] Optionally, in this embodiment, the target sticker special effect element may be, but is not limited to, a sticker special effect element that is more sensitive to mirror processing, such as text special effects, such as Chinese and English characters, numbers and other elements. Among them, the text special effects may be, but are not limited to, sticker special effects, which refers to drawing additional pictures or videos at specific positions on the screen. Therefore, the additional special effect data corresponding to the target sticker special effect element here will be pushed to the audience end for playback as separate business data and mirror video stream data.
[0095] In addition, in this embodiment, it is also possible but not limited to combine artificial intelligence (AI) technology to identify the facial feature information corresponding to the position of the key points of the face in the face area of the live video screen, and push the mirror video stream data generated by the face screen area determined based on the above facial feature information and the additional special effect data corresponding to the target sticker special effect element to the audience. This makes it easier for the audience to add the target sticker special effect element to the corresponding position in the face area more accurately when rendering it locally, and then in the dynamic video effect, make it compatible with various facial postures (such as frontal face posture, side face posture, etc.), and achieve the display effect of face stickers.
[0096] Optionally, in this embodiment, the sticker special effect elements configured in the above video screen may also include basic sticker special effect elements. Here, the basic sticker special effect elements are elements that are insensitive to mirror processing, such as sticker special effect elements that are symmetrical in themselves, for example, Figure 1 Sun sticker shown.
[0097] Through the embodiment provided by the present application, when the mirror video stream data and the additional special effect data corresponding to the target sticker special effect element sent by the anchor end are received, the mirror candidate screen is parsed from the mirror video stream data, and the above target sticker special effect element is rendered onto the mirror candidate screen according to the additional special effect data to obtain the mirror video screen, and the above mirror video screen is displayed on the audience end. Thus, the target sticker special effect element is rendered and displayed normally in the mirror video screen, avoiding the mirror processing of the target sticker special effect element that is sensitive to the mirror, resulting in the display error of the special effect element, thereby achieving the effect of improving the accuracy of content display.
[0098] As an optional solution, rendering the target sticker special effect element onto the mirror candidate screen according to the additional special effect data to obtain the mirror video screen includes:
[0099] S1, obtaining a first display area corresponding to a target sticker special effect element from the additional special effect data, wherein the first display area is a display area of the target sticker special effect element in a video screen displayed on the anchor end;
[0100] S2, determining a second display area on the viewer end corresponding to the first display area after the first display area is mirrored;
[0101] S3, when displaying the mirror candidate picture, rendering the target sticker special effect element in the second display area to obtain the mirror video picture.
[0102] It should be noted that, due to the different types of playback application clients used in the viewer end (i.e., the viewer end) and different display resolutions, in this embodiment, in order to ensure that the mirrored video image is not distorted, the position of the target sticker special effect element in the first display area may be, but is not limited to, mirrored, but the target sticker special effect element itself is not mirrored.
[0103] For example, the screen position of the anchor side during streaming is proportionally mapped to the video stream display position of the audience side. When mirror flipping exists, the X-axis can be flipped. The formula for horizontal flip coordinates is: Xnew = ScreenWidth-Xold. The calculation method for vertical flipping can refer to the above formula and is not limited here.
[0104] Through the embodiments provided by the present application, after obtaining the first display area corresponding to the target sticker special effect element from the additional special effect data, the second display area corresponding to the first display area after mirroring is determined on the audience side, thereby ensuring the mirroring accuracy of the display position of the target sticker special effect element when displaying the mirror candidate screen.
[0105] As an optional solution, when displaying the mirrored video screen, it includes:
[0106] S1, when receiving mirrored video stream data according to a preset transmission frame rate, performing linear interpolation processing between any two displayed mirrored video frames.
[0107] It should be noted that in the relevant technology, the standard video stream transmission rate is 15 frames per second. In this embodiment, in order to save transmission resources, it can be reduced to 5 to 10 frames per second. However, due to the reduction in the transmission frame rate, when rendering the target sticker special effect elements on the audience side, the display will be stuck and the update frequency will be low due to the slow update of the AI data that it relies on. To address this phenomenon, this embodiment will perform a linear difference or a 1 / 2 approximation difference between two frames based on the time difference.
[0108] Through the embodiment provided by the present application, by performing linear interpolation processing on two frames of mirrored video images, a smooth transition of the mirrored video images received by reducing the transmission frame rate is achieved, thereby avoiding the problem of display freeze.
[0109] According to another aspect of the embodiments of the present invention, a video processing device is also provided. Fig. 9 As shown, the device comprises:
[0110] 1) A display unit 902 is configured to display a live video screen on the host side, wherein the live video screen includes a scene screen input by the host side's camera and sticker special effect elements superimposed on the scene screen;
[0111] 2) A mirror processing unit 904 is configured to perform mirror processing on a candidate picture extracted from the live video picture in response to a mirror processing instruction on the live video picture to obtain a mirror candidate picture, wherein the candidate picture is a video picture obtained by removing a target sticker special effect element from the live video picture, and the difference between the mirror sticker special effect element obtained after the target sticker special effect element is mirrored and the target sticker special effect element is greater than a target threshold;
[0112] 3) Push unit 906 is configured to push the mirror video stream data corresponding to the mirror candidate picture and the additional special effect data corresponding to the target sticker special effect element to the audience end, so that the audience end can display the picture obtained by integrating the mirror candidate picture and the target sticker special effect element.
[0113] It should be noted that the video processing device here can be but is not limited to the anchor terminal of the anchor terminal, such as Figure 1 The terminal device 102 shown, its embodiment can refer to the above method embodiment, and will not be repeated here.
[0114] As an optional solution, it also includes:
[0115] The first determination unit is configured to perform mirror processing on the candidate picture extracted from the live video picture to obtain the mirror candidate picture, and when the sticker special effect element superimposed on the scene picture includes the target sticker special effect element and the basic sticker special effect element, merge the scene picture with the basic sticker special effect element to obtain the candidate picture, wherein the difference between the mirror sticker special effect element obtained after the basic sticker special effect element is mirrored and the basic sticker special effect element is less than or equal to the target threshold;
[0116] The second determination unit is configured to use the scene picture input by the camera of the anchor end as a candidate picture when the sticker special effect element superimposed on the scene picture is the target sticker special effect element.
[0117] Optionally, in this embodiment, the embodiments of the above-mentioned modules may refer to the above-mentioned method embodiments, which will not be repeated here.
[0118] As an optional solution, it also includes:
[0119] A third determination unit is configured to perform mirror processing on the candidate picture extracted from the live video picture to obtain the mirrored candidate picture, and then determine the position of the key points of the face associated with the target sticker special effect element from the face area presented in the live video picture;
[0120] An extraction unit is configured to extract features of key points of a face to obtain feature information of the face;
[0121] A fourth determining unit is configured to determine a face picture area corresponding to the face feature information in the mirror candidate picture;
[0122] The generating unit is configured to generate mirror video stream data based on the face picture area.
[0123] Optionally, in this embodiment, the embodiments of the above-mentioned modules may refer to the above-mentioned method embodiments, which will not be repeated here.
[0124] As an optional solution, it also includes:
[0125] The storage unit is configured to extract features from key facial points and obtain facial feature information, and then store the coordinate information of the pixel points corresponding to each key facial point in the data bits corresponding to the integer type variable.
[0126] Optionally, in this embodiment, the embodiments of the above-mentioned modules may refer to the above-mentioned method embodiments, which will not be repeated here.
[0127] As an optional solution, the storage unit includes:
[0128] The storage module is configured to store the horizontal coordinate information of the pixel point into the first sequence data bits of the integer type variable, and store the vertical coordinate information of the pixel point into the second sequence data bits of the integer type variable.
[0129] Optionally, in this embodiment, the embodiments of the above-mentioned modules may refer to the above-mentioned method embodiments, which will not be repeated here.
[0130] According to another aspect of the embodiments of the present invention, a video processing device is also provided. Fig.10 As shown, the device comprises:
[0131] 1) A receiving unit 1002 is configured to receive the mirrored video stream data and the additional special effect data corresponding to the target sticker special effect element sent by the anchor end, wherein the difference between the mirrored sticker special effect element obtained after the target sticker special effect element is mirrored and the target sticker special effect element is greater than a target threshold;
[0132] 2) A parsing unit 1004 is configured to parse the mirrored video stream data to obtain a mirrored candidate picture, wherein the mirrored candidate picture is a picture obtained by mirroring the candidate picture, and the candidate picture is a video picture obtained by removing the target sticker special effect element from the live video picture;
[0133] 3) A rendering unit 1006 is configured to render a target sticker special effect element onto a mirror candidate picture according to the additional special effect data to obtain a mirror video picture.
[0134] It should be noted that the video processing device here can be but is not limited to the audience end of the application host end, such as Figure 1 The terminal device 106 shown, its embodiment can refer to the above method embodiment, and will not be repeated here.
[0135] As an optional solution, the rendering unit 1006 includes:
[0136] An acquisition module, configured to acquire a first display area corresponding to a target sticker special effect element from the additional special effect data, wherein the first display area is a display area of the target sticker special effect element in a video screen displayed on a host end;
[0137] A determination module, configured to determine a second display area corresponding to the first display area in the viewer terminal after the first display area is mirrored;
[0138] The rendering module is configured to render the target sticker special effect element in the second display area when displaying the mirror candidate picture to obtain the mirror video picture.
[0139] Optionally, in this embodiment, the embodiments of the above-mentioned modules may refer to the above-mentioned method embodiments, which will not be repeated here.
[0140] As an optional solution, the rendering unit 1006 further includes:
[0141] The processing module is configured to perform linear interpolation processing between any two displayed mirror video frames when receiving the mirror video stream data according to a preset transmission frame rate.
[0142] Optionally, in this embodiment, the embodiments of the above-mentioned modules may refer to the above-mentioned method embodiments, which will not be repeated here.
[0143] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0144] In an exemplary embodiment, an electronic device is further provided, the electronic device comprising: a processor; and a memory for storing processor-executable instructions, wherein the processor is configured to execute the above instructions to implement the video processing method described in any one of the above embodiments.
[0145] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.
[0146] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:
[0147] S1, displaying a live video screen on the anchor end, wherein the live video screen includes a scene screen input by the anchor end's camera and sticker special effect elements superimposed on the scene screen;
[0148] S2, in response to a mirror processing instruction for a live video screen, mirror processing is performed on a candidate screen extracted from the live video screen to obtain a mirror candidate screen, wherein the candidate screen is a video screen obtained by removing a target sticker special effect element from the live video screen, and a difference between a mirror sticker special effect element obtained after the target sticker special effect element is mirrored and the target sticker special effect element is greater than a target threshold;
[0149] S3, pushing the mirror video stream data corresponding to the mirror candidate picture and the additional special effect data corresponding to the target sticker special effect element to the audience end, so that the audience end can display the picture obtained by integrating the mirror candidate picture and the target sticker special effect element.
[0150] Optionally, in this embodiment, the processor may also be configured to perform the following steps through a computer program:
[0151] S1, receiving the mirrored video stream data and the additional special effect data corresponding to the target sticker special effect element sent by the anchor end, wherein the difference between the mirrored sticker special effect element obtained after the target sticker special effect element is mirrored and the target sticker special effect element is greater than the target threshold;
[0152] S2, parsing the mirrored video stream data to obtain a mirrored candidate picture, wherein the mirrored candidate picture is a picture obtained by mirroring the candidate picture, and the candidate picture is a video picture obtained by removing the target sticker special effect element from the live video picture;
[0153] S3, rendering the target sticker special effect element onto the mirror candidate screen according to the additional special effect data to obtain a mirror video screen.
[0154] Alternatively, it will be appreciated by those skilled in the art that the electronic device may also be a terminal device such as a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, and a mobile Internet device (Mobile Internet Devices, MID), a PAD, etc. Among them, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof. In addition, other module units in the above-mentioned video processing device may also be included, but are not limited to, and will not be repeated in this example.
[0155] Optionally, the electronic device may further include a transmission device for receiving or sending data via a network. Specific examples of the network may include a wired network and a wireless network. In one example, the transmission device includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and a router via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0156] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the above instructions can be executed by a processor of the device to complete the above method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0157] Optionally, in this embodiment, the computer-readable storage medium may be configured to store computer instructions for performing the following steps:
[0158] S1, displaying a live video screen on the anchor end, wherein the live video screen includes a scene screen input by the anchor end's camera and sticker special effect elements superimposed on the scene screen;
[0159] S2, in response to a mirror processing instruction for a live video screen, mirror processing is performed on a candidate screen extracted from the live video screen to obtain a mirror candidate screen, wherein the candidate screen is a video screen obtained by removing a target sticker special effect element from the live video screen, and a difference between a mirror sticker special effect element obtained after the target sticker special effect element is mirrored and the target sticker special effect element is greater than a target threshold;
[0160] S3, pushing the mirror video stream data corresponding to the mirror candidate picture and the additional special effect data corresponding to the target sticker special effect element to the audience end, so that the audience end can display the picture obtained by integrating the mirror candidate picture and the target sticker special effect element.
[0161] Optionally, in this embodiment, the computer-readable storage medium may also be configured to store computer instructions for executing the following steps:
[0162] S1, receiving the mirrored video stream data and the additional special effect data corresponding to the target sticker special effect element sent by the anchor end, wherein the difference between the mirrored sticker special effect element obtained after the target sticker special effect element is mirrored and the target sticker special effect element is greater than the target threshold;
[0163] S2, parsing the mirrored video stream data to obtain a mirrored candidate picture, wherein the mirrored candidate picture is a picture obtained by mirroring the candidate picture, and the candidate picture is a video picture obtained by removing the target sticker special effect element from the live video picture;
[0164] S3, rendering the target sticker special effect element onto the mirror candidate screen according to the additional special effect data to obtain a mirror video screen.
[0165] In an exemplary embodiment, a computer program product is also provided, including a computer program, the computer program including computer instructions, the computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned video processing.
[0166] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.
[0167] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A video processing method, characterized in that: include: Displaying a live video screen on the host end, wherein the live video screen includes a scene screen input by the camera of the host end and a sticker special effect element superimposed on the scene screen; In response to a mirror processing instruction for the live video screen, a candidate screen extracted from the live video screen is mirrored to obtain a mirror candidate screen, wherein the candidate screen is a video screen obtained by removing a target sticker special effect element from the live video screen, and a difference between a mirror sticker special effect element obtained after the target sticker special effect element is mirrored and the target sticker special effect element is greater than a target threshold; The mirror video stream data corresponding to the mirror candidate picture and the additional special effect data corresponding to the target sticker special effect element are pushed separately to the audience end, so that the audience end can complete the rendering of the target sticker special effect element locally and display the picture obtained by integrating the mirror candidate picture and the target sticker special effect element.
2. The method according to claim 1, characterized in that Before performing mirror processing on the candidate picture extracted from the live video picture to obtain the mirror candidate picture, the method further includes: In the case where the sticker special effect element superimposed on the scene screen includes the target sticker special effect element and the basic sticker special effect element, the scene screen and the basic sticker special effect element are merged to obtain the candidate screen, wherein the difference between the mirror sticker special effect element obtained after the basic sticker special effect element is mirrored and the basic sticker special effect element is less than or equal to the target threshold; In the case where the sticker special effect element superimposed on the scene screen is the target sticker special effect element, the scene screen input by the camera of the anchor end is used as the candidate screen.
3. The method according to claim 1, characterized in that: After performing mirror processing on the candidate picture extracted from the live video picture to obtain the mirror candidate picture, the method further includes: Determining the position of key points of the face associated with the target sticker special effect element from the face area presented in the live video screen; Extracting features of key points of the face to obtain facial feature information; Determining a face picture area corresponding to the face feature information in the mirror candidate picture; The mirror video stream data is generated based on the face picture area.
4. The method according to claim 3, characterized in that After extracting the features of the key points of the face to obtain the face feature information, the method further includes: The coordinate information of the pixel points corresponding to each of the facial key points is stored in each data bit corresponding to the integer type variable.
5. The method according to claim 4, characterized in that The step of storing the coordinate information of the pixel points corresponding to each of the key points of the face into the data bits corresponding to the integer type variables includes: The horizontal coordinate information of the pixel point is stored in the first sequence data bits of the integer type variable, and the vertical coordinate information of the pixel point is stored in the second sequence data bits of the integer type variable.
6. A video processing method, characterized in that: include: Respectively receive the mirrored video stream data sent by the anchor end and the additional special effect data corresponding to the target sticker special effect element, wherein the difference between the mirrored sticker special effect element obtained after the target sticker special effect element is mirrored and the target sticker special effect element is greater than the target threshold; Parsing the mirrored video stream data to obtain a mirrored candidate picture, wherein the mirrored candidate picture is a picture obtained by mirroring the candidate picture, and the candidate picture is a video picture obtained by removing the target sticker special effect element from the live video picture; The target sticker special effect element is rendered onto the mirror candidate picture according to the additional special effect data to obtain a mirror video picture.
7. The method according to claim 6, characterized in that The step of rendering the target sticker special effect element onto the mirror candidate picture according to the additional special effect data to obtain the mirror video picture includes: Acquire a first display area corresponding to the target sticker special effect element from the additional special effect data, wherein the first display area is a display area of the target sticker special effect element in the video screen displayed on the anchor end; Determine a second display area on the viewer end corresponding to the first display area after the first display area is mirrored; When displaying the mirror candidate picture, the target sticker special effect element is rendered in the second display area to display the mirror video picture.
8. The method according to claim 6, characterized in that When displaying the mirrored video screen, it includes: When the mirror video stream data is received according to a preset transmission frame rate, linear interpolation processing is performed between any two displayed frames of the mirror video images.
9. A video processing device, characterized in that: include: A display unit, configured to display a live video screen on the anchor end, wherein the live video screen includes a scene screen input by the anchor end's camera and a sticker special effect element superimposed on the scene screen; A mirror processing unit is configured to perform mirror processing on a candidate picture extracted from the live video picture in response to a mirror processing instruction for the live video picture to obtain a mirror candidate picture, wherein the candidate picture is a video picture obtained by removing a target sticker special effect element from the live video picture, and a difference between a mirror sticker special effect element obtained after the target sticker special effect element is mirrored and the target sticker special effect element is greater than a target threshold; The push unit is configured to separately push the mirror video stream data corresponding to the mirror candidate picture and the additional special effect data corresponding to the target sticker special effect element to the audience end, so that the audience end can locally complete the rendering of the target sticker special effect element and display the picture obtained by integrating the mirror candidate picture and the target sticker special effect element.
10. The device according to claim 9, characterized in that Also includes: A first determining unit is configured to, before performing mirror processing on the candidate picture extracted from the live video picture to obtain the mirror candidate picture, merge the scene picture with the basic sticker special effect element when the sticker special effect element superimposed on the scene picture includes the target sticker special effect element and the basic sticker special effect element to obtain the candidate picture, wherein the difference between the mirror sticker special effect element obtained after the basic sticker special effect element is mirrored and the basic sticker special effect element is less than or equal to the target threshold; The second determination unit is configured to use the scene picture input by the camera of the anchor end as the candidate picture when the sticker special effect element superimposed on the scene picture is the target sticker special effect element.
11. The device according to claim 9, characterized in that Also includes: A third determination unit is configured to, after performing mirror processing on the candidate picture extracted from the live video picture to obtain the mirrored candidate picture, determine the position of the key points of the face associated with the target sticker special effect element from the face area presented in the live video picture; An extraction unit is configured to extract features of key points of the face to obtain feature information of the face; A fourth determining unit is configured to determine a face picture area corresponding to the face feature information in the mirror candidate picture; A generating unit is configured to generate the mirror video stream data based on the face picture area.
12. The device according to claim 11, characterized in that Also includes: The storage unit is configured to store the coordinate information of the pixel points corresponding to each of the facial key point positions into the data bits corresponding to the integer type variables after extracting the features of the facial key point positions to obtain the facial feature information.
13. The device according to claim 12, characterized in that The storage unit comprises: The storage module is configured to store the horizontal coordinate information of the pixel point in the first sequence data bit of the integer type variable, and store the vertical coordinate information of the pixel point in the second sequence data bit of the integer type variable.
14. A video processing device, characterized in that: include: A receiving unit is configured to respectively receive the mirror video stream data sent by the anchor end and the additional special effect data corresponding to the target sticker special effect element, wherein the difference between the mirror sticker special effect element obtained after the target sticker special effect element is mirrored and the target sticker special effect element is greater than a target threshold; A parsing unit, configured to parse the mirrored video stream data to obtain a mirrored candidate picture, wherein the mirrored candidate picture is a picture obtained by mirroring the candidate picture, and the candidate picture is a video picture obtained by removing the target sticker special effect element from the live video picture; A rendering unit is configured to render the target sticker special effect element onto the mirror candidate picture according to the additional special effect data to obtain a mirror video picture.
15. The device according to claim 14, characterized in that The rendering unit comprises: An acquisition module, configured to acquire a first display area corresponding to the target sticker special effect element from the additional special effect data, wherein the first display area is a display area of the target sticker special effect element in the video screen displayed on the anchor end; A determination module, configured to determine a second display area corresponding to the first display area in the viewer terminal after the first display area is mirrored; A rendering module is configured to render the target sticker special effect element in the second display area when displaying the mirror candidate picture to display the mirror video picture.
16. The device according to claim 14, characterized in that The rendering unit also includes: The processing module is configured to perform linear interpolation processing between any two displayed frames of the mirror video images when receiving the mirror video stream data according to a preset transmission frame rate.
17. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the video processing method according to any one of claims 1 to 8. 18 . A computer-readable storage medium, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the video processing method according to claim 1 .
19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the video processing method according to any one of claims 1 to 8 is implemented.
Citation Information
Patent Citations
Photographing method and mobile terminal
CN105959564A
Live broadcast method and apparatus
CN107105311A
Image processing method and device
CN107203964A
Video image display method and device
CN108076303A
Image processing method and apparatus
CN108848405A
