Video processing method, apparatus, device, and storage medium

Abstract

This disclosure relates to a video processing method, apparatus, device, and storage medium. The method includes: during video processing (shooting or playback), when a current triggering event is detected, determining the target style corresponding to the current triggering event; determining a target person image based on an initial person image in the shot video and the target style; and fusing the target person image with each frame of the shot video to be played, according to a fusion speed and fusion direction, to generate a fused video conforming to the target style and play it. This allows users to obtain a special effects video of the target style without editing the video themselves, improving the speed and efficiency of video processing, and the gradual fusion video effect increases the fun of the video playback process.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a video processing method, apparatus, device, and storage medium. Background Technology

[0002] With the development of internet technology and the widespread use of electronic devices, more and more users enjoy sharing their lives and expanding their social activities through videos. For example, people frequently shoot videos and post them on social networks for sharing and communication.

[0003] Current video shooting methods can only achieve personalized special effects videos through post-production video editing. However, this video processing method is complex, increasing the operational complexity for users to obtain special effects videos, reducing the convenience and efficiency of video shooting, and hindering its widespread adoption. Summary of the Invention

[0004] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, this disclosure provides a video processing method, apparatus, device and storage medium.

[0005] In a first aspect, this disclosure provides a video processing method, the method comprising:

[0006] If a current triggering event is detected, the target style corresponding to the current triggering event is determined;

[0007] Based on the initial human image in the captured video and the target style, the target human image is determined;

[0008] According to the fusion speed and fusion direction, the target person image is fused with the video frames to be played in the captured video to generate a fused video that conforms to the target style and then played.

[0009] Secondly, this disclosure provides a video playback device, the device comprising:

[0010] The target style determination module is used to determine the target style corresponding to the current triggering event if the current triggering event is detected.

[0011] The target person image determination module is used to determine the target person image based on the initial person image in the captured video and the target style;

[0012] The video playback module is used to merge the target person image with the video frames to be played in the captured video according to the fusion speed and fusion direction, so as to generate a fused video that conforms to the target style and play it.

[0013] Thirdly, this disclosure provides a video playback device, the device comprising:

[0014] processor;

[0015] Memory, used to store executable instructions;

[0016] The processor is configured to read executable instructions from memory and execute the executable instructions to implement the video processing method described in any embodiment of this disclosure.

[0017] Fourthly, this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the video processing method described in any embodiment of this disclosure.

[0018] The video processing method, apparatus, device, and storage medium of this disclosure can determine the target style corresponding to the current triggering event when a current triggering event is detected during video processing (shooting or playback); determine the target person image based on the initial person image in the shot video and the target style; and fuse the target person image with each video frame to be played in the shot video according to the fusion speed and fusion direction to generate a fused video that conforms to the target style and play it. This allows users to obtain special effects videos of the target style without editing the video themselves, improving the speed and efficiency of video processing, and the gradual fusion video effects increase the fun of the video playback process. Attached Figure Description

[0019] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0020] Figure 1 A flowchart illustrating a video processing method provided in an embodiment of this disclosure;

[0021] Figure 2 This is a schematic diagram illustrating the effect of gradually merging a target person image with a video frame to be played, provided in an embodiment of this disclosure.

[0022] Figure 3 This is a schematic diagram illustrating the effect of merging multiple styles of video frames according to an embodiment of the present disclosure;

[0023] Figure 4 This is a schematic diagram of the structure of a video processing apparatus provided in an embodiment of the present disclosure;

[0024] Figure 5 This is a schematic diagram of the structure of a video processing device provided in an embodiment of the present disclosure. Detailed Implementation

[0025] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0026] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0027] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0028] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0029] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0030] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0031] Currently, the main method for obtaining special effects videos is through post-production editing after shooting. However, this video processing method is relatively complex in its logic and steps, requires a certain level of video processing skills from the user, and has poor universality. In addition, this video processing method requires users to perform many operations, which is time-consuming and labor-intensive, reducing the convenience and efficiency of obtaining special effects videos.

[0032] Based on the above, this application provides a video processing solution to automatically determine at least one suitable style for the video during or after the user is shooting the video, and automatically generate a target person image that matches the style. As the video plays, the target person image is gradually merged with each video frame, thereby presenting the user with the effect of gradually switching the style of the person in the video. This not only greatly reduces the user's operation during the special effects video processing, improving the convenience and efficiency of special effects video processing, but also increases the fun of video playback.

[0033] The following is a combination of... Figure 1-3 The video processing method provided in the embodiments of this disclosure will be described.

[0034] In this embodiment of the disclosure, the video processing method can be executed by a video processing device, which can be implemented in hardware and / or software, and can be integrated into a video processing equipment. The video processing equipment may include, but is not limited to, mobile terminals such as smartphones, PDAs (Personal Digital Assistants), PADs (Tablet PCs), PMPs (Portable Multimedia Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable devices, and laptops, as well as fixed terminals such as digital TVs, desktop computers, and smart home devices.

[0035] Figure 1 A flowchart illustrating a video processing method provided in an embodiment of this disclosure is shown. Figure 1 As shown, the video processing method may include the following steps:

[0036] S110. If the current triggering event is detected, determine the target style corresponding to the current triggering event.

[0037] The current triggering event refers to the event that occurs at the current moment. A triggering event is an event that triggers a change in video style; it can be related to the video's visuals, audio, or other content, or it can be a pre-set condition unrelated to the video content, such as time or frame rate. Here, style refers to the image's drawing style and overall color style, tone style, and lighting style. Drawing style includes character design such as appearance, body shape, hairstyle, and clothing, as well as the spatial sense and layering of the image elements. For example, the style could be comic book style, crayon style, ink painting style, oil painting style, sketching style, etc. The target style is the style corresponding to the current triggering event.

[0038] Specifically, in this embodiment, the video processing automatically converts the initial video captured by the camera into a special effects video containing at least one style, and the style switching in the special effects video is triggered by a pre-set trigger event. Therefore, the video processing device detects trigger events on the video captured by the camera (which can be real-time video or video that has already been captured). If a current trigger event is detected, the target style corresponding to that current trigger event is determined. This target style can be a pre-set fixed style, or a style selected from multiple styles according to a filtering rule based on the current trigger event. The filtering rule here can be, for example, a random selection rule or a sorting selection rule, and the sorting basis can be a user-defined style order, or the frequency of style application, etc.

[0039] In some embodiments, the triggering event includes at least one of the following: an audio rhythm point, a video motion change point, a set time interval point, a preset body posture, and a preset voice. Here, an audio rhythm point refers to an accented point in the audio of the video. A video motion change point is the moment when one action segment in the video changes into another; it can be the start or end moment of an action segment in the video. A set time interval point is the moment when the playback / shooting duration of the video reaches a set time interval. A preset body posture refers to a pre-set human posture. A preset voice refers to a pre-set voice.

[0040] In one example, when the triggering event is an audio rhythm point, a rhythm detection algorithm can be used to detect the rhythm in the audio data contained in the video. If an audio rhythm point is detected in the audio data at the current moment, then the current triggering event is considered to have been detected. The rhythm detection algorithm can be a dynamic programming algorithm, an algorithm based on spectral analysis to identify rhythm points in the audio data, or an algorithm that performs windowing, Fourier transform, and difference processing on the original audio signal before further analysis to obtain the rhythm points. This allows the video style to be switched at each musical rhythm point, further enhancing the video's appeal and improving the user experience of shooting special effects videos.

[0041] In another example, when the triggering event is a point of change in video action, temporal action detection algorithms can be used to detect the start time, end time, and action category of action segments within the video. If the current moment is detected as the start time of an action segment, i.e., the current video frame is detected as the start frame of that action segment, then the current triggering event is considered to have been detected. Alternatively, if the current video frame is detected as the end frame of an action segment, the current triggering event can be manually detected.

[0042] It should be noted that when the trigger event is a point of change in video motion, the point of change in video motion remains unchanged throughout the entire video processing. For example, if the trigger event is the start point of motion, then the trigger event is considered to have been detected only when the start frame of motion is detected during the entire video processing; similarly, if the trigger event is the end point of motion, then the trigger event is considered to have been detected only when the end frame of motion is detected during the entire video processing.

[0043] The aforementioned settings for video action change points allow for switching the video style at the start or end of each action segment. For example, in a dance video, each action or group of actions can switch to a different style, enhancing user interaction and engagement in the special effects video, thereby further increasing the video's fun factor and improving the user experience of shooting special effects videos.

[0044] In another example, when the triggering event is a set time interval, a timer for the corresponding time interval can be set, and the style in the video will switch each time the timer expires.

[0045] In another example, when the triggering event is a preset body posture, the video style will be switched as soon as the user's action is detected to match the preset body posture.

[0046] In another example, when the triggering event is a preset voice, the video style will be switched as soon as the user's input voice matches the preset voice.

[0047] In another example, when the triggering events are preset body postures and preset voices, if the user's action matches the preset body posture and the user's voice input matches the preset voice, such as detecting the user's finger snapping action and the finger snapping sound, the video style will be switched.

[0048] In another example, the triggering event can also be a combination of at least two of the following: audio rhythm points, video motion change points, set time interval points, preset body postures, and preset voice. For example, detecting any of the triggering events in the combination can switch the style of the video, thereby increasing the number of times the video switches styles and further enhancing the video's appeal.

[0049] The settings of the various trigger event types used in the embodiments of this disclosure can be predetermined by the designer, or a trigger event type setting interface can be provided to the user so that the user can set personalized trigger events according to their own needs and interests.

[0050] S120. Based on the initial human image and target style in the captured video, determine the target human image.

[0051] Here, "video capture" refers to video recorded via a camera, which can be live or pre-recorded. "Initial human image" refers to a human image from a video captured by the camera; it is a real-life image without style conversion. "Target human image" refers to a stylized human image that has undergone style conversion. The human image here is an image that at least contains the facial region.

[0052] Specifically, to increase user interactivity and interest in the video during processing, this embodiment primarily focuses on style switching for the people in the video. Therefore, the video processing device can extract person images from the captured video to obtain initial person images. Then, after determining the target style, the initial person images are stylized, transforming them into target person images with the target style.

[0053] The above-mentioned process of processing the initial human image into the target human image can be achieved by inputting the initial human image into an image style transfer model corresponding to the target style, and obtaining the model output result after running the model. The model output result is the target human image.

[0054] The image style transfer model described above is a machine learning model pre-trained using multiple sets of sample images. Each set of training images includes the original image without style transfer and a styled image of the original image after style processing (such as hand-drawn or computer-generated). This image style transfer model can be a model capable of performing various style transfers, or it can be a model capable of performing only one style transfer. If the image style transfer model is a model capable of performing only one style transfer, then it is necessary to pre-train corresponding image style transfer models for various styles.

[0055] S130. According to the fusion speed and fusion direction, the target person image is fused with the video frames to be played in the captured video to generate a fused video that matches the target style and then played.

[0056] The fusion speed refers to how quickly images are fused, and it can be preset or selected by the user. In some embodiments, the fusion speed can be expressed as the number of preset pixels fused per unit time, that is, the number of preset pixels fused per unit time.

[0057] The unit of time mentioned above can be a set duration based on experience, such as 0.1ms.

[0058] The aforementioned unit time can also be a duration determined based on the video playback frame rate, in order to control the image fusion speed according to the video playback speed. For example, the duration of playing n frames can be determined as the unit time, where the duration of playing n frames = playback frame rate * n, and n is a positive integer greater than or equal to 1.

[0059] The aforementioned unit time can also be the interval between two adjacent audio beats, used to control the image fusion speed based on the audio playback speed. For audio data contained in a video, each audio beat is determined when the audio data is generated, and the interval between two adjacent audio beats can also be determined accordingly. This interval between two adjacent audio beats is defined as the unit time. Audio beats can be determined through the attribute information of the audio data or identified through relevant beat detection algorithms.

[0060] The number of pixels set above can be a preset number of pixels, such as 10 pixels; or it can be calculated based on the number of pixels contained in the target image and the set percentage of the set number of pixels in the target image. For example, if the target image contains 512 pixels and the set percentage is 1 / 4, then the set number of pixels is 128 pixels. In this way, the fusion of the target image can be completed by performing the fusion operation of the reciprocal of the set percentage.

[0061] The blending direction refers to the direction in which images are blended. It can be a single direction pointing from one direction to another, such as from top to bottom (or the opposite), from left to right (or the opposite), from upper left to lower right (or the opposite), from upper right to lower right (or the opposite), etc.; it can also be an all-around direction following a certain pattern from the inside to the outside (or the opposite), such as a 360° all-around direction following a circle from the outside to the inside. The blending direction can be preset or determined by the user.

[0062] Specifically, to enhance the video's appeal and present users with a gradual transition effect, this embodiment employs a step-by-step fusion method when blending the target person image with the corresponding area in the video. That is, following the fusion direction and fusion speed, the target person image is gradually blended with the corresponding area in each video frame (i.e., the video frame to be played) starting from the current moment, generating a fused video that conforms to the target style, which is then played through a video processing device. Here, the image fusion method can be pixel-value weighted fusion or pixel-value substitution fusion.

[0063] For example, if the video played between two styles has 10 frames per second, the determined fusion speed is to fuse 1 / 3 of the target person's image area (i.e., the number of pixels is set to 1 / 3 of the target person's image) for each frame played (i.e., the playback time of one frame). The determined fusion direction is from top to bottom, and the determined fusion method is pixel value substitution fusion. Then, for... Figure 2 (a) shows the video frame that triggered the style switching and Figure 2 (b) shows the target person image, starting from the video frame that triggers the style switch. In the first video frame, the upper 1 / 3 of the target person image is blended with the corresponding area of ​​that first video frame, with the blending effect as shown. Figure 2 As shown in (c); in the second video frame, the upper two-thirds of the target person's image is fused with the corresponding area of ​​the second video frame, and the fusion effect is as follows. Figure 2 As shown in (d); in the third video frame, the entire image of the target person is fused with the corresponding area of ​​that third video frame, and the fusion effect is as follows. Figure 2 As shown in (e), the third video frame is directly replaced with the image of the target person; from the fourth to the tenth video frame, the corresponding video frame is replaced with the image of the target person, and the fusion effect is as follows. Figure 2 As shown in (e).

[0064] It should be noted that when the playback area covered by the target person's image changes across different video frames, a corresponding target person image needs to be generated based on the target style for each video frame with changing playback area, and then the corresponding video frames are fused. For example, the above... Figure 2 In the example, the target person image covers the entire video playback area. Therefore, the playback frame changes in each video frame. Thus, for each video frame, a corresponding target person image needs to be generated based on the target style before the relevant video frames are merged. It is important to note that... Figure 2 This is only to illustrate the process of gradual image merging; it does not show any changes in the video playback screen.

[0065] In some embodiments, the fusion method in S130 is to directly replace pixel values ​​of the target person image in the face region, while using the target person image for pixel value weighting in the non-face region. This highlights the stylized effects of the face while ensuring smooth transitions between other regions and video frames. Furthermore, it allows for stylized transformation of only the image region containing the face, reducing the stylized image area and thus improving video processing efficiency. Specifically, S130 can be implemented as steps A and / or B as follows:

[0066] Step A: For any region to be merged, determined according to the fusion speed and fusion direction, if the region to be merged is a face region, replace the region to be merged in the video frame to be played with the region to be merged in the target person image, and generate the merged video corresponding to the region to be merged.

[0067] Specifically, the fusion process of the target person image is a continuous fusion process performed according to the fusion speed. In this process, each image fusion operation is performed on a local region, which is the region to be fused, determined based on the fusion progress (such as the already fused area), the fusion speed, and the fusion direction. When the region to be fused is identified as a face region, the pixel values ​​of the region to be fused in the target person image are directly used to replace the pixel values ​​of the region to be fused in the video frame to be played, generating the fusion result corresponding to that region.

[0068] Step B: For any region to be merged determined according to the fusion speed and fusion direction, if the region to be merged is not a face region, perform pixel value weighting processing on the region to be merged in the video frame to be played and the region to be merged in the target person image to generate the merged video corresponding to the region to be merged.

[0069] Specifically, for each region to be fused, if the region to be fused is identified as a non-face region, then according to a certain pixel value weighting algorithm, the pixel values ​​of the region to be fused in the target person image and the pixel values ​​of the region to be fused in the video frame to be played are weighted and calculated. The weighted calculation result is the fusion result corresponding to the region to be fused.

[0070] The weighted processing unit region in the aforementioned pixel-weighted algorithm can be a single pixel or an image patch composed of multiple pixels. For each weighted processing unit region, a weighting value can be assigned to each pixel, or a weighting value can be assigned to a statistical value (such as the mean) of the weighted processing unit region. The weighting values ​​can be preset empirically. For example, fixed weighting values ​​can be assigned to both the target person image and the video frame to be played. The weighting values ​​can also be determined based on the relationship between the weighted processing unit region and the face region. For example, the closer a weighted processing unit region is to the face region, the higher the weighting value of the target person image is compared to the weighting value of the corresponding video frame to be played; conversely, the farther away a weighted processing unit region is from the face region, the lower the weighting value of the target person image is compared to the corresponding video frame to be played, thus further enhancing the special effects of the target style in the face region. The weighting method can be to add the two weighted results for the corresponding weighted processing unit region.

[0071] The above steps A and B are executed as follows: when the area to be merged contains only face areas or non-face areas, step A or step B is executed accordingly; when the area to be merged contains both face areas and non-face areas, step A and step B are executed as needed.

[0072] The video processing methods described above can determine the target style corresponding to the current triggering event when a triggering event is detected during video processing (shooting or playback); determine the target person image based on the initial person image and the target style in the shot video; and merge the target person image with each video frame to be played in the shot video according to the fusion speed and fusion direction to generate a fused video that conforms to the target style and play it. This allows users to obtain special effects videos of the target style without editing the video themselves, improving the speed and efficiency of video processing, and the gradual fusion video effects increase the fun of the video playback process.

[0073] In some embodiments, the number of styles switched during video recording can be less than the number of trigger events detected during video playback; that is, the style corresponding to each trigger event can be repeated. For example, a mapping relationship between each trigger event and a style can be pre-established, so that as long as the same trigger event is detected, the target style can be determined as the style corresponding to that trigger event.

[0074] In other embodiments, the target style is different from the style corresponding to any historical trigger event detected during the playback of the captured video. That is, the number of styles switched during the processing of the captured video is equal to the number of trigger events detected during video playback. Regardless of whether the currently detected trigger event is the same type of trigger event as a previously detected trigger event, a new style is determined for the current trigger event. For example, if the trigger events are all preset body gestures and / or preset voices for snapping fingers, for the first detected snapping finger trigger event, the target style is determined to be Japanese anime style, and for the second detected snapping finger trigger event, the target style is determined to be American comic style. This can further increase the number of styles switched during video playback, thereby further increasing the fun of the special effects video.

[0075] It should be understood that, according to the above description, in this embodiment, a mapping relationship between each triggering event and a style cannot be established to avoid determining the same style when the same triggering event is detected multiple times.

[0076] In one example, based on the above embodiment, S110 includes: if a current triggering event is detected, determining the sorting number of the current triggering event among the detected triggering events; selecting the style corresponding to the sorting number from a preset style sequence and determining it as the target style.

[0077] Specifically, in this example, the selection rule for the target style is a sorting selection rule. Multiple styles are pre-selected in order based on user-defined style order or the frequency of application of each style (i.e., a preset style sequence). Then, after detecting the current trigger event, its sorting number among all trigger events detected during the video playback up to the current moment is determined. For example, if four historical trigger events were detected before the current moment, the sorting number of the current trigger event is determined to be 5. Next, the style corresponding to the sorting number is selected from the preset style sequence; for example, selecting the style ranked 5th in the preset style sequence is determined as the target style. This setting allows for the generation of blended videos that better meet individual user needs or cater to general viewing preferences in subsequent processes, thereby further enhancing the video's appeal and user experience.

[0078] In another example, based on the above embodiment, S110 includes: if a current triggering event is detected, randomly selecting a style from the style library other than the historical styles as the target style.

[0079] Specifically, in this example, the selection rule for the target style is random selection, but it is necessary to ensure that the randomly selected styles do not overlap. This setting can reduce user intervention while obtaining a wider variety of styles, increasing the user's sense of novelty for the special effects videos generated according to these styles, thereby further increasing the fun of the special effects videos and user engagement.

[0080] One implementation of this example is to randomly select a style from the style library after detecting the current triggering event, and then compare the selected style with any of the previously determined historical styles. If it overlaps with any historical style, a new style is randomly selected from the style library, and the comparison continues until a style that is different from all historical styles is selected, which is then identified as the target style.

[0081] Another way to implement this example is to mark or remove selected styles from the style library. For example, after each style is selected, it is marked in the style library. When the current triggering event is detected, a style is randomly selected from the unmarked styles in the style library as the target style. Alternatively, after each style is selected, it is removed from the style library, so that the style library only stores styles that have not been selected. When the current triggering event is detected, a style is randomly selected from the style library as the target style.

[0082] Taking a scenario where the triggering event is the point of change in video motion, the target person's image covers the entire image area, the fusion method is a whole-image replacement method, and the fusion speed is to fuse the entire image per frame, as an example, the following can be obtained: Figure 3 The diagram shows the effect of merging video frames.

[0083] like Figure 3 As shown, when the video processing device detects... Figure 3 When the video motion change point shown in (a) is determined to be a beautiful Japanese anime style, the target character image in the beautiful Japanese anime style shown in Figure (b) can be obtained. Figure 3 (a) The video motion change point shown corresponds to the end of the motion segment, i.e., before it is detected. Figure 3 Before the point of change in video motion shown in (c), each video frame in the merged video presents a beautiful, anime-style aesthetic, except that the content of each video frame changes with the video. When video processing detects... Figure 3 When the video motion change point shown in (c) is determined to be American comic style, the target character image in American comic style as shown in Figure (d) can be obtained. Figure 3 (c) The video motion change point shown corresponds to the end of the motion segment, i.e., before it is detected. Figure 3Before the point of change in video motion shown in (e), each video frame in the merged video presents an American comic book style, except that the content of each video frame changes with the video. Detected. Figure 3 When the video motion change point shown in (e) is determined to be a cute anime style, a target character image in the cute anime style as shown in Figure (f) can be obtained. Figure 3 Before the action segment corresponding to the video action change point shown in (e) ends, that is, before the next video action change point is detected, each video frame in the fused video presents a cute Japanese anime style, except that the playback content of each video frame changes with the video.

[0084] In some embodiments, the process of generating the target face image in S120 can be performed in a video processing device. That is, an image style transfer model is run in the video processing device to generate the target face image.

[0085] In other embodiments, considering that style transfer needs to be performed on multiple video frames, the image style transfer model needs to be run multiple times. Each run of the image style transfer model consumes a significant amount of resources, placing a heavy burden on the video processing device and affecting the normal operation of other functions of the video processing device. Therefore, in this embodiment, the process of generating the target person is set to be executed on the server side, and the video processing device only needs to receive the image of the target person, thereby reducing the power consumption of the video processing device. In addition, considering that transmitting images between the video processing device and the server consumes bandwidth and has a certain time consumption, in order to reduce transmission bandwidth and transmission time, this embodiment further sets the transmitted image to be the image of the face region.

[0086] Based on the above description, S120 can be implemented as follows: extracting faces from the captured video to generate an initial face image; sending the initial face image and the target style to the server so that the server can input the initial face image into the image style transfer model corresponding to the target style and output the target face image; and receiving the target face image sent by the server.

[0087] Specifically, the video processing device performs face extraction processing on the current video frame of the captured video to obtain an initial face image. This initial face image is then transmitted to the server. The server inputs the initial face image into an image style transfer model, runs the model to obtain its output, and feeds this output back to the video processing device. The video processing device then uses this model output to obtain the target face image.

[0088] Furthermore, based on the above embodiments, considering that the image style transfer model has a good style transfer effect on frontal face images, the embodiments of this disclosure add a face correction process during the face extraction process to correct the face in the current video frame and obtain the initial face image of the frontal face, thereby increasing the accuracy of the style transfer of the target person image.

[0089] In practice, the video processing device extracts faces from the current video frame to obtain the face extraction result. Then, it determines whether the extracted face result is a frontal face image. If so, it transmits the image to the server. If not, it further extracts facial feature points from the extracted face result. Afterward, it uses the extracted facial feature points to perform face correction on the extracted face result, obtaining an initial frontal face image, which is then transmitted to the server.

[0090] It should be understood that if the initial face image is obtained by face correction of the face extraction result, then after the video processing device obtains the model output result fed back from the server, it needs to perform the reverse processing of the above face correction process on the model output result, that is, to perform face inverse correction on the model output result to obtain the target face image. This can further improve the fusion effect of subsequent fused videos.

[0091] Figure 4 A schematic diagram of the structure of a video processing apparatus provided in an embodiment of this disclosure is shown. Figure 4 As shown, the video processing apparatus 400 may include:

[0092] The target style determination module 410 is used to determine the target style corresponding to the current triggering event if the current triggering event is detected.

[0093] The target person image determination module 420 is used to determine the target person image based on the initial person image and target style in the captured video;

[0094] The video playback module 430 is used to merge the target person image with the video frames to be played in the captured video according to the fusion speed and fusion direction, generate a fused video that conforms to the target style and play it.

[0095] The aforementioned video processing device can determine the target style corresponding to the current triggering event when a triggering event is detected during video processing (shooting or playback); determine the target person image based on the initial person image and the target style in the shot video; and merge the target person image with each video frame to be played in the shot video according to the fusion speed and fusion direction to generate a fused video that conforms to the target style and play it. This allows users to obtain special effects videos of the target style without editing the video themselves, improving the speed and efficiency of video processing. Furthermore, the gradual fusion video effects increase the fun of the video playback process.

[0096] In some embodiments, the video playback module 430 includes an image fusion submodule, used for:

[0097] For any region to be fused determined according to the fusion speed and fusion direction, if the region to be fused is a face region, the region to be fused in the video frame to be played is replaced with the region to be fused in the target person image, and a fused video corresponding to the region to be fused is generated.

[0098] For any region to be merged, determined according to the fusion speed and fusion direction, if the region to be merged is not a face region, pixel value weighting is performed on the region to be merged in the video frame to be played and the region to be merged in the target person image to generate the merged video corresponding to the region to be merged.

[0099] In some embodiments, the fusion speed is the number of pixels fused per unit time;

[0100] Units of time include any of the following:

[0101] Set duration;

[0102] The duration is determined based on the video playback frame rate;

[0103] The duration of the interval between two adjacent audio beats.

[0104] In some embodiments, the target style is different from the style corresponding to any historical triggering event detected during the playback of the captured video.

[0105] Optionally, the target style determination module 410 is specifically used for:

[0106] If the current triggering event is detected, determine the sort number of the current triggering event among the detected triggering events;

[0107] Select the style corresponding to the sorting number from the preset style sequence and determine it as the target style.

[0108] Optionally, the target style determination module 410 is specifically used for:

[0109] If a current triggering event is detected, a style is randomly selected from the style library, excluding all historical styles, as the target style.

[0110] In some embodiments, the triggering event includes at least one of the following: audio rhythm point, video motion change point, set time interval point, preset body posture, and preset voice.

[0111] Optionally, the target style determination module 410 includes a current triggering event detection submodule, used for:

[0112] When the triggering event is an audio rhythm point, a rhythm detection algorithm is used to detect the audio rhythm point in the captured video at the current moment.

[0113] Optionally, the currently triggered event detection submodule is also used for:

[0114] When the triggering event is a video motion change point, it detects that the current video frame is the start or end frame of an action segment in the captured video.

[0115] In some embodiments, the target person image determination module 420 is specifically used for:

[0116] Extract faces from the captured video to generate initial face images;

[0117] The initial face image and the target style are sent to the server so that the server can input the initial face image into the image style transfer model corresponding to the target style and output the target face image.

[0118] Receive the target face image sent by the server;

[0119] The image style transfer model is obtained by training a machine learning model with sample images, which include the original image and style images obtained by style processing the original image.

[0120] Furthermore, the target person image determination module 420 is also specifically used for:

[0121] Face extraction is performed on the captured video to obtain the face extraction results;

[0122] Feature points are extracted from the face extraction results, and face correction is performed based on the extracted feature points to generate an initial face image.

[0123] It should be noted that, Figure 4 The video processing device 400 shown can perform... Figures 1 to 3 The various steps in the method embodiment shown are implemented. Figures 1 to 3 The processes and effects in the method embodiments shown are not described in detail here.

[0124] This disclosure also provides a video processing apparatus, which may include a processor and a memory, the memory being used to store executable instructions. The processor may be used to read the executable instructions from the memory and execute the executable instructions to implement the video processing method described in any of the above embodiments.

[0125] Figure 5 A schematic diagram of the structure of a video processing device provided in an embodiment of this disclosure is shown. Figure 5 The video processing device 500 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0126] The following is a detailed reference. Figure 5 It shows a schematic diagram of the structure suitable for implementing the video processing device 500 in the embodiments of this disclosure.

[0127] like Figure 5 As shown, the video processing device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the information processing device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output interface (I / O interface) 505 is also connected to the bus 504.

[0128] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows video processing equipment 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 A video processing apparatus 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0129] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the video processing method described in any of the above embodiments.

[0130] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the video processing method of embodiments of this disclosure.

[0131] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0132] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP, and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0133] The aforementioned computer-readable medium may be included in the aforementioned video processing device; or it may exist independently and not assembled into the video processing device.

[0134] The aforementioned computer-readable medium carries one or more programs, which, when executed by the video processing device, cause the video processing device to perform the steps of the video processing method described in any embodiment of this disclosure.

[0135] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0136] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0137] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0138] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0139] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0140] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0141] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0142] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A video processing method, characterized in that, include: During video processing, if a current triggering event is detected, the target style corresponding to the current triggering event is determined. The target style is different from the style corresponding to any historical triggering event detected during the playback of the captured video, and the current triggering event is the event that triggers the video to switch styles; Based on the initial human image in the captured video and the target style, a target human image is determined; the target human image refers to a stylized human image whose image style is the target style after processing the initial human image. According to the fusion speed and fusion direction, the target person image is fused with the video frames to be played in the captured video to generate a fused video that conforms to the target style and then played. The step of fusing the target person image with the video frames to be played in the captured video according to the fusion speed and fusion direction to generate a fused video that conforms to the target style includes: For any region to be fused determined according to the fusion speed and the fusion direction, if the region to be fused is a face region, the region to be fused in the video frame to be played is replaced with the region to be fused in the target person image, and a fused video corresponding to the region to be fused is generated. For any region to be merged determined according to the fusion speed and the fusion direction, if the region to be merged is not a face region, pixel value weighting processing is performed on the region to be merged in the video frame to be played and the region to be merged in the target person image to generate a merged video corresponding to the region to be merged.

2. The method according to claim 1, characterized in that, The fusion speed is the number of pixels fused per unit time. The unit of time includes any of the following: Set duration; The duration is determined based on the playback frame rate of the video; The duration of the interval between two adjacent audio beats.

3. The method according to claim 1, characterized in that, If a current triggering event is detected, determining the target style corresponding to the current triggering event includes: If the current triggering event is detected, then the sorting number of the current triggering event among the detected triggering events is determined; Select the style corresponding to the sorting number from the preset style sequence and determine it as the target style.

4. The method according to claim 1, characterized in that, If a current triggering event is detected, the target style corresponding to the current triggering event is determined to include: If the current triggering event is detected, a style is randomly selected from the style library other than the historical styles, and this style is chosen as the target style.

5. The method according to claim 1, characterized in that, Triggering events include at least one of the following: audio rhythm points, video motion change points, set time interval points, preset body postures, and preset voice.

6. The method according to claim 5, characterized in that, When the triggering event is the audio rhythm point, detecting the current triggering event includes: A rhythm detection algorithm is used to detect the audio rhythm points in the captured video at the current moment.

7. The method according to claim 5, characterized in that, When the triggering event is a change point in the video motion, detecting the current triggering event includes: The current video frame is detected as either the start or end frame of an action segment in the captured video.

8. The method according to claim 1, characterized in that, The process of determining the target person image based on the initial person image in the captured video and the target style includes: Faces are extracted from the captured video to generate initial face images; The initial face image and the target style are sent to the server so that the server can input the initial face image into the image style transfer model corresponding to the target style and output the target face image. Receive the target face image sent by the server; The image style transfer model is obtained by training a machine learning model with sample images, which include the original image and style images obtained by style processing the original image.

9. The method according to claim 8, characterized in that, The step of extracting faces from the captured video and generating an initial face image includes: Face extraction is performed on the captured video to obtain the face extraction results; Feature points are extracted from the face extraction results, and face correction is performed on the face extraction results based on the extracted feature points to generate the initial face image.

10. A video processing apparatus, characterized in that, include: The target style determination module is used to determine the target style corresponding to the current triggering event if a current triggering event is detected during video processing. The target style is different from the style corresponding to any historical triggering event detected during the playback of the captured video, and the current triggering event is the event that triggers the video to switch styles; The target person image determination module is used to determine a target person image based on an initial person image in the captured video and the target style; the target person image refers to a stylized person image whose image style is the target style after processing the initial person image. The video playback module is used to merge the target person image with the video frames to be played in the captured video according to the fusion speed and fusion direction, generate a fused video that conforms to the target style, and play it. The video playback module includes an image fusion submodule, used for: For any region to be fused determined according to the fusion speed and the fusion direction, if the region to be fused is a face region, the region to be fused in the video frame to be played is replaced with the region to be fused in the target person image, and a fused video corresponding to the region to be fused is generated. For any region to be merged determined according to the fusion speed and the fusion direction, if the region to be merged is not a face region, pixel value weighting processing is performed on the region to be merged in the video frame to be played and the region to be merged in the target person image to generate a merged video corresponding to the region to be merged.

11. A video playback device, characterized in that, include: processor; Memory, used to store executable instructions; The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the video processing method according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, causes the processor to implement the video processing method according to any one of claims 1-9.

Images