Video processing method and related device

Abstract

The invention provides a video processing method and related equipment, which are used for improving the consistency of a target person in a style redrawn video and improving the quality of the redrawn video. The method comprises the following steps: acquiring a to-be-processed video, wherein the to-be-processed video comprises a target person; the feature portrait of the target person is obtained from a person feature library, the feature portrait of the target person corresponds to the feature area of the target person, and the person feature library comprises the feature portrait of at least one person. And inputting the feature portrait of the target person and the feature region of the target person in the to-be-processed video into a first redrawing model to obtain a first redrawn video.

Description

(19) State Intellectual Property Office (12) Invention Patent Application (10) Application Publication Number (43) Application Publication Date (21) Application Number 202411147449.0 (22) Application Date 2024.08.20 (66) Domestic Priority Data 202410799827.7 2024.06.19 CN (71) Applicant Huawei Cloud Computing Technology Co., Ltd. Address 550025, Guizhou Province, Guiyang City, Guian New District, Qianzhong Avenue, Xinggong Road, Huawei Cloud Data Center (72) Inventors Liu Letao, Li Likun, Jia Haozhe, Xu Di, Zhou Xuan, Zhang Dong (74) Patent Agency Shenzhen Shenjia Intellectual Property Agency (General Partnership) 44285 Patent Attorney Fan Youfei (51) Int.Cl. G06T 3 / 04 (2024.01) G06V 20 / 40 (2022.01) G06V 40 / 10 (2022.01) (54) Invention Title: A Method and Related Apparatus for Video Processing (57) Abstract: This application provides a method and related apparatus for video processing to improve the consistency of a target person in a style-redrawn video and improve the quality of the redrawn video. The method includes: acquiring a video to be processed, the video to be processed including a target person; acquiring a feature profile of the target person from a feature library, the feature profile of the target person corresponding to a feature region of the target person, the feature library including feature profiles of at least one person; inputting the feature profile of the target person and the feature region of the target person in the video to be processed into a first redrawing model to obtain a first redrawn video. Claims 3 pages, Description 16 pages, Drawings 6 pages, CN 121169667 A 2025.12.19 CN 1 21 16 96 67 A 1. A video processing method, characterized in that it includes: acquiring a video to be processed, the video to be processed including a target person; acquiring a feature portrait of the target person from a feature database, the feature portrait of the target person corresponding to a feature region of the target person, the feature database including feature portraits of at least one person; inputting the feature portrait of the target person and the feature region of the target person in the video to be processed into a first redrawing model to obtain a first redrawing video. 2. The method according to claim 1, characterized in that, the step of inputting the feature portrait of the target person and the feature region of the target person in the video to be processed into a first redrawing model to obtain a first redrawing video includes: inputting the feature portrait of the target person, the feature region of the target person in the video to be processed, and a target redrawing style into a first redrawing model to obtain a first redrawing video, the redrawing style being used to indicate the style of the feature region of the target person in the first redrawing video.3. The method according to claim 2, wherein the target redrawing style includes a two-dimensional image style and / or a three-dimensional image style. 4. The method according to any one of claims 1 to 3, wherein after acquiring the video to be processed, the method further includes: responding to a first operation instruction for a target video frame in the video to be processed, determining the target person, the first operation instruction being used to specify the target person included in the target video frame; storing a feature profile of the target person in the person feature database, the feature profile of the target person being included in a first image included in the video to be processed, wherein the confidence level of the target person in the first image is higher than the confidence level of the target person in other video frames. 5. The method according to any one of claims 1 to 3, characterized in that the method further comprises: acquiring a target image; responding to a second operation instruction for the target image, determining the target person, the second operation instruction being used to specify the target person included in the target image; storing the feature portrait of the target person in the person feature library, the feature portrait of the target person being included in a second image, the confidence level of the target person in the second image being higher than the confidence level of the target person in other images, the other images being a plurality of video frames included in the video to be processed and images in the target image other than the second image. 6. The method according to any one of claims 1 to 5, characterized in that the person feature library further comprises an identifier of at least one person, the identifier of the at least one person corresponding one-to-one with the feature portrait of the at least one person; acquiring the feature portrait of the target person from the person feature library comprises: acquiring the feature portrait of the target person based on the identifier of the target person. 7. The method according to any one of claims 1 to 6, characterized in that the method further comprises: inputting the video to be processed into a second redrawing model to obtain a second redrawing video; splicing the first redrawing video and the second redrawing video to obtain a third redrawing video. 8. The method according to any one of claims 1 to 6, characterized in that, acquiring the video to be processed includes: inputting an initial video into a second redrawing model to obtain the video to be processed; the method further includes: splicing the first redrawing video and the video to be processed to obtain a fourth redrawing video. 9. The method according to any one of claims 1 to 7, characterized in that, after acquiring the video to be processed, the method further includes: splitting the video to be processed into multiple video frames; starting from the first frame of the multiple video frames, determining all the people included in the multiple video frames based on a person feature detection algorithm; storing the feature portraits of all the people in the person feature library, wherein the feature portrait of each person is included in the feature library.The video to be processed includes each third image, wherein the confidence level of each person in each third image is higher than the confidence level of each person in other video frames. 10. The method according to any one of claims 1 to 9, wherein the feature region of the target person includes the facial feature region of the target person and / or the clothing feature region of the target person. 11. A video processing apparatus, comprising: an acquisition unit, configured to acquire a video to be processed, the video to be processed including a target person; the acquisition unit is further configured to acquire a feature portrait of the target person from a feature database, the feature portrait of the target person corresponding to the feature region of the target person, the feature database including feature portraits of at least one person; and a processing unit, configured to input the feature portrait of the target person and the feature region of the target person in the video to be processed into a first redrawing model to obtain a first redrawing video. 12. The apparatus according to claim 11, wherein the processing unit is specifically configured to: input the feature portrait of the target person, the feature region of the target person in the video to be processed, and the target redrawing style into a first redrawing model to obtain a first redrawing video, wherein the redrawing style is used to indicate the style of the feature region of the target person in the first redrawing video. 13. The apparatus according to claim 12, wherein the target redrawing style includes a two-dimensional image style and / or a three-dimensional image style. 14. The apparatus according to any one of claims 11 to 13, wherein the processing unit is further configured to: respond to a first operation instruction for a target video frame in the video to be processed, determine the target person, wherein the first operation instruction is used to specify the target person included in the target video frame; store the feature portrait of the target person in the person feature library, wherein the feature portrait of the target person is included in a first image included in the video to be processed, wherein the confidence level of the target person in the first image is higher than the confidence level of the target person in other video frames. 15. The apparatus according to any one of claims 11 to 13, wherein the acquiring unit is further configured to acquire a target image; the processing unit is further configured to, in response to a second operation instruction for the target image, determine the target person, the second operation instruction being used to specify the target person included in the target image; the processing unit is further configured to store a feature portrait of the target person in the person feature database, the feature portrait of the target person being included in a second image, wherein the confidence level of the target person in the second image is higher than the confidence level of the target person in other images, the other images being a plurality of video frames included in the video to be processed and images in the target image other than the second image. 16. The apparatus according to any one of claims 11 to 15, wherein the person feature database further includes...The device is missing one person's identifier, and the identifier of the at least one person corresponds one-to-one with the feature portrait of the at least one person; the acquisition unit is specifically used to acquire the feature portrait of the target person based on the identifier of the target person. Claims 2 / 3 Page 3 CN 121169667 A 17. The device according to any one of claims 11 to 16, wherein the processing unit is further used to: input the video to be processed into the second redrawing model to obtain the second redrawing video; and stitch the first redrawing video and the second redrawing video together to obtain the third redrawing video. 18. The device according to any one of claims 11 to 16, wherein the acquisition unit is specifically used to input the initial video into the second redrawing model to obtain the video to be processed; and the processing unit is further used to stitch the first redrawing video and the video to be processed together to obtain the fourth redrawing video. 19. The apparatus according to any one of claims 11 to 17, wherein the processing unit is further configured to: divide the video to be processed into a plurality of video frames; starting from the first frame of the plurality of video frames, determine all persons included in the plurality of video frames based on a person feature detection algorithm; store feature portraits of all persons in the person feature library, wherein the feature portrait of each person is included in each third image of the video to be processed, and the confidence level of each person in each third image is higher than the confidence level of each person in other video frames. 20. A video processing apparatus, comprising a processor coupled to a memory; instructions stored in the memory, which, when executed on the processor, cause the video processing apparatus to perform the method according to any one of claims 1 to 10. 21. A computing device cluster, comprising at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform the method according to any one of claims 1 to 10. 22. A computer program product comprising instructions, characterized in that, when the instructions are executed on a communication device, the communication device performs the method as described in any one of claims 1 to 10; or, when the instructions are executed by a cluster of computing devices, the cluster of computing devices performs the method as described in any one of claims 1 to 10. 23. A computer-readable storage medium, characterized in that the computer-readable storage medium comprises computer program instructions, which, when executed on a communication device, cause the communication device to perform the method as described in any one of claims 1 to 10; or, when the computer program instructions are executed by a cluster of computing devices, cause the cluster of computing devices to perform the method as described in any one of claims 1 to 10.Claims 3 / 3 Page 4 CN 121169667 A A Method and Related Device for Video Processing

[0001] This application claims priority to Chinese Patent Application No. 202410799827.7, filed with the State Intellectual Property Office on June 19, 2024, entitled "A Video Redrawing Method and Apparatus for Maintaining Consistent Object IDs in Different Scenarios", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of communications, and more particularly to a method and related device for video processing. Background Art

[0003] With the development of artificial intelligence (AI) technology, video processing methods have become increasingly diversified. Among them, video style redrawing has attracted widespread attention. Video style redrawing, also known as video redrawing, is a technique for generating new videos by changing the style of characters in the original video. The style of the characters in the new video is different from that in the original video. For example, the characters in the original video are real characters, while the characters in the new video are virtual characters, such as animated characters or simulated characters.

[0004] In related technical solutions, the video input by the user is redrawn based on an AI model, thereby changing the style of the video. In this technical solution, since the characters in the video may be in a dynamic state, the state of the characters changes before and after the video frame, making it impossible for the AI ​​tool to maintain the consistency of the character features in the video when redrawing the video, resulting in low quality of the redrawn video and a poor viewing experience for the user. Summary of the Invention

[0005] This application provides a video processing method and related equipment. In the video processing method, a feature portrait of the target character is obtained from a character feature library, and the feature portrait corresponds to the feature region of the target character. The feature portrait and the feature region of the target character are input into a first redrawing model. The feature portrait of the target character serves as a reference for the feature region and is used to adjust the feature region of the target character during the redrawing process, providing a more accurate reference for the redrawing of the feature region. Therefore, even in different video frames, the redrawing of the feature region of the same target character is based on the same feature portrait, thereby improving the consistency of the feature region of the target character in the first redrawn video output by the first redrawing model and improving the quality of the first redrawn video.

[0006] In a first aspect, this application provides a video processing method, comprising:

[0007] a video processing apparatus acquiring a video to be processed, the video to be processed including a target person. That is, the video to be processed includes video frames containing images of the target person. In this application, the video to be processed may also be referred to as the video to be redrawn. The video processing apparatus acquires a feature portrait of the target person from a person feature database, the feature portrait of the target person corresponding to the target person.The feature region of the object, the feature database of the person includes the feature portrait of at least one person. The feature region of the target person is the region that needs to be redrawn. The video processing device inputs the feature portrait of the target person and the feature region of the target person in the video to be processed into the first redrawing model to obtain the first redrawing video. Specifically, the aforementioned process can be understood as adjusting the feature region of the target person with the feature portrait of the target person as a reference to obtain the first redrawing video. In the first redrawing video, the style of the feature region of the target person is different from that of the feature region of the target person in the video to be processed.

[0008] In this application, the feature portrait of the target person is obtained from the feature database of the person, which corresponds to the feature region of the object in the target person specification 1 / 16 page 5 CN 121169667 A. The feature portrait of the target person and the feature region of the target person are input into the first redrawing model. The feature portrait of the target person is used as a reference for the feature region to adjust the feature region of the target person during the redrawing process, providing a more accurate reference for the redrawing of the feature region. Therefore, even in different video frames, the redrawing of the feature regions of the same target person is based on the same feature profile, thereby improving the consistency of the feature regions of the target person in the first redrawing video output by the first redrawing model and improving the quality of the first redrawing video.

[0009] In some optional implementations of the first aspect, in addition to the aforementioned feature profile of the target person and the feature regions of the target person in the video to be processed, the input of the first redrawing model may also include a target redrawing style. The target redrawing style is used to indicate the style of the feature regions of the target person in the first redrawing video. It can also be understood that the target redrawing style indicates the style of the target person in the video to be redrawn. The target redrawing style can be input into the first redrawing model simultaneously with the feature profile of the target person and the feature regions of the target person in the video to be processed, or it can be input into the first redrawing model before or after the feature profile of the target person and the feature regions of the target person in the video to be processed. The specific details are not limited here.

[0010] In this application, the input of the first redrawing model also includes a target redrawing style, which indicates the style in which the feature areas of the target character will be redrawn. This means that the style in which the feature areas of the target character are redrawn can be selected according to the needs of the actual application, providing users with personalized services and improving the flexibility of the technical solution of this application.

[0011] In some optional implementations of the first aspect, the target redrawing style includes a two-dimensional image style and / or a three-dimensional image style. Among them, the two-dimensional image style includes pixel style, painting style, and other two-dimensional image styles. Painting style includes comic style, ink painting style, oil painting style, meticulous brush painting style, printmaking style, gouache painting style, sketching style, etc. Three-dimensional image style includes clay style, origami style, building block style, doll style, ceramic style, bronze style, etc.

[0012] In this application, there are multiple possibilities for the target redrawing style, which can be flexibly selected, further enriching the application scenarios and implementation methods of the technical solution of this application, and improving the flexibility of the technical solution of this application.

[0013] In some optional implementations of the first aspect, the video processing device can determine the target person based on the video to be processed. Specifically, after acquiring the video to be processed, the video processing device can also respond to a first operation instruction for the target video frame in the video to be processed to determine the target person. The first operation instruction is used to specify the target person included in the target video frame. That is, after the video processing device acquires the video to be processed, it displays the target video frame. The user selects the target person in the target video frame based on touch operation or through an external device. The video processing device also stores the feature portrait of the target person in the person feature library. The feature portrait of the target person is included in the first video frame included in the video to be processed. Among them, the confidence level of the target person in the first image is higher than the confidence level of the target person in other video frames.

[0014] In this application, the video processing device can determine the target person based on the video to be processed, which also indicates that the data source processed by the video processing device is the video to be processed, and no other content needs to be acquired, making the operation simple. In addition, the feature profile of the target person is included in the first image included in the video to be processed, and the confidence level of the target person in the first image is higher than the confidence level of the target person in other video frames. Therefore, when redrawing the feature region of the target person, the reference of the feature profile of the target person is more accurate, improving the similarity between the redrawn and unredrawn feature region of the target person, and further improving the redrawing quality of the feature region of the target person.

[0015] In some optional implementations of the first aspect, the video processing device can also determine the target person in other ways. Specifically, the video processing device also acquires the target image and, in response to a second operation instruction for the target image, determines the target person. The second operation instruction is used to specify the target object included in the target image. That is, the target person is the person selected from the target image. The video processing device stores the feature portrait of the target person in a person feature database. The feature portrait of the target person is included in the second image. The confidence level of the target person in the second image is higher than the confidence level of the target person in other images. The other images are multiple video frames included in the video to be processed and images in the target image other than the second image.

[0016] In this application, the video processing device can also determine the target person from the target image, enriching the application scenarios of the technical solution of this application. In addition, the feature portrait of the target person is included in the second image. The confidence level of the target person in the second image is higher than the confidence level of the target person in other images. The other images are multiple video frames included in the video to be processed and images in the target image other than the second image.Images other than the second image in the image. This means that the feature profile of the target person is included in multiple video frames of the video to be processed and the image of the target person with the highest confidence among the target images. Therefore, when redrawing the feature region of the target person, the reference of the feature profile of the target person is more accurate, which improves the similarity between the redrawn and unredrawn feature region of the target person, and further improves the redrawing quality of the feature region of the target person.

[0017] In some optional implementations of the first aspect, the person feature library also includes at least one person's identifier, and at least one person's identifier corresponds one-to-one with at least one person's feature profile. Then, the video processing device obtains the feature profile of the target person from the person feature library, specifically based on the identifier of the target person. Specifically, after the video processing device determines the target person, it can obtain the identifier of the target person, determine the identifier in the person feature library that matches the identifier of the target person, and then find the feature profile of the person corresponding to the identifier, thus finding the feature profile of the target person.

[0018] In this application, in the character feature database, the character identifier corresponds one-to-one with the character feature portrait. After determining the identifier of the target character, the character feature portrait of the target character can be uniquely determined, thus improving the efficiency of the technical solution of this application.

[0019] In some optional implementations of the first aspect, the video to be processed can be an initial video. Then, the video processing device can also input the video to be processed into the second redrawing model to obtain the second redrawing video. The style of the second redrawing video can be the same as or different from the target redrawing style, which is not limited here. The video processing device can also splice the first redrawing video and the second redrawing video to obtain the third redrawing video. That is to say, compared with the video to be processed, each video frame in the third redrawing video has been redrawn.

[0020] In some optional implementations of the first aspect, the input of the second redrawing model includes, in addition to the video to be processed, the first redrawing style, which is used to indicate the style in which the video to be processed is redrawn. The first redrawing style can be input to the second redrawing model simultaneously with the video to be processed, or it can be input to the first redrawing model before or after the video to be processed; the specifics are not limited here. The first redrawing style is similar to the aforementioned target redrawing style and has multiple possibilities, which will not be elaborated here. The first redrawing style can be the same as or different from the target redrawing style; the specifics are not limited here.

[0021] In some optional implementations of the first aspect, the video to be processed can be a redrawn video. That is, the video processing device obtains the video to be processed by inputting the initial video into the second redrawing model to obtain the video to be processed. The video processing device will also stitch the first redrawing video and the video to be processed to obtain a fourth redrawing video.

[0022] In some optional implementations of the first aspect, when the initial video is input to the second redrawing model, the second redrawing style is also input to the second redrawing model. The second redrawing style is used to indicate the style at which the initial video will be redrawn. The second redrawing style can be input to the second redrawing model simultaneously with the initial video, or it can be input to the second redrawing model before or after the initial video. No specific limitation is made here. The second redrawing style is similar to the aforementioned target redrawing style and has multiple possibilities, which will not be elaborated here. The second redrawing style can be the same as or different from the target redrawing style. No specific limitation is made here.

[0023] In this application, there are multiple possibilities for the video to be processed. Therefore, there are multiple possibilities for the feature regions of the target person obtained from the video to be processed, which enriches the implementation methods and application scenarios of the technical solution of this application and further enhances the flexibility of the technical solution of this application. In addition, the video processing device will also stitch together the first redrawn video and the redrawn video obtained by redrawing other areas in the video to be processed or the initial video (page 3 / 16 of CN 121169667 A) (i.e., the second redrawn video or the third redrawn video), so that each video frame in the stitched video is a redrawn video frame, that is, the entire video is redrawn.

[0024] In some optional implementations of the first aspect, after acquiring the video to be processed, the video processing device will also split the video to be processed into multiple video frames. Then, starting from the first frame of the multiple video frames, all the people included in the multiple video frames are determined based on the person feature detection algorithm. Then, the feature portraits of all the people are stored in the person feature library, wherein the feature portrait of each person is included in each third image of the video to be processed, wherein the confidence of each person in each third image is higher than the confidence of each person in other video frames. For example, assuming that the video to be processed has a total of 600 frames, wherein person 1 is included in frames 1 to 400, and person 2 is included in frames 200 to 600. If the confidence level of Person 1 in frame 300 is higher than the confidence level of Person 1 in other video frames from frame 1 to frame 400, then the third image of Person 1 is frame 300. If the confidence level of Person 2 in frame 505 is higher than the confidence level of Person 2 in other video frames from frame 200 to frame 600, then the third image of Person 2 is frame 505.

[0025] In this application, the person feature library may also include the feature portrait of each person in the video to be processed. Then, when redrawing the style of people other than the target person, the corresponding feature portrait can be found from the person feature library, simplifying the operation process.

[0026] In some optional implementations of the first aspect, the feature region of the target person includes the facial feature region of the target person and / or the clothing feature region of the target person.

[0027] In this application, the feature regions of the target person can be multiple, which enriches the application scenarios of the technical solution of this application and further enhances the flexibility of the technical solution of this application.

[0028] In a second aspect, this application provides a video processing apparatus, including:

[0029] an acquisition unit, used to acquire a video to be processed, the video to be processed including a target person. Acquire a feature portrait of the target person from a feature database, the feature portrait of the target person corresponds to the feature region of the target person, the feature database includes feature portraits of at least one person.

[0030] a processing unit, used to input the feature portrait of the target person and the feature region of the target person in the video to be processed into a first redrawing model to obtain a first redrawing video.

[0031] The video processing apparatus is used to implement the aforementioned first aspect or any possible implementation of the first aspect, which will not be repeated here.

[0032] In a third aspect, this application provides a video processing apparatus, including a processor and a memory, the processor storing instructions, when the instructions stored in the memory are executed on the processor, to implement the method shown in the aforementioned first aspect or any possible implementation of the first aspect.

[0033] In a fourth aspect, this application provides a computing device, the computing device including a processor and a memory. The processor of the computing device is used to execute instructions stored in memory, causing the computing device to implement the method shown in the first aspect or any possible implementation of the first aspect.

[0034] In a fifth aspect, this application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster implements the method disclosed in the first aspect and any possible implementation of the first aspect.

[0035] In a sixth aspect, this application provides a computer program product containing instructions that, when executed on a processor, implement the method shown in the first aspect or any possible implementation of the first aspect; or, when the instructions are run by a computer device cluster, cause the computer device cluster to implement the method disclosed in the first aspect and any possible implementation of the first aspect.

[0036] In a seventh aspect, this application provides a computer-readable storage medium storing computer program instructions that, when executed on a processor, implement the method shown in the first aspect or any possible implementation of the first aspect; or, when executed by a cluster of computer devices, cause the cluster of computer devices to implement the method disclosed in the first aspect and any possible implementation of the first aspect.

[0037] The beneficial effects shown in any one of the second to seventh aspects are related to the first aspect or any possible implementation of the first aspect.Similar possible implementations are not described here. Brief Description of the Drawings

[0038] Figure 1 is a schematic diagram of the system architecture provided in an embodiment of this application;

[0039] Figure 2 is another schematic diagram of the system architecture provided in an embodiment of this application;

[0040] Figure 3 is a flowchart of a video processing method provided in an embodiment of this application;

[0041] Figure 4 is another flowchart of a video processing method provided in an embodiment of this application;

[0042] Figure 5 is another flowchart of a video processing method provided in an embodiment of this application;

[0043] Figure 6 is a structural schematic diagram of a video processing device provided in an embodiment of this application;

[0044] Figure 7 is a structural schematic diagram of a computing device provided in an embodiment of this application;

[0045] Figure 8 is a structural schematic diagram of a computing device cluster provided in an embodiment of this application;

[0046] Figure 9 is another structural schematic diagram of a computing device cluster provided in an embodiment of this application. Detailed Description of the Embodiments

[0047] This application provides a video processing method and related equipment. In the video processing method, a feature portrait of a target person is obtained from a person feature database, and the feature portrait corresponds to the feature region of the target person. The feature profile and feature region of the target person are input into the first redrawing model. The feature profile of the target person serves as a reference for the feature region, and is used to adjust the feature region of the target person during the redrawing process, providing a more accurate reference for the redrawing of the feature region. Therefore, even in different video frames, the redrawing of the feature region of the same target person is based on the same feature profile, thereby improving the consistency of the feature region of the target person in the first redrawing video output by the first redrawing model and improving the quality of the first redrawing video.

[0048] The embodiments of this application are described below with reference to the accompanying drawings. It is known to those skilled in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0049] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be interchanged where appropriate, and this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not expressly listed or inherent to those processes, methods, products, or apparatuses. Additionally, "at least one" means one or more, and "more than" means two or more. "And / or" describes the relationship between related objects, indicating that they may exist...In three relationships, for example, A and / or B, it can mean: A exists alone, A and B exist simultaneously, or B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0050] First, please refer to Figures 1 and 2, which are schematic diagrams of the system architecture provided in the embodiments of this application.

[0051] In the embodiment shown in Figure 1, an application is running on the client 101, which is used to implement the video processing method provided in the embodiments of this application. In other words, the user selects to launch the application, inputs a video into the client 101, the application processes the video to obtain a redrawn video, and displays the redrawn video on the screen of the client 101. Optionally, the computing power of the application can be provided by the server 102.

[0052] The video processing method provided in this application embodiment can also be applied to cloud scenarios. In cloud scenarios, the AI ​​model capabilities used for video processing can be deployed in the cloud, and the high computing power of the cloud platform can be used to achieve better processing results for long videos.

[0053] As shown in Figure 2, the user logs into the cloud service platform 203 through the client 201 via the Internet 202 using the account and password registered on the cloud service platform 203. The cloud service platform 203 manages the infrastructure, which includes multiple data centers set up in different regions. For example, region 1 shown in Figure 1 includes cloud data center 1 and cloud data center 2, and region 2 includes cloud data center 3 and cloud data center 4. Each cloud data center is equipped with multiple servers, and business instances (including at least one of virtual machines, containers, and dedicated hosts) run on the servers.

[0054] In this embodiment of the application, a video service is deployed in the business instance. The user purchases cloud services through the client 201 on the cloud service platform 203. The user sends a call request to the cloud service platform 203, which is used to request cloud services from the cloud service platform 203. The specific content of the cloud service includes providing video processing services, which will be described later, to the tenant.

[0055] Please refer to Figure 3 below. Figure 3 is a flowchart of the video processing method provided in this embodiment of the application.

[0056] 301. Obtain the video to be processed, which includes the target person.

[0057] The video to be processed obtained by the video processing device may be of various kinds, including the initial video and the redrawn video. Regardless of the possibility, the video to be processed includes the target person, which means that the target person is in the video to be processed.At least two video frames include images of the target person.

[0058] In the scheme where the video to be processed is the initial video, the video processing device can acquire the video to be processed in various ways, such as through an external device, from internal storage or a database, or through other methods, which are not limited here. Optionally, there are also various possibilities for acquisition through an external device, such as storing the video to be processed stored on the external device into the video processing device through a transmission medium, or receiving the video to be processed uploaded by the user, etc. Here, the initial video refers to a video that includes real people, or a video taken of real people.

[0059] In the scheme where the video to be processed is a redrawn video, after the video processing device acquires the initial video, it inputs the initial video into the second redrawing model to obtain the video to be processed. Here, the style of the video to be processed is different from the style of the initial video. For example, the initial video is a video of real people, and the image of the person in the video to be processed is in the redrawn style, such as: two-dimensional portrait style, or three-dimensional portrait style, etc.

[0060] 302. Obtain the feature portrait of the target person from the feature database. The feature portrait of the target person corresponds to the feature region of the target person. The feature database includes the feature portrait of at least one person.

[0061] The feature database stores the feature portrait of at least one person, wherein the feature portrait of each person corresponds to the feature region of each person. That is, the feature portrait of a person is the portrait of the feature region of the person. For example, the feature portrait of person 1 is the portrait of the feature region of person 1. In the embodiments of this application, the feature region of a person includes the facial region of the person and / or the clothing region of the person. That is, the feature region of a person includes any one of the following: the facial region of the person, the clothing region of the person, the facial region of the person, and the clothing region of the person. Similarly, the feature region of the target person also includes the facial feature region of the target person and / or the clothing feature region of the target person. Specification 6 / 16 pages 10 CN 121169667 A

[0062] Accordingly, the feature portrait of a person includes the portrait of the facial region of the person and / or the clothing region of the person.

[0063] It should be noted that the facial region of a person may include the face portion of the person, specifically including the person's eyebrows, glasses, nose, and mouth. Optionally, in addition to the aforementioned face portion, the facial region of a person may also include the person's ears.

[0064] In the embodiments of this application, the feature regions of the target person have multiple possibilities, enriching the application scenarios of the technical solution of this application and further improving the flexibility of the technical solution of this application.

[0065] In addition to storing the feature portrait of at least one person, the feature database also stores the identifier of at least one person. The identifier of at least one person corresponds one-to-one with the feature portrait of at least one person. That is to say, a person's identifier corresponds one-to-one with the feature portrait of at least one person.Each identifier uniquely corresponds to a feature portrait of a person, and each feature portrait of a person uniquely corresponds to an identifier of a person.

[0066] Then, the video processing device obtains the feature portrait of the target person from the feature database. Specifically, based on the identifier of the target person, it determines the feature portrait of the target person from the feature portraits of at least one person in the feature database. The specific implementation process is described in detail below and will not be elaborated here.

[0067] In this embodiment of the application, the identifier of a person in the feature database corresponds one-to-one with the feature portrait of a person. After determining the identifier of the target person, the feature portrait of the target person can be uniquely determined, which improves the efficiency of the technical solution of this application.

[0068] 303. Input the feature portrait of the target person and the feature region of the target person in the video to be processed into the first redrawing model to obtain the first redrawing video.

[0069] The video processing device inputs the feature portrait of the target person and the feature region of the target person in the video to be processed into the first redrawing model. The first redrawing model is used to redraw the feature region of the target person. It can also be understood as using the feature portrait of the target person as a reference to adjust the feature region of the target person, and finally obtain the first redrawing video.

[0070] When the video processing device redraws the feature regions of the target person using the first redrawing model, it also needs to obtain the target redrawing style. This target redrawing style indicates the style of the feature regions of the target person, that is, the style in which the feature regions of the target person will be redrawn, or the style of the feature regions of the target person in the first redrawing video. Therefore, this means that the input to the first redrawing model includes not only the feature portrait of the target person and the feature regions of the target person in the video to be processed, but also the target redrawing style. The target redrawing style can be input to the first redrawing model simultaneously with the feature portrait of the target person and the feature regions of the target person in the video to be processed, or it can be input before or after the feature portrait of the target person and the feature regions of the target person in the video to be processed; the specific method is not limited here.

[0071] There are several possibilities for the video processing device to obtain the target redrawing style. It can be obtained in response to user operation commands, it can be obtained by obtaining the default redrawing style, or it can be other methods, such as using the default redrawing style for the initial video as the target redrawing style; the specific method is not limited here.

[0072] In the embodiments of this application, the target redrawing style includes various possibilities, generally including two-dimensional image styles and / or three-dimensional image styles. Two-dimensional image styles refer to two-dimensional image styles, including pixel style, painting style, etc. Painting styles include animation style, comic style, ink painting style, oil painting style, meticulous brushwork painting style, printmaking style, gouache painting style, sketching style, etc. Three-dimensional image styles refer to three-dimensional image styles, including clay style, origami style, building block style, doll style, ceramic style, bronzeware style, etc.

[0073] It should also be noted that in the embodiments of this application, there can be one or more target persons. In the case of multiple target persons, the target redrawing style of each target person can be the same or different, and no specific limitation is made here. In the case of multiple target persons with different redrawing styles, the video processing device obtains the target redrawing style corresponding to each target person.

[0074] In the embodiments of this application, the input of the first redrawing model also includes the target redrawing style, which is used to indicate the style in which the feature area of ​​the target person's specification 7 / 16 page 11 CN 121169667 A will be redrawn. At the same time, there are multiple possible target redrawing styles, which means that the style in which the feature area of ​​the target person is redrawn can be selected according to the needs of actual application, providing personalized services for users and improving the flexibility of the technical solution of this application.

[0075] As can be seen from the description of Figure 3 above, in the embodiments of this application, the feature portrait of the target person is obtained from the feature database, and the feature portrait corresponds to the feature area of ​​the target person. The feature profile and feature region of the target person are input into the first redrawing model. The feature profile of the target person serves as a reference for the feature region, which is used to adjust the feature region of the target person during the redrawing process, providing a more accurate reference for the redrawing of the feature region. Therefore, even in different video frames, the redrawing of the feature region of the same target person is based on the same feature profile, thereby improving the consistency of the feature region of the target person in the first redrawing video output by the first redrawing model and improving the quality of the first redrawing video.

[0076] The process of the video processing device determining the target person is described in detail below. The target person can be understood as the person who needs to be continuously tracked in the video to be processed, or the person whose consistency is the focus during the video redrawing process. In the embodiments of this application, there are several possible ways for the video processing device to determine the target person, which are described below:

[0077] In some optional embodiments, the target person can be determined from the video to be processed. In this technical solution, the target person can be specified by the user or set by the system, which are described below.

[0078] Optionally, after acquiring the video to be processed, the video processing device responds to a first operation instruction for a target video frame in the video to be processed to determine the target person. The first operation instruction is used to specify the target person included in the target video frame. The target video frame is any frame in the video to be processed that includes the target person.

[0079] Optionally, the target video frame can be a video frame selected by the user or a default video frame. The default video frame can have various possibilities; it can be the video frame containing the most people, the video frame containing people of the same gender, or other types of video frames, such as the video frame containing people whose ages are all greater than or all less than [a certain age].The threshold video frame, etc., can be set according to the needs of actual application, and are not specifically limited here.

[0080] The video processing device responds to the first operation instruction for the target video frame in the video to be processed. This can be understood as the video processing device displaying the target video frame, the user performing touch operation on the display screen, or inputting the first operation instruction to the video processing device through an external device (such as a mouse, keyboard, etc.), so that the video processing device determines the target person.

[0081] Optionally, after acquiring the video to be processed, the video processing device can determine the target person based on the default settings. The default settings can be all people in the video to be processed, or people whose gender in the video frame to be processed is female, or people in the video to be processed whose age is greater than or less than the threshold, etc. There are many possibilities for the default settings, which are used to select people with common characteristics as target people. They can be set according to the needs of actual application, and are not specifically limited here.

[0082] It should be noted that the number of target people can be one or more, and is not specifically limited here. For example, after a user selects a person, the video processing device can display a prompt box to ask the user whether to continue selecting people, until the user no longer selects any new people, and the video processing device determines the number of target people.

[0083] After determining the target people, the video processing device also stores the feature portrait of the target people in the person feature database. The feature portrait of the target people is included in the first image of the video to be processed, and the confidence level of the target people in the first image is higher than the confidence level of the target people in other video frames. For example, suppose the video to be processed includes 600 video frames, of which 400 video frames include the target people. Among them, the confidence level of the target people in the 300th frame is higher than the confidence level of the target people in the other 399 frames. Then, the 300th frame is the first image, and the feature portrait of the target people is the portrait of the feature regions of the target people in the first image. Instruction manual, page 8 / 16, 12 CN 121169667 A

[0084] It is understood that the feature profile of the target person is included in the first image of the video to be processed, and the confidence level of the target person in the first image is higher than that of the target person in other video frames. This means that the target person in the video frame to be processed is closest to the appearance of the target person, so using this as the feature profile of the target person provides a more accurate reference when redrawing the video.

[0085] It should also be noted that in the scheme where there are multiple target persons, the feature profile of each target person is determined from the video frames containing each target person. For example, assuming that the target persons include Person 1 and Person 2, and of the 600 video frames to be processed, 500 video frames include Person 1 and 300 video frames include Person 2. Then, the personThe feature profile of object 1 is determined from 500 video frames including person 1; the feature profile of person 2 is determined from 300 video frames including person 2.

[0086] It should also be noted that the feature region of the target person includes the facial region and / or clothing region of the target person. Accordingly, there are multiple possibilities for the feature profile of the target person, which can be a profile of the facial region of the target person, a profile of the clothing region of the target person, or a profile of both the facial region and clothing region of the target person. Optionally, the feature profile of the facial region and clothing region of the target person can also be directly used as the feature profile of the target person. In this way, no matter what the feature region of the target person is, the feature profile of the target person can be used as a reference to provide prior experience.

[0087] In this embodiment of the application, the video processing device can determine the target person based on the video to be processed, which also shows that the data source processed by the video processing device is the video to be processed, and no other content needs to be obtained, making the operation simple. In addition, the feature profile of the target person is included in the first image included in the video to be processed, and the confidence level of the target person in the first image is higher than the confidence level of the target person in other video frames. Therefore, when redrawing the feature regions of the target person, the reference of the feature portrait of the target person is more accurate, which improves the similarity between the redrawn and unredrawn feature regions of the target person, and further improves the redrawing quality of the feature regions of the target person.

[0088] In some optional embodiments, the target person can be determined from the target image. Specifically, in addition to acquiring the video to be processed, the video processing device also acquires the target portrait. Then, in response to the second operation instruction for the target image, the target person is determined. The second operation instruction is used to specify the target person included in the target image. The feature portrait of the target person is stored in the person feature library. The feature portrait of the target person is included in the multiple video frames and target images included in the video to be processed, and the image with the highest confidence of the target person is the image.

[0089] The target image can be an image of a real person or a redrawn image, which is not limited here.

[0090] The video processing device determines the target person in response to the second operation instruction for the target image. This can be understood as follows: the video processing device displays a target image, and the user performs touch operations on the display screen or inputs a second operation command to the video processing device through an external device (such as a mouse, keyboard, etc.), causing the video processing device to determine the target person. Optionally, the target person can be some or all of the people in the target image; that is, the number of target people can be one or more. For example, when the user selects a person, the video processing device can display a prompt box, prompting the user whether to continue selecting people, until the user no longer selects new people, at which point the video processing device determines the number of target people.

[0091] After the video processing device identifies the target person, it also stores the target person's feature portrait in the person feature database. The target person's feature portrait is included in the second image, and the confidence level of the target person in the second image is higher than that of the target person in other images. Other images refer to the target portrait and all video frames in the video to be processed that contain the target person, excluding the second image. For example, suppose the video to be processed includes 600 video frames, of which 400 video frames include the target person, and the confidence level of the target person in the 305th video frame is higher than that of the target person in the other 399 video frames, and also higher than that of the target person in the target portrait. Then, the 305th video frame is the first image, the feature portrait of the target person (page 9 / 16 of the specification, CN 121169667 A), the portrait of the feature region of the target person in the first image.

[0092] It should also be noted that in the scheme where there are multiple target persons, the feature portrait of each person is determined based on the confidence level of each person. For example, assuming the target person includes person 3 and person 4, among the 100 video frames in the video to be processed, 50 video frames include person 3 and 40 video frames include person 4. Then, the feature profile of person 3 is determined from the target profile and the 50 video frames including person 3; the feature profile of person 4 is determined from the target profile and the 40 video frames including person 4.

[0093] In this embodiment of the application, the video processing device can also determine the target person from the target image, enriching the application scenarios of the technical solution of this application. In addition, the feature profile of the target person is included in the first image, and the confidence level of the target person in the first image is higher than the confidence level of the target person in other images. Other images refer to the target image and all video frames in the video to be processed that include the target person, excluding the first image. Therefore, when redrawing the feature region of the target person, the reference of the feature profile of the target person is more accurate, improving the similarity between the redrawn and unredrawn feature region of the target person, and further improving the redrawing quality of the feature region of the target person.

[0094] In this embodiment, the target person can be one or more. When there are multiple target persons, the target redrawing styles of different target persons can be the same or different. The method for determining the target redrawing style of a target person is described below:

[0095] In some optional embodiments, after determining a target person, the video processing device can display a candidate box near the display area of ​​the target person. The candidate box includes at least one candidate redrawing style. The user can slide the options in the candidate box to browse the candidate redrawing styles provided by the video processing device. The video processing device responds to a hit operation for a certain candidate redrawing style and determines that the candidate redrawing style is the target redrawing style corresponding to the target person.

[0096] In some optional embodiments, after all target persons are determined, the video processing device displays a redraw style matching interface, which includes a person selection area and a redraw style selection area. The person selection area displays candidate persons, and the user can slide the control in the person selection area to browse all target persons. The redraw style selection area displays candidate redraw styles, and the user can slide the control in the redraw style selection area to browse all candidate redraw styles. The video processing device responds to a hit operation for a target person in the person selection area and selects a target redraw style for that target person. Based on this, responding to a hit operation for a redraw style in the redraw style selection area, the device determines that the redraw style is the target redraw style corresponding to the target person.

[0097] In some optional embodiments, the user can also input the target redraw style of the target person.

[0098] In the foregoing embodiments, the redrawing of the feature areas of the target person was emphasized. In practical applications, other areas in the initial video can also be redrawn, which will be described below.

[0099] In the scheme where the video to be processed is the initial video, the video processing device can also input the video to be processed into the second redrawing model to obtain the second redrawing video. The style of the characters in the second redrawing video is different from that in the video to be processed, and all areas in the second redrawing video are redrawn. The video processing device splices the first redrawing video and the second redrawing video to obtain the third redrawing video. Each video frame in the third redrawing video is redrawn.

[0100] Optionally, the input to the second redrawing model includes the first redrawing style in addition to the video to be processed. The first redrawing style can be input into the second redrawing model at the same time as the video to be processed, or it can be input into the first redrawing model before or after the video to be processed, which is not limited here. The first redrawing style is used to indicate the style in which the video to be processed is redrawn. The first redrawing style is similar to the aforementioned target redrawing style and has multiple possibilities, which will not be elaborated here. The first redrawing style can be the same as or different from the target redrawing style, which is not limited here.

[0101] In the scheme where the video to be processed is a redrawn video, it means that the video processing device obtains the video to be processed (page 10 / 16 of the instruction manual, 14 CN 121169667 A) by inputting the initial video into the second redraw model. The character style in the model to be processed is different from the character style in the initial video, and all areas in the video to be processed are redrawn. The video processing device can also stitch together the first redraw video and the video to be processed to obtain a fourth redraw video.

[0102] Optionally, the input to the second redraw model includes, in addition to the initial video, a second redraw style. The second redraw style can be input to the second redraw model simultaneously with the initial video, or it can be input before or after the initial video.The specific details of the dual-paint model are not limited here. The second repaint style is used to indicate the style of the initial video being repainted. Similar to the aforementioned target repaint style, there are multiple possibilities. The second repaint style can be the same as or different from the target repaint style. The specific details are not limited here.

[0103] In the embodiments of this application, there are multiple possibilities for the video to be processed. Therefore, there are multiple possibilities for the feature regions of the target person obtained from the video to be processed, which enriches the implementation methods and application scenarios of the technical solution of this application and further improves the flexibility of the technical solution of this application. In addition, the video processing device will also splice the first repainted video and the repainted video obtained by repainting other regions in the video to be processed or the initial video (i.e., the second repainted video or the third repainted video), so that each video frame in the spliced ​​video is a repainted video frame, that is, the complete repainting of the video is achieved.

[0104] Similarly, the way in which the video processing device determines the first repaint style and the second repaint style is similar to the way in which the target repaint style of the target person is determined, as shown above, and will not be repeated here.

[0105] In the embodiment described above, the character feature library stores the feature portrait of the target character. In practical applications, the character feature library can store the feature portraits of all characters in the video to be processed. Specifically, after acquiring the video to be processed, the video processing device can split the video to be processed into multiple video frames. Starting from the first frame of the multiple video frames, all characters included in the multiple video frames are determined based on the character feature detection algorithm. The feature portraits of all characters are stored in the character feature library, wherein the feature portrait of each character is included in each third image included in the video to be processed, and the confidence of each character in each third image is higher than the confidence of each character in other video frames. That is to say, the feature portrait of character 1 is included in the video frame with the highest confidence of character 1 among the multiple video frames; the feature portrait of character 2 is included in the video frame with the highest confidence of character 2 among the multiple video frames.

[0106] In this application, the character feature library can also include the feature portrait of each character in the video to be processed. Then, when redrawing the style of characters other than the target character in the subsequent process, the corresponding feature portraits can be found from the character feature library, simplifying the operation process.

[0107] Next, the video processing method provided in this application embodiment will be described in detail with reference to examples.

[0108] For example, please refer to FIG4, which is a flowchart of the video processing method provided in this application embodiment. It should be noted that in the embodiment shown in FIG4, the video to be processed is the video to be redrawn.

[0109] As shown in FIG4, the video processing device acquires the initial video and divides the initial video into multiple video frames according to the frame rate of the initial video itself. Then, starting from the first frame, based on the human feature detection algorithm, human detection is performed on multiple video frames.The algorithm detects the features of people in each video frame. If the difference between two features exceeds a specified threshold, it means that the two features correspond to different people, and they are recorded as People 1 and People 2 respectively. If the difference between two features is less than the specified threshold, it means that the two features correspond to the same person and the person's identifier is the same. If the difference between a third feature and the features of the already identified people is greater than the specified threshold, it means that the third feature corresponds to a new person, and it is recorded as People 3. And so on, identifying N people in multiple video frames. The difference between the features can be calculated as a certain distance based on the algorithm.

[0110] Based on the person detection algorithm, the N people in the initial video are identified, and the identifiers of the N people and the feature portraits of these N people are stored in the person feature library. The feature portrait of each person is included in each third image of the video to be processed, and the confidence of each person in each third image is higher than the confidence of each person in other video frames.

[0111] In the video redrawing stage, based on the aforementioned person detection, after identifying the target person, the target person's identifier can be determined. Specifically, the target person's features are queried and matched with the features in the person feature database. When the difference between the features of two people is less than a threshold, the match is considered successful. The target person's identifier is successfully matched, and the target person's feature profile is obtained.

[0112] Multiple video frames obtained by splitting the initial video are input into the second redrawing model for redrawing to obtain the video to be processed. The style of the video to be processed is the second redrawing style.

[0113] Based on the person detection algorithm, the feature region of the target person is obtained from the video to be processed. The feature region of the target person and the feature profile of the target person are input into the first redrawing model to obtain the first redrawing video. In the first redrawing video, the redrawing style of the target person's feature region is the target redrawing style. The target redrawing style and the second redrawing style can be the same or different, and no specific limitation is made here.

[0114] Finally, the first redrawing video and the video to be processed are spliced ​​together to obtain the fourth redrawing video, which is the output video. Specifically, the splicing operation involves pasting the first redrawn video back onto the feature region of the target person in the video to be processed.

[0115] For example, please refer to Figure 5, which is a flowchart illustrating the video processing method provided in this application embodiment. It should be noted that in the embodiment shown in Figure 5, the video to be processed is taken as the initial video.

[0116] As shown in Figure 5, the video processing device acquires the initial video and splits it into multiple video frames according to the frame rate of the initial video itself. Then, starting from the first frame, based on the person feature detection algorithm, person detection is performed on multiple video frames.The video processing device detects and identifies all the people included in the video to be processed. The specific implementation process is similar to that shown in Figure 4, and will not be repeated here.

[0117] In the embodiment shown in Figure 5, the video processing device also acquires a target image and, in response to a second operation instruction for the target image, determines the target people in the target image. Assume that the target people are person 1 and person 2. Then, the video processing device compares all the previously identified people with the target people to determine the feature profile of the target people.

[0118] In the embodiment shown in Figure 5, the target people are person 1 and person 2. The identifier and feature profile of person 1, as well as the identifier and feature profile of person 2, are stored in the person feature library. The feature profile of each person is included in each third image included in the video to be processed, and the confidence of each person in each third image is higher than the confidence of each person in other video frames.

[0119] The person detection algorithm of the video processing device determines the identifier of the target person, which is used to obtain the feature profile of the target person from the person feature library when redrawing the feature region of the target person. In addition, in the video redrawing stage, the feature region of the target person is obtained from the video to be processed based on the person detection algorithm. The feature regions and feature images of the target person are input into the first redrawing model to obtain the first redrawing video. In the first redrawing video, the redrawing style of the feature regions of the target person is the target redrawing style.

[0120] The video processing device can also input multiple video frames obtained by splitting the initial video into the second redrawing model for redrawing to obtain the second redrawing video. The style of the second redrawing video is the first redrawing style. The target redrawing style and the first redrawing style can be the same or different, and no specific limitation is made here.

[0121] Finally, the first redrawing video and the second redrawing video are spliced ​​together to obtain the third redrawing video, which is the output video. Specifically, the splicing operation is to paste the first redrawing video back into the feature regions of the target person in the second redrawing video.

[0122] It should be noted that in the embodiments of this application, the specific person feature detection algorithm is not limited, as long as it can identify the facial area or clothing area of ​​the person. The algorithm used in the first redraw model is not limited, as long as it can preserve the characteristic regions of the person, such as the roop algorithm, instant ID algorithm, image prompt adapter (IP adapter) algorithm, low-rank adapter (LoRA) algorithm, etc. The types of the first and second redraw models are also not limited, as long as they are models used for video redrawing, such as diffusion models, autoregressive models, etc. Specific limitations are not specified here.

[0123] Please refer to Figure 6, which is a schematic diagram of the structure of the video processing device provided in the embodiment of this application. As shown in Figure 6, the video processing device 600 includes an acquisition unit 601 and a processing unit 602.

[0124] In some optional embodiments, the acquisition unit 601 is used to acquire a video to be processed, which includes a target person. The feature portrait of the target person is acquired from a feature database, and the feature portrait of the target person corresponds to the feature region of the target person. The feature database includes the feature portrait of at least one person.

[0125] The processing unit 602 is used to input the feature portrait of the target person and the feature region of the target person in the video to be processed into a first redrawing model to obtain a first redrawing video.

[0126] In some optional embodiments, the processing unit 602 is specifically used to input the feature portrait of the target person, the feature region of the target person in the video to be processed, and the target redrawing style into the first redrawing model to obtain a first redrawing video. The redrawing style is used to indicate the style of the feature region of the target person in the first redrawing video.

[0127] In some optional embodiments, the target redrawing style includes a two-dimensional image style and / or a three-dimensional image style.

[0128] In some optional embodiments, the processing unit 602 is further configured to, in response to a first operation instruction for a target video frame in the video to be processed, determine a target person, wherein the first operation instruction is used to specify the target person included in the target video frame. The feature portrait of the target person is stored in a person feature library, wherein the feature portrait of the target person is included in a first image included in the video to be processed, and the confidence level of the target person in the first image is higher than the confidence level of the target person in other video frames.

[0129] In some optional embodiments, the acquisition unit 601 is further configured to acquire a target image.

[0130] The processing unit 602 is further configured to, in response to a second operation instruction for a target image, determine a target person, wherein the second operation instruction is used to specify the target person included in the target image. The feature portrait of the target person is stored in a person feature library, wherein the feature portrait of the target person is included in a second image, wherein the confidence level of the target person in the second image is higher than the confidence level of the target person in other images, wherein the other images are multiple video frames included in the video to be processed and images in the target image other than the second image.

[0131] In some optional embodiments, the character feature database further includes at least one character identifier, and the identifier of at least one character corresponds one-to-one with the feature portrait of at least one character. The acquisition unit 601 is specifically used to acquire the feature portrait of the target character based on the identifier of the target character.

[0132] In some optional embodiments, the processing unit 602 is further used to input the video to be processed into the second redraw model to obtain the second redraw video. The first redraw video and the second redraw video are spliced ​​together to obtain the third redraw video.

[0133] In some optional embodiments, the processing unit 602 is specifically used to combine the first redraw style and the video to be processedThe initial video is input to the second redrawing model, and the first redrawing style is used to indicate the style in which the video to be processed will be redrawn.

[0134] In some optional embodiments, the acquisition unit 601 is specifically used to input the initial video to the second redrawing model to obtain the video to be processed.

[0135] The processing unit 602 is also used to splice the first redrawing video and the video to be processed to obtain the fourth redrawing video.

[0136] In some optional embodiments, the acquisition unit 601 is specifically used to input the initial video and the second redrawing style to the second redrawing model to obtain the video to be processed. The second redrawing style is used to indicate the style in which the initial video will be redrawn.

[0137] In some optional embodiments, the processing unit 602 is also used to split the video to be processed into multiple video frames. Starting from the first frame of the multiple video frames, all the people included in the multiple video frames are determined based on the person feature detection algorithm. The specification, pages 13 / 16, CN 121169667 A, stores the feature portraits of all people in a character feature database, wherein the feature portrait of each person is included in each third image of the video to be processed, and the confidence level of each person in each third image is higher than the confidence level of each person in other video frames.

[0138] In some optional embodiments, the feature region of the target person includes the facial feature region of the target person and / or the clothing feature region of the target person.

[0139] The acquisition unit 601 and the processing unit 602 can be implemented by software or by hardware. For example, the implementation of the processing unit 602 will be described below. Similarly, the implementation of the acquisition unit 601 can refer to the implementation of the processing unit 602.

[0140] As an example of a software functional unit, the processing unit 602 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above computing instance may be one or more. For example, processing unit 602 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0141] Similarly, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same virtual private cloud (VPN).In a cloud (VPC), it can also be distributed across multiple VPCs. Typically, one VPC is set up within one region. Communication between two VPCs within the same region, and between VPCs in different regions, requires a communication gateway within each VPC to achieve interconnection between VPCs.

[0142] As an example of a hardware functional unit, the processing unit 602 may include at least one computing device, such as a server. Alternatively, the processing unit 602 may be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0143] The multiple computing devices included in the processing unit 602 can be distributed in the same region or in different regions. The multiple computing devices included in the processing unit 602 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the processing unit 602 can be distributed in the same VPC or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0144] It should be noted that the video processing device 600 achieves all its functions by implementing different steps in the video processing method through the acquisition unit 601 and the processing unit 602. The video processing device 600 is used to execute the operations performed by the client in the embodiment shown in FIG1, the cloud service platform in the embodiment shown in FIG2, or the video processing device in the embodiments shown in FIG3 to 5, which will not be elaborated here.

[0145] Please refer to FIG7, which is a schematic diagram of the structure of a computing device provided in an embodiment of this application. The computing device 700 includes a processor 701, a communication interface 702, a bus 703, and a memory 704. The processor 701, the communication interface 702, and the memory 704 communicate through the bus 703. In practical applications, communication can also be achieved through other means such as wireless transmission, which is not limited here. Instruction manual, pages 14 / 16, 18 CN 121169667 A

[0146] The computing device 700 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 700.

[0147] The processor 701 may include any one or more of processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0148] The communication interface 702 uses transceiver modules such as, but not limited to, network interface cards and transceivers to realize communication between the computing device 700 and other devices or communication networks.

[0149] The bus 703 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into address bus, data bus, control bus, etc. For ease of representation, only one line is used in Figure 7, but this does not mean that there is only one bus or one type of bus. Bus 703 may include a path for transmitting information between various components of computing device 700 (e.g., memory 704, processor 701, communication interface 702).

[0150] Memory 704 may include volatile memory, such as random access memory (RAM). Memory 704 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0151] Executable program code is stored in memory 704, and processor 701 executes the executable program code to implement the functions of the aforementioned acquisition unit 601 and processing unit 602, thereby implementing the video processing method. That is, instructions for executing the video processing method are stored in memory 704.

[0152] Embodiments of this application also provide a computing device cluster, which includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some alternative implementations, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0153] Please refer to Figures 8 and 9, which are schematic diagrams of the structure of the computing device cluster provided in the embodiments of this application.

[0154] As shown in Figure 8, the computing device cluster includes at least one computing device 700. The memory 704 of one or more computing devices 700 in the computing device cluster may store the same instructions for executing the video processing method provided in the embodiments of this application.

[0155] In some possible implementations, the memory 704 of one or more computing devices 700 in the computing device cluster may also store some instructions for executing the video processing method. In other words, the combination of the memory 704 of one or more computing devices can jointly execute the instructions for executing the video processing method.

[0156] It should be noted that the memory 704 of different computing devices 700 in the computing device cluster may store different instructions, which are used to execute some functions of the video processing device. That is, the instructions stored in the memory 704 of different computing devices 700 can realize the functions of one or more units in the acquisition unit 601 and the processing unit 602.

[0157] In some possible implementations, one or more computing devices in the computing device cluster can be connected through a network. The network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 9 shows one possible implementation. As shown in Figure 9, two computing devices 700A and 700B are connected via a network. Specifically, they are connected to the network through the communication interface in each computing device. In this type of possible implementation, the memory 704 in computing device 700A stores instructions for executing the function of the acquisition unit 601. At the same time, the memory 704 in computing device 700B stores instructions for executing the function of unit 602.

[0158] The connection method between the computing device cluster shown in Figure 9 can be considered in the video processing method provided in this application, where the processing operation and the operation other than the processing operation are executed separately. That is, the function of the acquisition unit 601 is considered to be executed by computing device 700A, and the function of the processing unit 602 is executed by computing device 700B.

[0159] It should be understood that the function of computing device 700A shown in FIG. 9 can also be performed by multiple computing devices 700. Similarly, the function of computing device 700B can also be performed by multiple computing devices 700.

[0160] This application embodiment also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similarly referred to the connection method of the computing device cluster shown in FIG. 8 and FIG. 9, and will not be repeated here.

[0161] This application embodiment also provides a computer program product containing instructions. The computer program product can...This refers to a software or program product containing instructions that can run on a computing device or be stored in any available medium. When the computer program product is run on at least one computer device, it causes the at least one computer device to perform the video processing method described above.

[0162] This application embodiment also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc. The computer-readable storage medium includes instructions that instruct the computing device to perform the video processing method described above.

[0163] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the systems, devices, and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application. Instruction Manual 16 / 16 Page 20 CN 121169667 A Figure 1 Figure 2 Instruction Manual Appendix 1 / 6 Page 21 CN 121169667 A Figure 3 Instruction Manual Appendix 2 / 6 Page 22 CN 121169667 A Figure 4 Instruction Manual Appendix 3 / 6 Page 23 CN 121169667 A Figure 5 Figure 6 Instruction Manual Appendix 4 / 6 Page 24 CN 121169667 A Figure 7 Figure 8 Instruction Manual Appendix 5 / 6 Page 25 CN 121169667 A Figure 9 Instruction Manual Appendix 6 / 6 Page 26 CN 121169667 A Our Ref.: C065 / P1031 / HK VIDEO PROCESSING METHOD AND RELATED DEVICE Abstract The present application provides a video processing method and a related device, used for improving the consistency of a target character ina video that has undergone style redrawing, and improving the quality of a redrawn video. The method includes: acquiring a video to be processed, where the video to be processed includes a target character; acquiring a feature portrait of the target character from a character feature library, where the feature portrait of the target character corresponds to a feature region of the target character, and the character feature library includes a feature portrait of at least one character; and inputting the feature portrait of the target character and the feature region of the target character in the video to be processed into a first redrawing model to obtain a first redrawn video.

Claims

1. A video processing method, characterized in that, include: Acquire a video to be processed, the video including the target person; The feature profile of the target person is obtained from the feature database, and the feature profile of the target person corresponds to the feature region of the target person. The feature database includes feature profiles of at least one person. The feature profile of the target person and the feature region of the target person in the video to be processed are input into the first redraw model to obtain the first redraw video.

2. The method according to claim 1, characterized in that, The step of inputting the feature profile of the target person and the feature region of the target person in the video to be processed into the first redraw model to obtain the first redraw video includes: The feature profile of the target person, the feature region of the target person in the video to be processed, and the target redrawing style are input into the first redrawing model to obtain the first redrawing video. The redrawing style is used to indicate the style of the feature region of the target person in the first redrawing video.

3. The method according to claim 2, characterized in that, The target redrawing style includes a two-dimensional image style and / or a three-dimensional image style.

4. The method according to any one of claims 1 to 3, characterized in that, After acquiring the video to be processed, the method further includes: In response to a first operation instruction for a target video frame in the video to be processed, the target person is determined, wherein the first operation instruction is used to specify the target person included in the target video frame; The feature profile of the target person is stored in the feature database. The feature profile of the target person is included in the first image of the video to be processed. The confidence level of the target person in the first image is higher than that of the target person in other video frames.

5. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Acquire the target image; In response to a second operation instruction for the target image, the target person is determined, wherein the second operation instruction is used to specify the target person included in the target image; The feature profile of the target person is stored in the feature database. The feature profile of the target person is included in the second image. The confidence level of the target person in the second image is higher than that of the target person in other images. The other images are multiple video frames included in the video to be processed and images in the target image other than the second image.

6. The method according to any one of claims 1 to 5, characterized in that, The character feature database also includes the identifier of at least one character, and the identifier of the at least one character corresponds one-to-one with the feature portrait of the at least one character; The step of obtaining the feature profile of the target person from the feature database includes: Based on the identifier of the target person, obtain a feature profile of the target person.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: The video to be processed is input into the second redraw model to obtain the second redraw video; The first redraw video and the second redraw video are spliced ​​together to obtain the third redraw video.

8. The method according to any one of claims 1 to 6, characterized in that, The step of obtaining the video to be processed includes: inputting the initial video into the second redraw model to obtain the video to be processed; The method further includes: splicing the first redrawn video and the video to be processed to obtain a fourth redrawn video.

9. The method according to any one of claims 1 to 7, characterized in that, After acquiring the video to be processed, the method further includes: The video to be processed is split into multiple video frames; Starting from the first frame of the plurality of video frames, all the people included in the plurality of video frames are determined based on a person feature detection algorithm; The feature profiles of all the individuals are stored in the individual feature database, wherein the feature profile of each individual is included in each third image of the video to be processed, and the confidence level of each individual in each third image is higher than the confidence level of each individual in other video frames.

10. The method according to any one of claims 1 to 9, characterized in that, The characteristic regions of the target person include the facial characteristic regions of the target person and / or the clothing characteristic regions of the target person.

11. A video processing apparatus, characterized in that, include: An acquisition unit is used to acquire a video to be processed, wherein the video to be processed includes the target person; The acquisition unit is further configured to acquire the feature portrait of the target person from the feature database, wherein the feature portrait of the target person corresponds to the feature region of the target person, and the feature database includes the feature portrait of at least one person. The processing unit is used to input the feature portrait of the target person and the feature region of the target person in the video to be processed into the first redraw model to obtain the first redraw video.

12. The apparatus according to claim 11, characterized in that, The processing unit is specifically used for: The feature profile of the target person, the feature region of the target person in the video to be processed, and the target redrawing style are input into the first redrawing model to obtain the first redrawing video. The redrawing style is used to indicate the style of the feature region of the target person in the first redrawing video.

13. The apparatus according to claim 12, characterized in that, The target redrawing style includes a two-dimensional image style and / or a three-dimensional image style.

14. The apparatus according to any one of claims 11 to 13, characterized in that, The processing unit is further configured to: In response to a first operation instruction for a target video frame in the video to be processed, the target person is determined, wherein the first operation instruction is used to specify the target person included in the target video frame; The feature profile of the target person is stored in the person feature database. The feature profile of the target person is included in the first image of the video to be processed. The confidence level of the target person in the first image is higher than that of the target person in the other video frames.

15. The apparatus according to any one of claims 11 to 13, characterized in that, The acquisition unit is also used to acquire the target image; The processing unit is further configured to respond to a second operation instruction for the target image to determine the target person, wherein the second operation instruction is configured to specify the target person included in the target image; The processing unit is further configured to store the feature portrait of the target person in the person feature database. The feature portrait of the target person is contained in the second image. The confidence level of the target person in the second image is higher than that of the target person in other images. The other images are multiple video frames included in the video to be processed and images in the target image other than the second image.

16. The apparatus according to any one of claims 11 to 15, characterized in that, The character feature database also includes the identifier of at least one character, and the identifier of the at least one character corresponds one-to-one with the feature portrait of the at least one character; The acquisition unit is specifically used to acquire a feature profile of the target person based on the target person's identifier.

17. The apparatus according to any one of claims 11 to 16, characterized in that, The processing unit is further configured to: The video to be processed is input into the second redraw model to obtain the second redraw video; The first redraw video and the second redraw video are spliced ​​together to obtain the third redraw video.

18. The apparatus according to any one of claims 11 to 16, characterized in that, The acquisition unit is specifically used to input the initial video into the second redraw model to obtain the video to be processed. The processing unit is also used to stitch together the first redrawn video and the video to be processed to obtain a fourth redrawn video.

19. The apparatus according to any one of claims 11 to 17, characterized in that, The processing unit is further configured to: The video to be processed is split into multiple video frames; Starting from the first frame of the plurality of video frames, all the people included in the plurality of video frames are determined based on a person feature detection algorithm; The feature profiles of all the individuals are stored in the individual feature database, wherein the feature profile of each individual is included in each third image of the video to be processed, and the confidence level of each individual in each third image is higher than the confidence level of each individual in other video frames.

20. A video processing apparatus, characterized in that, Includes a processor, which is coupled to a memory; The memory stores instructions that, when executed on the processor, cause the video processing apparatus to perform the method of any one of claims 1 to 10.

21. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 10.

22. A computer program product containing instructions, characterized in that, When the instruction is executed on the communication device, it causes the communication device to perform the method as described in any one of claims 1 to 10; or, When the instruction is executed by the computing device cluster, the computing device cluster performs the method as described in any one of claims 1 to 10.

23. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer program instructions that, when executed on a communication device, cause the communication device to perform the method as described in any one of claims 1 to 10; or, When the computer program instructions are executed by the computing device cluster, the computing device cluster performs the method as described in any one of claims 1 to 10.

Images