Video processing method and device, intelligent agent, equipment, medium and product

Abstract

The invention provides a video processing method and device, an intelligent agent, equipment, a medium and a product, belongs to the technical field of artificial intelligence, and particularly relates to the technical fields of large models, deep learning, virtual images, digital humans, meta universe and the like. According to the specific implementation scheme, the method comprises the steps of determining a first original video frame and a second original video frame from a plurality of original video frames based on first display mode information of original objects included in the plurality of original video frames in an original video; performing object replacement on an original object in the first original video frame by using the reference image to obtain a first target video frame; performing object replacement on an original object in the second original video frame by using the first target video frame to obtain a second target video frame; and obtaining a target video based on the first target video frame and the second target video frame.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of large models, deep learning, virtual avatars, digital humans, and metaverse, specifically to video processing methods, devices, intelligent agents, electronic devices, storage media, and program products. Background Technology

[0002] With the development of generative technologies, video generation technology has been widely applied in many fields such as film and television production, virtual interaction, and content creation. For example, video generation technology can be used to replace original objects in the original video. Currently, how to improve the realism of the video after object replacement has become a key research focus. Summary of the Invention

[0003] This disclosure provides a video processing method, apparatus, intelligent agent, electronic device, storage medium, and program product.

[0004] According to one aspect of this disclosure, a video processing method is provided, comprising: determining a first original video frame and a second original video frame from a plurality of original video frames based on first display method information of original objects included in each of the original video frames in the original video; performing object replacement on the original objects in the first original video frame using a reference image to obtain a first target video frame, wherein the first target video frame includes a first target object, the identity information of the first target object matches the identity information of a reference object in the reference image, and the action information of the first target object matches the action information of the original object in the first original video frame; performing object replacement on the original objects in the second original video frame using the first target video frame to obtain a second target video frame, wherein the second target video frame includes a second target object, the identity information of the second target object matches the identity information of a reference object, and the action information of the second target object matches the action information of the original object in the second original video frame; and obtaining a target video based on the first target video frame and the second target video frame.

[0005] According to another aspect of this disclosure, a video processing apparatus is provided, comprising: a video frame determination module, configured to determine a first original video frame and a second original video frame from a plurality of original video frames based on first display method information of original objects included in each of the plurality of original video frames in the original video; a first processing module, configured to perform object replacement on the original objects in the first original video frame using a reference image to obtain a first target video frame, wherein the first target video frame includes a first target object, the identity information of the first target object matches the identity information of a reference object in the reference image, and the action information of the first target object matches the action information of the original object in the first original video frame; a second processing module, configured to perform object replacement on the original objects in the second original video frame using the first target video frame to obtain a second target video frame, wherein the second target video frame includes a second target object, the identity information of the second target object matches the identity information of a reference object, and the action information of the second target object matches the action information of the original object in the second original video frame; and a video generation module, configured to obtain a target video based on the first target video frame and the second target video frame.

[0006] According to another aspect of this disclosure, an intelligent agent is provided, comprising: an input module for receiving input information; a processing module for determining a target task based on the input information received by the input module, determining a large model based on the target task, and obtaining output information by calling the large model to execute the method described above; and an output module for outputting the output information obtained by the processing module.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods described above.

[0009] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described above.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0012] Figure 1 This illustration schematically shows an exemplary system architecture to which video processing methods and apparatus can be applied according to embodiments of the present disclosure;

[0013] Figure 2 A flowchart illustrating a video processing method according to an embodiment of the present disclosure is shown schematically.

[0014] Figure 3A This illustration shows a schematic diagram of dividing the original video frames according to an embodiment of the present disclosure;

[0015] Figure 3B This illustration schematically shows a diagram of splitting the original video frame according to an embodiment of the present disclosure;

[0016] Figure 4 This illustration shows a schematic diagram of segmenting the original video according to an embodiment of the present disclosure;

[0017] Figure 5 This schematic diagram illustrates the determination of first display mode information according to an embodiment of the present disclosure;

[0018] Figure 6A This illustration schematically shows a diagram of determining a first target video frame according to an embodiment of the present disclosure;

[0019] Figure 6B A schematic diagram illustrating the determination of a second target video frame according to an embodiment of the present disclosure is shown.

[0020] Figure 7 A schematic diagram illustrating the determination of a reference image according to an embodiment of the present disclosure is shown.

[0021] Figure 8 The illustration schematically shows a diagram of generating a plurality of candidate reference images according to an embodiment of the present disclosure;

[0022] Figure 9 A block diagram of a video processing apparatus according to an embodiment of the present disclosure is shown schematically;

[0023] Figure 10 A schematic diagram illustrating the structure of an intelligent agent according to embodiments of the present disclosure is shown; and

[0024] Figure 11 A block diagram of an electronic device suitable for implementing a video processing method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0025] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0026] Video character replacement is a key technology in Artificial Intelligence Generated Content (AIGC) for film and television production, short video creation, and virtual human interaction in the metaverse. With the explosion of the short video industry, creators' demand for high-frequency, high-quality video content remakes has surged.

[0027] However, in the field of video post-production and derivative works, video character replacement technology currently suffers from technical problems such as inconsistencies in character appearance between multiple video frames, as well as inconsistencies in lighting, background, and other information.

[0028] In view of the above, embodiments of this disclosure provide a video processing method, including: determining a first original video frame and a second original video frame from multiple original video frames based on first display method information of original objects included in each of multiple original video frames in an original video; replacing the original objects in the first original video frame with objects using a reference image to obtain a first target video frame, wherein the first target video frame includes a first target object, the identity information of the first target object matches the identity information of a reference object in the reference image, and the action information of the first target object matches the action information of the original object in the first original video frame; replacing the original objects in the second original video frame with objects using the first target video frame to obtain a second target video frame, wherein the second target video frame includes a second target object, the identity information of the second target object matches the identity information of a reference object, and the action information of the second target object matches the action information of the original object in the second original video frame; and obtaining a target video based on the first target video frame and the second target video frame.

[0029] Using the video processing method provided in this disclosure, based on a reference object in a reference image, the original object of the first original video frame in the original video is replaced to obtain a first target video frame. Then, based on the first target object of the first target video frame, other original video frames of the original video, such as the second original video frame, are replaced with objects. This allows the reference image to be used as a trigger condition, improving the accuracy of object replacement. Using the first target video frame as the basis for object replacement of the second original video frame can utilize the visual consistency between the lighting, background, and other pixel information of the first target video frame and the lighting, background, and other pixel information of the second original video frame, improving the visual consistency of the target video after character replacement, and thus improving the realism of object replacement in the target video.

[0030] Figure 1 The illustration schematically depicts an exemplary system architecture to which video processing methods and apparatus can be applied according to embodiments of the present disclosure.

[0031] It is important to note that Figure 1 The examples shown are merely examples of system architectures applicable to embodiments of this disclosure, intended to help those skilled in the art understand the technical content of this disclosure. They do not imply that embodiments of this disclosure cannot be used in other devices, systems, environments, or scenarios. For instance, in another embodiment, an exemplary system architecture to which video processing methods and apparatus can be applied may include a terminal device. However, the terminal device can implement the video processing methods and apparatus provided in the embodiments of this disclosure without interacting with a server.

[0032] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0033] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (for example only).

[0034] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0035] Server 105 can be a server that provides various services, such as a backend management server that supports the content browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0036] A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system. It solves the shortcomings of traditional physical hosts and VPS services ("Virtual Private Server", or simply "VPS"), such as high management difficulty and weak business scalability. A server can also be a server for a distributed system or a server that incorporates blockchain technology.

[0037] It should be noted that the video processing method provided in the embodiments of this disclosure can generally be executed by terminal devices 101, 102, or 103. Correspondingly, the video processing apparatus provided in the embodiments of this disclosure can also be disposed in terminal devices 101, 102, or 103.

[0038] Alternatively, the video processing method provided in this embodiment can generally be executed by server 105. Correspondingly, the video processing apparatus provided in this embodiment can generally be located in server 105. The video processing method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the video processing apparatus provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0039] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0040] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of any type of information, such as user personal information, comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0041] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0042] It should be noted that the sequence numbers of the operations in the following methods are for descriptive purposes only and should not be considered as indicating the execution order of the operations. Unless explicitly stated otherwise, the method does not need to be executed in the exact order shown.

[0043] Figure 2 A flowchart illustrating a video processing method according to an embodiment of the present disclosure is shown schematically.

[0044] like Figure 2 As shown, the method includes operations S210~S240.

[0045] In operation S210, based on the first display mode information of the original object included in each of the multiple original video frames in the original video, the first original video frame and the second original video frame are determined from the multiple original video frames.

[0046] In operation S220, using the reference image, the original object in the first original video frame is replaced to obtain the first target video frame.

[0047] In operation S230, the original object in the second original video frame is replaced using the first target video frame to obtain the second target video frame.

[0048] In operation S240, the target video is obtained based on the first target video frame and the second target video frame.

[0049] The original video can be live-action, but it's not limited to that; it can also be animated. It just needs to be composed of multiple original video frames.

[0050] If multiple original video frames contain the same original object A, but the display method of original object A may differ in each of the multiple original video frames, then the first display method information will be different. The first display method information may include at least one of the following: display category, the amount of display information of the entire object area, and the amount of information of the target object area.

[0051] For example, if original object A is standing in original video frame 1, sitting in original video frame 2, and lying down in original video frame 3, then the amount of information displayed across the entire object area differs in the three original video frames. As another example, if original object A is displayed as both a distant view and a close-up view in different original video frames, then the display category differs across the multiple original video frames.

[0052] Based on the first display method information of the original object included in each original video frame, the original video frame with the most display information of the original object can be determined from multiple original video frames as the first original video frame, and the other original video frames in the original video can be used as the second original video frames.

[0053] Optionally, the number of frames in the second original video frame and the first original video frame is not limited. It can be one frame, but it is not limited to this and can also include multiple frames.

[0054] A reference image includes a reference object. The reference object can be different from the original object. For example, the original object is object A, a male with short black hair, while the reference object is object B, a female with long blonde hair.

[0055] By using a reference image, the original object in the first original video frame can be replaced to obtain a first target video frame that includes the first target object. The first target video frame can include the first target object, the identity information of the first target object matches the identity information of the reference object in the reference image, and the action information of the first target object matches the action information of the original object in the first original video frame.

[0056] Motion information can include at least one of posture, facial expression, and lip movement. For example, posture information can be used to describe the spatial posture of the original object's head and / or body. Facial expression information can be used to describe the deformation of the original object's facial muscles and / or facial features. Lip movement information can be used to describe the opening and closing and shape changes of the original object's lips.

[0057] Identity information can include characteristic information used to uniquely identify an object. For example, it can include information such as skin color and facial features that identify an object.

[0058] For example, the first target video frame may include a first target object that is represented as object B and whose actions still retain the actions of object A in the first original video frame.

[0059] By using the first target video frame to replace the original object in the second original video frame, a second target video frame including the second target object can be obtained.

[0060] The identity information of the second target object can be matched with the identity information of the reference object, and the action information of the second target object can be matched with the action information of the original object in the second original video frame.

[0061] For example, the second target video frame may include a second target object that is represented as object B and whose actions still retain the actions of object A in the second original video frame.

[0062] The video processing method provided in this disclosure uses a reference object in a reference image as a basis to replace the original object in the first original video frame of the original video to obtain a first target video frame. Then, using the first target object of the first target video frame as a basis, other original video frames of the original video, such as the second original video frame, are replaced with objects. This allows the reference image to be used as a trigger condition, improving the accuracy of object replacement. Furthermore, using the first target video frame as the basis for object replacement in the second original video frame can utilize the visual consistency between the lighting, background, and other pixel information of the first target video frame and the lighting, background, and other pixel information of the second original video frame, improving the visual consistency of the target video after character replacement, thereby enhancing the realism of object replacement in the target video.

[0063] In one example, characters in the original video can be replaced using manual masking, motion tracking, and 3D model synthesis.

[0064] Compared with the method of synthesis using three-dimensional models, the video object replacement method provided by the embodiments of this disclosure can reduce labor costs and the requirements for professional skills, and improve processing efficiency.

[0065] In another example, the reference image can be used to perform object replacement on the first original video frame and the second original video frame respectively to obtain the first target video frame and the second target video frame.

[0066] Compared with the method of replacing objects in multiple original video frames using the same reference image, the video object replacement method provided by the present disclosure can combine the correlation between multiple original video frames to avoid the problem of flickering video images and sudden changes in the image of the person at different angles, thereby ensuring the consistency of background pixels and object actions in time sequence.

[0067] According to embodiments of this disclosure, for example, Figure 2 The operation S210 shown, which determines a first original video frame and a second original video frame from multiple original video frames based on the first display mode information of the original objects included in each of the multiple original video frames in the original video, may include: if it is determined that the multiple original video frames meet predetermined conditions based on the first display mode information of the original objects included in each of the multiple original video frames, then determining the first original video frame from the multiple original video frames based on the temporal relationship information of the multiple original video frames in the original video, wherein the predetermined conditions include the difference in the amount of information of the target object regions of the original objects displayed in each of the multiple original video frames being less than or equal to a predetermined threshold. The other video frames from the multiple original video frames, excluding the first original video frame, are then used as the second original video frame.

[0068] Since the target object region can be used to distinguish the identity information of the original object, the region that contributes most significantly to distinguishing the object's identity can be selected as the target object region. However, information such as body shape and movement is less relevant to distinguishing the object's identity than facial features; therefore, the target object region can refer to the face of the original object. The amount of information in the target object region of the original object displayed in each original video frame can be used to represent the position and size of the original object's face within that original video frame.

[0069] If multiple original video frames meet predetermined conditions, it can be determined that the facial regions of the original objects included in each of the multiple original video frames are close in position and have stable facial sizes. Therefore, the original objects in multiple original video frames can be replaced, improving the replacement accuracy.

[0070] For multiple original video frames that meet predetermined conditions, the first original video frame can be selected as the first original video frame based on the temporal relationship information of the multiple original video frames in the original video, and the other video frames can be selected as the second original video frames. However, this is not a limitation. The last original video frame can also be selected as the first original video frame, and the other video frames can be selected as the second original video frames. As long as the original video frame is easy to select, it can be used as the first original video frame.

[0071] Figure 3A The illustration shows a schematic diagram of the division of the original video frame according to an embodiment of the present disclosure.

[0072] like Figure 3A As shown, after splitting the original video into frames, N original video frames are obtained, P1, P2, ..., PN. Based on the first display mode information of the original objects included in each of the N original video frames, if the difference in the amount of information of the facial regions of the original objects included in the multiple original video frames is less than or equal to a predetermined threshold, then it can be determined that the N original video frames of the original video meet the predetermined conditions.

[0073] Based on the temporal relationship information between the N original video frames, it can be determined that P1 is the first original video frame. Therefore, P1 can be used as the first original video frame, and P2, ..., PN can be used as the second original video frames.

[0074] According to embodiments of this disclosure, when multiple original video frames meet predetermined conditions, it can be determined that the display states of the original objects in the multiple original video frames are similar, and their postures and regions are stable, thus providing a reference for object replacement. By selecting a first original video frame based on its temporal relationship within the original video, and using the remaining original video frames as second original video frames, the first target video frame can be used to replace the second original video frame after processing the first original video frame. This reduces redundant calculations, improves processing efficiency, maintains consistency in replacement results, avoids instability caused by excessive inter-frame differences, and improves the overall video processing effect.

[0075] According to embodiments of this disclosure, for example, Figure 2 The operation S210 shown may further include: when it is determined that multiple original video frames do not meet predetermined conditions based on the first display mode information of the original objects included in each of the multiple original video frames, the multiple original video frames are divided based on the amount of information of the target object region of the original objects included in each of the multiple original video frames and the temporal relationship information of the multiple original video frames to obtain multiple original video frames that meet predetermined conditions.

[0076] If, based on the first display method information of the original objects included in each of the original video frames, it is determined that multiple original video frames do not meet the predetermined conditions, it can be determined that there are original video frames in the original video where the differences between the facial regions of the original objects are large. In this case, if the first original video frame and the second original video frame are still directly divided, and the second video frame is processed using the processed first target video frame, the replacement effect between the original video frames where the differences between the facial regions of the original objects are large will be poor, and it will be difficult to ensure authenticity.

[0077] Therefore, based on the amount of information about the target object region of the original object included in each of the multiple original video frames and the temporal relationship information of the multiple original video frames, original video frames that are temporally adjacent and whose difference in the amount of information about the target object region of the original object is greater than a predetermined threshold can be divided into different video segments, so that multiple original video frames in each video segment meet the predetermined conditions.

[0078] Figure 3B The illustration shows a schematic diagram of splitting the original video frame according to an embodiment of the present disclosure.

[0079] like Figure 3B As shown, after splitting the original video into frames, N original video frames are obtained, P1, P2, ..., PN. Based on the first display method information of the original objects included in each of the N original video frames, if the difference in the amount of information of the facial regions of the original objects included in the multiple original video frames is greater than a predetermined threshold, then it can be determined that the N original video frames do not meet the predetermined conditions.

[0080] For example, P1, ..., PM meet the predetermined conditions, and PM+1, ..., PN also meet the predetermined conditions, but PM and PM+1 do not meet the predetermined conditions. In this case, the original video frames can be divided into two segments. The first segment includes the original video frames P1, ..., PM, and the second segment includes the original video frames PM+1 to PN.

[0081] For the two original video frames after splitting, the first original video frame P1 in the first video frame can be used as the first original video frame, and P2, ..., PM can be used as the second original video frames. The first original video frame PM+1 in the second video frame can be used as the first original video frame, and PM+2, ..., PN can be used as the second original video frames.

[0082] According to embodiments of this disclosure, by jointly dividing the target object region information and timing, original video frames with large differences in posture, region, and display mode can be divided into different stable video frame segments, making the display state of the target object region similar among multiple original video frames within each segment. This avoids problems such as replacement failure and target object distortion caused by excessive differences in the information content of the target object region in the original video frames, thereby improving the robustness and adaptability of video processing.

[0083] According to embodiments of this disclosure, when performing such Figure 2 Before operation S210, the video processing method may further include: dividing the original video into segments to determine multiple original video segments, wherein each original video frame in the original video segment includes the same scene and original object. The target object region of the original object included in each original video frame in the original video segment is detected to obtain first display mode information of the original object included in each original video frame.

[0084] The scenes and original objects included in each of the multiple original video frames in the original video segment can be determined. Based on the temporal relationship information of the multiple original video frames and the scenes and original objects included in each of the multiple original video frames, the multiple original video frames are divided into multiple original video segments. Among them, the multiple original video frames in each original video segment are adjacent in the original video.

[0085] Figure 4 The illustration shows a schematic diagram of segmenting the original video according to an embodiment of the present disclosure.

[0086] like Figure 4 As shown, after splitting the original video into frames, we obtain N original video frames: P1, P2, ..., PN. We then determine the scenes and original objects included in each of P1, ..., PN, and divide the original video into segments based on these scenes and original objects.

[0087] For example, P1, ..., PX are indoor scenes, and the original video frames include the original object A; PX+1, ..., PY are outdoor scenes, and the original video frames include the original object A; PY+1, ..., PZ are indoor scenes, and the original video frames include the original object A; PZ+1, ..., PN are indoor scenes, and the original video frames include the original object B.

[0088] Based on the temporal relationship information of multiple original video frames, and the scenes and original objects included in each of the multiple original video frames, the original video can be divided into 4 original video segments. The first original video segment includes original video frames P1, ..., PX; the second original video segment includes original video frames PX+1, ..., PY; the third original video segment includes original video frames PY+1, ..., PZ; and the fourth original video segment includes original video frames PZ+1, ..., PN.

[0089] As can be seen, although the scenes and original objects of multiple original video frames in the first and third original video segments are the same, the original video frames can be divided into different original video segments because the first and third original video segments are not temporally adjacent.

[0090] According to embodiments of this disclosure, segmenting the original video into segments can yield multiple original video segments with the same scene and original object. This avoids interference caused by inconsistencies in scene or original object during subsequent replacement processing based on the original video segments, improving the success rate and efficiency of video processing. Detecting the target object region of each original video frame within each original video segment yields first display mode information. This ensures that subsequent operations such as video frame splitting and object replacement based on the first display mode are all built upon stable and continuous video segments with the same scene and object, thereby improving the accuracy of video processing.

[0091] According to embodiments of this disclosure, detecting the target object region of the original object included in each original video frame of the original video segment to obtain first display mode information of the original object included in each original video frame may include: performing image detection on the original video frame to determine the position identification information of the target object region of the original object in the original video frame; and determining the information content ratio of the target object region of the original object in the original video frame based on the position identification information of the target object region of the original object to obtain the first display mode information.

[0092] The target object region identifies the face region of the original object. Therefore, a face detection model can be used to perform image detection on the original video frame to determine the location identification information of the target object region of the original object in the original video frame.

[0093] For example, object detection can be performed on the original video frame, and the output video frame with bounding boxes superimposed can be obtained. Based on the position of the bounding boxes in the original video frame, the location identification information of the target object region can be determined.

[0094] Location identification information can be used to identify the area of ​​a target object. Therefore, based on the location identification information of the target object area of ​​the original object, the amount of information in the target object area can be calculated, and the proportion of information in the target object area within the original video frame can be determined. The location identification information and the proportion of information are then used as the primary display information.

[0095] Figure 5 A schematic diagram illustrating the determination of first display mode information according to an embodiment of the present disclosure is shown.

[0096] like Figure 5 As shown, for any original video frame 501 in the original video, target detection can be performed on the original video frame 501. The original video frame output after target detection has a bounding box 502 used to represent the target object region. Based on the position of the bounding box 502 in the original video frame 501, the position identification information of the target object region of the original object in the original video frame 501 can be determined.

[0097] The area of ​​the target object region can be used as the information content of the target object region in the original video. Therefore, the area of ​​the geometry corresponding to the bounding box can be calculated as the information content of the target object region, and the area of ​​the original video frame can be calculated as the information content of the original video frame. The ratio of the area of ​​the geometry corresponding to the bounding box to the area of ​​the original video frame is used as the proportion of information content of the target object region in the original video frame.

[0098] For example, if the bounding box 502 used to define the target object region has a size of 600px * 800px, and the original video frame 501 has a size of 1920px * 1080px, then the information content of the target object region of the original object in the original video frame is approximately (600px * 800px) / (1920px * 1080px) ≈ 23%.

[0099] According to embodiments of this disclosure, image detection is performed on the original video frame to determine the location identifier information of the target object region of the original object, ensuring accurate region positioning. Based on the location identifier, the information content ratio of the target object region in the entire frame is calculated, allowing the object display method to be represented numerically. By using the location identifier information and information content ratio as the first display method information, the display method of the original object in the original video frame can be quantified, facilitating comparison between multiple original video frames. This provides a reliable numerical basis for subsequent operations such as predetermined condition judgment and segment division, improving the standardization and accuracy of video processing.

[0100] According to embodiments of this disclosure, segmenting the original video to determine multiple original video segments may include: segmenting the original video into scenes to obtain multiple initial original video segments; performing object recognition on each video frame in the initial original video segments to obtain object category information for each of the multiple video frames; and segmenting the initial original video segments based on the object category information for each of the multiple video frames to obtain multiple original video segments.

[0101] In one embodiment, the original video can be split into multiple original video frames, and the original video can be divided into scenes according to the scenes included in each of the multiple original video frames, so that the scenes included in the multiple original video frames in each initial original video segment are consistent.

[0102] In another embodiment, the original video can be parsed using a video scene detection-based shot segmentation tool to identify and locate the switching boundaries of different scenes in the original video, and a content detector can be used to determine the timestamp information of the original video frames corresponding to the switching boundaries from the original video.

[0103] The video frame rate of the original video can be obtained by using video stream information analysis tools, and the frame number of the original video frame corresponding to the switching boundary can be calculated by combining the timestamp information of the original video frame corresponding to the switching boundary in the original video.

[0104] Based on the frame sequence number of the original video frame corresponding to the switching boundary, the segmentation point for splitting the original video can be determined. Video filtering tools can then be used to extract original video frames from the original video according to the segmentation point, resulting in multiple initial original video segments. Alternatively, the original video stream can be re-encapsulated through methods such as stream copying and hardware-accelerated re-encoding. During the encapsulation process, the video can be divided according to multiple segmentation points to obtain multiple initial original video segments.

[0105] Facial recognition tools can be used to identify objects in each frame of each initial video segment. For example, a lightweight facial recognition tool can be used to determine the location of objects in each video frame, and then a facial feature extraction tool can be used to determine the high-dimensional object features based on the object's location. These high-dimensional object features can then be used as the object category information for the video frame.

[0106] The object category information of each video frame in the initial original video segment can be used to divide the initial original video segment into objects, so that the object category information of the video frames divided into the same original video segment is consistent.

[0107] For example, unsupervised spatial clustering algorithms can be used to process the initial original video segments. The object category information of the earliest video frame in the initial original video segment is used as reference information. Subsequent video frames are processed sequentially according to their chronological order within the initial original video segment. The object category information of each video frame is compared with the reference information to obtain the similarity between the two. Similarity can be constructed using methods such as cosine similarity or Euclidean distance. If the similarity is less than a preset threshold, the object category information of the current video frame is considered inconsistent with that of the previous video frame. If the similarity is greater than or equal to the preset threshold, the object category information of the current video frame is considered consistent with that of the previous video frame.

[0108] According to embodiments of this disclosure, the original video is divided into multiple initial original video segments based on the scene, thereby achieving scene differentiation. Object recognition is performed on each video frame within the initial original video segments, and the initial original video segments are further divided based on the obtained object category information to obtain original video segments. Through this two-layer division, the complex division task can be decoupled into two simple division tasks, improving processing efficiency and accuracy, and ensuring that the scenes and objects included in the video frames of the original video segments are consistent.

[0109] According to embodiments of this disclosure, for example, Figure 2 The operation S220 shown, which uses a reference image to replace the original object in the first original video frame to obtain a first target video frame, may include: masking the original object in the first original video frame to obtain a first mask image; and processing the first mask image based on the reference image and the first action information of the original object in the first original video frame to obtain the first target video frame.

[0110] Object detection can be performed on the first original video frame to obtain the object detection result. Based on the object detection result, the original objects in the first original video frame are masked to obtain the first mask image.

[0111] Object detection models such as YOLO (You Only Live Once, an object detection model) can be used to detect objects in the first original video frame, but it is not limited to this. Other object detection models can also be used, as long as they are capable of detecting the outlines of the original objects in a fine-grained manner.

[0112] The object detection results can include the object category information of the original object and the position information of the original object in the first original video frame, such as the outline information of the original object.

[0113] Based on the object detection results, the global object region containing the original object in the first original video frame can be determined. The original object is then masked based on this global object region.

[0114] The first target video frame can be obtained by processing the first mask image using the Diffusion Transformer (DiT) model based on the reference image and the first action information of the original object in the first original video frame. However, it is not limited to this. A multimodal large model based on the diffusion model framework can also be used, as well as an improved multimodal large model trained with low-rank matrix fine-tuning.

[0115] According to embodiments of this disclosure, masking the first mask image can clearly define the area of ​​the object to be replaced, improve the guiding effect, and thus improve the object replacement effect and precision. In addition, using the first action information of the original object in the first original video frame as reference information can ensure that the action of the first target object is consistent with the action of the original object, thereby improving the consistency of the action timing between the first target video frame after object replacement and other video frames.

[0116] According to embodiments of this disclosure, the first action information may include first posture information, first facial expression information, and first lip movement information.

[0117] A pose estimation algorithm can be used to extract skeletal features from the original objects in the original video frame to obtain the first pose information. Alternatively, a feature extraction model can be used to extract facial features from the first original video frame to determine the first expression information and the first lip movement information.

[0118] There are no restrictions on the methods for determining the first posture information, the first facial expression information, and the first lip movement information, as long as the corresponding information can be accurately extracted.

[0119] The object replacement model can be used to process the first mask image based on the motion information of the original object in the first original video frame, using the reference image, to obtain the first target video frame.

[0120] The following will be through Figure 6A The method of obtaining the first target video frame using the object replacement model is explained.

[0121] Figure 6A A schematic diagram illustrating the determination of a first target video frame according to an embodiment of the present disclosure is shown.

[0122] like Figure 6A As shown, the object replacement model can include a splicing module, a global driving module, and a detail driving module.

[0123] like Figure 6AAs shown, the stitching module can be used to stitch together the reference image, the first mask image, and the noise image to obtain the first stitched image. The global driving module uses the first pose information to globally drive the first stitched image, resulting in the first global motion-driven image. The detail driving module uses the first facial expression information and the first lip movement information to perform detail driving on the first global motion-driven image, resulting in the first target video frame.

[0124] Noisy images may include, but are not limited to, pure Gaussian noise images, and may also include Gaussian noise images carrying basic information about the facial contours of a reference object.

[0125] In practical applications, the noisy image, the first mask image, and the reference image can be stitched together according to a preset channel dimension to obtain the first stitched image.

[0126] Optionally, the first stitched image can be input into a preset video encoder. The video encoder can adopt a variational autoencoder (VAE) structure, which can perform layer-by-layer feature extraction and encoding transformation on the first stitched image, mapping the first stitched image containing noise interference and constraint information to a high-dimensional feature space, generating image features that combine the basic visual features of the noisy image with the semantic guidance of the first mask image and the reference image.

[0127] The global driving module can prioritize performing global-driven denoising on the first stitched image, filtering out random interference introduced by the noisy image, and retaining the basic semantic and visual features brought by the first pose information to obtain the first global action-driven image. In some embodiments, the global driving module can be implemented based on a self-attention mechanism and normalization, but it is not limited to this. It can also be implemented based on a cross-attention mechanism and a fully connected neural network such as a multi-layer perceptron (MLP).

[0128] The detail-driven module can perform deep denoising on the first global action-driven image based on first facial expression information and first lip movement information. Through a cross-attention mechanism and normalization, the first facial expression information and first lip movement information guide the optimization direction of the first global action-driven image. In some embodiments, the detail-driven module may include a first detail-driven module and a second detail-driven module. The first detail-driven module performs deep denoising on the first global action-driven image based on the first facial expression information to obtain a first initial detail-driven image. The second detail-driven module can perform deep denoising on the first initial detail-driven image based on the first lip movement information to obtain a second initial detail-driven image. The first and second detail-driven modules can be implemented using self-attention mechanisms and normalization, but are not limited to these; they can also be implemented using cross-attention mechanisms and fully connected neural networks such as multi-layer perceptrons (MLPs).

[0129] Multiple processing steps, such as N times, can be performed using global and detail-driven modules to improve the fineness of object replacement.

[0130] Optionally, after completing the final round of processing, a preset video decoder can be used to decode the second initial detail-driven image from the final round to generate the target image. The video encoder can be symmetrical to the video encoder used to encode the first stitched image.

[0131] According to embodiments of this disclosure, using a diffusion-based object replacement model to process the first original video frame can combine the first pose information, first expression information, and first lip movement information in the first original video frame, thereby improving the richness and comprehensiveness of the first action information and thus improving the realism of the first target video frame after object replacement.

[0132] Optionally, the global driving module, the first detail driving module, and the second detail driving module can be obtained by adding new modules to the open-source video generation model and then training it. Furthermore, low-rank matrices can be added to fine-tune the original model parameters of the open-source video generation model to accommodate the object replacement function.

[0133] According to embodiments of this disclosure, for example, Figure 2 The operation S230 shown, which uses the first target video frame to replace the original object in the second original video frame to obtain the second target video frame, may include: masking the original object in the second original video frame to obtain a second mask image; and processing the second mask image based on the first target video frame and the action information of the original object in the second original video frame to obtain the second target video frame.

[0134] Object detection can be performed on the second original video frame using an object detection model to obtain the object detection results. Based on the object detection results, the original objects in the second original video frame are masked to obtain the second mask image.

[0135] The object detection results can include the object category information of the original object and the position information of the original object in the second original video frame, such as the outline information of the original object.

[0136] Based on the object detection results, the global object region containing the original object in the second original video frame can be determined. The original object is then masked based on this global object region.

[0137] DiT can be used to process the second mask image based on the second action information of the original object in the second video frame and the first target video frame to obtain the second target video frame. However, it is not limited to this. A multimodal large model based on a diffusion model framework can also be used, as well as an improved multimodal large model trained with low-rank matrix fine-tuning.

[0138] According to embodiments of this disclosure, masking the second mask image can clearly define the area of ​​the object to be replaced, improving the guiding effect and thus enhancing the object replacement effect and precision. Furthermore, using the second action information of the original object in the second original video frame as reference information ensures that the action of the second target object is consistent with the action of the original object, thereby improving the consistency of the action timing between the second target video frame after object replacement and other video frames. Moreover, using the first target video frame to replace the original object in the second original video frame fully utilizes the consistency of the scene between the first target video frame and the second original video frame, as well as the coherence between the action of the target object in the first target video frame and the action of the original object in the second original video frame, improving the realism of the object replacement in the target video.

[0139] The following will be through Figure 6B The method of obtaining the second target video frame using the object replacement model is explained.

[0140] Figure 6B A schematic diagram illustrating the determination of a second target video frame according to an embodiment of the present disclosure is shown.

[0141] like Figure 6B As shown, the object replacement model can include a splicing module, a global driving module, and a detail driving module.

[0142] like Figure 6BAs shown, the stitching module can be used to stitch together the first target video frame, the second mask image, and the noise image to obtain the second stitched image. The global driving module uses the second pose information to globally drive the second stitched image, resulting in the second global motion-driven image. The detail driving module uses the second facial expression information and the second lip movement information to perform detail driving on the second global motion-driven image, resulting in the second target video frame.

[0143] In practical applications, the noisy image, the second mask image, and the reference image can be stitched together according to a preset channel dimension to obtain the second stitched image.

[0144] Optionally, the second stitched image can be input into a preset video encoder to perform layer-by-layer feature extraction and encoding transformation on the second stitched image, mapping the second stitched image containing noise interference and constraint information to a high-dimensional feature space, generating image features that combine the basic visual features of the noisy image with the semantic guidance of the second mask image and the reference image.

[0145] The global driving module can prioritize global denoising processing on the second stitched image, filtering out random interference introduced in the noisy image, and retaining the basic semantic and visual features brought by the second pose information to obtain the second global action-driven image.

[0146] The detail-driven module can perform deep denoising on the second global action-driven image based on the second facial expression information and the second lip movement information. Through a cross-attention mechanism and normalization, the second facial expression information and the second lip movement information guide the optimization direction of the second global action-driven image. In some embodiments, the detail-driven module may include a first detail-driven module and a second detail-driven module. The first detail-driven module performs deep denoising on the second global action-driven image based on the second facial expression information to obtain a third initial detail-driven image. The second detail-driven module can perform deep denoising on the second initial detail-driven image based on the second lip movement information to obtain a fourth initial detail-driven image. The first and second detail-driven modules can be implemented using a self-attention mechanism and normalization, respectively, but are not limited to these; they can also be implemented using a cross-attention mechanism and a fully connected neural network such as an MLP.

[0147] Multiple processing steps can be performed using global and detail-driven modules to improve the fineness of object replacement.

[0148] Optionally, after completing the final round of processing, a preset video decoder can be used to decode the second initial detail-driven image from the final round to generate the second target video frame. The video encoder can be symmetrical to the video encoder used to encode the second stitched image.

[0149] According to embodiments of this disclosure, using a diffusion-based object replacement model to process the second original video frame can combine the second pose information, second expression information, and second lip movement information in the second original video frame, thereby improving the richness and comprehensiveness of the second action information and thus improving the realism of the second target video frame after object replacement.

[0150] According to embodiments of this disclosure, for example, Figure 2 The operation S240 shown, which obtains a target video based on a first target video frame and a second target video frame, includes: updating the first original video frame and the second original video frame in the original video based on the first target video frame and the second target video frame to obtain the target video.

[0151] According to embodiments of this disclosure, the first and second target video frames can be timestamped based on their respective occurrence times in the original video, such as timestamp information. This allows for updating, for example, replacing, the first and second original video frames in the original video based on their respective timestamp information, thus obtaining the target video.

[0152] According to embodiments of this disclosure, updates are performed using an interpolation method to maintain the smoothness of the editing rhythm, improve processing efficiency, and maintain video continuity and motion consistency.

[0153] As a preferred embodiment of this disclosure, the video processing method may further include: performing watermark recognition on the original video frame, and, if a watermark is determined to exist, performing a watermark removal operation. For example, erasing station logos, subtitles, or watermarks from the original video, and restoring obscured image details.

[0154] Video processing methods may also include at least one of: timbre cloning and translation.

[0155] For example, the speaker embedding of the target object in the original video can be extracted, and cross-language text translation and speech synthesis (such as converting Chinese dialogue into English) technologies can be used to generate speech in other languages ​​while preserving the original object's timbre and rhythm.

[0156] Another example is reconstructing the speech of the original object into the target timbre using a stream matching model (such as changing a male voice into a female voice).

[0157] According to embodiments of this disclosure, subtitles can be added to the target video, and updates such as timbre changes and language changes can be combined to encapsulate the target video, multilingual audio streams, and subtitle streams into a final product. This improves the functionality and user experience of the video processing method.

[0158] According to embodiments of this disclosure, when performing such Figure 2 Prior to operation S220, the video processing method may further include: determining a reference image from the multiple reference images that matches the display mode of the original object in the first original video frame, based on the second display mode information of the reference object included in each of the multiple candidate reference images.

[0159] Figure 7 A schematic diagram of a reference image determined according to an embodiment of the present disclosure is shown.

[0160] like Figure 7 As shown, the display methods of the reference objects in the multiple candidate reference images are all different. For example, they include a front view of the reference object standing in front of the candidate reference image, a side view of the reference object standing sideways, and a back view of the reference object facing away from the candidate reference image. They may also include a side view of the distant view, a back view of the distant view, and a front view of the distant view.

[0161] like Figure 7 As shown, when the original object in the first original video frame is displayed as a close-up front view, the close-up front view can be determined as a reference image from multiple candidate reference images.

[0162] Optionally, the values ​​of the first display mode information and the second display mode information can be compared, and the candidate reference images corresponding to the second display mode information whose ratio meets a predetermined threshold can be used as reference images, thereby improving the richness of the reference images.

[0163] According to embodiments of this disclosure, a reference image is determined from multiple candidate reference images with different display methods, thereby increasing the range of filterable reference images and thus expanding the application scope. Furthermore, filtering reference images based on display methods improves the effectiveness of object replacement when using reference images.

[0164] According to embodiments of this disclosure, before determining the reference image, the video processing method may further include: generating an image based on a reference object in a standard reference image to obtain multiple candidate reference images with different display modes.

[0165] The second display mode information of the reference object in the standard reference image indicates the target display mode.

[0166] Optionally, the target display method may include a close-up front view. It may also include a distant front view, as long as it can include facial information of the reference object.

[0167] Figure 8 The illustration shows a schematic diagram of generating a plurality of candidate reference images according to an embodiment of the present disclosure.

[0168] like Figure 8 As shown, a standard reference image and a prompt message can be input into the image generation model to obtain multiple candidate reference images.

[0169] Optionally, the image generation model is not limited; for example, it can include a large image generation model generated by a diffusion model, but it is not limited to this. It can also include a large image generation model generated by a encoder-decoder. As long as the image can generate multiple candidate reference images, it is acceptable.

[0170] Optionally, there is no limit to the number of standard reference images; for example, it may include one, but it is not limited to this and may include multiple images.

[0171] The prompt message can be used to describe the expected result. For example, the prompt message may include asking you to generate a target number of candidate reference images (Character Sheets) of the reference object from different angles (front view, side view, back view, etc.) based on a standard reference image.

[0172] According to embodiments of this disclosure, using artificial intelligence generation technology to automatically generate multiple candidate reference images with different display modes can improve the generation efficiency and effect of candidate reference images and reduce generation costs.

[0173] Figure 9 A block diagram of a video processing apparatus according to an embodiment of the present disclosure is shown schematically.

[0174] like Figure 9 As shown, the video processing device 900 includes a video frame determination module 910, a first processing module 920, a second processing module 930, and a video generation module 940.

[0175] The video frame determination module 910 is used to determine the first original video frame and the second original video frame from multiple original video frames based on the first display method information of the original objects included in each of the multiple original video frames in the original video.

[0176] The first processing module 920 is used to replace the original object in the first original video frame with an object using a reference image to obtain a first target video frame. The first target video frame includes a first target object, the identity information of the first target object matches the identity information of the reference object in the reference image, and the action information of the first target object matches the action information of the original object in the first original video frame.

[0177] The second processing module 930 is used to replace the original object in the second original video frame with the first target video frame to obtain the second target video frame. The second target video frame includes a second target object, the identity information of the second target object matches the identity information of the reference object, and the action information of the second target object matches the action information of the original object in the second original video frame.

[0178] The video generation module 940 is used to obtain the target video based on the first target video frame and the second target video frame.

[0179] According to embodiments of this disclosure, the video frame determination module 910 includes a first determination submodule and a second determination submodule.

[0180] The first determining submodule is used to determine the first original video frame from the multiple original video frames based on the temporal relationship information of the multiple original video frames in the original video, when it is determined that the multiple original video frames meet the predetermined conditions based on the first display mode information of the original objects included in the multiple original video frames, wherein the predetermined conditions include the difference in the amount of information of the target object area of ​​the original objects displayed by the multiple original video frames is less than or equal to a predetermined threshold.

[0181] The second determining submodule is used to take the other video frames from the multiple original video frames, excluding the first original video frame, as the second original video frame.

[0182] According to embodiments of this disclosure, the video processing apparatus 900 further includes a first video segmentation module.

[0183] The first video segmentation module is used to segment multiple original video frames based on the information of the target object region of the original object included in each of the multiple original video frames and the temporal relationship information of the multiple original video frames, when it is determined that multiple original video frames do not meet the predetermined conditions based on the first display mode information of the original objects included in each of the multiple original video frames. This results in multiple original video frames that meet the predetermined conditions.

[0184] According to embodiments of this disclosure, the video processing apparatus 900 further includes a second video segmentation module and an information determination module.

[0185] The second video segmentation module is used to divide the original video into segments and determine multiple original video segments. Each original video frame in an original video segment contains the same scene and original object.

[0186] The information determination module is used to detect the target object region of the original object included in each original video frame in the original video segment, and obtain the first display mode information of the original object included in each original video frame.

[0187] According to embodiments of this disclosure, the information determination module includes a location determination submodule and an information quantity determination submodule.

[0188] The location determination submodule is used to perform image detection on the original video frame and determine the location identification information of the target object region of the original object in the original video frame.

[0189] The information content determination submodule is used to determine the information content ratio of the target object region in the original video frame based on the location identification information of the target object region of the original object, and to obtain the first display mode information.

[0190] According to embodiments of this disclosure, the second video segmentation module includes a scene segmentation submodule, an object recognition submodule, and an object segmentation submodule.

[0191] The scene segmentation submodule is used to segment the original video into scenes, resulting in multiple initial original video clips.

[0192] The object recognition submodule is used to perform object recognition on each video frame in the initial original video segment to obtain object category information for each of the multiple video frames.

[0193] The object segmentation submodule is used to segment the initial original video segment based on the object category information of each of the multiple video frames, resulting in multiple original video segments.

[0194] According to embodiments of this disclosure, the video processing apparatus 900 further includes an image determination module.

[0195] The image determination module is used to determine a reference image from multiple candidate reference images that matches the display mode of the original object in the first original video frame, based on the second display mode information of the reference object included in each of the multiple candidate reference images.

[0196] According to embodiments of this disclosure, the video processing apparatus 900 further includes an image generation module.

[0197] The image generation module is used to generate an image based on a reference object in a standard reference image, and obtain multiple candidate reference images with different display modes. The second display mode information of the reference object in the standard reference image indicates the target display mode.

[0198] According to embodiments of this disclosure, the first processing module 920 includes a first mask submodule and a first processing submodule.

[0199] The first masking submodule is used to mask the original objects in the first original video frame to obtain the first mask image.

[0200] The first processing submodule is used to process the first mask image based on the reference image and the first action information of the original object in the first original video frame to obtain the first target video frame.

[0201] According to embodiments of this disclosure, the first processing submodule includes a first splicing unit, a first driving unit, and a second driving unit.

[0202] The first stitching unit is used to stitch together the reference image, the first mask image, and the noise image to obtain the first stitched image.

[0203] The first driving unit is used to globally drive the first stitched image based on the first pose information to obtain the first global motion-driven image.

[0204] The second driving unit is used to perform detail driving on the first global motion driving image based on the first facial expression information and the first lip movement information to obtain the first target video frame.

[0205] According to embodiments of this disclosure, the second processing module 930 includes a second mask submodule and a second processing submodule.

[0206] The second masking submodule is used to mask the original objects in the second original video frame to obtain the second mask image.

[0207] The second processing submodule is used to process the second mask image based on the first target video frame and the action information of the original object in the second original video frame to obtain the second target video frame.

[0208] According to embodiments of this disclosure, the second processing submodule includes a second splicing unit, a third driving unit, and a fourth driving unit.

[0209] The second stitching unit is used to stitch together the first target video frame, the second mask image, and the noise image to obtain the second stitched image.

[0210] The third driving unit is used to drive the second stitched image by utilizing the second pose information to obtain the second global motion-driven image.

[0211] The fourth driving unit is used to perform detail driving on the second global motion driving image based on the second facial expression information and the second lip movement information to obtain the second target video frame.

[0212] According to embodiments of this disclosure, the video generation module 940 includes a video update submodule.

[0213] The video update submodule is used to update the first and second original video frames in the original video based on the first and second target video frames to obtain the target video.

[0214] Figure 10 A schematic block diagram of a smart agent according to an embodiment of the present disclosure is shown.

[0215] like Figure 10 As shown, the intelligent agent 1000 may include an input module 1010, a processing module 1020, and an output module 1030.

[0216] Input module 1010 is used to receive input information;

[0217] Processing module 1020 is used to determine the target task based on the input information received by the input module, determine the large model based on the target task, and obtain output information by calling the large model to execute the video generation method provided in the embodiments of this disclosure;

[0218] Output module 1030 is used to output the output information obtained by the processing module.

[0219] According to embodiments of this disclosure, the input module 1010 is responsible for receiving or sensing information such as queries, requests, instructions, signals, or data from the outside world (e.g., users or the external environment), and converting it into a format that the intelligent agent 1000 can understand and process. The input module 1010 is the primary link for the intelligent agent 1000 to interact with the outside world, enabling the intelligent agent 1000 to efficiently and accurately obtain necessary "sensory" information from the outside world and respond to this information.

[0220] In the example, the input module 1010 can input the demand speech or sample demand speech, sample demand speech features and demand speech features described above.

[0221] In the example, processing module 1020 is the core support for the ability of agent 1000 to handle complex tasks. Processing module 1020 can execute the video processing methods described above.

[0222] In the example, the performance of the processing module 1020 is closely related to the large model on which the agent 1000 is based. To fully leverage the capabilities of the large model, the internal structure of the processing module 1020 can be designed to be highly configurable and scalable to handle various types of tasks and requirements in real-world scenarios.

[0223] In the example, after acquiring the original video and the reference image, the processing module 1020 can process the original video using a large video generation model. Based on the first display method information of the original objects included in each of the multiple original video frames in the original video, the first original video frame and the second original video frame are determined from the multiple original video frames. Using the reference image, the original objects in the first original video frame are replaced to obtain a first target video frame, wherein the first target video frame includes a first target object, the identity information of the first target object matches the identity information of the reference object in the reference image, and the action information of the first target object matches the action information of the original object in the first original video frame. Using the first target video frame, the original objects in the second original video frame are replaced to obtain a second target video frame, wherein the second target video frame includes a second target object, the identity information of the second target object matches the identity information of the reference object, and the action information of the second target object matches the action information of the original object in the second original video frame. Based on the first target video frame and the second target video frame, a target video is obtained and transmitted to the output module 1030.

[0224] Understandably, while large models possess excellent language understanding and generation capabilities, like humans, their ability to solve tasks is limited without the aid of any tools. However, once Agent 1000 is given the ability to invoke tools, it can perform tasks such as using a calculator to complete mathematical calculations, using Python to perform data analysis, and using a search engine to create weather forecasts.

[0225] In the example, output module 1030 can output the target video described above.

[0226] The intelligent agent 1000 according to the embodiments of this disclosure can simply and effectively improve the level of intelligence, and enhance flexibility and versatility.

[0227] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0228] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0229] According to embodiments of the present disclosure, a non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions are used to cause a computer to perform the method described above.

[0230] According to an embodiment of this disclosure, a computer program product includes a computer program that, when executed by a processor, implements the method described above.

[0231] Figure 11 A schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0232] like Figure 11 As shown, the electronic device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 11011 into a random access memory (RAM) 1103. The RAM 1103 may also store various programs and data required for the operation of the device 1100. The computing unit 1101, ROM 1102, and RAM 1103 are interconnected via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0233] Multiple components in device 1100 are connected to input / output (I / O) interface 1105, including: input unit 1106, such as a keyboard, mouse, etc.; output unit 1107, such as various types of displays, speakers, etc.; storage unit 11011, such as a disk, optical disk, etc.; and communication unit 1109, such as a network card, modem, wireless transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0234] The computing unit 1101 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as video processing methods. For example, in some embodiments, the video processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 11011. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by the computing unit 1101, one or more steps of the video processing method described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to perform video processing methods by any other suitable means (e.g., by means of firmware).

[0235] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0236] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0237] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0238] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0239] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0240] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0241] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0242] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A video processing method, comprising: Based on the first display method information of the original objects included in each of the multiple original video frames in the original video, the first original video frame and the second original video frame are determined from the multiple original video frames. Using a reference image, the original object in the first original video frame is replaced to obtain a first target video frame. The first target video frame includes a first target object. The identity information of the first target object matches the identity information of the reference object in the reference image. The action information of the first target object matches the action information of the original object in the first original video frame. Using the first target video frame, object replacement is performed on the original object in the second original video frame to obtain a second target video frame. The second target video frame includes a second target object, the identity information of the second target object matches the identity information of the reference object, and the action information of the second target object matches the action information of the original object in the second original video frame. The target video is obtained based on the first target video frame and the second target video frame.

2. The method according to claim 1, wherein, The step of determining the first original video frame and the second original video frame from the multiple original video frames based on the first display method information of the original objects included in each of the multiple original video frames in the original video includes: When it is determined that the original video frames meet predetermined conditions based on the first display method information of the original objects included in each of the original video frames, the first original video frame is determined from the original video frames based on the temporal relationship information of the original video frames in the original video, wherein the predetermined conditions include the difference in the amount of information of the target object regions of the original objects displayed in each of the original video frames being less than or equal to a predetermined threshold; and The other video frames among the multiple original video frames, excluding the first original video frame, are used as the second original video frame.

3. The method according to claim 2, further comprising: If, based on the first display method information of the original objects included in each of the original video frames, it is determined that the original video frames do not meet the predetermined conditions, the original video frames are divided based on the amount of information of the target object region of the original objects included in each of the original video frames and the temporal relationship information of the original video frames, to obtain the original video frames that meet the predetermined conditions.

4. The method according to any one of claims 1 to 3, further comprising: The original video is divided into segments to determine multiple original video segments, wherein each original video frame in each original video segment includes the same scene and original object; and The target object region of the original object included in each original video frame in the original video segment is detected to obtain the first display mode information of the original object included in each original video frame.

5. The method according to claim 4, wherein, The step of detecting the target object region of the original object included in each original video frame of the original video segment to obtain the first display mode information of the original object included in each original video frame includes: Image detection is performed on the original video frame to determine the location identification information of the target object region of the original object in the original video frame; and Based on the location identification information of the target object region of the original object, the information content ratio of the target object region of the original object in the original video frame is determined, and the first display mode information is obtained.

6. The method according to claim 4 or 5, wherein, The process of dividing the original video into segments and determining multiple original video segments includes: The original video is divided into scenes to obtain multiple initial original video segments; Object identification is performed on each video frame in the initial original video segment to obtain object category information for each of the multiple video frames; and Based on the object category information of each of the multiple video frames, the initial original video segment is divided into multiple original video segments.

7. The method according to any one of claims 1 to 6, further comprising: Based on the second display mode information of the reference objects included in each of the multiple candidate reference images, a reference image that matches the display mode of the original object in the first original video frame is determined from the multiple candidate reference images.

8. The method according to claim 7, further comprising: Based on the reference object in the standard reference image, image generation is performed to obtain multiple candidate reference images with different display modes, wherein the second display mode information of the reference object in the standard reference image indicates the target display mode.

9. The method according to any one of claims 1 to 8, wherein, The step of replacing the original object in the first original video frame with an object using a reference image to obtain the first target video frame includes: The original objects in the first original video frame are masked to obtain a first mask image; and Based on the reference image and the first action information of the original object in the first original video frame, the first mask image is processed to obtain the first target video frame.

10. The method according to claim 9, wherein, The first action information includes first posture information, first facial expression information, and first lip movement information; The step of processing the first mask image based on the reference image and the action information of the original object in the first original video frame to obtain the first target video frame includes: The reference image, the first mask image, and the noise image are stitched together to obtain the first stitched image; Based on the first pose information, the first stitched image is globally driven to obtain a first global motion-driven image; and Based on the first facial expression information and the first lip movement information, detail-driven processing is performed on the first global motion-driven image to obtain the first target video frame.

11. The method according to any one of claims 1 to 9, wherein, The step of replacing the original object in the second original video frame with the first target video frame to obtain the second target video frame includes: The original objects in the second original video frame are masked to obtain a second mask image; and Based on the first target video frame and the action information of the original object in the second original video frame, the second mask image is processed to obtain the second target video frame.

12. The method according to claim 10, wherein, The second action information includes second posture information, second facial expression information, and second lip movement information; The step of processing the second mask image based on the first target video frame and the action information of the original object in the second original video frame to obtain the second target video frame includes: The first target video frame, the second mask image, and the noise image are stitched together to obtain a second stitched image; The second pose information is used to perform pose-driven processing on the second stitched image to obtain a second global motion-driven image; and Based on the second facial expression information and the second lip movement information, the second global motion-driven image is subjected to detail-driven processing to obtain the second target video frame.

13. The method according to any one of claims 1 to 12, wherein, The step of obtaining the target video based on the first target video frame and the second target video frame includes: Based on the first target video frame and the second target video frame, the first original video frame and the second original video frame in the original video are updated to obtain the target video.

14. A video processing apparatus, comprising: The video frame determination module is used to determine a first original video frame and a second original video frame from the multiple original video frames based on the first display method information of the original objects included in each of the multiple original video frames in the original video. The first processing module is used to replace the original object in the first original video frame with an object using a reference image to obtain a first target video frame. The first target video frame includes a first target object, the identity information of the first target object matches the identity information of the reference object in the reference image, and the action information of the first target object matches the action information of the original object in the first original video frame. The second processing module is configured to use the first target video frame to perform object replacement on the original object in the second original video frame to obtain a second target video frame, wherein the second target video frame includes a second target object, the identity information of the second target object matches the identity information of the reference object, and the action information of the second target object matches the action information of the original object in the second original video frame; and The video generation module is used to obtain a target video based on the first target video frame and the second target video frame.

15. An intelligent agent, comprising: The input module is used to receive input information; The processing module is configured to determine a target task based on the input information received by the input module, determine a large model based on the target task, and execute the method of any one of claims 1 to 13 by calling the large model to obtain output information; An output module is used to output the output information obtained by the processing module.

16. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 13.

17. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 13.

18. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 13.

Images