Video content replacement method, program product, electronic device and chip system

Abstract

The application provides a video content replacement method, a program product, an electronic device and a chip system, relates to the technical field of image processing, and can realize automatic replacement of a specified object in a video. The method can include the following processes: obtaining a first three-dimensional model corresponding to a to-be-processed video; determining a to-be-replaced region in the first three-dimensional model in response to first semantic information input by a user; the to-be-replaced region is a region corresponding to a first object; the first natural semantic information is used for describing the first object in the to-be-processed video, and the first object is a replaced object; generating a second three-dimensional model corresponding to a second object in response to second semantic information input by the user; the second object is a target object used for replacing the first object; replacing or covering the to-be-replaced region with the second three-dimensional model in the first three-dimensional model to obtain a third three-dimensional model after fusion; generating a target video corresponding to the third three-dimensional model; and the first object in the target video is replaced with the second object.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a video content replacement method, program product, electronic device and chip system. Background Technology

[0002] With the rise of some social short video platforms, users' participation and demand for video creation have increased significantly.

[0003] There are many types of business needs for video creation. One of these needs is video content replacement, which requires replacing specific objects or elements in the video. For example, replacing a person with another person or thing, or replacing a certain object with another object.

[0004] Currently, a convenient video character replacement solution involves identifying the skeleton of a person and their pose in 3D space based on the input video, binding the character model to the identified skeleton, removing the person from the video, and finally replacing it with the bound target character model and rendering the final video. However, this method currently only applies to character replacement; it cannot edit or replace other elements in the video besides the person. Summary of the Invention

[0005] This application provides a video content replacement method, program product, electronic device, and chip system that can quickly replace specified objects in a video.

[0006] In a first aspect, embodiments of this application provide a video content replacement method, the method comprising: obtaining a first three-dimensional model corresponding to a video to be processed; determining a region to be replaced in the first three-dimensional model in response to first semantic information input by a user; the region to be replaced being a region corresponding to a first object; a first natural semantic meaning used to describe the first object in the video to be processed, the first object being the object to be replaced; generating a second three-dimensional model corresponding to a second object in response to second semantic information input by a user; the second object being the target object used to replace the first object; replacing or covering the region to be replaced in the first three-dimensional model to obtain a fused third three-dimensional model; generating a target video corresponding to the third three-dimensional model; and replacing the first object with the second object in the target video.

[0007] This method can automatically generate a 3D model of the replacement object by inputting information such as text or voice, so as to achieve smooth replacement of items in the video, while ensuring the spatial and temporal continuity of the replacement object, and without requiring users to build the model themselves, thus lowering the barrier to entry for users.

[0008] In one possible implementation, obtaining a first three-dimensional model corresponding to the video to be processed includes: responding to a user instruction, performing camera pose estimation and depth estimation on each frame of the first image in the video to be processed to obtain a camera pose and a first depth image corresponding to each frame of the first image; and performing three-dimensional reconstruction based on the camera pose and the first depth image corresponding to each frame of the first image to obtain a first three-dimensional model.

[0009] In one possible implementation, in response to first semantic information input by a user, determining the region to be replaced in a first three-dimensional model includes: in response to the first semantic information input by a user, segmenting each frame of a first image in the video to be processed, and determining the boundary of a first object in each frame of the first image; generating a mask corresponding to each frame of the first image based on the boundary of the first object; and performing a mapping from the mask to the first three-dimensional model to determine the boundary of the region to be replaced in the first three-dimensional model.

[0010] In one possible implementation, a mapping from the mask to the first three-dimensional model is performed to determine the boundary of the region to be replaced in the first three-dimensional model, including: converting the coordinates of each 0 or 1 pixel in the mask into the three-dimensional coordinates of the corresponding pixel in the first three-dimensional model through a preset mapping function; and segmenting the pixel cloud in the first three-dimensional model according to the three-dimensional coordinates to determine the boundary of the region to be replaced.

[0011] In one possible implementation, before generating a second three-dimensional model corresponding to the second object in response to second semantic information input by the user, the method further includes: extracting keyframes from the video to be processed.

[0012] In one possible implementation, extracting keyframes from the video to be processed includes: determining the angular offset of the camera pose relative to the first object for each frame based on the camera pose corresponding to the first image of each frame in the video to be processed; and extracting a frame as a keyframe at every predetermined angular offset.

[0013] In one possible implementation, in response to second semantic information input by the user, a second three-dimensional model corresponding to a second object is generated, including: in response to the second semantic information input by the user, merging the mask and first depth image corresponding to each keyframe to obtain a merged image; obtaining an image sequence corresponding to the second object based on the merged image and the second semantic information; performing depth estimation on each frame of the second image in the image sequence to obtain a second depth image corresponding to each frame of the second image; and performing three-dimensional reconstruction based on the second depth image corresponding to each frame of the second image and the camera pose to obtain a second three-dimensional model.

[0014] In one possible implementation, obtaining an image sequence corresponding to a second object based on the merged graph and the second semantic information includes: sending a request message to a target server, the request message carrying the merged graph and the second semantic information; receiving data returned by the target server; the returned data containing the image sequence.

[0015] In one possible implementation, generating a target video corresponding to a third 3D model includes: performing a mapping from the third 3D model to an image sequence to obtain a target image sequence; performing at least rendering processing on each frame of the target image sequence; and combining the rendered frames into a target video.

[0016] Secondly, embodiments of this application also provide a computer program product, which includes a program that, when run by an electronic device, causes the electronic device to perform the method as described in any one of the first aspects above.

[0017] Thirdly, this application also provides an electronic device for transmitting data via a data transmission interface. The electronic device includes: one or more processors; a memory; at least one application program; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the electronic device, cause the electronic device to perform the method as described in any one of the first aspects above.

[0018] Fourthly, the present application also provides a storage medium including computer instructions, which, when executed on an electronic device, cause the electronic device to perform the method described in any one of the first aspects above.

[0019] Fifthly, the present application also provides a chip system, including: a communication interface for inputting and / or outputting data; and a processor for executing a computer-executable program, causing a device equipped with the chip system to perform the method as described in any one of the first aspects above. Attached Figure Description

[0020] Figure 1 and Figure 2 This is an example diagram of related technology two;

[0021] Figure 3 This is a schematic diagram illustrating the application scenario of the method provided in the embodiments of this application;

[0022] Figure 4 This is a flowchart illustrating one embodiment of the method provided in this application.

[0023] Figure 5 This is a schematic diagram illustrating the mapping of an image sequence to a three-dimensional model in one embodiment of the method provided in this application.

[0024] Figure 6 This is a schematic diagram of the camera pose and normal vector in one embodiment of the method provided in this application;

[0025] Figure 7 This is a schematic diagram of keyframe extraction in one embodiment of the method provided in this application;

[0026] Figure 8 This is a schematic diagram illustrating the mapping of a 3D model to an image sequence in one embodiment of the method provided in this application.

[0027] Figure 9 This is a flowchart illustrating the full-frame operation and keyframe operation in one embodiment of the method provided in this application.

[0028] Figure 10 This is an example diagram of a frame of an image from a video to be processed in one embodiment of the method provided in this application.

[0029] Figure 11 This is an example image of a frame from a target video in one embodiment of the method provided in this application.

[0030] Figure 12 This is a schematic diagram comparing images from different angles before and after replacement in one embodiment of the method provided in this application;

[0031] Figure 13 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] The terminology used in the implementation section of this application is for the purpose of explaining specific embodiments of this application only, and is not intended to limit this application.

[0034] In recent years, with the development of computer vision, deep learning and machine learning technologies, artificial intelligence (AI) has been widely used in the field of video processing. For example, with the help of AI technology, massive amounts of video footage can be analyzed automatically and quickly, and the video content can be divided into different categories, thus providing important assistance for the next step of data processing; with the help of deep learning technology, it is possible to quickly and accurately identify different people and objects in the video.

[0035] Currently, more and more people are participating in video creation, especially short video creation. AI-based video and image editing tools (such as replacement, removal, and addition) are in high demand. However, current video editing tools generally have a high user barrier, requiring a complex workflow involving 3D modeling, model rendering, and rendering compositing, making them cumbersome and unfriendly to non-professional users. This is partly because video editing needs to consider the spatial and temporal continuity of the frames before and after the edited section. Simply applying AI image editing methods to each frame of a video will result in randomness in the generated results, leading to discontinuities in the three-dimensional space and jitter, making effective video editing impossible. Therefore, there is a relative lack of convenient tools that can achieve the desired video processing results while lowering the user barrier.

[0036] In related technologies, some solutions are dedicated to solving such problems. To facilitate understanding of the solutions proposed in the embodiments of this application, two related technologies are described below.

[0037] One proposed solution utilizes Stable Diffusion image editing technology to edit all frames of the video one by one, and uses ControlNet technology to achieve a certain degree of temporal stability between frames. However, due to the randomness of generative AI, visually perceptible image jitter still occurs between frames. Furthermore, this approach typically stylizes the overall image, similar to a filter effect, and cannot achieve spatially and temporally continuous removal and replacement of specific objects in the video.

[0038] Related technology two provides a convenient method for replacing people in videos. For example... Figure 1 and Figure 2 As shown, the skeleton of the person and its pose in three-dimensional space are first identified based on the input video. Then, the person model is bound to the identified skeleton. Subsequently, the person in the video is removed. Finally, the bound person model is replaced and rendered to obtain the final video.

[0039] This video editing method currently only applies to replacing people, and its overall approach is to identify the pose of the person's skeleton in 3D space. However, it cannot edit or replace other elements in the video scene besides people, and the 3D model used to replace the person needs to be created by the user, which increases the barrier to entry to some extent.

[0040] Therefore, the solutions in the relevant technologies either cannot achieve stable replacement of objects in the video, or can only replace a single type of person in the video, and the models need to be made by themselves. None of them can achieve editing and replacement of arbitrary objects in the product.

[0041] In view of this, this application proposes a video content replacement method. This method can automatically generate a 3D model of the object to be replaced by inputting information such as text or voice, achieving smooth replacement of items within the video while ensuring the spatial and temporal continuity of the replaced object. Furthermore, this method only requires the user to input semantic information such as keywords, eliminating the need to create 3D models or for the user to be familiar with model rendering, rendering compositing, and other technologies, thus lowering the user threshold and making it more user-friendly for ordinary users.

[0042] like Figure 3 As shown, this method can be applied to various application scenarios where users edit videos using electronic devices, such as creating or editing videos using mobile phones or computers. In some embodiments, this method is suitable for replacing content such as objects in a video. The objects can be static objects, and the processing effect is better for videos captured by multi-angle camera poses. In other embodiments, it is possible that this method can also be applied to replacing moving objects in a video, such as replacing moving objects like a moving car.

[0043] This method can be implemented on either the terminal device side or the server side. That is, it can be applied to either the terminal device or the server. The terminal device can be a device with a display screen that supports user interaction, such as a smartphone, large-screen TV, smartwatch, computer, tablet, augmented reality (AR) device, virtual reality (VR) device, projector, or learning machine. When implemented on a terminal device, the corresponding product can be an application or an application upgrade package, plugin, etc.

[0044] When this method is implemented on the server side, the corresponding product can be a platform that provides video processing services, such as a platform that provides video content replacement services in response to client requests. The video processing service platform can be implemented based on a standalone server (e.g., an application server) or a server cluster.

[0045] Specifically, the video content replacement method provided in this application embodiment, such as... Figure 4 As shown, the process may include the following:

[0046] S11, obtain the first 3D model corresponding to the video to be processed.

[0047] For ease of description, in this embodiment, the object to be replaced in the video is defined as the first object, which can be an object or other element in the video; the second object is the replacement target. For example, if a white packaging box in a video is replaced with a wine glass, the white packaging box is the first object and the wine glass is the second object; if a television set in a video is replaced with a mural, the television set is the first object and the mural is the second object. The following examples will primarily use a white packaging box as the first object and a wine glass as the second object.

[0048] To prevent confusion, the image containing the first object is defined as the first image, the 3D model generated based on the image containing the first object is defined as the first 3D model, and the image and 3D model corresponding to the second object are defined as the second image and the second 3D model.

[0049] In some embodiments, the first three-dimensional model can be obtained in the following manner:

[0050] In response to user commands, camera pose estimation and depth estimation are performed on the first image of each frame in the image sequence of the video to be processed, resulting in the camera pose and first depth image corresponding to each frame. Then, based on the camera pose and first depth image corresponding to each frame, 3D reconstruction is performed to obtain the first 3D model.

[0051] Specifically, a 3D model is constructed based on camera pose and depth images. In some embodiments, this can be achieved by installing a module, application, or plugin with 3D reconstruction capabilities locally. After generating the camera pose and depth images, the 3D model is reconstructed by calling the corresponding module or plugin to obtain the first 3D model.

[0052] In another embodiment, 3D reconstruction often requires a large amount of memory space and has certain requirements for hardware configuration. For lightweight terminal devices, a 3D reconstruction service can be requested by sending a request to a remote server. That is, the 3D reconstruction function is deployed on the server side, the terminal device sends a service request to the server, and after the server reconstructs the 3D model, it feeds back the data required by the terminal device.

[0053] Optionally, the three-dimensional reconstruction process can be achieved through a three-dimensional reconstruction model, such as at least one of NeRF (Neural Radiance Fields) and SFM (Structure From Motion).

[0054] S12, in response to the first semantic information input by the user, determine the region to be replaced in the first three-dimensional model.

[0055] The first semantic information is used to describe the first object. Specifically, and exemplaryly, the first semantic information is generally a keyword or statement describing the first object. The input method can be at least one of text, voice, gesture, etc. For example, if a user inputs the keyword "white cardboard box," it means that the white cardboard box in the current video needs to be replaced with another item; or, if a user inputs the voice command "Please replace the white cardboard box with a wine glass" through a smart voice assistant, then through semantic recognition, the white cardboard box can be identified as the first object, and the wine glass as the second object. Alternatively, the user can first click to select an object in the video, and then make a gesture indicating that the object should be replaced with another object. The user interface can provide multiple candidate replacement target objects for the user to choose from, and the user can click to complete the selection of the target object.

[0056] In some embodiments, the region to be replaced in the first three-dimensional model can be determined in the following manner:

[0057] The first image of each frame in the video to be processed is segmented, the boundary of the first object in each frame is determined, and a mask corresponding to each frame is generated based on the boundary of the first object. The mapping from the mask to the first 3D model is performed to determine the boundary of the region to be replaced in the first 3D model.

[0058] Specifically, after determining the semantics of the first object, semantic segmentation needs to be performed on each frame of the video to be processed. That is, the region representing the white cardboard box in the first image needs to be segmented, which means determining the boundary of the region of the white cardboard box in each frame of the first image, specifically determining the coordinates of each pixel point of the boundary.

[0059] Next, a masking process is performed on the segmented first image. This masking can be implemented using the `mask` function in OpenCV. Specifically, the region containing the first object is occluded using the `mask` function, indicating that the occluded region is the area that needs to be replaced in subsequent processing. Alternatively, in other embodiments, other regions in the first image besides the first object can be masked, indicating that the occluded regions are background regions that do not need to be replaced in subsequent processing.

[0060] After obtaining the mask corresponding to the first image of each frame, a mapping from the mask to the first 3D model (image to point cloud mapping) is performed. The specific mapping process is as follows: Figure 5 As shown:

[0061] The image sequence of the video to be processed includes frames 1 to n (the first image), and the boundaries of the first object in each frame have been determined, such as... Figure 5 The cube shown is the first object, and the coordinates of each pixel on the boundary of the first object have been obtained in the above steps.

[0062] At this point, using a pre-defined first mapping function, the coordinates of the boundary pixels in each frame of the image sequence (the first image) are mapped to three-dimensional space, obtaining the corresponding pixel coordinates in three-dimensional space. Specifically, the boundary pixels in the two-dimensional image can be used as independent variables, and the coordinate values ​​of the pixels in three-dimensional space can be calculated using the first mapping function. This determines the boundary of the area to be replaced. Once the boundary is determined, the spatial range within the boundary is also determined. The area to be replaced is the spatial area occupied by the first object in three-dimensional space, and the boundary of this area is the boundary of the spatial area that needs to be replaced.

[0063] For example, such as Figure 6 As shown, the specific mapping process can be as follows:

[0064] Based on the mask, assuming the coordinates of a pixel on the boundary of the first object are (x, y), and assuming the horizontal viewing angle of the image acquisition device (e.g., camera or webcam) during video capture is V1, the vertical viewing angle is V2, and the horizontal resolution of the image acquisition device is r1 and the vertical resolution is r2, then the mapping equation (i.e., mapping function) for the boundary pixel to the 3D model is:

[0065]

[0066] Where (x0, y0, z0) represents the initial coordinates of the pixel points mapped to the 3D model; k1 and k2 are intermediate variables;

[0067] Based on the initial coordinate values, continue with the rotation and translation operations on the boundary pixels:

[0068] Assume the normal vector of the image acquisition device at the time of shooting is It can be a vector after normalization;

[0069] Based on the initial coordinate values ​​mentioned above, rotate the boundary pixels according to the following calculation formula (2):

[0070]

[0071] in, is the rotation angle along the z-axis; arcsinn3 is the rotation angle along the y-axis.

[0072] The position of the camera relative to the object (the first physical object) is known to be (θ, φ, r);

[0073] Continue translating the boundary pixels according to the following calculation formula (3):

[0074]

[0075] That is, offset the boundary pixels in the x-axis direction. Offset in the y-axis direction Offset in the z-axis direction

[0076] In this way, the specific location of the final boundary pixel in the three-dimensional model space can be obtained. By colliding with the point cloud in the three-dimensional model through the above calculation, the intersection point is obtained, realizing the mapping from the pixel in the image to the pixel in the three-dimensional space.

[0077] There is no explicit order of execution for rotation and translation operations. In some embodiments, rotation can be performed before translation; in other embodiments, translation can be performed before rotation.

[0078] S13, in response to the second semantic information input by the user, generates a second three-dimensional model corresponding to the second object.

[0079] The second semantic information is used to describe the second object. Similarly, the second semantic information can also be input in the form of text or voice.

[0080] It should be noted that, in some embodiments, before generating the second three-dimensional model corresponding to the second object, keyframes are first extracted from the image sequence corresponding to the video to be processed, and the second three-dimensional model is generated based on the extracted keyframes.

[0081] In some embodiments, keyframes can be extracted as follows: based on the camera poses corresponding to each frame of the first image in the video to be processed, the angular offsets of the camera poses relative to the first object for each frame are determined, and a frame is extracted as a keyframe at predetermined angular offsets. For example, as shown... Figure 7As shown, by estimating the camera pose, we can simulate how image acquisition devices such as cameras and webcams capture images of the first object at different angles along the direction indicated by the arc arrows in the diagram, thus obtaining the video to be processed. The angular offset of the camera pose relative to the first object is the angle between the normals pointing from the image acquisition device to the object. Specifically, the normal refers to the directed vector from the optical center O in the camera coordinate system to the center point O' of the pixel plane (physical imaging plane). The angle between the normal vectors at different shooting angles is the angular offset of the camera pose relative to the first object. For example... Figure 7 In this context, θ represents the angular offset, also known as the offset angle.

[0082] For example, at every predetermined threshold offset angle, such as θ∈(2-15°), a corresponding frame is selected as a keyframe to achieve keyframe extraction. For example, Figure 7 In this example, assuming θ = 5°, a frame is extracted for every 5° shift of the normal vector of the optical center O relative to the pixel plane. The 4th frame is the image frame obtained after shifting the initial normal vector by 5°. The 1st and 4th frames are then extracted as keyframes, and so on, resulting in multiple keyframes. The predetermined threshold can be chosen from various options, such as 5°, 4°, 2°, 6°, or 10°, etc., and can be set according to the actual situation. This specification does not limit this setting.

[0083] In other embodiments, keyframes can be extracted by extracting one frame every predetermined number of frames, such as every 5 frames, every 3 frames, etc., without considering the camera pose during shooting.

[0084] It should be noted that the extraction method based on a predetermined number of frames has a certain degree of randomness and may miss some image features from key shooting angles. In contrast, selecting keyframes based on the offset angle of the normal vector can evenly extract image features from different shooting angles. This provides image data from different shooting angles during 3D model construction, avoiding the loss of images from certain shooting angles and facilitating the acquisition of target videos with multi-angle variations.

[0085] After extracting the keyframes, the second 3D model corresponding to the second object can be generated in the following way:

[0086] In response to the second semantic information input by the user, the mask and the first depth image corresponding to each keyframe are merged to obtain a merged image (large image). Based on the merged image and the second semantic information, the image sequence corresponding to the second object is obtained. Then, depth estimation is performed on each frame of the second image in the obtained image sequence to obtain the second depth image corresponding to each frame of the second image. Next, three-dimensional reconstruction is performed based on the second depth image and the camera pose corresponding to each frame to obtain the second three-dimensional model.

[0087] The process of obtaining the image sequence corresponding to the second object based on the merged graph and the second semantic information can be achieved using an AI-based image generation model. Typically, an AI video processing tool that supports text or voice input can be chosen to output the image sequence corresponding to the second object. For example, one or more of the Stable Diffusion model, Dall-E model, and Midjourney model can be used. The following explanation primarily uses the Stable Diffusion model as an example.

[0088] Specifically, the merged image and the second semantic information can be input together into the Stable Diffusion model, which then outputs the image sequence corresponding to the second object. The Stable Diffusion model can be installed locally or accessed through a third-party service platform to call the video processing services provided by Stable Diffusion.

[0089] For example, if the user wants to generate a second object as a wine glass, then in response to the user's input keyword "wine glass", the extracted keyframe corresponding mask and depth map are combined, and the merged map is provided to the video processing service platform along with the keyword "wine glass". Then, the platform receives the image sequence, which is the image sequence corresponding to the second object.

[0090] Specifically, in some embodiments, AI models such as Stable Diffusion can be deployed locally, for example, on a terminal device, as a callable functional module.

[0091] In other embodiments, a request can be initiated through a third-party platform that provides services such as Stable Diffusion to obtain the corresponding services. For example, a request message is sent to the target server, carrying data information of the merged graph and second semantic information, and then the server waits to receive data returned by the target server; the returned data at least contains the image sequence corresponding to the second object.

[0092] In another embodiment, when the product corresponding to the method provided in this application is a standalone application (App), a corresponding functional module can be integrated into the App to achieve the following function: inputting the merged graph and second semantic information, and outputting the image sequence corresponding to the second object. This solves the dependency problem on models such as StableDiffusion.

[0093] After obtaining the image sequence, camera pose and depth estimation are performed again to obtain multiple second depth images corresponding to the second object, which are then used to reconstruct the 3D model of the second object. The method for reconstructing the 3D model here can refer to the reconstruction process of the first 3D model, and will not be repeated here.

[0094] It should be noted that the replaced video will not change the camera pose corresponding to each frame of the video. Therefore, when performing 3D reconstruction of the image sequence of the second object, in some embodiments, the previously estimated camera pose information can be used, and there is no need to perform camera pose estimation again.

[0095] S14, in the first three-dimensional model, replace or cover the area to be replaced by the second three-dimensional model to obtain the fused third three-dimensional model.

[0096] In step S12 above, the area to be replaced is determined, which is equivalent to determining the background area in the 3D model that does not need to be replaced (other areas besides the first object). In step S13, the 3D model of the replacement target (second object) is obtained. The two are combined (fused) to obtain the complete 3D model. For ease of description and to prevent confusion, the fused 3D model is defined as the third 3D model.

[0097] The fusion of the first 3D model and the second 3D model can specifically involve using the second 3D model to replace, cover, or fill the area to be replaced.

[0098] In the space of a 3D model, image data can be stored in the form of point clouds. The attribute information of each pixel in the point cloud includes 3D coordinate information and pixel value information. For example, the pixel value can be RGB three-color pixel value.

[0099] In some embodiments, after determining the area to be replaced in the first 3D model, the pixels within that area can be deleted, that is, the pixels corresponding to the first object can be deleted. This is equivalent to removing the first object from the first 3D model, leaving the remaining background area that does not need to be replaced. During fusion, the second 3D model covers or replaces the area to be replaced.

[0100] It should be noted that in practical applications, the boundary shapes of the first and second objects are often inconsistent. That is, the boundary of the area to be replaced and the boundary of the second 3D model may not perfectly match in some cases, such as replacing a sphere with a cube, or replacing other irregularly shaped objects. When the area or volume of the second object is larger than that of the first object, the 3D model of the second object can directly cover the area to be replaced during fusion, generally without the need for filling. However, in other cases, this may lead to missing local pixels, requiring an impaint operation during fusion, such as filling or patching near the boundaries.

[0101] In another embodiment, after determining the area to be replaced in the first 3D model, instead of deleting pixels, the pixel values ​​of each pixel in the area to be replaced can be directly replaced with the pixel values ​​of the corresponding pixels in the second 3D model. That is, the pixel values ​​of each pixel corresponding to the second object cover the pixel values ​​of the corresponding pixels of the first object. When the resolution of the first image in the video to be processed is consistent with that of each second image in the image sequence corresponding to the second object, there is a one-to-one correspondence between the pixels in the first image and the pixels in the second image, and also a one-to-one correspondence with the pixels in the 3D model. Therefore, pixel-level replacement can be achieved, that is, pixel value replacement corresponding to the same coordinate point.

[0102] In the fused third 3D model, the first object has been replaced with the second object. Thus, a 3D model containing the second object is first obtained. Next, a mapping from the third 3D model to the target image sequence is performed to obtain the target video containing the second object.

[0103] S15, generate the target video corresponding to the third 3D model.

[0104] like Figure 8 As shown, after obtaining the third 3D model, it is necessary to perform a mapping from the third 3D model to the target image sequence (mapping from point cloud to image). This mapping process can be understood as the inverse mapping from the first image sequence to the 3D model.

[0105] Specifically, a second mapping function can be used here to perform the mapping from the pixel coordinates in the 3D model to the pixel coordinates in each image of the target image sequence.

[0106] The mapping from point cloud to image can be understood as the inverse mapping process from image to point cloud. Therefore, based on the mapping process from image to point cloud in S12 above, the mapping process from point cloud to image can be obtained accordingly, which will not be elaborated here.

[0107] After the mapping process is completed, the target image sequence is obtained. Each frame in the target image sequence is rendered. The rendered frames are then combined into the target video.

[0108] It should be noted that extracting keyframes in step S13 is not a necessary step. In some implementations, keyframe extraction can be omitted. Instead, the mask and first depth image corresponding to each full-frame image in the video to be processed can be merged. Then, the merged image and the second semantic information are input into Stable Diffusion or other AI models to output the image sequence corresponding to the second object. This full-frame processing method places a greater data processing burden on models such as Stable Diffusion, which will extend the processing time.

[0109] Therefore, in some other embodiments, keyframe operations are implemented for certain operations to reduce data processing pressure and improve processing efficiency. For example, such as... Figure 9 As shown in this embodiment, the overall processing can be divided into two main parts: full-frame operation and keyframe operation.

[0110] Full-frame operation

[0111] Camera pose and depth estimation: First, the original input video (video to be processed) is received, and the pose of all frames in space is calculated. Based on these poses and the depth estimate of each frame, the original 3D scene is reconstructed.

[0112] Input and extraction of objects to be replaced: Based on the user's text input information (first semantic information), the object to be replaced in the scene (i.e., the first object) is obtained. In each original video frame, the object to be replaced (the first object) is automatically selected and segmented, and mapped to a 3D space to obtain the part to be replaced in the 3D space (the area to be replaced).

[0113] Keyframe Operation

[0114] Keyframe extraction: Using the camera pose as a threshold, a keyframe is recorded whenever the camera pose rotates by 5°.

[0115] Generating replacement images based on keyframe depth maps: First, keyframes are converted into depth estimation maps. Then, based on previous user input (second semantic information), the region to be replaced (the Mask region) is selected. The masks and depth maps of all keyframes are then combined, along with the second semantic information, and input into the Stable Diffusion model. Based on the Mask, depth map, and text information of the replacement object, a sequence of generated images containing the new object is obtained. This method generates a sequence of replacement images with high content consistency and conforms to the pose changes of the replaced object in space.

[0116] Depth estimation of keyframes after replacement: The depth of the newly generated image sequence is re-estimated. It should be noted that in some embodiments, the camera pose does not need to be re-estimated; instead, the camera pose information corresponding to each frame in the video to be processed can be directly used, because the replacement operation does not change the video's shooting angle. In other embodiments, it is possible to re-estimate the camera pose at this point.

[0117] Replacement model generation: Based on the multi-angle images of the newly generated object (second object) and its depth estimation information, a three-dimensional model of the new object (second three-dimensional model) can be reconstructed.

[0118] Full-frame operation

[0119] New Model Replacement: Replace the existing model in 3D space that needs to be replaced with a new model (second 3D model). Obtain a new 3D scene (third 3D model).

[0120] Render all frames: Render all frames and combine them into a new video (target video) for output.

[0121] The following is a specific example.

[0122] like Figure 10 As shown, Figure 10 The image shown is a frame from the video to be processed. After receiving the original input video, the user can issue a command to replace the video content while editing the video. For example, a "Replace" button can be set in the user interface. When the user clicks this button, it is considered that a replacement command has been issued. In response to this command, the following operations are performed:

[0123] Camera pose and depth estimation: Nerf (Neural Radiance Fields) or SFM (Structure from Motion) techniques can be used to calculate the inverse kinematics of the poses in space for all frames, i.e., the camera poses corresponding to each frame. Depth estimation is then performed on each frame, and the original 3D scene is reconstructed based on these poses and the depth estimation images of each frame.

[0124] Input and extraction of the object to be replaced: At this point, the user can be prompted to input keywords describing the first object. The user will then input text, voice, or gestures, such as keywords like "white square box" or "white box." Through semantic recognition, the white packaging box in the video to be processed is identified as the first object to be replaced, specifying the name of the object to be replaced. Based on the semantics, a mask is applied to the content to be replaced in each frame of the input, and the white packaging box is segmented. That is, the white packaging box in each frame of the video to be processed is segmented, the coordinates of its boundary pixels are recorded, and the corresponding mask is generated.

[0125] By mapping the segmented image mask into three-dimensional space, the area or spatial range corresponding to the "white box" can be selected in three-dimensional space and marked as the part that needs to be replaced.

[0126] Next, we will perform the keyframe extraction operation:

[0127] Using the camera pose as a threshold, whenever the camera pose rotation angle shifts by ≥5°, the actual detected angle shift is often difficult to precisely control within 5°. Instead, if the detected angle shift is slightly greater than 5°, a keyframe is recorded.

[0128] A replacement image is generated based on the keyframe depth map: The user is prompted to enter semantic information describing the second object again, such as text or voice. The user will then enter again, such as text keywords, specifying the content to be generated, which is "wine glass" in this embodiment.

[0129] The mask containing the "white box" of all keyframes is selected and stitched together with the depth estimation images of each keyframe into a single large image. This stitched image, along with the keyword "wine glass," is then input into the Stable Diffusion model to obtain a sequence of generated images containing the new object (wine glass). For example, a frame from this sequence of images containing the new object, the wine glass, is shown below. Figure 11 As shown.

[0130] The image sequences generated by this method have high content consistency and will conform to the pose changes of the replaced object in space.

[0131] Then, the new image sequence containing the wine glass is subjected to depth estimation and camera pose estimation again. Based on the obtained depth image and camera pose information, 3D reconstruction is performed to obtain a 3D model containing the wine glass.

[0132] The 3D model containing the wine glass (local, containing only the wine glass) is used to replace the parts of the original 3D scene that need to be replaced, resulting in a full-scene 3D model (global, containing both the wine glass and the background) with the new object, the wine glass. This global 3D model is then mapped onto a 2D image sequence, resulting in a global image sequence containing the wine glass. After rendering and other processing, these images are combined to obtain a new video containing the wine glass.

[0133] like Figure 12 As shown, (a) and (b) are two frames from different shooting angles in the original video (the video to be processed); (a') and (b') are two frames from different shooting angles in the target video after replacement. It can be seen that, based on the method provided in this application embodiment, continuous replacement of objects shot from different angles in a video can be achieved.

[0134] It should be noted that the method provided in this application embodiment can be implemented on smart devices such as mobile phones and computers, enabling fast and convenient video editing capabilities. It achieves smooth replacement of objects within the video while ensuring the spatial and temporal continuity of the replaced object. Furthermore, it can automatically generate a 3D model of the replaced object through text input.

[0135] In summary, the embodiments of this application provide a dynamic video content replacement method based on artificial intelligence. First, it can achieve smooth replacement of items in the video and ensure the spatial and temporal continuity of the replaced object. Second, based on input such as text and voice, it can automatically generate a three-dimensional model of the replaced object without requiring the user to perform 3D modeling.

[0136] Through actual testing, the method provided in this application embodiment can achieve the following technical effects:

[0137] The video edited using the method described in this application will not experience shaky footage. Individual items within the video can be arbitrarily replaced, and the replacement items do not require manual modeling; a 3D model can be automatically generated simply through text or voice commands. This significantly lowers the barrier to entry for users engaged in video editing and creation, improving the convenience and effectiveness of video editing.

[0138] This application also provides a computer program product, which includes a program that, when run by an electronic device, causes the electronic device to perform the method described in any of the above embodiments. For example, the program product may be an application program, or an upgrade package or feature plugin for upgrading the application program, etc.

[0139] This application also provides an electronic device, the electronic device including: one or more processors; a memory; at least one application program; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the electronic device, cause the electronic device to perform the method as described in any of the above embodiments.

[0140] Specifically, the electronic device can be a server or a cluster of servers that provides operator sequence optimization services. See also Figure 13 The hardware structure of electronic devices can include: processor, memory, and chipset.

[0141] The processor is the core that handles all computing tasks and requests, and includes at least the CPU.

[0142] Storage: This includes RAM and hard disk. The larger the capacity, the more requests and files can be processed simultaneously.

[0143] Chipset: Includes the Northbridge chip and the Southbridge chip, which connect various I / O devices. The Northbridge chip connects to the CPU, memory, and PCI bus, while the Southbridge chip connects to I / O devices such as hard drives and network devices.

[0144] I / O Bus: Connects various I / O devices, such as network cards and USB ports. The larger the I / O bus bandwidth of the server, the more requests it can handle. I / O Devices: Include network cards, USB ports, etc. The network card is the main device for connecting the server to the external network, and its speed and performance directly affect the server's response speed and transmission speed.

[0145] Power supply: Provides stable power to the server to ensure its normal operation.

[0146] Chassis: The outer casing of server hardware, protecting it from dust, vibration, and temperature.

[0147] For example, a processor may include one or more processing units, such as a neural network processing unit (NPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a digital signal processor (DSP), a baseband processor, etc. The different processing units may be independent devices or integrated into one or more processors. The controller can generate operation control signals based on the instruction opcode and timing signals to control instruction fetching and execution.

[0148] The memory can be used to store executable program code, including instructions. Internal memory may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc. The data storage area may store data created during the use of the electronic device (such as input data, output data, etc.). Furthermore, internal memory may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. The processor executes various functional applications and data processing of the electronic device by running instructions stored in the internal memory and / or instructions stored in memory located within the processor.

[0149] Alternatively, electronic devices can also be terminal devices with a certain computing power, such as mobile phones, personal computers (PCs) (e.g., laptops), tablet PCs, wearable electronic devices, etc. Terminal devices may include processors, external memory interfaces, internal memory, universal serial bus (USB) interfaces, charging management modules, power management modules, batteries, etc.

[0150] It is understood that the structures illustrated in the embodiments of the present invention are merely examples and do not constitute a limitation on the electronic device. The electronic device in the embodiments of this application may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0151] This application also provides a storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the method described in any of the above embodiments.

[0152] This application also provides a chip system, including a communication interface for inputting and / or outputting data; and a processor for executing a computer-executable program, causing a device equipped with the chip system to perform the methods described in any of the above embodiments.

[0153] It should be noted that the solutions provided in the embodiments of this application include any one or more of video content replacement methods, program products, electronic devices, and chip systems, or may be other forms of software products, hardware products, or virtual service products obtained based on the technical concept of the methods described in this application specification.

[0154] In the description of the apparatus in the embodiments of this application, various modules are constructed in the apparatus to implement the function of the method flow. The division of modules is only a logical division of functions. When implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware.

[0155] Specifically, in actual implementation, the various functional modules can be fully or partially integrated into a single physical entity, or they can be physically separated. These modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented in software via processing element calls, while others are implemented in hardware. For example, the detection module can be a separate processing element or integrated into a chip within the electronic device. The implementation of other modules is similar. Furthermore, these modules can be fully or partially integrated together, or implemented independently. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0156] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs). Alternatively, these modules can be integrated together as a system-on-a-chip (SOC).

[0157] Specifically, in one embodiment of this application, the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the device, cause the device to perform the method steps described in the embodiment of this application.

[0158] Specifically, in one embodiment of this application, the processor of the electronic device may be a device-on-a-chip (SoC), which may include a central processing unit (CPU) and may further include other types of processors, such as a graphics processing unit (GPU). Specifically, in one embodiment of this application, the processor of the electronic device may be a PWM control chip.

[0159] Specifically, in one embodiment of this application, the processor may include, for example, a CPU, DSP, microcontroller, or digital signal processor, and may also include a GPU, embedded neural network processing units (NPUs), and image signal processors (ISPs). The processor may also include necessary hardware accelerators or logic processing hardware circuits, such as ASICs, or one or more integrated circuits for controlling the execution of the program of the technical solution of this application. Furthermore, the processor may have the function of operating one or more software programs, which may be stored in a storage medium.

[0160] Specifically, in one embodiment of this application, the memory of the electronic device may be a read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), or other types of dynamic storage devices capable of storing information and instructions. It may also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices. Alternatively, it may be any computer-readable medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer.

[0161] Specifically, in one embodiment of this application, the processor and memory can be combined into a single processing device, or more commonly, they are independent components. The processor executes program code stored in the memory to implement the method described in the embodiments of this application. In specific implementations, the memory can also be integrated into the processor, or it can be independent of the processor.

[0162] Furthermore, the devices, apparatuses, and modules described in the embodiments of this application may be implemented by computer chips or physical entities, or by products with certain functions.

[0163] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media containing computer-usable program code.

[0164] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0165] The embodiments described in this application are described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0166] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0167] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0168] It should also be noted that in the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0169] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0170] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0171] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0172] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments of this application can be implemented using electronic hardware, computer software, or a combination of electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0173] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0174] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method for replacing video content, characterized in that, The method includes: Obtain the first 3D model corresponding to the video to be processed; In response to the first semantic information input by the user, a region to be replaced in the first three-dimensional model is determined; the region to be replaced is the region corresponding to the first object, and the region to be replaced is the spatial region occupied by the first object in the three-dimensional space of the first three-dimensional model; the first semantic information is used to describe the first object in the video to be processed, and the first object is the object to be replaced; In response to the second semantic information input by the user, a second three-dimensional model corresponding to the second object is generated; the second object is the target object used to replace the first object. In the first three-dimensional model, the second three-dimensional model is used to replace or cover the area to be replaced to obtain the fused third three-dimensional model; Generate a target video corresponding to the third 3D model; in the target video, the first object is replaced with the second object; The first three-dimensional model is a model obtained by three-dimensional reconstruction based on the camera pose and first depth image corresponding to each frame of the first image in the video to be processed; The second three-dimensional model is a model obtained by three-dimensional reconstruction based on the camera pose and second depth image corresponding to each frame of the second image in the image sequence corresponding to the second object; The image sequence corresponding to the second object is: The image sequence obtained based on the second semantic information, the mask corresponding to each keyframe extracted from the video to be processed, and the first depth image, wherein the mask corresponding to each keyframe is the mask corresponding to the first object; or... The image sequence is obtained based on the second semantic information, the mask corresponding to the first image of each frame in the video to be processed, and the first depth image. The mask corresponding to the first image of each frame is the mask corresponding to the first object.

2. The method as described in claim 1, characterized in that, The process of obtaining the first 3D model corresponding to the video to be processed includes: In response to user instructions, camera pose estimation and depth estimation are performed on the first images of each frame in the video to be processed to obtain the camera pose and first depth images corresponding to each first image. Based on the camera pose and first depth image corresponding to each frame of the first image, three-dimensional reconstruction is performed to obtain the first three-dimensional model.

3. The method as described in claim 1, characterized in that, The step of determining the region to be replaced in the first 3D model in response to the first semantic information input by the user includes: In response to the first semantic information input by the user, the first image of each frame in the video to be processed is segmented to determine the boundary of the first object in each frame of the first image; Based on the boundary of the first object, generate a mask corresponding to each frame of the first image; Perform a mapping from the mask corresponding to each frame of the first image to the first 3D model to determine the boundary of the region to be replaced in the first 3D model.

4. The method as described in claim 3, characterized in that, The step of performing the mapping from the mask corresponding to each frame of the first image to the first 3D model, and determining the boundary of the region to be replaced in the first 3D model, includes: The coordinates of each 0 or 1 pixel in the mask corresponding to each frame of the first image are converted into the 3D coordinates of the corresponding pixel in the first 3D model through a preset mapping function. Based on the three-dimensional coordinates, the pixel point cloud in the first three-dimensional model is segmented to determine the boundary of the region to be replaced.

5. The method according to any one of claims 1-4, characterized in that, Before generating the second 3D model corresponding to the second object in response to the second semantic information input by the user, the method further includes: Extract the keyframe from the video to be processed; Extracting the keyframes from the video to be processed includes: Based on the camera poses corresponding to the first images of each frame in the video to be processed, determine the angular offset of the camera poses relative to the first object for each frame. At predetermined angular offsets, one frame is extracted and used as the keyframe.

6. The method according to any one of claims 1-4, characterized in that, The step of generating a second three-dimensional model corresponding to the second object in response to second semantic information input by the user includes: In response to the second semantic information input by the user, the mask corresponding to each key frame and the first depth image are merged to obtain the merged image; Based on the merged graph and the second semantic information, the image sequence corresponding to the second object is obtained; For each frame of the second image in the image sequence corresponding to the second object, depth estimation is performed to obtain the second depth image corresponding to each frame of the second image; Based on the second depth image and camera pose corresponding to each frame of the second image, three-dimensional reconstruction is performed to obtain a second three-dimensional model.

7. The method as described in claim 6, characterized in that, The step of obtaining the image sequence corresponding to the second object based on the merged graph and the second semantic information includes: A request message is sent to the target server, the request message carrying the merged graph and the second semantic information; Receive data returned by the target server; the returned data contains an image sequence corresponding to the second object.

8. The method according to any one of claims 1-4, characterized in that, The generation of the target video corresponding to the third 3D model includes: Perform a mapping from the third 3D model to the image sequence to obtain the target image sequence; Each frame in the target image sequence is rendered at least once; the rendered frames are then combined into a target video.

9. A computer program product, characterized in that, The program product includes a program that, when run by an electronic device, causes the electronic device to perform the method as described in any one of claims 1-8.

10. An electronic device, characterized in that, The electronic device includes: One or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the electronic device, cause the electronic device to perform the method as described in any one of claims 1-8.

11. A storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-8.

12. A chip system, characterized in that, include: A communication interface for inputting and / or outputting data; A processor for executing a computer-executable program, causing a device having the chip system mounted to perform the method as described in any one of claims 1-8.

Images