An e-commerce video intelligent replication method and system

Abstract

The present application relates to the technical field of data processing, and more particularly to an e-commerce video intelligent replication method and system. The method comprises: obtaining a reference video, a target product image and a target model image; extracting relationship triples and camera motion trajectories from the reference video; inputting the interactive relationship triples and the camera motion trajectories into a large language model to generate a structured script; performing feature extraction on the target product image and the target model image respectively to obtain product texture features and model facial features; processing the structured script using a diffusion model to generate a plurality of video segments; wherein when generating each video frame of the video segment, the product texture features and the model facial features are respectively injected into product mask regions and character mask regions in the video frame through a spatial decoupling attention mechanism; and the video segments are spliced in chronological order to obtain a target video. The method of the present application can improve the accuracy of video replication.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for intelligent replication of e-commerce videos. Background Technology

[0002] In e-commerce marketing, short videos have become a core medium for improving product conversion rates, showcasing product features, and attracting user attention. Traditional manual shooting and editing methods are costly and time-consuming, making it difficult to meet the e-commerce industry's demand for rapid video content iteration. In response, existing technologies have emerged that utilize artificial intelligence to assist in video production.

[0003] For example, Chinese patent application CN119963255A discloses a video advertisement generation method and system. The method includes: receiving advertisement text and extracting themes and keywords from the advertisement text; selecting case videos with exposure based on the extracted themes from media videos within a preset time range, and selecting materials from a preset material library based on the extracted keywords; calculating the individual exposure of the selected materials based on the case videos, combining materials based on the individual exposure, and generating a draft video; sending the draft video to a verification terminal, receiving evaluation information from the verification terminal, and filtering the draft video; wherein, the material selection process is connected to an AI module, and the application conditions of the AI ​​module are: comparing the individual exposure of the selected materials with a preset exposure threshold, recording the number of materials with an individual exposure greater than the preset exposure threshold, and when the number of materials is less than a preset quantity threshold, applying the AI ​​module to supplement materials.

[0004] This solution can replicate the exposure of viral videos by generating videos containing footage highly similar to those in viral videos. However, it only selects and combines footage with high similarity to viral videos to create the target video, ignoring the camera movement, pacing, and interactive logic of the original viral videos. This results in the replicated video lacking core appeal. Furthermore, existing AI (artificial intelligence) generation solutions are prone to issues such as facial drift and product distortion, leading to discrepancies between the advertised product and the actual product, failing to meet usage standards.

[0005] Therefore, how to accurately replicate high-quality e-commerce videos is a technical problem that urgently needs to be solved. Summary of the Invention

[0006] To address the issues of existing technologies neglecting camera movement and interactive logic, and the resulting videos exhibiting facial drift and product distortion, this invention provides solutions in the following aspects.

[0007] In a first aspect, the present invention provides an intelligent replication method for e-commerce videos, comprising: acquiring a reference video, a target product image, and a target model image; extracting relation triples and camera motion trajectories from the reference video; the relation triples being used to describe the interaction relationships between people and objects in the reference video; inputting the relation triples and the camera motion trajectory into a large language model to generate a structured script; extracting features from the target product image and the target model image to obtain product texture features and model facial features; processing the structured script using a diffusion model to generate several video segments; wherein, when generating each video frame of a video segment, a spatial decoupling attention mechanism is used to inject the product texture features and model facial features into the product mask region and the person mask region of the video frame, respectively; and splicing the video segments in temporal order to obtain the target video.

[0008] Furthermore, feature extraction is performed on the target product image and the target model image to obtain product texture features and model facial features, including: extracting product texture features from the target product image using IP-Adapter Plus; and extracting model facial features from the target model image using FaceID-Adapter.

[0009] Furthermore, the method further includes: when generating the (i+1)th video segment, extracting the last frame of the i-th video segment; using the last frame of the i-th video segment as a reference for the first frame of the (i+1)th video segment, generating other video frames in the (i+1)th video segment.

[0010] Furthermore, the process of injecting product texture features and model facial features into the product mask region and person mask region in the video frame, respectively, is represented as follows:

[0011]

[0012] In the formula, Attention after feature injection This is the original self-attention calculation result of the diffusion model. The preset first injection intensity, The preset second injection intensity, The results of cross-attention calculation are based on the target model image. Masking the person The results of cross-attention calculation are based on the target product graph. For product mask, For Hadamaji.

[0013] Furthermore, the method also includes generating product masks and person masks using a SAM model.

[0014] Furthermore, before obtaining the target video, the method further includes: scoring the video segment using a preset multi-granularity closed-loop review model to obtain a comprehensive score for the video segment; determining whether the comprehensive score of the video segment is greater than a preset threshold; if not, partially redrawing the video segment until the comprehensive score of the redrawn video segment is greater than the preset threshold.

[0015] Furthermore, the formula for calculating the comprehensive score is as follows:

[0016] In the formula, For comprehensive scoring, , , All are preset weighting coefficients. The cosine similarity function is used. The facial feature vectors extracted from the generated video frames. To reference the facial feature vector in the model image, This is a visual feature similarity calculation module based on the DINOv2 model. For the generated video frames, For the target product image, Visual artifact penalty item Furthermore, the method also includes: using OCR technology to detect the integrity of the text content of the target product in the video clip.

[0017] Furthermore, the method also includes: determining the duration of each video segment according to a preset adaptive duration allocation algorithm.

[0018] In a second aspect, the present invention also provides an e-commerce video intelligent replication system, including a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the e-commerce video intelligent replication method described in the first aspect is implemented.

[0019] The beneficial effects of this invention are as follows: By extracting the human-object interaction logic and camera motion trajectory from the reference video to generate a structured script, the subsequently generated target video can replicate the interaction logic and camera movement rhythm of the reference video, thereby improving the accuracy of video replication; through a spatial decoupling attention mechanism, product texture features and model facial features are injected into the corresponding areas of the video frames, respectively, which can avoid mutual contamination of features, thereby solving the problem of mismatch between the product and the description in the prior art, and thus improving the quality of video replication; by redrawing video segments that do not meet the requirements of the comprehensive score, the quality of video replication is further improved, and finally, low-cost, high-fidelity, and scalable e-commerce video replication is achieved. Attached Figure Description

[0020] Figure 1 This is a flowchart of the e-commerce video intelligent replication method in an embodiment of the present invention; Figure 2 This is a structural block diagram of the e-commerce video intelligent replication system in an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] To highlight the key technical points of the present invention, the specification and accompanying drawings do not elaborate on or omit structures, steps, or operational features that are not directly related to solving the technical problems of this application when describing the embodiments. Such omissions do not affect the understanding of the technical solution of the present invention by those skilled in the art, nor do they constitute a limitation on the embodiments. Those skilled in the art can supplement or replace the relevant content based on conventional technical knowledge without affecting the completeness and feasibility of the technical solution of the present invention.

[0023] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0024] Figure 1 This is a flowchart of the e-commerce video intelligent replication method in an embodiment of the present invention.

[0025] In a first aspect, the present invention provides a method for intelligent replication of e-commerce videos. Specifically, as... Figure 1 As shown, the method of the present invention includes the following steps.

[0026] S1. Obtain reference videos, target product images, and target model images.

[0027] Specifically, users upload reference videos, target product images, and target model images to the platform to obtain these resources. Reference videos are viral videos, popular videos, or videos that users want to replicate. Viral videos are those with high likes, comments, saves, or shares. Target product images are images of the products users want to showcase on the video platform, and target model images are images of the people / characters users want to portray.

[0028] S2. Extract relation triples and camera motion trajectories from the reference video.

[0029] Specifically, the Spatiotemporal Scene Atlas (STSG) technique is used to deconstruct the reference video, thereby extracting the "subject-action-object" triples (i.e., relationship triples / HOI triples) and camera motion trajectories for each keyframe in the reference video. The relationship triples describe the interaction relationships between people and objects in the reference video and can be defined as follows:<Subject,Predicate,Object> The semantic structure, for example, the relation triple extracted from a reference video is <model, holding, face cream>.

[0030] In one embodiment, the reference video can be segmented into multiple video segments based on scene change detection, and then keyframes can be extracted from each video segment.

[0031] In one embodiment, the SfM (Structure of Motion) algorithm can be used to calculate the camera motion trajectory of the reference video. Since spatiotemporal scene mapping and the SfM algorithm are existing technologies, the specific processing steps will not be described in detail here.

[0032] By extracting the relationship triples from the reference video, it can be ensured that the interaction between the model and the product in the generated target video strictly corresponds to the viral video; by extracting the camera motion trajectory of the reference video, the subsequent generated target video can replicate the push-pull, pan, and tilt camera movements of the reference video.

[0033] S3. Input the interaction triplet and camera motion trajectory into the large language model to generate a structured script.

[0034] In this embodiment, the structured script includes action constraints, control parameters, CameraListy (storyboard list), and Mask (mask) planning, etc.

[0035] Specifically, relation triples and camera motion trajectories are input into the large language model as prompt words. Simultaneously, a pre-defined rule engine constrains the output of the large language model to obtain a structured script. The large language model maps the camera motion trajectory to Camera LoRA control parameters of the diffusion model. These control parameters constrain the camera's physical rules during video clip generation, such as shooting angle, focal length, and zoom movement, ensuring that the generated target video maintains the same camera movement rhythm as the reference video.

[0036] By extracting relation triples and camera motion trajectories, and then generating structured scripts, it is possible to ensure that the human-object interaction logic and camera movement rhythm of the subsequently generated target video are consistent with those of the reference video. This solves the problem in existing technologies that neglect interaction logic and camera movement rhythm when replicating reference videos, making it difficult to replicate them deeply, and improves the accuracy of video replication.

[0037] S4. Extract features from the target product image and the target model image respectively to obtain product texture features and model facial features.

[0038] Specifically, the IP-Adapter Plus is used to extract features from the target product image to obtain high-definition product texture features; at the same time, the FaceID-Adapter is used to extract features from the target model image to obtain the model's facial features.

[0039] In other optional embodiments, those skilled in the art can use other methods to extract product texture features and model facial features, such as extracting product texture features using the CLIP (Contrastive Language-Image Pre-Training) model and extracting model facial features using the ArcFace model.

[0040] S5. The structured script is processed using a diffusion model to generate several video clips.

[0041] In this embodiment, the diffusion model can be the Dual-Stable Diffusion model.

[0042] Specifically, a pre-defined parsing engine is used to parse the structured script, and the parsing results are input into a pre-trained diffusion model for processing to obtain several video segments. When the diffusion model generates each video frame of a video segment, the SAM (Segment Anything Model) model generates the person mask and product mask for the current video frame in real time. Then, through a spatial decoupling attention mechanism, product texture features are injected into the product mask region of the current video frame, and model facial features are injected into the person mask region of the current video frame.

[0043] In one embodiment, the process of injecting product texture features and model facial features into the product mask region and person mask region in the generated video frame, respectively, can be represented as:

[0044] In the formula, Attention after feature injection This is the original self-attention calculation result of the diffusion model. The preset first injection intensity, The preset second injection intensity, The results of cross-attention calculation are based on the target model image. Masking the person The results of cross-attention calculation are based on the target product graph. For product mask, This is the Hadama product (element-by-element multiplication).

[0045] In one embodiment, , and The calculation expression is:

[0046]

[0047]

[0048] in, For attention calculation module, The query vector is obtained by linear transformation of the feature map of the current video frame; The key vector is obtained by linear transformation of the feature map of the current video frame; The value vector is obtained by linear transformation of the feature map of the current video frame; The key vector extracted from the target model image. This is the value vector extracted from the target model image. The key vector extracted from the target product image. This is the value vector extracted from the target product image.

[0049] By using a spatial decoupling attention mechanism, product texture features are forcibly confined to the product mask area, and model facial features are confined to the person mask area. This effectively avoids mutual contamination of features, thereby ensuring the fixation of the model's identity and the zero-distortion restoration of the product's appearance. It solves the problem of facial and product deformation, thus improving the quality of the generated target video.

[0050] In one embodiment, when generating video segments, the method of the present invention may further include: determining the duration of each video segment using an adaptive duration allocation algorithm. Specifically, the calculation expression for the duration of each video segment is:

[0051] In the formula, For the first The duration of each video clip For the first The weight of each video segment, The target video duration This represents the number of video clips.

[0052] in, This is obtained by segmenting the reference video using scene change detection technology. Specifically, frame-level analysis is performed on the reference video to detect key scene transition nodes, such as camera transitions, background changes, and subject / action changes. Using these key scene transition nodes as boundaries, the reference video is split into multiple independent video segments. The total number of video segments obtained after this segmentation is... .

[0053] The total duration of the target video can be set by the user, or the total duration of the reference video can be used as the target video's total duration. It can be determined based on expert experience, the motion amplitude of reference video clips, or the complexity of each scene.

[0054] In one embodiment, after generating a video segment, the method of the present invention further includes: performing a comprehensive consistency review on each video frame in the currently generated video segment. Specifically, the video frames of the currently generated video segment are input into a multi-granularity closed-loop review model to calculate a comprehensive score for the video frame; it is determined whether the comprehensive score is greater than a preset threshold. If so, it indicates that the currently generated video frame is qualified, and the review or generation of the next video frame is performed; if not, it indicates that the currently generated video frame is unqualified, for example, the generated video frame has problems such as facial drift, product deformation, or screen flickering, and a local redraw is triggered. The above process is repeated until the comprehensive score of the redrawn video frame is greater than the preset threshold. The preset threshold can be set to 0.85 or other values, which are not limited here.

[0055] Specifically, the process involves locating / extracting masked areas with low scores, then using Inpainting / FaceSwap tools to repair these areas, resulting in repaired video frames. The comprehensive score of these repaired video frames is calculated, and it is determined whether the comprehensive score exceeds a preset threshold. If so, the next video frame is reviewed or generated; otherwise, a local redraw is triggered. The multi-granularity closed-loop review model can employ a DCRN (Dual-Check Consistency Review Network) model, which acts as an AI quality inspector, performing three-dimensional scoring on the generated video frames for person identity, product details, and visual artifacts. These three-dimensional scores are then weighted and summed to obtain a comprehensive score. In other optional embodiments, only person identity and product details may be scored, or at least one of these three categories (person identity, product details, and visual artifacts) may be scored and reviewed.

[0056] In one embodiment, the formula for calculating the overall score is:

[0057] In the formula, For comprehensive scoring, , , All are preset weighting coefficients. The cosine similarity function is used. The facial feature vectors extracted from the generated video frames. The facial feature vectors extracted from the reference model image. This is a visual feature similarity calculation module based on a pre-trained DINOv2 model. For the generated video frames, For the target product image, This is a visual artifact penalty, calculated based on light trail anomaly detection or screen flicker.

[0058] The process involves extracting visual feature vectors from the generated video frames and the target product image using the DINOv2 model, and then calculating the similarity between the visual feature vectors of the generated video frames and the visual feature vectors of the target product image using cosine similarity. The generated video frame can be processed using the NIQE (Naturalness Image Quality Evaluator) algorithm, or other existing algorithms that can evaluate the degree of visual artifacts in the video frame can be used to obtain the visual artifact penalty term for the video frame. No restrictions are imposed here.

[0059] By conducting a comprehensive consistency review, targeted repairs can be performed on video frames with facial drift, product deformation, or screen flickering, thus preventing the generation of videos with facial drift, product deformation, or screen flickering, thereby improving the quality of the target video and achieving a closed-loop consistency review.

[0060] In one embodiment, when generating the next video segment, the last frame of the current video segment is extracted and used as the reference frame for the first frame of the next video segment. Then, the other video frames in the next video segment are generated. By using the last frame of the current video segment as the reference frame for the first frame of the next video segment, the visual continuity of long videos can be ensured, thereby improving the quality of video replication.

[0061] In one embodiment, the method of the present invention further includes: using OCR (Optical Character Recognition) technology to detect and verify whether the brand text on the product packaging in the generated video frame is complete; if it is incomplete, triggering a local redraw.

[0062] S6. Segment the video clips in chronological order to obtain the target video.

[0063] The generated target video can be displayed directly on the video platform; to further improve the quality of the target video, it can also be reviewed and adjusted by experts before being displayed.

[0064] In one embodiment ， To ensure no jumps at the junctions of segments, an inter-frame continuity constraint loss function is introduced. Specifically, this loss function... for:

[0065] In the formula, This refers to the feature extraction function (perceptual feature extractor or encoder). For the first The last frame of a video clip, For the first The first frame of a video clip. It is an L2 norm.

[0066] The loss function constrains the minimization of visual feature distance at the seams of adjacent segments, and this constraint changes with the training or generation process. Through minimization optimization, adjacent frames will be infinitely similar in visual features, thereby eliminating the visually noticeable jumps and ensuring the smoothness of long videos in terms of sequence.

[0067] Optionally, the method of the present invention operates on a multi-agent collaborative architecture based on LangGraph, which supports user intervention at any node such as script, storyboard, and review, and supports dual-mode switching of "reproduction / creation" and fine-grained version rollback.

[0068] Specifically, the architecture includes an initialization node, a deep parsing node, a main inference node, a DCRN review node, a state management node, a completed segment node, and a video merging node. The initialization node performs initialization; the deep parsing node extracts HOI interaction relationships and camera motion trajectories, and generates structured scripts; the main inference node schedules the order of generation, review, and correction tools, and manages the token budget; the state management node stores the global context, including the URL of the currently generated last frame (used as a reference for the first frame of the next segment), historical review scoring records, and HOI structured data; the completed segment node calls Dual-Stream Diffusion to generate video segments; and the video merging node splices / merges video segments that have passed review.

[0069] Figure 2 This is a structural block diagram of the e-commerce video intelligent replication system in an embodiment of the present invention.

[0070] In a second aspect, the present invention also provides an intelligent e-commerce video replication system. Specifically, as... Figure 2As shown, the system includes a processor and a memory, the memory storing computer program instructions, which, when executed by the processor, implement the e-commerce video intelligent replication method described in the first aspect of the present invention.

[0071] The system also includes other components well known to those skilled in the art, such as communication interfaces, the settings and functions of which are known in the art and will not be described in detail here.

[0072] In this invention, the aforementioned memory can be any tangible medium containing or storing a program that can be used or combined with an instruction execution system, apparatus, or device. For example, a computer-readable storage medium can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc., or any other medium that can be used to store desired information and can be accessed by an application, module, or both. Any such computer storage medium can be part of a device or accessible to or connected to a device. Any application or module described in this invention can be implemented using computer-readable / executable instructions that can be stored or otherwise maintained by such a computer-readable medium.

[0073] In the description of this specification, "multiple" means at least two, such as two, three or more, etc., unless otherwise explicitly specified. Furthermore, the division of steps in the above method is only for clarity of description; in implementation, it can be combined into one step or some steps can be split into multiple steps, as long as they include the same logical relationship. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0074] While this specification has shown and described numerous embodiments of the invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and essence of the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in the practice of this invention.

Claims

1. A method for intelligent replication of e-commerce videos, characterized in that, include: Obtain reference videos, target product images, and target model images; Extract relation triples and camera motion trajectories from reference videos; The relation triples are used to describe the interaction relationships between people and objects in the reference video; The interaction triples and the camera motion trajectory are input into the large language model to generate a structured script; Feature extraction was performed on the target product image and the target model image to obtain product texture features and model facial features; The structured script is processed using a diffusion model to generate several video clips. During the generation of each video frame, a spatial decoupling attention mechanism is used to inject product texture features and model facial features into the product mask region and person mask region of the video frame, respectively. This also includes... A pre-defined multi-granularity closed-loop review model is used to score the video frames of a video segment, resulting in a comprehensive score for each video frame. Determine whether the overall score of the video frame is greater than the preset threshold. If not, perform local redrawing of the video frame until the overall score of the redrawn video frame is greater than the preset threshold. Among them, the large language model maps the camera motion trajectory to the Camera LoRA control parameters of the diffusion model, which are used to constrain the camera physical rules in the video clip generation process; The video segments are spliced ​​together in chronological order to obtain the target video.

2. The e-commerce video intelligent replication method according to claim 1, characterized in that, Feature extraction was performed on the target product image and the target model image respectively to obtain product texture features and model facial features, including: The IP-Adapter Plus is used to extract product texture features from the target product image; The FaceID-Adapter is used to extract the model's facial features from the target model image.

3. The e-commerce video intelligent replication method according to claim 1, characterized in that, The method further includes: When generating the (i+1)th video segment, extract the last frame of the i-th video segment; Use the last frame of the i-th video segment as a reference for the first frame of the (i+1)-th video segment to generate the other video frames in the (i+1)-th video segment.

4. The e-commerce video intelligent replication method according to claim 1, characterized in that, The process of injecting product texture features and model facial features into the product mask region and person mask region in the video frame, respectively, is represented as: In the formula, Attention after feature injection This is the original self-attention calculation result of the diffusion model. The preset first injection intensity, The preset second injection intensity, The results of cross-attention calculation are based on the target model image. Masking the person The results of cross-attention calculation are based on the target product graph. For product mask, For Hadamaji.

5. The e-commerce video intelligent replication method according to claim 1 or 4, characterized in that, Also includes: The SAM model is used to generate product masks and person masks.

6. The e-commerce video intelligent replication method according to claim 1, characterized in that, The formula for calculating the overall score is: In the formula, For comprehensive scoring, , , All are preset weighting coefficients. The cosine similarity function is used. The facial feature vectors extracted from the generated video frames. To reference the facial feature vector in the model image, This is a visual feature similarity calculation module based on the DINOv2 model. For the generated video frames, For the target product image, This is a penalty for visual artifacts.

7. The e-commerce video intelligent replication method according to claim 1, characterized in that, The method also includes: using OCR technology to detect the integrity of the text content of the target product in the video clip.

8. The e-commerce video intelligent replication method according to claim 1, characterized in that, The method further includes: determining the duration of each video segment according to a preset adaptive duration allocation algorithm.

9. An intelligent e-commerce video replication system, characterized in that, It includes a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the e-commerce video intelligent replication method according to any one of claims 1-8 is implemented.

Images