A commodity detail page multimodal text and video collaborative generation method

Abstract

This invention discloses a method for collaboratively generating multimodal images, text, and video on a product details page, relating to the field of image and video generation technology. The method includes: Step 1: Generating a set of product selling point units; Step 2: Generating a set of image selling point evidence areas; Step 3: Generating a sequence of video selling point image blocks; Step 4: Performing a selling point evidence weaving operation to construct a selling point evidence continuity diagram; Step 5: Inputting the selling point evidence continuity diagram into an improved TimeSformer model, introducing a selling point temporal residency operator, and setting an evidence continuity calibration layer to generate a selling point spatiotemporal continuity result; Step 6: Generating the image and text generation result and the video generation result for the details page; Step 7: Performing consistency verification to generate selling point misalignment records; Step 8: Generating the collaborative generation result of images, text, and video. This invention combines the selling point evidence weaving operation and the improved TimeSformer model to achieve consistent collaborative generation of images, text, and video on a product details page.

Description

Technical Field

[0001] This invention relates to the field of image and video generation technology, and in particular to a method for collaborative generation of multimodal images, text, and video for product detail pages. Background Technology

[0002] With the increasing demand for short video displays, automatic product detail page generation, and multimedia content arrangement on e-commerce platforms, the collaborative generation technology of text and video for basic product information, product image materials, and product video materials has received widespread attention. Current methods for generating product detail pages mainly rely on templated text splicing, single product image recognition, or simple extraction of video clips. However, these methods commonly suffer from the following problems in practical applications:

[0003] Product selling points are typically derived from text descriptions, but images and videos lack a stable correspondence between selling points and product areas. This leads to issues such as unclear descriptions, insufficient visual evidence, and selling points being disconnected from the visible product area in the product details page. Product video footage usually contains multiple frames, multiple parts, and multiple viewing angles. Existing video processing methods struggle to continuously track the spatial image blocks of the same product selling point in adjacent video frames, resulting in issues such as scene jumps, incomplete selling point displays, and desynchronization between subtitles and the video after video clip extraction. Existing multimodal generation methods typically process text and image generation separately from video generation, lacking verification and correction based on a unified evidence connection. When misalignments occur between text / image selling point descriptions, video selling point segments, and video subtitle text, it is difficult to promptly locate the type of misalignment and the corresponding correction position, thus affecting the consistency between the text / image content, video footage, and subtitle content on the product details page.

[0004] Therefore, how to provide a method for collaboratively generating multimodal images, text, and video for product detail pages is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a method for the collaborative generation of multimodal images, text, and video on a product details page. This invention extracts product selling point units, binds product image regions and video frame spatial image blocks with selling point identity markers, forming an image selling point evidence set and a video selling point image block sequence; constructs a selling point evidence continuity graph through selling point evidence weaving operations, and inputs it into an improved TimeSformer model to generate a spatiotemporal continuity result for selling points; then performs image, text, and video generation, consistency verification, and misalignment correction to achieve collaborative and consistent generation of images, text, and video on the product details page.

[0006] A method for collaboratively generating multimodal images, text, and video for a product details page according to an embodiment of the present invention includes the following steps:

[0007] Step 1: Obtain product selling point units based on the basic product information, configure selling point identity tags for the product selling point units, and generate a set of product selling point units;

[0008] Step 2: Identify the product image region corresponding to the product selling point unit based on the product image material, and bind the product image region with the selling point identification mark to generate an image selling point evidence region set;

[0009] Step 3: Perform frame-level block processing on the product video material, and write the selling point identification mark into the spatial image block of the corresponding video frame to generate a video selling point image block sequence;

[0010] Step 4: Perform a selling point evidence weaving operation based on the product selling point unit set, the image selling point evidence area set, and the video selling point image block sequence to construct a selling point evidence connection diagram;

[0011] Step 5: Input the aforementioned selling point evidence connection diagram into the improved TimeSformer model, introduce the selling point temporal residence operator in the temporal self-attention layer, and set an evidence connection calibration layer between the temporal self-attention layer and the spatial self-attention layer to generate the selling point spatiotemporal connection result;

[0012] Step Six: Based on the spatiotemporal connection results of the aforementioned selling points, generate the detailed page image and text generation results and the detailed page video generation results;

[0013] Step 7: Based on the aforementioned selling point evidence connection diagram, perform consistency verification on the image and text generation results and the video generation results of the details page, and generate a selling point misalignment record;

[0014] Step 8: Based on the misalignment record of the selling points, correct the image and text generation results and the video generation results of the details page, and generate a collaborative image and text and video generation result.

[0015] Optionally, step one specifically includes:

[0016] Obtain the basic product information from the product details page generation task, and extract the product name, product category, specification description, function description, and usage scenario description from the basic product information to generate basic product description text;

[0017] The basic description text of the product is semantically segmented into short sentences to obtain a set of product description sentences;

[0018] The product description phrase set is filtered for consistency with the product subject based on the product name and product category. Product description phrases that do not point to the current product subject are deleted, and a set of valid product description phrases is generated.

[0019] Product selling point units are extracted from the set of effective product description phrases. These product selling point units include appearance selling point units, structural selling point units, functional selling point units, and scenario selling point units.

[0020] Configure a selling point identity tag for each product selling point unit according to product category, selling point type, and selling point order;

[0021] The product selling point units after configuring the selling point identity tags are summarized to generate a product selling point unit set.

[0022] Optionally, step two specifically involves:

[0023] Obtain product image materials from the product details page generation task, record the image number of each product image, and perform product body segmentation on each product image to obtain the product body area;

[0024] Extract the appearance outline area, local structure area, functional component area, and scene contact area within the main body area of ​​the product;

[0025] Based on the appearance selling point units, structural selling point units, functional selling point units and scene selling point units in the product selling point unit set, the candidate product image areas are obtained by corresponding them to the appearance outline area, local structure area, functional component area and scene contact area respectively.

[0026] The candidate product image regions are matched with the selling point descriptions of the corresponding product selling point units to determine the product image regions;

[0027] Read the selling point identification tag corresponding to the product selling point unit, and write the selling point identification tag into the region attribute field of the product image area;

[0028] The product image areas with the added selling point identification tags are summarized according to image number and area location to generate an image selling point evidence area set.

[0029] Optionally, step three specifically includes:

[0030] Obtain the product video footage from the product details page generation task and record the video ID of each product video footage;

[0031] The product video material is extracted frame by frame according to the video frame order to generate a product video frame sequence, and a frame number is assigned to each video frame.

[0032] Each video frame in the product video frame sequence is divided into several non-overlapping spatial image blocks, and a block position marker is configured for each spatial image block.

[0033] Based on the selling point identification tags written in the image selling point evidence area, extract the regional visual features of the corresponding product image area;

[0034] The visual features of each spatial image block are matched with the visual features of the region to determine the video frame and spatial image block corresponding to the product selling point unit.

[0035] Read the selling point identification tag corresponding to the product selling point unit, and write the selling point identification tag into the spatial image block attribute field of the corresponding video frame;

[0036] The spatial image blocks, after being labeled with selling points, are sorted according to video number, frame number, and block position label to generate a video selling point image block sequence.

[0037] Optionally, step four specifically includes:

[0038] Read the product selling point units and selling point identity tags from the product selling point unit set, and set the product selling point unit as the receiving master node;

[0039] Read the image selling point evidence area from the image selling point evidence area set and set the image selling point evidence area as a static evidence node;

[0040] Read spatial image blocks from the video selling point image block sequence and set the spatial image blocks as dynamic evidence nodes;

[0041] By combining static and dynamic evidence nodes corresponding to the same selling point identity marker, a continuous evidence trajectory of the product selling point unit is generated between the image area and the video frame image block.

[0042] Breakpoint detection is performed on the continuous evidence trajectory between adjacent product selling point units. The main node that lacks static evidence nodes or dynamic evidence nodes is marked as a weak main node, and the main node that has both static and dynamic evidence nodes, and the dynamic evidence nodes are continuously distributed according to frame number, is marked as a strong main node.

[0043] Based on the strong receiving nodes, weak receiving nodes are connected to each other in a sequential manner, and the weak receiving nodes are connected to the strong receiving nodes that are closest to them in the order of the product selling point unit set.

[0044] The strength of the receiving main node is obtained by averaging and weighting the static evidence existence value, dynamic evidence existence value, dynamic evidence continuity value and adjacent traction value.

[0045] The static evidence existence value is 1 when the main node has a static evidence node, and 0 otherwise.

[0046] The dynamic evidence existence value is 1 when the receiving master node has a dynamic evidence node, and 0 otherwise.

[0047] The continuous value of dynamic evidence is the number of dynamic evidence nodes with the largest continuous distribution divided by the total number of dynamic evidence nodes under the corresponding main node.

[0048] The adjacency traction value is 1 when the main node and the strong node are directly connected, and 0 otherwise.

[0049] Based on the main node, static evidence nodes, dynamic evidence nodes, and the strength of the connection, a selling point evidence connection diagram is generated.

[0050] Optionally, the improved TimeSformer model specifically includes a graph node embedding layer, a temporal self-attention layer, an evidence acceptance calibration layer, a spatial self-attention layer, a feedforward mapping layer, and an acceptance output layer;

[0051] The graph node embedding layer reads the main node, static evidence node, dynamic evidence node and bearing strength in the selling point evidence bearing graph, maps the static evidence node, dynamic evidence node and bearing strength to static evidence features, dynamic image block features and strength embedding features respectively, and splices the dynamic image block features and strength embedding features to generate a selling point image block embedding sequence.

[0052] The temporal self-attention layer groups the embedded sequences of selling point image blocks according to the receiving master node, performs inter-frame association processing on the embedded sequences of selling point image blocks under the same receiving master node, and generates selling point temporal residence features through the selling point temporal residence operator.

[0053] The evidence receiving calibration layer reads the selling point time-series residency features and static evidence features, concatenates the selling point time-series residency features with the corresponding static evidence features, and generates evidence calibration gate values ​​through linear mapping and the Sigmoid function;

[0054] The evidence calibration gate value is used to gate and retain the time-series residence features of the selling points, and the residuals of the gated features are added to the static evidence features to generate the evidence calibration features;

[0055] The spatial self-attention layer performs intra-frame spatial correlation processing on the dynamic image block features under the same frame number in the evidence calibration features to obtain spatial attention features. The spatial attention features are then added to the evidence calibration features by residual addition and layer normalization to generate the selling point spatial correlation features.

[0056] The feedforward mapping layer sequentially performs linear mapping and GELU activation processing on the spatial correlation features of selling points to generate spatiotemporal fusion features of selling points.

[0057] The receiving output layer pools the spatiotemporal fusion features of selling points under the same receiving master node to generate receiving discrimination features. The receiving discrimination features are then input into the text and image receiving output head, the video receiving output head, and the subtitle receiving output head, respectively. The Sigmoid function is used to generate text and image receiving scores, video receiving scores, and subtitle receiving scores, respectively.

[0058] The scores for text and image integration, video integration, and subtitle integration are used as the temporal and spatial integration results of the selling points.

[0059] Optionally, the selling point time-series dwell operator is specifically:

[0060] Based on the embedded sequence of selling point image blocks under the same main node, read the frame number and block position mark corresponding to the features of each dynamic image block;

[0061] Establish the temporal connection relationship between dynamic image block features according to the ascending order of frame number, and determine the dynamic image block features with adjacent block position markers in adjacent frame numbers as continuously resident image block features;

[0062] A resident mask matrix is ​​generated based on the temporal connection relationship and the features of consecutive resident image blocks. The matrix positions that belong to the same host node and constitute the features of consecutive resident image blocks are set as reserved values, while the matrix positions that do not belong to the same host node or do not constitute the features of consecutive resident image blocks are set as negative suppression values.

[0063] The sequence of embedded image blocks representing selling points is mapped into a query matrix, a key matrix, and a value matrix, respectively.

[0064] Multiply the query matrix by the transpose of the key matrix to obtain the initial temporal correlation score matrix between the features of each dynamic image block under the same host node. Then, scale the initial temporal correlation score matrix according to the feature dimension of the key matrix to obtain the temporal attention score matrix.

[0065] The residency mask matrix is ​​superimposed on the time attention score matrix to generate the selling point residency attention score matrix.

[0066] The selling point retention attention score matrix is ​​subjected to Softmax normalization to obtain the selling point retention attention weight matrix;

[0067] The selling point retention attention weight matrix and value matrix are multiplied together to generate the selling point time-series retention features.

[0068] Optionally, step six specifically includes:

[0069] Read the image and text acceptance score, video acceptance score, and subtitle acceptance score from the spatiotemporal acceptance results of the selling points;

[0070] Product selling point units with image and text connection scores greater than the set image and text connection threshold are identified as image and text generation selling point units, and the corresponding product selling point units and image selling point evidence areas are called to generate image and text selling point description segments.

[0071] The image and text description segments of the product's selling points are sorted according to the arrangement order of the product's selling point units, and the image and text generation results of the details page are generated in combination with the corresponding image selling point evidence area;

[0072] Product selling point units whose video acceptance score is greater than the set video acceptance threshold are identified as video generation selling point units, and the corresponding product video materials are called to generate video selling point clips.

[0073] Product selling point units with subtitle acceptance scores greater than the set subtitle acceptance threshold are identified as subtitle generation selling point units, and video subtitle text is generated based on the corresponding product selling point units.

[0074] Based on the frame numbering order corresponding to the video selling point segments, the video subtitle text is superimposed onto the corresponding video selling point segments to generate the video for the details page.

[0075] Optionally, step seven specifically includes:

[0076] Read the main node, static evidence node, and dynamic evidence node in the aforementioned selling point evidence connection diagram;

[0077] Extract the image and text description segments and corresponding image selling point evidence areas from the image and text generation results of the details page, and then look up the corresponding static evidence nodes based on the image selling point evidence areas;

[0078] Extract video selling point segments and video subtitle text from the video generation results on the details page, and look up the corresponding dynamic evidence nodes based on the frame numbers corresponding to the video selling point segments;

[0079] Verify the corresponding text and image selling point descriptions, static evidence nodes, video selling point clips, dynamic evidence nodes, and video subtitle texts under the same main node.

[0080] When the text and image description of selling points does not correspond to a static evidence node under the same main node, or the video selling point segment does not correspond to a dynamic evidence node under the same main node, or the video subtitle text is inconsistent with the product selling point unit corresponding to the text and image description of selling points, it is determined that there is a selling point misalignment.

[0081] Record the product's selling point unit, misalignment type, corresponding image number, and corresponding frame number where there is a misalignment, and generate a selling point misalignment record. The misalignment types include misalignment of textual and graphic evidence, misalignment of video evidence, and misalignment of subtitle transitions.

[0082] Optionally, step eight specifically includes:

[0083] Read the product selling point unit, misalignment type, corresponding image number, and corresponding frame number from the selling point misalignment record;

[0084] When the misalignment type is image and text evidence misalignment, the static evidence node under the same receiving main node is retrieved according to the corresponding image number, and the image and text selling point description section in the image and text generation result of the details page is rebound to the corresponding image selling point evidence area.

[0085] When the misalignment type is video evidence misalignment, the dynamic evidence node under the same receiving main node is retrieved according to the corresponding frame number, and the video selling point segment in the video generation result of the details page is replaced with the video frame segment where the corresponding dynamic evidence node is located.

[0086] When the misalignment type is subtitle continuation misalignment, the video subtitle text is regenerated based on the product selling point unit with selling point misalignment, and the regenerated video subtitle text is superimposed on the corresponding video selling point segment.

[0087] The revised results of generating text and images and the revised results of generating videos on the details page are summarized to generate a collaborative result of text, images and videos.

[0088] The beneficial effects of this invention are:

[0089] This invention extracts product selling point units and binds them to product image areas and video frame spatial image blocks to form a product selling point unit set, an image selling point evidence area set, and a video selling point image block sequence. This establishes a stable correspondence between text selling points, image evidence, and video footage, improving the consistency and traceability of multimodal material organization.

[0090] This invention constructs a selling point evidence connection diagram through a selling point evidence weaving operation, forming a unified topological connection relationship between the main connection node, static evidence nodes, and dynamic evidence nodes. The connection strength expresses the completeness of the product's selling point evidence, thereby reducing problems such as exaggerated text and image descriptions, inconsistent images, and lack of evidence support from video footage, and improving the stability of cross-modal connection of product selling points.

[0091] This invention improves the TimeSformer model by introducing a selling point temporal dwelling operator and an evidence continuation calibration layer. This enhances the continuous dwelling relationship of the same product's selling points in adjacent video frames and uses static image evidence to calibrate the video temporal features, generating selling point spatiotemporal continuation results in three directions: text, video, and subtitles. Furthermore, it uses selling point misalignment records to specifically correct misalignment of text evidence, video evidence, and subtitle continuation, resulting in a final text and video co-generation result with higher consistency, controllability, and content reliability. Attached Figure Description

[0092] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0093] Figure 1 This is an overall flowchart of a method for collaboratively generating multimodal images, text, and video for product detail pages, as proposed in this invention.

[0094] Figure 2 This is a schematic diagram of the improved TimeSformer model structure for a multimodal image and video collaborative generation method for product detail pages proposed in this invention;

[0095] Figure 3 This is a schematic diagram illustrating the frame-level segmentation and key selling point identification tagging of product video materials in a multimodal image and video collaborative generation method for product detail pages proposed in this invention. Detailed Implementation

[0096] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0097] refer to Figures 1-3 A method for collaboratively generating multimodal images, text, and video for product detail pages, comprising the following steps:

[0098] Step 1: Obtain product selling point units based on the basic product information, configure selling point identity tags for each product selling point unit, and generate a set of product selling point units;

[0099] Step 2: Identify the product image region corresponding to the product selling point unit based on the product image material, and bind the product image region with the selling point identification mark to generate an image selling point evidence region set;

[0100] Step 3: Perform frame-level block processing on the product video footage, and write the selling point identification tags into the spatial image blocks of the corresponding video frames to generate a video selling point image block sequence;

[0101] Step 4: Perform a selling point evidence weaving operation based on the product selling point unit set, the image selling point evidence area set, and the video selling point image block sequence to construct a selling point evidence connection diagram;

[0102] Step 5: Input the selling point evidence connection diagram into the improved TimeSformer model, introduce the selling point temporal residence operator in the temporal self-attention layer, and set an evidence connection calibration layer between the temporal self-attention layer and the spatial self-attention layer to generate the selling point spatiotemporal connection result;

[0103] Step Six: Based on the temporal and spatial context of the selling points, generate the detailed page image and text output and the detailed page video output;

[0104] Step 7: Based on the selling point evidence connection diagram, perform consistency verification on the generated text and image results and the generated video results of the details page, and generate a selling point misalignment record;

[0105] Step 8: Based on the record of misaligned selling points, correct the image and text generation results and video generation results on the details page, and generate a collaborative image and text and video generation result.

[0106] In this embodiment, step one specifically includes:

[0107] Obtain the basic product information from the product details page generation task, and extract the product name, product category, specification description, function description, and usage scenario description from the basic product information to generate the basic product description text;

[0108] The basic product description text is semantically segmented into short sentences to obtain a set of product description sentences;

[0109] The product description phrase set is filtered for consistency with the product subject based on the product name and product category. Product description phrases that do not point to the current product subject are deleted, and a set of valid product description phrases is generated.

[0110] Product selling point units are extracted from the set of effective product description phrases. These product selling point units include appearance selling point units, structural selling point units, functional selling point units, and scenario selling point units.

[0111] Configure a selling point identity tag for each product selling point unit according to product category, selling point type, and selling point order;

[0112] Summarize the product selling point units after configuring selling point identity tags to generate a product selling point unit set;

[0113] In the specific implementation process, the basic product description text is first uniformly converted to UTF-8 encoded text, and duplicate spaces, line breaks, emojis, and platform promotional words are removed. When segmenting semantic short sentences, commas, semicolons, periods, and line breaks are used as the first segmentation boundaries. Short sentences longer than 30 Chinese characters are further segmented into verb phrases, so that each short sentence corresponds to a product feature expression. When screening for consistency of product subjects, the product name and product category are used as subject words, and content in the product description short sentences that does not contain subject words, category synonyms, or product component words is removed. The selling point identity tag is generated by combining product category code, selling point type code, and three-digit sequence number. The three-digit sequence number increments from 001 according to the order of appearance of short sentences in the set of valid product description short sentences, so that image region binding and video image block writing can be performed according to the same tag, reducing cross-modal mismatch between different selling points.

[0114] In this embodiment, step two specifically includes:

[0115] Obtain product image materials from the product details page generation task, record the image number of each product image, and perform product body segmentation on each product image to obtain the product body area;

[0116] Extract the appearance outline area, local structure area, functional component area, and scene contact area within the main product area;

[0117] Based on the appearance selling points, structural selling points, functional selling points and scene selling points in the product selling points unit set, the candidate product image areas are obtained by matching them with the appearance outline area, local structure area, functional component area and scene contact area respectively.

[0118] The candidate product image regions are matched with the selling point descriptions of the corresponding product selling point units to determine the product image regions;

[0119] Read the selling point identification tag corresponding to the selling point unit of the product, and write the selling point identification tag into the area attribute field of the product image area;

[0120] The product image areas with the added selling point identification tags are summarized according to image number and area location to generate an image selling point evidence area set;

[0121] In the specific implementation process, product image materials are first screened for size uniformity and clarity, eliminating images that are severely blurry, lack a main subject, or have excessive repetition, ensuring that the images entering subsequent processing can stably reflect the product's appearance and usage status. The main product area is saved using segmented masks, and the image number, area coordinates, and area category are recorded simultaneously. The appearance outline area is determined based on the overall boundary of the main product, the local structure area is determined based on the structural changes and local details of the product's surface, the functional component area is determined based on the parts of the product that perform operation, connection, opening and closing, support, or display functions, and the scene contact area is determined based on the position where the product comes into contact with the usage environment or the user. Through the above area division method, different types of product selling points can be placed into corresponding visible areas, forming a stable binding relationship between selling point identification marks and image evidence, providing a reliable static evidence basis for video selling point image block sequences and selling point evidence connection diagrams.

[0122] In this embodiment, step three specifically includes:

[0123] Obtain the product video footage from the product details page generation task and record the video ID of each product video footage;

[0124] The product video material is extracted frame by frame according to the video frame order to generate a product video frame sequence, and a frame number is assigned to each video frame.

[0125] Each video frame in the product video frame sequence is divided into several non-overlapping spatial image blocks, and a block position marker is configured for each spatial image block.

[0126] Based on the selling point identification tags written in the image selling point evidence area, extract the regional visual features of the corresponding product image area;

[0127] The visual features of each spatial image block are matched with the visual features of the region to determine the video frame and spatial image block corresponding to the product selling point unit.

[0128] Read the selling point identification tag corresponding to the product selling point unit, and write the selling point identification tag into the spatial image block attribute field of the corresponding video frame;

[0129] The spatial image blocks, after being labeled with selling points, are sorted according to video number, frame number, and block position label to generate a video selling point image block sequence.

[0130] In the specific implementation process, the product video materials are first uniformly converted to MP4 format, and the video frame rate is uniformly set to 25 frames per second. When extracting frame by frame, consecutive frame numbers are generated according to the playback time sequence. Each video frame is uniformly scaled to 1024×1024 pixels and then spatially divided. The size of the spatial image block is set to 64×64 pixels. Each frame is divided into 256 non-overlapping spatial image blocks in the horizontal direction from left to right and in the vertical direction from top to bottom. The block position is marked by the pixel coordinates of the upper left corner and the lower right corner of the spatial image block, so that each spatial image block has a definite position range in the corresponding video frame.

[0131] In visual feature processing, the product image region and spatial image block are first converted to the HSV color space, and the pixel distribution of the H, S, and V channels is statistically analyzed to obtain color histogram features. Then, the product image region and spatial image block are grayscaled, and the horizontal and vertical gradients are extracted using the Sobel operator. The edge direction distribution is statistically analyzed based on the gradient direction to obtain edge direction features. Furthermore, the grayscale image is encoded using local binary mode, and the distribution of encoded values ​​is statistically analyzed to obtain local texture features. The color histogram features, edge direction features, and local texture features are concatenated in sequence to obtain the region visual features and image block visual features.

[0132] L2 normalization is performed on both the regional visual features and the image patch visual features. This involves calculating the square root of the sum of the squares of the values ​​in each dimension of the feature vector, and then dividing each dimension value by this square root to obtain a normalized feature vector of length 1. For cosine similarity calculation, the normalized image patch visual features and the regional visual features are multiplied item by item along their corresponding dimensions and summed. The resulting value is used as the cosine similarity between the two. Spatial image patches with a cosine similarity of at least 0.80 are identified as matching image patches. When the same spatial image patch corresponds to multiple selling point identification tags, the selling point identification tag with the highest cosine similarity is retained and written into the spatial image patch attribute field. Through this processing, the product image area in the video can form a stable correspondence with the selling point evidence area in the preceding image, allowing selling point identification tags to be continuously written sequentially along the video frames. This facilitates the identification of the location and duration of the same selling point in the video using selling point evidence weaving operations.

[0133] In this embodiment, step four specifically includes:

[0134] Read the product selling point units and selling point identity tags from the product selling point unit set, and set the product selling point unit as the receiving master node;

[0135] Read the image selling point evidence area from the image selling point evidence area set and set the image selling point evidence area as a static evidence node;

[0136] Read the spatial image blocks in the video's selling point image block sequence and set the spatial image blocks as dynamic evidence nodes;

[0137] By combining static and dynamic evidence nodes corresponding to the same selling point identity marker, a continuous evidence trajectory of the product selling point unit is generated between the image area and the video frame image block.

[0138] Breakpoint detection is performed on the continuous evidence trajectory between adjacent product selling point units. The main node that lacks static evidence nodes or dynamic evidence nodes is marked as a weak main node, and the main node that has both static and dynamic evidence nodes, and the dynamic evidence nodes are continuously distributed according to frame number, is marked as a strong main node.

[0139] Based on the strong receiving nodes, weak receiving nodes are connected to each other in a sequential manner, and the weak receiving nodes are connected to the strong receiving nodes that are closest to them in the order of the product selling point unit set.

[0140] The strength of the receiving main node is obtained by averaging and weighting the static evidence existence value, dynamic evidence existence value, dynamic evidence continuity value and adjacent traction value.

[0141] The static evidence existence value is 1 if the main node has a static evidence node, otherwise it is 0.

[0142] The value of dynamic evidence existence is 1 if the main node has a dynamic evidence node, otherwise it is 0.

[0143] The continuous value of dynamic evidence is the number of dynamic evidence nodes with the largest continuous distribution divided by the total number of dynamic evidence nodes under the corresponding main node.

[0144] The adjacency pull value is 1 when the primary node and the strong node are directly connected; otherwise, it is 0.

[0145] Based on the main node, static evidence node, dynamic evidence node, and the strength of the connection, generate a selling point evidence connection diagram;

[0146] In the specific implementation process, the selling point evidence connection diagram adopts a topological structure for storage. Each connection master node corresponds to a product selling point unit. Each static evidence node records the image number, region location, and selling point identification mark, while each dynamic evidence node records the video number, frame number, block location mark, and selling point identification mark. When performing evidence trajectory weaving, the static evidence nodes with the same selling point identification mark in the image selling point evidence area are first assigned to the corresponding connection master node using the selling point identification mark as an index. Then, the dynamic evidence nodes with the same selling point identification mark in the video selling point image block sequence are connected to the connection master node in frame number order. Subsequently, static evidence edges are established between the receiving master node and static evidence nodes, and dynamic evidence edges are established between the receiving master node and dynamic evidence nodes. Furthermore, time-series edges are established between dynamic evidence nodes under the same selling point identity marker, in ascending order of frame number, so that the same product selling point unit can simultaneously connect image area evidence and video frame image block evidence. For weak receiving nodes, traction edges are also established between the weak receiving node and the nearest strong receiving node, thereby forming a continuous evidence trajectory from the product selling point unit to image evidence and then to video evidence, so that selling point evidence from different sources maintains a traceable topological connection relationship in the same graph structure.

[0147] When calculating the strength of connection, the static evidence existence value, dynamic evidence existence value, dynamic evidence continuity value, and adjacent traction value are averaged and weighted, that is, the sum of the four values ​​is divided by 4 to obtain the connection strength of the main node. This connection strength reflects the completeness of the corresponding product selling point unit in image evidence, video evidence, and adjacent selling point connection relationships, enabling the model to distinguish between selling points with complete evidence and those with missing evidence. In this way, the selling point evidence connection diagram can uniformly constrain the cross-modal evidence of product selling points in the pre-generation stage, reducing the problems of exaggerated descriptions, inconsistent images, and misaligned subtitles caused by generating detail page content based solely on text selling points. At the same time, weak connection nodes are connected to strong connection nodes through traction edges, so that selling points with insufficient evidence can form traceable compensation relationships along adjacent strong evidence selling points, thereby improving the continuity, consistency, and verifiability between the image and text generation results and the video generation results of the detail page.

[0148] In this embodiment, the improved TimeSformer model specifically includes a graph node embedding layer, a temporal self-attention layer, an evidence acceptance calibration layer, a spatial self-attention layer, a feedforward mapping layer, and an acceptance output layer.

[0149] The graph node embedding layer reads the main node, static evidence node, dynamic evidence node, and strength of the selling point evidence connection graph. It maps the static evidence node, dynamic evidence node, and strength of the connection graph to static evidence features, dynamic image block features, and strength embedding features, respectively. Then, it concatenates the dynamic image block features with the strength embedding features to generate a selling point image block embedding sequence.

[0150] The temporal self-attention layer groups the embedded sequences of image blocks of selling points according to the receiving master node, performs inter-frame correlation processing on the embedded sequences of image blocks of selling points under the same receiving master node, and generates selling point temporal residence features through the selling point temporal residence operator.

[0151] The evidence receiving calibration layer reads the selling point time-series residency features and static evidence features, concatenates the selling point time-series residency features with the corresponding static evidence features, and generates evidence calibration gate values ​​through linear mapping and the Sigmoid function;

[0152] The evidence calibration gate value is used to gate and retain the time-series residence features of the selling points, and the residuals of the gated features are added to the static evidence features to generate the evidence calibration features;

[0153] The spatial self-attention layer performs intra-frame spatial correlation processing on the dynamic image block features under the same frame number in the evidence calibration features to obtain spatial attention features. The spatial attention features are then added to the evidence calibration features by residual addition and layer normalization to generate the selling point spatial correlation features.

[0154] The feedforward mapping layer sequentially performs linear mapping and GELU activation on the spatial correlation features of selling points to generate spatiotemporal fusion features of selling points.

[0155] The output layer pools the spatiotemporal fusion features of selling points under the same main node to generate acceptance discrimination features. These features are then input into the text and image acceptance output head, video acceptance output head, and subtitle acceptance output head, respectively. The Sigmoid function is used to generate text and image acceptance scores, video acceptance scores, and subtitle acceptance scores, respectively.

[0156] The scores for text and image integration, video integration, and subtitle integration will be used as the temporal and spatial integration results of the selling points;

[0157] In the specific implementation process, the graph node embedding layer processes the data in the selling point evidence connection graph in groups based on the receiving master node. One receiving master node corresponds to one product selling point unit. For dynamic evidence nodes under the same receiving master node, the visual features, frame number features, and block position marker features of their corresponding spatial image blocks are extracted and uniformly mapped to 256-dimensional dynamic image block features. For the connection strength under the same receiving master node, it is mapped to 64-dimensional strength embedding features. Then, each dynamic image block feature is concatenated with the strength embedding feature corresponding to the receiving master node to obtain 320-dimensional concatenated features, which are then linearly mapped to 256-dimensional embedding features. The resulting selling point image block embedding sequence is a sequence of embedding features corresponding to multiple spatial image blocks under the same product selling point unit, arranged by frame number and block position marker, rather than a single image block feature. This sequence simultaneously carries video image block content, video frame temporal sequence, spatial location, and selling point connection strength, providing input for the temporal self-attention layer oriented towards product selling points.

[0158] In the temporal self-attention layer, the embedded sequence of selling point image blocks is grouped according to the receiving master node for inter-frame association processing. Each group of sequences corresponds to only one product selling point unit. The selling point temporal dwelling operator enhances the association of image blocks on the continuous evidence trajectory within the group, generating a 256-dimensional selling point temporal dwelling feature. The evidence receiving calibration layer is a new structure added to the standard TimeSformer model in this invention. It is set between the temporal self-attention layer and the spatial self-attention layer and is used to introduce static image evidence to calibrate the video temporal features. The evidence receiving calibration layer concatenates the selling point temporal dwelling features with the 256-dimensional static evidence features under the same receiving master node to form a 512-dimensional concatenated feature. After being compressed to 256 dimensions by linear mapping, it is input into the Sigmoid function to obtain an evidence calibration gate value with a value range of 0 to 1. This evidence calibration gate value is applied to the selling point temporal dwelling features dimension by dimension, retaining the inter-frame video features consistent with the static evidence and weakening the video features lacking image evidence support. Then, the residual is added with the static evidence features to obtain the 256-dimensional evidence calibration feature.

[0159] The spatial self-attention layer adopts the intra-frame spatial self-attention processing method already present in the standard TimeSformer model. Specifically, the spatial self-attention layer uses the same frame number as the processing unit, and treats the evidence calibration features corresponding to different block position markers within the same video frame as spatial association objects, enabling adjacent or related local areas of products in the same frame to interact in terms of features. During the calculation process, the evidence calibration features are transformed according to the standard self-attention method and spatial attention weighted to obtain 256-dimensional spatial attention features. Subsequently, the spatial attention features are added to the evidence calibration features input to this layer by residual addition, and layer normalization is performed, so that the model can supplement the spatial relationship between product areas within the same frame while retaining the previous evidence calibration results, generating 256-dimensional selling point spatial association features. The feedforward mapping layer also adopts a conventional feedforward network structure, first linearly mapping the 256-dimensional selling point spatial association features to 512 dimensions, then linearly mapping them back to 256 dimensions after GELU activation, to obtain the spatiotemporal fusion features of selling points. This processing belongs to the conventional spatial modeling and feature mapping process in the TimeSformer model. Its role is to inherit the results of previous time dwell and evidence calibration, complete the spatial information fusion within the same video frame, and ensure that the output features simultaneously include the temporal continuity of selling points, static image evidence constraints, and intra-frame spatial correlation.

[0160] In the output layer, the image / text output head, video output head, and subtitle output head are all composed of a linear mapping layer and a Sigmoid function, but they act on different output directions. The output layer first performs average pooling on the spatiotemporal fusion features of selling points under the same main output node to obtain 256-dimensional output discriminant features. The image / text output head maps these discriminant features to an image / text output score, which indicates the reliability of the product's selling point unit as the basis for the image / text content of the details page. The video output head maps these discriminant features to a video output score, which indicates the reliability of the product's selling point unit as the basis for the video clips of the details page. The subtitle output head maps these discriminant features to a subtitle output score, which indicates the reliability of the product's selling point unit as the basis for the video subtitle content. All three scores are limited to between 0 and 1, with higher scores indicating more sufficient evidence acceptance in the corresponding output direction. This improved TimeSformer model outputs not just ordinary video classification features, but also spatiotemporal connections of selling points across three generation directions: text / images, video, and subtitles. This allows subsequent generation processes to select more stable selling point evidence based on different connections, reducing misalignment between text / image descriptions and video / subtitle content.

[0161] In this embodiment, the selling point timing residency operator is specifically as follows:

[0162] Based on the embedded sequence of selling point image blocks under the same main node, read the frame number and block position mark corresponding to the features of each dynamic image block;

[0163] Establish the temporal connection relationship between dynamic image block features according to the ascending order of frame number, and determine the dynamic image block features with adjacent block position markers in adjacent frame numbers as continuously resident image block features;

[0164] A resident mask matrix is ​​generated based on the temporal connection relationship and the features of consecutive resident image blocks. The matrix positions that belong to the same host node and constitute the features of consecutive resident image blocks are set as reserved values, while the matrix positions that do not belong to the same host node or do not constitute the features of consecutive resident image blocks are set as negative suppression values.

[0165] The sequence of embedded image blocks representing selling points is mapped into a query matrix, a key matrix, and a value matrix, respectively.

[0166] Multiply the query matrix by the transpose of the key matrix to obtain the initial temporal correlation score matrix between the features of each dynamic image block under the same host node. Then, scale the initial temporal correlation score matrix according to the feature dimension of the key matrix to obtain the temporal attention score matrix.

[0167] The residency mask matrix is ​​superimposed on the temporal attention score matrix to generate the selling point residency attention score matrix.

[0168] The selling point attention score matrix is ​​normalized using Softmax to obtain the selling point attention weight matrix;

[0169] Multiply the selling point retention attention weight matrix and the value matrix to generate the selling point time-series retention features;

[0170] In the specific implementation process, the embedded sequence of selling point image blocks under the same host node is arranged in ascending order of frame number. The feature dimension of each dynamic image block is set to 256 dimensions, and the feature dimensions of the query matrix, key matrix, and value matrix are all set to 256 dimensions. The block position mark is represented by the pixel coordinates of the upper left and lower right corners of the spatial image block. When two dynamic image blocks are located in adjacent frames, and the horizontal difference between their upper left and lower right corner pixel coordinates does not exceed 64 pixels and the vertical difference does not exceed 64 pixels, the corresponding block position marks are determined to be adjacent, and they are used as continuous resident image block features. The number of rows and columns of the resident mask matrix is ​​consistent with the number of dynamic image block features under the same host node. The retention value is set to 0, and the negative suppression value is set to -10000, so that the attention weight corresponding to the negative suppression position after Softmax normalization is close to 0. The initial temporal association score matrix is ​​divided by the square root of the feature dimension of the key matrix to avoid the attention score being too large due to the feature dimension being too high. Through the above processing, the selling point temporal dwell operator can prioritize retaining the image block associations of the same product selling point appearing continuously in adjacent video frames, reducing the interference of frame skipping, misaligned areas and other selling point image blocks on temporal attention calculation, so that the generated selling point temporal dwell features more stably reflect the continuous display process of product selling points in the video, thereby improving the accuracy of image-text continuity score, video continuity score and subtitle continuity score.

[0171] The improved TimeSformer model is based on the standard TimeSformer model, retaining its basic processing framework of video image block embedding, temporal self-attention, spatial self-attention, residual connections, layer normalization, and feedforward mapping. Compared to the standard TimeSformer model, the improved TimeSformer model introduces a selling point temporal dwelling operator into the temporal self-attention layer. This means that temporal attention is no longer calculated solely based on general visual similarity between video frames, but rather combines the host node, frame number, and block position markers to enhance the continuous dwelling relationship of the same product selling point in adjacent video frames. Simultaneously, an evidence-acceptance calibration layer is added between the temporal and spatial self-attention layers. This layer performs gating calibration on the video temporal features using static evidence features, ensuring that the image blocks of interest in the video are consistent with the static evidence in the product image material. This improvement transforms the model from a general video understanding model into a graphic-video collaborative generation model oriented towards product selling point evidence acceptance, reducing the risk of misalignment between video frames, graphic descriptions, and subtitle content.

[0172] The model training data comes from historical generated samples of product detail pages, including basic product information, product image materials, product video materials, manually reviewed and approved detail page text and image content, video clips, and subtitle text. During annotation, each product selling point unit is labeled with its corresponding image evidence region, video frame, spatial image block position, text-image transition result, video transition result, and subtitle transition result. The text-image transition result, video transition result, and subtitle transition result are labeled with 0 or 1, where 1 indicates that the selling point can be used as the basis for the corresponding generation direction, and 0 indicates that it cannot. During training, the text-image transition output head, video transition output head, and subtitle transition output head each use a binary cross-entropy loss, and the average of the three losses is used as the total loss. Training parameters are set as follows: batch size of 16, initial learning rate of 0.0001, 80 training epochs, AdamW optimizer, and weight decay set to 0.01. The model is considered converged when the total loss on the validation set decreases by less than 0.001 for 8 consecutive epochs. Through the above training methods, the model can learn the correspondence between product selling points and static image evidence, video dynamic evidence, and subtitle expression, thereby improving the consistency and controllability of the generated product details page content.

[0173] In this embodiment, step six specifically includes:

[0174] Read the image and text connection score, video connection score, and subtitle connection score from the selling point's temporal and spatial connection results;

[0175] Product selling point units with image and text connection scores greater than the set image and text connection threshold are identified as image and text generation selling point units, and the corresponding product selling point units and image selling point evidence areas are called to generate image and text selling point description segments.

[0176] The image and text description segments of the product's selling points are sorted according to the arrangement order of the product's selling point units, and the image and text generation results of the details page are generated in combination with the corresponding image selling point evidence area;

[0177] Product selling point units whose video acceptance score is greater than the set video acceptance threshold are identified as video generation selling point units, and the corresponding product video materials are called to generate video selling point clips.

[0178] Product selling point units with subtitle acceptance scores greater than the set subtitle acceptance threshold are identified as subtitle generation selling point units, and video subtitle text is generated based on the corresponding product selling point units.

[0179] Based on the frame numbering order corresponding to the video selling point segments, the video subtitle text is superimposed onto the corresponding video selling point segments to generate the video for the details page;

[0180] In the specific implementation process, the thresholds for image / text integration, video integration, and subtitle integration are all set to 0.70 to uniformly control the generation basis of image / text content, video clips, and subtitle text. When generating image / text selling point description segments, each image / text selling point generation unit corresponds to a description of no more than 60 Chinese characters. The description content is directly derived from the selling point description in the product selling point unit and is bound to the image number and area position of the corresponding image selling point evidence area, so that the image / text content can point to a specific product image area. When generating image / text generation results for the details page, each image / text selling point description segment is grouped with the corresponding image selling point evidence area to avoid the text selling point from being separated from the image area. When generating video selling point clips, the product video material corresponding to the video selling point generation unit is used as the source, and the clip is extracted according to the consecutive frame number of the selling point in the video. The duration of a single video selling point clip is set to 2 to 6 seconds. When generating video subtitle text, each subtitle does not exceed 20 Chinese characters, and the display duration is consistent with the corresponding video selling point clip. Through the above processing, images, videos, and subtitles are generated based on their corresponding connection scores, ensuring that selling points with sufficient connection are prioritized in the details page content, and reducing issues such as exaggerated descriptions, inconsistent video footage, and misaligned subtitles caused by selling points with weak evidence.

[0181] In this embodiment, step seven specifically includes:

[0182] Read the main node, static evidence node, and dynamic evidence node in the selling point evidence connection diagram;

[0183] Extract the image and text description segments and corresponding image selling point evidence areas from the image and text generation results on the details page, and then look up the corresponding static evidence nodes based on the image selling point evidence areas;

[0184] Extract video selling point clips and video subtitle text from the video generation results on the details page, and look up the corresponding dynamic evidence nodes based on the frame number corresponding to the video selling point clips;

[0185] Verify the corresponding text and image selling point descriptions, static evidence nodes, video selling point clips, dynamic evidence nodes, and video subtitle texts under the same main node.

[0186] When the text and image description of selling points does not correspond to a static evidence node under the same main node, or the video selling point segment does not correspond to a dynamic evidence node under the same main node, or the video subtitle text is inconsistent with the product selling point unit corresponding to the text and image description of selling points, it is determined that there is a selling point misalignment.

[0187] Record the product's selling point unit, misalignment type, corresponding image number, and corresponding frame number where there is a misalignment of selling points, and generate a selling point misalignment record. Misalignment types include misalignment of text and image evidence, misalignment of video evidence, and misalignment of subtitle transitions.

[0188] In practice, the corresponding verification is based on the main node. For image-based selling point evidence areas, static evidence nodes are retrieved using image numbers and region locations. For video selling point segments, dynamic evidence nodes are retrieved using start and end frame numbers. Furthermore, misalignment of image and text evidence indicates that the image / text selling point description segment could not be retrieved to a static evidence node under the same main node; misalignment of video evidence indicates that the video selling point segment could not be retrieved to a dynamic evidence node under the same main node; and misalignment of subtitles indicates that the video subtitle text is inconsistent with the corresponding product selling point unit in the image / text selling point description segment. Through the above classification of misalignment types, the selling point misalignment record can clearly indicate whether the error occurred on the image / text side, the video side, or the subtitle side, facilitating targeted correction according to the misalignment type.

[0189] In this embodiment, step eight specifically includes:

[0190] Read the product selling point unit, misalignment type, corresponding image number, and corresponding frame number from the selling point misalignment record;

[0191] When the misalignment type is image and text evidence misalignment, the static evidence node under the same main node is retrieved according to the corresponding image number, and the image and text selling point description section in the image and text generation result of the details page is rebound to the corresponding image selling point evidence area.

[0192] When the misalignment type is video evidence misalignment, the dynamic evidence node under the same main node is retrieved according to the corresponding frame number, and the video selling point segment in the video generation result of the details page is replaced with the video frame segment where the corresponding dynamic evidence node is located.

[0193] When the misalignment type is subtitle continuation misalignment, the video subtitle text is regenerated based on the product selling point unit with misaligned selling points, and the regenerated video subtitle text is superimposed on the corresponding video selling point segment.

[0194] The revised results of generating text and images and the revised results of generating videos on the details page are summarized to generate a collaborative result of text, images and videos.

[0195] In the specific implementation process, the misalignment record of selling points is used as a correction index, the misalignment of text and image evidence corresponds to the correction of the binding relationship between text and image areas, the misalignment of video evidence corresponds to the correction of the source of video clips, and the misalignment of subtitles corresponds to the correction of subtitle text content. By processing them separately according to the misalignment type, it is possible to avoid the content fluctuation caused by overall regeneration and improve the consistency of the generated details page.

[0196] Example 1: To verify the feasibility of this invention in practice, it was applied to the product detail page generation scenario of a small home appliance brand's blender. This brand needs to publish a 1.5L glass blender on an e-commerce platform. It already has one set of basic product information, 12 product images, and 3 product video clips, with a total video length of 54 seconds. Previously, the detail page creation mainly involved operations staff manually selecting images, extracting video clips, and adding subtitles based on the product description. This often resulted in issues such as the text emphasizing "high borosilicate glass body," but the video showing the touch panel, or the subtitle stating "gradations are clearly visible," but the image focusing on the base knob area, leading to misalignment between the text / image content, video footage, and subtitles on the detail page.

[0197] refer to Figure 3 As shown, in this scenario, the appearance selling point unit, structural selling point unit, functional selling point unit, and scene selling point unit are first extracted from the basic product information, and a selling point identity tag is configured for each product selling point unit; then, the product image material is segmented into the main body of the product to obtain the glass cup area, scale area, touch panel area, cup lid structure area, and fruit and vegetable stirring scene area, forming an image selling point evidence area set. Figure 3 The left side shows the frame-by-frame extraction process of the product video footage. The video footage is split into consecutive video frames according to the playback order, and each video frame is assigned a frame number. The middle section shows the normalization and block division process after selecting representative frames. The representative frames are uniformly processed into 1024×1024 pixels and divided into 256 non-overlapping spatial image blocks of 64×64 pixels. The position range of each spatial image block is determined by the pixel coordinates of the upper left corner and the lower right corner. The right side shows the process of extracting matching spatial image blocks from the video frames and writing them into selling point identification tags. Different block positions correspond to local areas of the product such as the glass body, capacity scale, and touch panel. After successful matching, the corresponding selling point identification tag is written, and the blocks are arranged in the scanning order of their positions to form a video selling point image block sequence.

[0198] In practical applications, product video materials are processed, unifying the video to MP4 format, setting the frame rate to 25 frames per second, and normalizing the representative frame to 1024×1024 pixels. Each frame is divided into 256 spatial image blocks of 64×64 pixels, with the block position marked using the pixel coordinates of the top left and bottom right corners. The system matches the regional visual features of the image selling point evidence area with the visual features of the image blocks in the video spatial image blocks, identifying spatial image blocks with a cosine similarity of not less than 0.80 as matching image blocks and writing corresponding selling point identification tags to form a video selling point image block sequence. Subsequently, a selling point evidence connection diagram is constructed through selling point evidence weaving operations and input into an improved TimeSformer model to obtain image-text connection scores, video connection scores, and subtitle connection scores; product selling point units with scores greater than 0.70 participate in image, text, video, and subtitle generation.

[0199] To verify the practical effect of this invention, two comparative schemes were set up. Comparative scheme one was a manual template generation scheme, where operators manually selected images, video clips, and subtitle content. Comparative scheme two was a standard TimeSformer scheme, where product video materials were divided into spatial image blocks and input into a standard TimeSformer model for conventional temporal and spatial self-attention processing. However, this scheme did not perform the selling point evidence weaving operation, did not construct a selling point evidence connection graph, and did not set a selling point temporal residency operator or evidence connection calibration layer. The standard TimeSformer scheme could only select video clips based on the visual features of video frames and could not uniformly constrain product selling points, image evidence areas, video image blocks, and subtitle content to the same connection master node. The present invention's scheme includes both selling point evidence weaving operation and an improved TimeSformer model, enabling the establishment of cross-modal evidence connection relationships before generation and consistency verification and correction through selling point misalignment records after generation. The test sample included 80 product selling points, 12 product images, and 3 product videos. The results are shown in Table 1 below.

[0200] Table 1. Comparison of the collaborative generation effects of text, images, and videos on the details page using different schemes: Comparison Option 1 83.1% 78.4% 75.6% 17 times 44.8 minutes / page 10.7 times / page Comparison Option 2 84.7% 82.6% 78.9% 14 times 17.4 minutes / page 8.1 times / page Invention Solution 94.6% 92.1% 90.8% 4 times 11.5 minutes / page 2.6 times / page

[0201] As can be seen from the data in Table 1 above, Comparison Scheme 1 relies on manual or template-based methods for the arrangement of text, images, videos, and subtitles, lacking unified constraints on product selling points, image evidence, and video clips. Therefore, the accuracy rate of matching text selling points is 83.1%, the accuracy rate of matching video clips is 78.4%, the consistency rate of subtitles and images is only 75.6%, the number of selling point misalignments reaches 17, and the number of manual modifications reaches 10.7 times per page. Although Comparison Scheme 2 improves the efficiency of video clip selection through standard video spatiotemporal feature modeling, reducing the average generation time from 44.8 minutes / page to 17.4 minutes / page, it still cannot guarantee the consistency of text descriptions and video images because it does not perform selling point evidence weaving operations or establish a selling point evidence connection diagram. Since both text and subtitle content correspond to the same product selling point, the consistency rate of subtitles and images only improved to 78.9%, and the number of selling point misalignments was still 14. The solution of this invention establishes a cross-modal evidence connection relationship through selling point evidence weaving operation, and uses an improved TimeSformer model to generate connection scores in three directions: text, video, and subtitles. This allows product selling points with sufficient evidence to be prioritized in the generated results. The accuracy rate of text and image selling point matching is improved to 94.6%, the accuracy rate of video segment matching is improved to 92.1%, the consistency rate of subtitles and images is improved to 90.8%, the number of selling point misalignments is reduced to 4, and the number of manual modifications is reduced to 2.6 times / page. This shows that the present invention can significantly improve the consistency and generation efficiency of text, images, and videos in the collaborative generation of product detail pages.

[0202] This embodiment uses the generation of a food processor product details page as an application scenario, focusing on the collaborative generation of text, images, and videos related to basic product information, product images, and product videos. By extracting product selling point units, identifying product image regions, dividing video frames into spatial image blocks, and performing selling point evidence weaving operations, a unified connection can be established between textual selling points, image evidence, and video footage. An improved TimeSformer model is used to generate spatiotemporal connection results for selling points, ensuring that textual descriptions, video clips, and subtitles are all generated based on corresponding evidence. Finally, consistency checks and misalignment corrections reduce issues such as inconsistencies between text and images, video clip offsets, and subtitle misalignments, improving the stability and credibility of the generated product details page.

[0203] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for collaboratively generating multimodal images, text, and video for product detail pages, characterized in that, Includes the following steps: Step 1: Obtain product selling point units based on the basic product information, configure selling point identity tags for the product selling point units, and generate a set of product selling point units; Step 2: Identify the product image region corresponding to the product selling point unit based on the product image material, and bind the product image region with the selling point identification mark to generate an image selling point evidence region set; Step 3: Perform frame-level block processing on the product video material, and write the selling point identification mark into the spatial image block of the corresponding video frame to generate a video selling point image block sequence; Step 4: Perform a selling point evidence weaving operation based on the product selling point unit set, the image selling point evidence area set, and the video selling point image block sequence to construct a selling point evidence connection diagram; Step 5: Input the aforementioned selling point evidence connection diagram into the improved TimeSformer model, introduce the selling point temporal residence operator in the temporal self-attention layer, and set an evidence connection calibration layer between the temporal self-attention layer and the spatial self-attention layer to generate the selling point spatiotemporal connection result; Step Six: Based on the spatiotemporal connection results of the aforementioned selling points, generate the detailed page image and text generation results and the detailed page video generation results; Step 7: Based on the aforementioned selling point evidence connection diagram, perform consistency verification on the image and text generation results and the video generation results of the details page, and generate a selling point misalignment record; Step 8: Based on the misalignment record of the selling points, correct the image and text generation results and the video generation results of the details page, and generate a collaborative image and text and video generation result.

2. The method for collaborative generation of multimodal images, text, and video for a product details page according to claim 1, characterized in that, Step one specifically involves: Obtain the basic product information from the product details page generation task, and extract the product name, product category, specification description, function description, and usage scenario description from the basic product information to generate basic product description text; The basic description text of the product is semantically segmented into short sentences to obtain a set of product description sentences; The product description phrase set is filtered for consistency with the product subject based on the product name and product category. Product description phrases that do not point to the current product subject are deleted, and a set of valid product description phrases is generated. Product selling point units are extracted from the set of effective product description phrases. These product selling point units include appearance selling point units, structural selling point units, functional selling point units, and scenario selling point units. Configure a selling point identity tag for each product selling point unit according to product category, selling point type, and selling point order; The product selling point units with configured selling point identity tags are aggregated to generate a product selling point unit set.

3. The method for collaborative generation of multimodal images, text, and video for a product details page according to claim 1, characterized in that, Step two specifically involves: Obtain product image materials from the product details page generation task, record the image number of each product image, and perform product body segmentation on each product image to obtain the product body area; Extract the appearance outline area, local structure area, functional component area, and scene contact area within the main body area of ​​the product; Based on the appearance selling point units, structural selling point units, functional selling point units and scene selling point units in the product selling point unit set, the candidate product image areas are obtained by corresponding them to the appearance outline area, local structure area, functional component area and scene contact area respectively. The candidate product image regions are matched with the selling point descriptions of the corresponding product selling point units to determine the product image regions; Read the selling point identification tag corresponding to the product selling point unit, and write the selling point identification tag into the region attribute field of the product image area; The product image areas with the added selling point identification tags are summarized according to image number and area location to generate an image selling point evidence area set.

4. The method for collaborative generation of multimodal images, text, and video for a product details page according to claim 1, characterized in that, Step three specifically involves: Obtain the product video footage from the product details page generation task and record the video ID of each product video footage; The product video material is extracted frame by frame according to the video frame order to generate a product video frame sequence, and a frame number is assigned to each video frame. Each video frame in the product video frame sequence is divided into several non-overlapping spatial image blocks, and a block position marker is configured for each spatial image block. Based on the selling point identification tags written in the image selling point evidence area, extract the regional visual features of the corresponding product image area; The visual features of each spatial image block are matched with the visual features of the region to determine the video frame and spatial image block corresponding to the product selling point unit. Read the selling point identification tag corresponding to the product selling point unit, and write the selling point identification tag into the spatial image block attribute field of the corresponding video frame; The spatial image blocks, after being labeled with selling points, are sorted according to video number, frame number, and block position label to generate a video selling point image block sequence.

5. The method for collaborative generation of multimodal images, text, and video for a product details page according to claim 1, characterized in that, Step four specifically involves: Read the product selling point units and selling point identity tags from the product selling point unit set, and set the product selling point unit as the receiving master node; Read the image selling point evidence area from the image selling point evidence area set and set the image selling point evidence area as a static evidence node; Read spatial image blocks from the video selling point image block sequence and set the spatial image blocks as dynamic evidence nodes; By weaving together static and dynamic evidence nodes corresponding to the same selling point identity mark, a continuous evidence trajectory of the product selling point unit is generated between the image area and the video frame image block. Breakpoint detection is performed on the continuous evidence trajectory between adjacent product selling point units. The main node that lacks static evidence nodes or dynamic evidence nodes is marked as a weak main node, and the main node that has both static and dynamic evidence nodes, and the dynamic evidence nodes are continuously distributed according to frame number, is marked as a strong main node. Based on the strong receiving nodes, weak receiving nodes are connected to each other in a sequential manner, and the weak receiving nodes are connected to the strong receiving nodes that are closest to them in the order of the product selling point unit set. The strength of the receiving main node is obtained by averaging and weighting the static evidence existence value, dynamic evidence existence value, dynamic evidence continuity value and adjacent traction value. The static evidence existence value is 1 when the main node has a static evidence node, and 0 otherwise. The dynamic evidence existence value is 1 when the receiving master node has a dynamic evidence node, and 0 otherwise. The continuous value of dynamic evidence is the number of dynamic evidence nodes with the largest continuous distribution divided by the total number of dynamic evidence nodes under the corresponding main node. The adjacency traction value is 1 when the main node and the strong node are directly connected, and 0 otherwise. Based on the main node, static evidence nodes, dynamic evidence nodes, and the strength of the connection, a selling point evidence connection diagram is generated.

6. The method for collaborative generation of multimodal images, text, and video for a product details page according to claim 1, characterized in that, The improved TimeSformer model specifically includes a graph node embedding layer, a temporal self-attention layer, an evidence acceptance calibration layer, a spatial self-attention layer, a feedforward mapping layer, and an acceptance output layer. The graph node embedding layer reads the main node, static evidence node, dynamic evidence node and bearing strength in the selling point evidence bearing graph, maps the static evidence node, dynamic evidence node and bearing strength to static evidence features, dynamic image block features and strength embedding features respectively, and splices the dynamic image block features and strength embedding features to generate a selling point image block embedding sequence. The temporal self-attention layer groups the embedded sequences of selling point image blocks according to the receiving master node, performs inter-frame association processing on the embedded sequences of selling point image blocks under the same receiving master node, and generates selling point temporal residence features through the selling point temporal residence operator. The evidence receiving calibration layer reads the selling point time-series residency features and static evidence features, concatenates the selling point time-series residency features with the corresponding static evidence features, and generates evidence calibration gate values ​​through linear mapping and the Sigmoid function; The evidence calibration gate value is used to gate and retain the time-series residence features of the selling points, and the residuals of the gated features are added to the static evidence features to generate the evidence calibration features; The spatial self-attention layer performs intra-frame spatial correlation processing on the dynamic image block features under the same frame number in the evidence calibration features to obtain spatial attention features. The spatial attention features are then added to the evidence calibration features by residual addition and layer normalization to generate the selling point spatial correlation features. The feedforward mapping layer sequentially performs linear mapping and GELU activation processing on the spatial correlation features of selling points to generate spatiotemporal fusion features of selling points. The receiving output layer pools the spatiotemporal fusion features of selling points under the same receiving master node to generate receiving discrimination features. The receiving discrimination features are then input into the text and image receiving output head, the video receiving output head, and the subtitle receiving output head, respectively. The Sigmoid function is used to generate text and image receiving scores, video receiving scores, and subtitle receiving scores, respectively. The scores for text and image integration, video integration, and subtitle integration are used as the temporal and spatial integration results of the selling points.

7. The method for collaborative generation of multimodal images, text, and video for a product details page according to claim 6, characterized in that, The specific selling point time-series residency operator is as follows: Based on the embedded sequence of selling point image blocks under the same main node, read the frame number and block position mark corresponding to the features of each dynamic image block; Establish the temporal connection relationship between dynamic image block features according to the ascending order of frame number, and determine the dynamic image block features with adjacent block position markers in adjacent frame numbers as continuously resident image block features; A resident mask matrix is ​​generated based on the temporal connection relationship and the features of consecutive resident image blocks. The matrix positions that belong to the same host node and constitute the features of consecutive resident image blocks are set as reserved values, while the matrix positions that do not belong to the same host node or do not constitute the features of consecutive resident image blocks are set as negative suppression values. The sequence of embedded image blocks representing selling points is mapped into a query matrix, a key matrix, and a value matrix, respectively. Multiply the query matrix by the transpose of the key matrix to obtain the initial temporal correlation score matrix between the features of each dynamic image block under the same host node. Then, scale the initial temporal correlation score matrix according to the feature dimension of the key matrix to obtain the temporal attention score matrix. The residency mask matrix is ​​superimposed on the time attention score matrix to generate the selling point residency attention score matrix. The selling point retention attention score matrix is ​​subjected to Softmax normalization to obtain the selling point retention attention weight matrix; The selling point retention attention weight matrix and value matrix are multiplied together to generate the selling point time-series retention features.

8. The method for collaborative generation of multimodal images, text, and video for a product details page according to claim 1, characterized in that, Step six specifically involves: Read the image and text acceptance score, video acceptance score, and subtitle acceptance score from the spatiotemporal acceptance results of the selling points; Product selling point units with image and text connection scores greater than the set image and text connection threshold are identified as image and text generation selling point units, and the corresponding product selling point units and image selling point evidence areas are called to generate image and text selling point description segments. The image and text description segments of the product's selling points are sorted according to the arrangement order of the product's selling point units, and the image and text generation results of the details page are generated in combination with the corresponding image selling point evidence area; Product selling point units whose video acceptance score is greater than the set video acceptance threshold are identified as video generation selling point units, and the corresponding product video materials are called to generate video selling point clips. Product selling point units with subtitle acceptance scores greater than the set subtitle acceptance threshold are identified as subtitle generation selling point units, and video subtitle text is generated based on the corresponding product selling point units. Based on the frame numbering order corresponding to the video selling point segments, the video subtitle text is superimposed onto the corresponding video selling point segments to generate the video for the details page.

9. The method for collaborative generation of multimodal images, text, and video for a product details page according to claim 1, characterized in that, Step seven specifically involves: Read the main node, static evidence node, and dynamic evidence node in the aforementioned selling point evidence connection diagram; Extract the image and text description segments and corresponding image selling point evidence areas from the image and text generation results of the details page, and then look up the corresponding static evidence nodes based on the image selling point evidence areas; Extract video selling point segments and video subtitle text from the video generation results on the details page, and look up the corresponding dynamic evidence nodes based on the frame numbers corresponding to the video selling point segments; Verify the corresponding text and image selling point descriptions, static evidence nodes, video selling point clips, dynamic evidence nodes, and video subtitle texts under the same main node. When the text and image description of selling points does not correspond to a static evidence node under the same main node, or the video selling point segment does not correspond to a dynamic evidence node under the same main node, or the video subtitle text is inconsistent with the product selling point unit corresponding to the text and image description of selling points, it is determined that there is a selling point misalignment. Record the product's selling point unit, misalignment type, corresponding image number, and corresponding frame number where there is a misalignment, and generate a selling point misalignment record. The misalignment types include misalignment of textual and graphic evidence, misalignment of video evidence, and misalignment of subtitle transitions.

10. The method for collaborative generation of multimodal images, text, and video for a product details page according to claim 1, characterized in that, Step eight specifically involves: Read the product selling point unit, misalignment type, corresponding image number, and corresponding frame number from the selling point misalignment record; When the misalignment type is image and text evidence misalignment, the static evidence node under the same receiving main node is retrieved according to the corresponding image number, and the image and text selling point description section in the image and text generation result of the details page is rebound to the corresponding image selling point evidence area. When the misalignment type is video evidence misalignment, the dynamic evidence node under the same receiving main node is retrieved according to the corresponding frame number, and the video selling point segment in the video generation result of the details page is replaced with the video frame segment where the corresponding dynamic evidence node is located. When the misalignment type is subtitle continuation misalignment, the video subtitle text is regenerated based on the product selling point unit with selling point misalignment, and the regenerated video subtitle text is superimposed on the corresponding video selling point segment. The revised results of generating text and images and the revised results of generating videos on the details page are summarized to generate a collaborative result of text, images and videos.

Images