Model training method, visual copywriting generation method and model training platform

Abstract

The invention provides a model training method, a visual copywriting generation method and a model training platform, and the method comprises the steps: obtaining initial sample data which comprises visual content description of visual data and an initial copywriting related to the visual data; performing semantic rewriting on a target entity in the initial copywriting based on the visual content description through a semantic reconstruction model to obtain a reconstructed copywriting; through a topic extraction model, on the basis of the visual content description, copywriting topic information is extracted from the reconstructed copywriting, target sample data are obtained, and the target sample data comprise the visual content description, the reconstructed copywriting and the copywriting topic information; and training a natural language model based on the target sample data to obtain a copywriting generation model. The illusion problem can be solved from a training data source, so that the model quality is fundamentally improved, and the model can generate a natural copywriting conforming to user habits.

Description

Model training methods, visual text generation methods, and model training platforms Technical Field

[0001] This specification relates to the technical field of text generation, and in particular to a model training method, a visual text generation method, and a model training platform. Background Technology

[0002] With the rapid development of short video platforms, self-media creation, and digital marketing, the production and consumption of video content have experienced explosive growth. Against this backdrop, users' demands for video editing tools are no longer limited to basic editing functions; they further expect the ability to automatically generate high-quality accompanying scripts based on the video content to improve creation efficiency and optimize content dissemination.

[0003] Currently, generating corresponding text based on video content is mainly achieved through large language models. However, during the training process of large language models, there are often technical problems such as the original text containing a large number of proper nouns that cannot be obtained from the video description, and the original text often containing emotional expressions and background information that cannot be obtained from the visual features of the video. This leads to poor training results for the model and low matching degree of the text generated based on the video content. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a model training method. One or more embodiments of this specification also relate to a visual text generation method, a model training platform, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a model training method is provided, comprising:

[0006] Initial sample data is obtained, including a visual content description of the visual data and initial text related to the visual data. Using a semantic reconstruction model, target entities in the initial text are semantically rewritten based on the visual content description to obtain reconstructed text. Using a topic extraction model, text topic information is extracted from the reconstructed text based on the visual content description to obtain target sample data, including the visual content description, the reconstructed text, and the text topic information. Based on the target sample data, a natural language model is trained to obtain a text generation model.

[0007] According to a second aspect of the embodiments of this specification, a visual text generation method is provided, comprising:

[0008] Obtain a visual content description of the target visual data; generate target text based on the visual content description using a text generation model, wherein the text generation model is trained according to any of the model training methods described herein.

[0009] According to a third aspect of the embodiments of this specification, a model training platform is provided, including a database interface and a training module:

[0010] The database interface is used to obtain initial sample data from the database, wherein the initial sample data includes a visual content description of the visual data and initial text related to the visual data;

[0011] The training module is configured to: semantically rewrite the target entities in the initial copy based on the visual content description using a semantic reconstruction model to obtain reconstructed copy; extract copy topic information from the reconstructed copy based on the visual content description using a topic extraction model to obtain target sample data, wherein the target sample data includes the visual content description, the reconstructed copy, and the copy topic information; and train a natural language model based on the target sample data to obtain a copy generation model.

[0012] According to a fourth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0013] A memory and a processor; the memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above method.

[0014] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0015] According to a sixth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0016] In one embodiment of this specification, initial sample data is obtained, which includes a visual content description of the visual data and initial text related to the visual data, providing data support for obtaining target sample data subsequently. Through a semantic reconstruction model, based on the visual content description, the target entities in the initial text are semantically rewritten to obtain reconstructed text. This allows for the semantic rewriting of entities in the initial text that do not conform to the visual content description, ensuring the accuracy of the training data. Through a topic extraction model, based on the visual content description, the topic information of the text is extracted from the reconstructed text to obtain target sample data. The target sample data includes the visual content description, the reconstructed text, and the topic information of the text. This allows for the extraction of potential topic semantic information in the reconstructed text, providing data support for training the natural language model subsequently. Based on the target sample data, a natural language model is trained to obtain a text generation model, enabling the trained text generation model to have stronger content matching capabilities. By employing a dual-module collaborative mechanism of semantic reconstruction and topic extraction models, mismatched specific information is eliminated while retaining reasonable background information. This ensures that the training data is both accurate and rich, addressing the illusion problem at the training data source rather than requiring post-processing corrections during inference. This fundamentally improves model quality and avoids additional computational overhead during inference. Selective rewriting of the text based on visual content descriptions avoids simplistic full replacement, maintaining the information integrity and naturalness of the training data, improving model training effectiveness and generalization ability, and enhancing user experience. Attached Figure Description

[0017] Figure 1 is a flowchart of a model training method provided in one embodiment of this specification;

[0018] Figure 2 is a schematic diagram of a model training method framework provided in one embodiment of this specification;

[0019] Figure 3 is a flowchart of a visual text generation method provided in one embodiment of this specification;

[0020] Figure 4 is a schematic diagram illustrating an embodiment of this specification for generating target text based on visual content description;

[0021] Figure 5 is a schematic diagram of an embodiment of this specification providing a method for generating a visual content description based on target visual data;

[0022] Figure 6a is a schematic diagram of a target visual data selection interface provided in one embodiment of this specification;

[0023] Figure 6b is a schematic diagram of a video loading interface provided in one embodiment of this specification;

[0024] Figure 6c is a schematic diagram of an updated video loading interface provided in one embodiment of this specification;

[0025] Figure 6d is a schematic diagram of an updated video loading interface provided in one embodiment of this specification;

[0026] Figure 6e is a schematic diagram of a video browsing interface provided in one embodiment of this specification;

[0027] Figure 6f is a flowchart illustrating the application of the model training method provided in an embodiment of this specification in the text generation module of an intelligent editing system;

[0028] Figure 7 is a schematic diagram of the structure of a model training platform provided in one embodiment of this specification;

[0029] Figure 8 is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0030] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0031] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0032] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0033] Furthermore, it should be noted that the data involved in one or more embodiments of this specification (including but not limited to data used for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0034] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0035] Natural Language Models (NLPs): NLPs are computer models that can understand and generate human language. They learn language patterns from text data using statistical and machine learning methods. NLPs are trained on large-scale text and can handle tasks such as text generation, classification, and translation, generating coherent responses based on input text.

[0036] Large Language Models (LLMs) are a class of deep learning models trained on massive amounts of data that are capable of understanding and generating natural language. They typically contain hundreds of millions of parameters and are pre-trained on a wide range of texts, thus enabling them to handle a variety of natural language tasks.

[0037] Transformer model: A neural network architecture based on self-attention mechanism. It abandons the structure of traditional recurrent neural networks and convolutional neural networks, can be trained in parallel, and significantly improves the performance of processing long sequence information. It is the core of modern natural language tasks.

[0038] Bidirectional Encoder Representations from Transformers (BERT model): A Transformer architecture model that allows bidirectional modeling of the context of each word, that is, considering information from both the left and right sides simultaneously, which greatly improves the performance of Transformer models in natural language understanding tasks.

[0039] Pre-trained Generative Transformer (GPT): A Transformer architecture model that uses an autoregressive decoder structure. It is pre-trained in unsupervised mode using a large amount of text data and can then be fine-tuned for specific tasks to generate coherent text output.

[0040] General industry semantic tags: General industry semantic tags are standardized semantic identifiers for common concepts or objects across different industries. They are used to uniformly describe common characteristics in different industries, facilitating model understanding and generalization. For example, "finance," "healthcare," and "education" can be used as general industry semantic tags to indicate the field to which video content belongs.

[0041] Spatial feature semantic description: Spatial feature semantic description is a textual representation of the scene, object layout, and spatial relationships in a video or image. It extracts the position, direction, distance, and other relationships of visual elements and transforms them into structured semantic information. For example, "the background is a mountain range" is an example of spatial feature semantic description.

[0042] Functional attribute semantic description: A functional attribute semantic description is a textual definition of the purpose, function, or characteristics of an object or scene in a video. It describes the object's function or attribute, rather than its appearance or spatial information. For example, "knife for cutting" or "folding chair" are examples of functional attribute semantic descriptions.

[0043] Role identity semantic identifiers: Role identity semantic identifiers are textual annotations of the social roles, professions, or identities of people or entities in a video, used to distinguish the responsibilities or identity characteristics of different subjects. For example, "doctor," "teacher," and "driver" are role identity semantic identifiers.

[0044] Currently, video content analysis and text generation based on artificial intelligence technology has become a hot research area in the industry. In certain application scenarios, it is necessary to build a dedicated large language model for text generation. This model takes video descriptions as input and outputs corresponding text content. Building a dedicated large language model for video text generation requires supervised fine-tuning training using a large amount of real user data. However, there are instances where the videos and texts in the platform's user data do not match. Directly using this data for training will cause the model to learn incorrect mapping relationships, resulting in numerous illusion problems.

[0045] Current mainstream solutions primarily rely on multimodal extensions of large language models (such as Generative Pre-trained Transformer (GPT) and Bidirectional Encoder Representations from Transformers (BERT)). By combining video visual features (such as object recognition, scene classification, and action detection) with text generation capabilities, they achieve end-to-end output from video to text. However, this technical approach faces the following key challenges in practical applications:

[0046] 1. Training data mismatch: The text in the original user data often contains a large amount of proper noun information (brand names, place names, numerical values, specific events, etc.) that cannot be obtained from the corresponding video descriptions. This information exceeds the scope of visual content. If used directly for training, the training objective will contain additional information beyond the input information, affecting the model's learning paradigm.

[0047] 2. Illusory learning due to data quality: The model may incorrectly learn the ability to infer specific names from generalized descriptions, leading to illusions in real-world applications. For example, the training data may visually describe "a certain shopping mall," but the corresponding text may be "Shopping Mall A," which has a specific name.

[0048] 3. Over-association of background information: Text often contains rich background information and emotional expressions that cannot be obtained from purely visual descriptions. Direct training can lead to over-association in the model, adding non-existent background information during inference. How to construct a precise mapping relationship between video content and text generation has become a key technical bottleneck in improving the practicality of AI-assisted creative tools. Future research needs to seek breakthroughs in cross-modal understanding, knowledge injection, and controllable generation to achieve more accurate automatic generation of video text.

[0049] To address the aforementioned issues, this specification provides a model training method in one or more embodiments. This method involves using a semantic reconstruction model to semantically rewrite target entities in initial text based on visual content description, obtaining reconstructed text; using a topic extraction model to extract topic information from the reconstructed text based on visual content description, obtaining target sample data; and training a natural language model based on the target sample data to obtain a text generation model. This approach solves the illusion problem from the training data source, fundamentally improving model quality, avoiding additional computational overhead during inference, and enabling the trained model to possess stronger content matching capabilities. This ensures that the trained model can generate natural text that conforms to user habits, aligns with user preferences, and enhances the user experience.

[0050] This specification provides a model training method, and also relates to a visual text generation method, a model training platform, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail in the following embodiments.

[0051] Referring to Figure 1, which shows a flowchart of a model training method provided in one embodiment of this specification, the method includes the following specific steps:

[0052] Step 102: Obtain initial sample data, which includes visual content descriptions of the visual data and initial text related to the visual data.

[0053] The initial sample data is unprocessed sample data. This initial sample data can be obtained from open-source sample libraries, constructed from historical data in historical databases, or manually constructed; no limitation is made here. For example, user data collected from within the platform includes, but is not limited to: publicly available video files from the short video platform and their metadata (such as titles, tags, and upload times), user-generated video description text, and associated comment content.

[0054] Visual data refers to digital media files containing visual information, such as dynamic video files (e.g., Audio Video Interleaved (AVI) format), still image sequences (e.g., Joint Photographic Experts Group (JPEG) format, Portable Network Graphics (PNG) format).

[0055] Visual content descriptions are brief textual descriptions of image or video content. For example, they can be objective descriptive text generated by pre-trained visual language models (such as Contrastive Language–Image Pre-training (CLIP)) or manually annotated structured content summaries (containing the three elements of subject, action, and scene).

[0056] The initial copy is the original, unmodified copy related to visual data, such as titles or descriptions written by users when uploading videos, hashtags or mentions added by users on social media platforms, or background information added by video creators in the comments section.

[0057] Optionally, initial sample data can be obtained by writing web crawler scripts to scrape publicly available data from short video platforms. For example, the requests library can be used to send Hypertext Transfer Protocol (HTTP) requests, BeautifulSoup can be used to parse Hypertext Markup Language (HTML) pages, or Selenium can be used to simulate browser operations to obtain dynamically loaded content. This method can obtain rich data such as video titles, descriptions, tags, and comment content. Optionally, sample data can be manually constructed. For example, a video can be shot as visual data, the content in the video (such as people, animals, and the environment appearing in the video) can be described in language as visual content description, and a text can be written based on the content in the video as initial text. This specification does not limit the method for obtaining initial sample data.

[0058] For example, referring to Figure 2, Figure 2 shows a schematic diagram of a model training method framework provided in one embodiment of this specification. As shown in Figure 2, the visual content is described as "[0001, 0002] The image depicts a portrait in a museum-like building, with the name of artist A next to the portrait. [0003, 0004]...". The initial text is "[0001, 0002] Finally met artist A in person at an art museum! Smaller than I imagined, but with magical eyes. Even surrounded by a crowd, I could still see her mysterious smile. Suddenly I understand why she has been famous for 500 years. The power of art is truly healing. [0003, 0004]...".

[0059] In step 102, initial sample data is obtained, which includes visual content descriptions of visual data and initial text related to visual data. This provides a data foundation for subsequent semantic rewriting of target entities in the initial text based on semantic transformation rules to obtain reconstructed text.

[0060] Step 104: Using a semantic reconstruction model, based on visual content description, semantically rewrite the target entities in the initial copy to obtain the reconstructed copy.

[0061] The semantic reconstruction model is a pre-trained text generation model. For example, the text generation model can be a GPT series model (such as GPT-3, GPT-4), which generates text word by word through autoregression and excels at tasks such as creative writing and dialogue generation. The text generation model can also be a BERT model, suitable for text conversion tasks (such as translation and summarization). This specification does not restrict the type of text generation model. The semantic reconstruction model is built based on high-quality manually annotated corpus. The annotated corpus adopts a triple structure of <multi-segment visual content description, original text, reconstructed text>, where the reconstructed text is the result of precise semantic rewriting of proper noun entities beyond the perceptual range in the initial text based on the visual content description. It learns the complex mapping relationship from "visual content description + initial text" to "semantic reconstruction text," deeply mastering the intelligent recognition and semantic conversion mechanism of proper noun entities.

[0062] The target entities are proper noun entities in the initial copy that do not match the visual content description. For example, the visual content description is "[0001, 0002] The image depicts a portrait inside a museum-like building, with the name of artist A next to the portrait. [0003, 0004]...", while the initial copy is "[0001, 0002] Finally met artist A in person at an art gallery! Smaller than I imagined, but with magical eyes. Even surrounded by a crowd, I could still see her mysterious smile. Suddenly I understood why she's been famous for 500 years; the power of art is truly healing. [0003, 0004]...". Since the visual content description does not include "art gallery" and "500 years," the target entities in the initial copy are "art gallery" and "500 years."

[0063] For example, referring to Figure 2, the semantic reconstruction model is obtained by pre-training the text generation model using training data. The visual content description and the initial text are input into the semantic reconstruction model. Based on the visual content description, the semantic reconstruction model semantically rewrites the target entities in the initial text, resulting in the reconstructed text: "[0001, 0002]Finally met artist A here! Smaller than I imagined, but her eyes are magical. Even surrounded by crowds, I could still see her mysterious smile. Suddenly I understand why she's been so popular for so many years~ The power of art is truly healing. [0003, 0004]...". Compared to the initial text, the reconstructed text rewrites the target entity "a certain art museum" in the initial text as "here," and "500 years" as "so many years."

[0064] In step 104, the semantic reconstruction model is used to semantically rewrite the target entities in the initial copy based on the visual content description to obtain the reconstructed copy, which provides a data foundation for extracting the copy theme information from the reconstructed copy through the topic extraction model.

[0065] Step 106: Using a topic extraction model, extract the topic information of the reconstructed copy from the visual content description to obtain target sample data, which includes visual content description, reconstructed copy, and topic information of the copy.

[0066] The topic extraction model is a pre-trained text generation model. The types of text generation models have been introduced above and will not be repeated here. The topic information extraction model is built on a carefully designed, manually annotated corpus. The annotated corpus adopts a structured format of <multi-segment visual content description, reconstructed text, and text topic information>.

[0067] The thematic information in the text is background knowledge that can be extracted from visual descriptions but requires in-depth understanding, manually extracted from the text. It is structured information with semantic hierarchy extracted from the text, covering multiple semantic dimensions such as emotional attitude semantics, event background semantics, temporal flow semantics, and experiential semantics. Annotation experts strictly adhere to the dual standards of semantic rationality and natural expression in the precise extraction and semantic description of thematic information.

[0068] The target sample data is used to train the natural language model. Each set of target sample data can include visual content descriptions, reconstructed text, and text theme information from the visual data.

[0069] For example, referring to Figure 2, the visual content description and reconstructed copywriting input theme extraction model yields the following copywriting theme information: "Level 1 Theme (Detailed): The excitement of finally seeing artist A's original work! Although the painting is smaller than I imagined, I was awestruck by her mysterious gaze. A pilgrimage to see that healing smile even in the crowd." "Level 2 Theme (Medium): Visiting artist A's original work, being awestruck by her mysterious gaze and smile, and feeling the healing power of art." "Level 3 Theme (Brief):"

[0070] "Checking in at the original artwork of artist A, I was awestruck by her mysterious and captivating art experience."

[0071] In step 106, the topic extraction model extracts the topic information of the copywriting from the reconstructed copywriting based on the visual content description, and obtains target sample data, which provides a data foundation for subsequent training of the natural language model and obtaining the copywriting generation model.

[0072] Step 108: Based on the target sample data, train a natural language model to obtain a copywriting generation model.

[0073] Natural language models are mathematical models used to process, understand, and generate human language. Their core objective is to capture the statistical regularities, grammatical structures, and semantic information of language through probabilistic or deep learning techniques. Natural language models can be large language models, such as the GPT series of models.

[0074] The copy generation model is an optimization of the natural language model, specifically designed to generate text content that conforms to a specific style and scenario. It is a model that generates corresponding copy based on the input visual content description. For example, the visual content description is "documenting an adventure to City C, featuring snow-capped mountains, temples, and local cultural experiences," and the generated copy reads: "This is a journey of courage and devotion. Crossing snow-capped mountains at an altitude of 5000 meters, the whistling of prayer flags in the cold wind seems like whispers from the gods. The golden roof of Palace X gleams in the sunlight, while figures prostrating themselves before Temple E measure the power of faith with their bodies. Following the camera, we enter the warm tent of a L ethnic family, where the aroma of butter tea mingles with the sweetness of barley cakes, listening to villagers tell stories of circumambulating the mountain. Beneath the white walls of Temple B, we unexpectedly encounter the smile of a red-robed lama; at that moment, the noisy world suddenly falls silent. As the drone sweeps across the snow line of Mountain D, and the sound of prayer wheels turning blends with chanting, you will discover—the magic of City C lies not only in its breathtaking scenery, but also in its ability to make you confront the insignificance and vastness of life."

[0075] Alternatively, one method for training a natural language model based on target sample data to obtain a copywriting generation model is as follows:

[0076] Using a natural language model, predictive copy is generated based on the topic information and visual content description of the copy; based on the predicted copy and the reconstructed copy, the copy loss value is calculated; based on the copy loss value, the natural language model is trained to obtain the copy generation model.

[0077] For example, the input visual content is described as "a vast field of purple flowers," and the text theme is "tranquility and healing in the flower field." A natural language model (such as a GPT-structured model) learns to generate predicted text based on these two conditions. The generated predicted text is compared with the actual reconstructed text, "In this boundless lavender field...", and a text loss value is calculated. Based on this loss value, the error is backpropagated using optimization algorithms such as gradient descent to update the parameters of the natural language model, enabling it to generate higher-quality text that more closely resembles the reconstructed text in the next iteration. After iterative training with a large number of samples, a high-performance text generation model is finally obtained.

[0078] In step 108, a natural language model is trained based on the target sample data to obtain a copy generation model. This enables the trained copy generation model to have stronger content matching capabilities, generate natural copy that conforms to user habits, align with user preferences, and improve user experience.

[0079] In the embodiments of this specification, a semantic reconstruction model is used to semantically rewrite target entities in the initial text based on visual content description, resulting in reconstructed text. This allows for the semantic rewriting of entities in the initial text that do not conform to the visual content description, ensuring the accuracy of the training data. A topic extraction model is used to extract text topic information from the reconstructed text based on visual content description, obtaining target sample data. This target sample data includes visual content description, reconstructed text, and text topic information, enabling the retrieval of potential topic semantic information within the reconstructed text, providing data support for subsequent training of the natural language model. The collaborative mechanism of the semantic reconstruction model and the topic extraction model eliminates mismatched specific information while retaining reasonable background information, ensuring that the training data is both accurate and rich. This addresses the illusion problem from the training data source, rather than requiring post-processing corrections during inference, thereby fundamentally improving model quality and avoiding additional computational overhead during inference. Selective rewriting of the text based on visual content description avoids simplistic full replacement, maintaining the information integrity and naturalness of the training data, improving model training effectiveness and generalization ability, and enhancing user experience.

[0080] In one optional embodiment of this specification, step 104 includes the following specific steps:

[0081] The semantic reconstruction model identifies target entities based on the semantic relevance between multiple entities in the visual content description and the initial copy. The target entities are then semantically rewritten based on semantic transformation rules to obtain the reconstructed copy. The semantic transformation rules are learned by the semantic reconstruction model during the pre-training phase.

[0082] Multiple entities are named units with specific semantics, such as brand names (e.g., a car brand), geographical names (e.g., a city), and time expressions (e.g., a year, a month, a day).

[0083] Semantic relevance is an indicator of the semantic association between entities in the visual content description and entities in the initial copy. For example, the visual content description is "The image depicts a desert at sunrise, with a golden ridge in the desert." The initial copy is "At 7:10 AM, the golden Mount B stretches out in Desert A, a magnificent sight to behold." The entities in the visual content description are: "sunrise," "desert," and "golden ridge." The entities in the initial copy are: "7:10 AM," "Desert A," and "Mount B." Analyzing the semantic relevance of each entity in the initial copy with the entities in the visual content description reveals that "7:10 AM" and "sunrise" are both entities describing time, and their semantic relevance is high. "Desert" and "Desert A" are both entities describing space, and their semantic relevance is high. "Golden ridge" and "Mount B" are both entities describing mountain ranges, and their semantic relevance is high.

[0084] Optionally, the determination of the semantic association index between each entity in the visual content description and each entity in the initial copy can be achieved by: determining it based on a dictionary and knowledge base (calculating the semantic similarity of entities through synsets and hyponymy / hypernymy), determining it through a context-related model (e.g., BERT / GPT, using the Transformer architecture to generate dynamic word vectors, where the same word has different vectors in different contexts), constructing a word relationship graph through a graph neural network (nodes = words, edges = co-occurrence or grammatical relationships), and aggregating neighbor information through graph convolution, for example, "marathon" and "long-distance running" are directly connected in the sports knowledge graph and have a high degree of association.

[0085] The semantic transformation rules are pre-trained entity rewriting strategies. For example, brand entity rewriting: rewriting "a specific model of mobile phone of brand A" as "this high-end smartphone", and rewriting "a specific model of sports shoes of brand A" as "a classic sports shoe of a well-known sports brand"; geographical entity rewriting: rewriting "a famous iron tower in city A" as "this iconic metal building", and rewriting "a specific commercial street in city B" as "a bustling commercial center"; time entity rewriting: rewriting "winter of 2023" as "a cold season of a certain year", and rewriting "a specific period of revival" as "that prosperous era".

[0086] For example, the visual content description is "The image depicts a desert at sunrise, with a golden mountain ridge in the middle of the desert." The initial copy is "At 7:10 in the morning, the dawn breaks, and the sunlight, like a blanket, covers Desert A. The golden mountain B, as if just awakened, stretches its body in Desert A. We are awestruck by this magnificent scene." The entities "7:10 in the morning," "Desert A," and "Mount B" in the initial copy can be found in the visual content description with high semantic relevance: "sunrise," "desert," and "golden mountain ridge," respectively. Therefore, by rewriting the entities in the initial copy based on semantic transformation rules, the reconstructed copy becomes: "One early morning, the dawn breaks, and the sunlight, like a blanket, covers a vast desert. The golden mountain, as if just awakened, stretches its body in this vast desert. We are awestruck by this magnificent scene."

[0087] In the embodiments of this specification, a semantic reconstruction model is used to determine target entities based on the semantic relevance between multiple entities in the visual content description and the initial copy. The target entities are then semantically rewritten based on semantic transformation rules to obtain reconstructed copy. This model can identify entities with high semantic relevance between the entities in the initial copy and the entities in the visual content description. Information in the initial copy that does not match the visual content description with the aforementioned entities with high semantic relevance is rewritten. This preserves the reasonable background information contained in the initial copy, ensuring that the training data is both accurate and rich. This provides a data foundation for the subsequent topic extraction model to extract accurate copy topic information and for the copy generation model to generate copy that matches the visual content description.

[0088] In one optional embodiment of this specification, the semantic transformation rule includes at least one of the following:

[0089] Rewrite brand entities as general industry semantic tags; rewrite geographical entities as spatial feature semantic descriptions; rewrite time entities as time feature semantic descriptions; rewrite product model entities as functional attribute semantic representations; rewrite personal name entities as role identity semantic identifiers.

[0090] Brand entities are proper nouns that serve as commercial identifiers, such as "Brand A" for watches, "Brand B" for cars, and "Brand C" for mobile phones. General industry semantic tags are neutral descriptions of product categories, such as "famous coffee chain brands," "electric vehicle manufacturers," and "luxury leather goods brands." Geographic entities are proper nouns with spatial location characteristics, such as "National Park A," "Shopping Mall B," and "Opera House C." Spatial feature semantic descriptions are visual representations of geographic entities, such as "this volcanic landscape scenic area," "a bustling commercial district," and "a cultural landmark by the sea." Temporal entities are proper nouns with temporal characteristics, such as "20XX," "the Nth hour," and "the Nth day." Temporal feature semantic descriptions are relative representations of temporal entities, such as "the most recent football tournament," "a prosperous dynasty in ancient times," and "a stage of mechanized production revolution." Product model entities are identification codes for specific products, such as "processor 1XX" and "camera EXX." Functional attribute semantic representations are abstract descriptions of product performance, such as "high-performance computer processor," "professional full-frame camera," and "wide-body twin-aisle passenger aircraft." Personal name entities are the names of specific individuals, such as "Zhang San" or "Li Mou". Role identity semantic identifiers are descriptions of a person's social attributes, such as "a genius artist of a certain period", "a pop music diva", or "a well-known technology entrepreneur".

[0091] In the embodiments described in this specification, by constructing different semantic transformation rules corresponding to various entities, the training of the semantic reconstruction model can be guided by semantic transformation rules. This allows the model to learn during training to semantically rewrite entities in the initial text that cannot be visually represented, autonomously identify proper noun entities in the text that exceed the boundaries of visual content, and perform intelligent reconstruction, thereby eliminating the illusion of entity nouns unrelated to the visual content description. Furthermore, based on deep training on a large-scale manually annotated corpus, the model learns a rich system of semantic transformation rules during training. The model possesses accurate semantic boundary judgment capabilities and natural text generation capabilities, maintaining the linguistic style characteristics and emotional semantic color of the original text while performing semantic reconstruction.

[0092] In one optional embodiment of this specification, the pre-training phase of the semantic reconstruction model includes:

[0093] Obtain first training data, wherein the first training data includes a first visual content description of the first visual data, a first initial text, and a first tag text, wherein the first tag text is obtained by semantically rewriting the first tag entity in the first initial text based on semantic transformation rules;

[0094] Using the first text generation model, based on the first visual content description, the first predicted entity in the first initial text is semantically rewritten to obtain the predicted reconstructed text; based on the first predicted entity and the first tag entity, as well as the predicted reconstructed text and the first tag text, the first text generation model is trained to obtain the semantic reconstruction model.

[0095] The first training data is the data used to train the first text generation model. Any set of the first training data may include the first visual content description, the first initial text, and the first tag text of the first visual data.

[0096] The definitions of first visual data, first visual content description, and first initial copy are the same as those in the above text, and will not be repeated here.

[0097] The first tag text is obtained by semantically rewriting the first tag entity in the first initial text based on semantic transformation rules. It is used to supervise the training process of the first text generation model and serves as the standard text after semantically rewriting the target entity in the first initial text based on semantic transformation rules.

[0098] The first text generation model is a pre-trained deep learning model with general semantic understanding and generation capabilities. The initial model includes, but is not limited to, large language models such as the Transformer, BERT, and GPT. The model parameters include, but are not limited to, model layers, weight matrices, bias terms, and positional encoding parameters. Before domain-specific training, this model has been pre-trained on a large-scale general corpus and possesses basic language processing capabilities. The predicted reconstructed text is obtained by the first text generation model semantically rewriting the first predicted entity in the first initial text based on the first visual content description.

[0099] For example, 10,000 sets of first user data are input into the first text generation model for training. Suppose that the first visual content description in one set of first user data is "The picture depicts a desert at sunrise, with a golden mountain ridge in the desert.", the first initial text is "At 7:10 in the morning, the dawn is breaking, and the sunlight covers the A desert like a blanket. The golden B mountain looks like it has just woken up, stretching its body in the A desert. I am amazed by this magnificent scene.", and the first tag text is "One morning, the dawn is breaking, and the sunlight covers a vast desert like a blanket. The golden B mountain looks like it has just woken up, stretching its body in this vast desert. I am amazed by this magnificent scene.". One set of user data was input into the first text generation model. The first text generation model rewrote the entities "7:10 AM", "Desert A", and "Mount B" in the initial text based on semantic transformation rules. After multiple iterations of training, the predicted reconstructed text was obtained as: "One morning, as dawn broke, the sunlight covered Desert A like a blanket. The famous golden mountain range, as if just awakened, stretched its body in this vast desert. I was awestruck by this magnificent scene." The first text generation model compared the predicted reconstructed text from this iteration with the first tag text. It can be seen that the first text generation model successfully rewrote "7:10 AM" as "one morning" and "Mount B" as "famous mountain range," but failed to rewrite "Desert A" as "in this vast desert." Therefore, the first text generation model continued to iterate and adjust its parameters until the loss value between the predicted reconstructed text and the first tag text was less than a preset threshold.

[0100] In the embodiments of this specification, the first visual content description is used as a constraint condition for the first text generation model to perform semantic rewriting on the first predicted entity in the first initial text during the training process. This allows the trained semantic reconstruction model to selectively rewrite the initial text when performing semantic rewriting based on semantic transformation rules, avoiding a simple and crude full replacement. This maintains the information integrity and naturalness of the training data, and improves the model training effect and generalization ability.

[0101] In one optional embodiment of this specification, a first text generation model is used to semantically rewrite the first predicted entity in the first initial text based on a first visual content description, thereby obtaining a predicted reconstructed text, including:

[0102] Using the first text generation model, based on the semantic relevance between multiple entities in the first visual content description and the first initial copy, the first predicted entity is determined, and the first predicted entity is semantically rewritten to obtain the predicted reconstructed copy.

[0103] Based on the first predicted entity and the first labeled entity, as well as the predicted reconstructed text and the first labeled text, a first text generation model is trained to obtain a semantic reconstruction model, including:

[0104] Based on the first predicted entity and the first labeled entity, determine the entity loss value; based on the predicted reconstructed text and the first labeled text, determine the text loss value; based on the entity loss value and the text loss value, determine the total loss value; based on the total loss value, train the first text generation model to obtain the semantic reconstruction model.

[0105] The semantic relevance between multiple entities refers to the degree of association between the meanings of multiple entities. It is used by the first text generation model to determine whether there are corresponding entities in the first visual content description for multiple entities in the first initial text. One possible approach is to calculate the semantic difference value between the i-th entity in the first initial text and each entity in the first visual content description. If the semantic difference value is less than a preset threshold, it is determined whether there is a corresponding entity in the first visual content description for the i-th entity in the first initial text.

[0106] The first predicted entity is an entity determined by the first text generation model based on the semantic relevance between multiple entities in the first visual content description and the first initial text. One optional approach is to determine entities in the first initial text whose semantic relevance to entities in the first visual content description is greater than a preset threshold as the first predicted entity. For example, if there are 5 entities in the first visual content description and 7 entities in the first initial text, and the semantic relevance of entities 1, 2, 4, 5, and 7 in the first initial text to the 5 entities in the first visual content description is greater than the preset threshold, then the number of first predicted entities is 5. Referring to the above example, another optional approach is: if the semantic relevance of entities 3 and 6 in the first initial text to the 5 entities in the first visual content description is less than the preset threshold, then these two entities do not appear in the first visual content description, and these entities not appearing in the first visual content description can also be determined as first predicted entities, resulting in a number of first predicted entities of 5 + 2 = 7. The first label entity is a reference entity for the first text generation model to semantically rewrite the first predicted entity, used for supervision during the semantic rewriting process.

[0107] The entity loss is the difference between the first predicted entity and the first labeled entity calculated by the first text generation model. It is used by the first text generation model to adjust its parameters during training. For example, the entity loss is the difference between the vector corresponding to the first predicted entity and the vector corresponding to the first labeled entity (e.g., cross-entropy loss).

[0108] The first-label text is a reference text generated by the first text generation model during training, used to supervise the training process of the first text generation model in generating the predicted reconstructed text.

[0109] The copywriting loss is the difference between the predicted reconstructed copy and the first-label copy, used by the first text generation model to adjust its parameters during training. For example, the copywriting loss is the difference between the vector corresponding to the predicted reconstructed copy and the vector corresponding to the first-label copy. The total loss is a weighted sum of the entity loss and the copywriting loss. For example, if the entity loss is 0.2 with a weight of 0.5, and the copywriting loss is 0.3 with a weight of 0.5, then the total loss is 0.2 × 0.5 + 0.3 × 0.5 = 0.25.

[0110] For example, the first visual content description, the first initial text, and the first tag text are input into the first text generation model. Based on the semantic relevance between multiple entities in the first visual content description and the first initial text, the first text generation model determines the first predicted entities in the first initial text as "7:10 a.m.", "Desert A", and "Mount B". The first predicted entities are semantically rewritten to obtain the predicted reconstructed text as "One morning, as dawn broke, the sunlight covered Desert A like a blanket. The famous golden mountain range, as if just awakened, stretched its body in this vast desert. I was awestruck by this magnificent scene." The first text generation model determines the entity loss value based on the first predicted entity and the first labeled entity, determines the text loss value based on the predicted reconstructed text and the first labeled text, determines the total loss value L1_total based on the entity loss value and the text loss value, and calculates the gradient based on the total loss value L1_total using the backpropagation algorithm. The parameters of the first text generation model are updated using an adaptive moment estimation optimizer. After each training cycle, the generalization performance index on the validation set is evaluated. When the generalization performance index reaches the preset training termination condition, the trained semantic reconstruction model is obtained.

[0111] In the embodiments described in this specification, a dual constraint mechanism is introduced. On the one hand, the entity loss value is determined using the first predicted entity and the first labeled entity to measure the accuracy of the first text generation model in semantically rewriting entities. On the other hand, the copy loss value between the reconstructed copy and the first labeled copy is calculated to measure the overall naturalness of the predicted reconstructed copy obtained by the first text generation model. This dual approach optimizes the training process of the first text generation model from both the overall (copy level) and local (entity level) perspectives, thereby achieving fine-tuning of the model's parameters. Furthermore, adjusting the model parameters through the total loss value further enhances training efficiency and effectiveness, enabling the final semantic reconstruction model to be applied more stably and efficiently to practical tasks.

[0112] In one optional embodiment of this specification, step 106 includes the following specific steps:

[0113] By using a topic extraction model, semantic aggregation is performed on the reconstructed copy to obtain the copy background. Based on the semantic relevance between the visual content description and the copy background, the copy topic information is extracted from the copy background to obtain the target sample data.

[0114] Semantic aggregation is the process of combining scattered semantic information into higher-level semantic units through association or summarization. It can improve the completeness and abstractness of semantic expression and reduce redundancy. For example, "running," "jumping," and "playing ball" can be aggregated into the high-level semantic word "sports."

[0115] The background information in the copywriting is the contextual information implied in the visual content that requires deep understanding to extract.

[0116] The semantic relevance between visual content description and text background is an indicator of the degree of correlation between the visual content description and text background in terms of textual meaning.

[0117] For example, suppose the restructured copy is: "In this boundless lavender field, it feels like being in summer in City A. A gentle breeze carries the faint fragrance of flowers, and all worries seem to be healed. I wish time could just stand still at this moment."

[0118] Semantic aggregation is performed through a topic extraction model: the model can identify key information points such as "lavender field", "city A", "summer", "breezy", "fragrance", "worries are cured", and "time stands still".

[0119] Obtaining the copywriting context: After aggregating these scattered semantic information, a higher-level copywriting context is obtained: "An experience of healing the mind and body and enjoying tranquil moments in a romantic sea of ​​flowers."

[0120] Calculate semantic relevance and extract themes: The visual content description might be: "A vast field of purple flowers, with mountains in the distance, and the flowers swaying gently in the breeze under a clear sky." Calculate the semantic relevance of this description to the copywriting background "body-mind healing and enjoying tranquil moments." Finally, extract the most core copywriting theme information from the copywriting background that is strongly related to the visual content, such as: "Tranquility and healing in a flower field."

[0121] In the embodiments of this specification, by performing semantic aggregation on the reconstructed text, the text background in the reconstructed text can be obtained, making the generated text theme information more consistent with the visual content description, thereby obtaining higher quality target sample data and providing a data foundation for the subsequent training of natural language models.

[0122] In one optional embodiment of this specification, semantic aggregation is performed on the reconstructed text to obtain the text background, including:

[0123] The reconstructed copy is semantically aggregated in at least one semantic dimension to obtain the copy background. The at least one semantic dimension includes the emotional attitude semantic dimension, the event background semantic dimension, the temporal movement semantic dimension, and the role experience semantic dimension.

[0124] The emotional attitude semantic dimension refers to the user's subjective emotional expression towards the video content, such as "shocking" or "healing".

[0125] The event background semantic dimension refers to contextual information that is not directly present in the rewritten copy but is implied within it, such as "an art pilgrimage." The temporal movement semantic dimension refers to the logical progression of actions and / or time implied in the rewritten copy, such as "first notice the eyes, then observe the smile." The role experience semantic dimension refers to the user's perceptual feedback contained in the rewritten copy, such as "smaller than expected" or "surrounded by a crowd."

[0126] For example, when the reconstructed copy reads, "Finally, I met artist A in person here! Smaller than I imagined, but her eyes are magical. Even surrounded by a crowd, I could still see her mysterious smile. Suddenly, I understand why she's been so popular for so many years; the power of art is truly healing.", semantic aggregation is performed on the reconstructed copy to obtain the following semantic dimensions: emotional attitude "healing," event background "visiting a museum with a portrait of artist A," chronological flow "noticing the eyes first, then observing the smile," and role experience "smaller than I imagined" and "surrounded by a crowd."

[0127] In the embodiments of this specification, by performing semantic aggregation on the reconstructed text in at least one semantic dimension, it is possible to obtain a text background containing information in multiple dimensions, making the generated text theme information more consistent with the visual content description, thereby obtaining higher quality target sample data, providing a data foundation for the subsequent training of the natural language model, enabling the text generation model obtained by the natural language model training to generate text that is more in line with the visual content description and user habits, aligning with user preferences, and improving user experience and acceptance.

[0128] In one optional embodiment of this specification, the pre-training phase of the topic extraction model includes:

[0129] Obtain second training data, which includes the second visual content description of the second visual data, the first reconstructed text and the tag text theme information. The tag text theme information is extracted from the first text background of the first reconstructed text based on the second visual content description.

[0130] Using a second text generation model, semantic aggregation is performed on the first reconstructed text based on the second visual content description to obtain the predicted text background. Based on the semantic relevance between the second visual content description and the predicted text background, information is extracted from the predicted text background to obtain the predicted text theme information. Based on the predicted text theme information and the tag text theme information, as well as the predicted text background and the first text background, the second text generation model is trained to obtain the theme extraction model.

[0131] The second training data is the data used to train the second text generation model. Any set of second training data includes the second visual content description of the second visual data, the first reconstructed text, and the tag text theme information.

[0132] The definitions of second visual data and second visual content description are the same as those of visual data and visual content description above, and will not be repeated here. The definition of first copywriting background is the same as that of copywriting background above, and will not be repeated here.

[0133] The first restructuring of the copy has been explained above and will not be repeated here.

[0134] The tag text theme information is extracted from the first text background of the first reconstructed text by the second text generation model based on the second visual content description. It is used to supervise the training process of the second text generation model and serves as reference text theme information for predicting the text theme information.

[0135] The second text generation model is of the same type as the first text generation model, and will not be described further here.

[0136] The predicted text background is generated by a second text generation model. Based on the second visual content description, it performs semantic aggregation on the first reconstructed text to obtain predicted text containing contextual information implicit in the first reconstructed text that requires deep understanding to extract. The predicted text topic information is also generated by the second text generation model, based on the second visual content description, and extracted from the first text background of the first reconstructed text.

[0137] For example, 10,000 sets of second user data are input into a second text generation model for training. Assume that the second visual content description in one set of second user data is "The image depicts a desert at sunrise, with a golden mountain ridge in the desert." The first reconstructed text is "One morning, as dawn breaks, the sunlight blankets a vast desert, and the golden, famous mountain range, as if just awakened, stretches its form in this vast desert. I am awestruck by this magnificent scene." The tag text theme information is "The morning sun shines on the desert, and I am awestruck by the mountains in the desert." One set of the aforementioned second user data is input into the second text generation model. Based on the second visual content description, the second text generation model performs semantic aggregation on the first reconstructed text to obtain the predicted text background. Based on the semantic relevance between the second visual content description and the predicted text background, information is extracted from the predicted text background. After multiple iterations of training, the predicted text theme information is obtained as "The morning sun shines on the desert, and I am awestruck by the mountains in the desert." The second text generation model compares the predicted text topic information with the tag text topic information in this iteration. It finds that the predicted text topic information obtained by the second text generation model is consistent with the tag text topic information, therefore the second text generation model stops training on this sample. The training of the second text generation model ends when the loss value between the predicted text topic information and the tag text topic information obtained by the second text generation model using each set of second user data is less than a preset threshold, and the loss value between the predicted text background and the first text background is less than a preset threshold.

[0138] In this embodiment, a second text generation model is used to semantically aggregate the first reconstructed text based on a second visual content description to obtain a predicted text background. Based on the semantic relevance between the second visual content description and the predicted text background, information is extracted from the predicted text background to obtain the predicted text theme information. Based on the predicted text theme information, the tagged text theme information, the predicted text background, and the first text background, the second text generation model is trained to obtain a theme extraction model. This model employs a fusion of multimodal semantic understanding and text generation to learn how to accurately identify and extract potential theme semantic information from the input. Through end-to-end training on large-scale labeled samples, the model has acquired the ability to analyze the correlation between visual content and deep semantics, mastered the inductive mechanism for semantic aggregation and theme extraction of visual information, and the ability to generate theme text that conforms to the user's natural expression paradigm.

[0139] In one optional embodiment of this specification, a second text generation model is trained based on predicted text topic information and tagged text topic information, as well as predicted text background and first text background, to obtain a topic extraction model, including:

[0140] Based on the predicted text background and the first text background, determine the text background loss value; based on the predicted text topic information and the tagged text topic information, determine the topic information loss value; based on the text background loss value and the topic information loss value, determine the total loss value; based on the total loss value, train the second text generation model to obtain the topic extraction model.

[0141] The definitions of predicted text background, first text background, predicted text topic information, and tag text topic information have been described above and will not be repeated here. The text background loss is the difference between the predicted text background calculated by the second text generation model and the first text background. It is used by the second text generation model to adjust its parameters during training. For example, the text background loss is the difference between the vector corresponding to the predicted text background and the vector corresponding to the first text background. The topic information loss is the difference between the predicted text topic information and the tag text topic information calculated by the second text generation model. It is used by the second text generation model to adjust its parameters during training. For example, the topic information loss is the difference between the vector corresponding to the predicted text topic information and the vector corresponding to the tag text topic information (e.g., cross-entropy loss).

[0142] The total loss value is the weighted sum of the copywriting background loss value and the topic information loss value. For example, if the copywriting background loss value is 0.1 and the weight of the copywriting background loss value is 0.5, and the topic information loss value is 0.2 and the weight of the topic information loss value is 0.5, then the total loss value is 0.1×0.5+0.2×0.5=0.15.

[0143] For example, in each iteration of training, the second text generation model calculates the text background loss value between the predicted text background and the first text background, and calculates the topic information loss value between the predicted text topic information and the tagged text topic information. Finally, the text background loss value and the topic information loss value are weighted and summed to obtain the total loss value L2_total. Based on the total loss value L2_total, the gradient is calculated through the backpropagation algorithm, and the parameters of the second text generation model are updated using the adaptive moment estimation optimizer. After each training cycle, the generalization performance index on the validation set is evaluated. When the generalization performance index reaches the preset training termination condition, the trained topic extraction model is obtained.

[0144] In this embodiment, on the one hand, the predicted text background and the first text background are used to determine the text background loss value, which can measure the consistency between the predicted text background and the first text background from multiple semantic dimensions. On the other hand, by calculating the topic information loss value between the predicted text topic information and the tag text topic information, the consistency of the predicted text topic information and the tag text topic information in summarizing the content in the visual data is measured. The training process of the second text generation model is optimized from both the consistency of text background and the consistency of text topic information, thereby achieving fine-tuning of the parameters of the second text generation model. Furthermore, adjusting the model parameters through the total loss value further improves training efficiency and effectiveness, enabling the final topic extraction model to be applied more stably and efficiently in practical tasks.

[0145] In one optional embodiment of this specification, the copywriting topic information includes multiple levels of copywriting subtopic information, and the semantic granularity between the multiple levels of copywriting subtopic information is different.

[0146] Semantic granularity refers to the level of detail in semantic description, and is divided into two categories: coarse-grained (general) and fine-grained (specific). It determines the level of detail in the model's output and affects the accuracy of the generated copy. For example, a coarse-grained description might be "animal," while a fine-grained description might be "a brown adult Labrador Retriever."

[0147] For example, referring to Figure 2, the copywriting topic information includes three levels of copywriting subtopic information, namely:

[0148] Level 1 Theme (Detailed): The excitement of finally seeing an original work by artist A. Although the painting was smaller than I imagined, I was awestruck by her mysterious gaze. A pilgrimage to art, where I had to make sure I could spot her healing smile even in the crowd. Level 2 Theme (Medium): Visiting an original work by artist A, being awestruck by her mysterious gaze and smile, and feeling the healing power of art. Level 3 Theme (Brief): Visiting an original work by artist A, a breathtaking art experience filled with her mysterious charm.

[0149] In the embodiments of this specification, by dividing the text theme information into multiple levels of text sub-theme information, and with different semantic granularities between the text sub-theme information at each level, the background information and emotional expression contained in the text can be extracted from different levels of detail, providing data support for the training of the subsequent text generation model, and avoiding the technical problem of the text generation model generating excessive association during training and adding non-existent background information and emotional expression during inference.

[0150] In one optional embodiment of this specification, step 108 includes the following specific steps:

[0151] Guided by the theme information of the copywriting, a predictive copywriting is generated based on the visual content description using a natural language model. Based on the predicted copywriting and the reconstructed copywriting, the copywriting loss value is calculated. Based on the copywriting loss value, the natural language model is trained to obtain the copywriting generation model.

[0152] The predicted text is the output of a natural language model, generated by taking the aforementioned text theme information as conditions and the aforementioned visual content description as input, and simulating the writing style and expression habits of the target user group. For example, suppose the text theme information is "the comfort and relaxation of outdoor camping," and the visual content description is "At sunset, on a tranquil lakeside, a campfire burns beside a tent, and a person sits in a folding chair looking at the lake." Guided by this theme information, the natural language model might generate the predicted text as: "The afterglow of the setting sun spills onto the lake, and the campfire beside the tent dispels the chill of the night. A chair, a moment of tranquility—this is what life should be like."

[0153] The copywriting loss value is a scalar numerical value used to quantify the difference between the predicted copy and the reconstructed copy, which serves as a supervision label. It is used to adjust model parameters during model training via backpropagation to minimize the gap between the prediction and the true value. For example, the reconstructed copy might be "The campfire illuminates the campers' smiling faces, the lake reflects the magnificent sunset, escaping the hustle and bustle of the city, feeling the tranquility of nature," while the predicted copy might be "The campers watch the campfire, the lake is bathed in sunset, it's very quiet." By calculating the differences between the two text sequences in terms of vocabulary distribution, semantic similarity, etc., a higher loss value can be obtained, indicating that the prediction effect is poor and the model parameters need significant adjustment. Optionally, the copywriting loss value can be calculated using either the cross-entropy loss function or the mean squared error loss function; there are no restrictions on the method used to calculate the copywriting loss value.

[0154] For example, the input visual content is described as "a vast field of purple flowers," and the textual theme is "tranquility and healing in the flower field." A natural language model (such as a GPT-structured model) learns to generate predicted text based on these two conditions. The generated predicted text is compared with the actual reconstructed text, "In this boundless lavender field...", and a text loss value is calculated. Based on this loss value, the error is backpropagated using optimization algorithms such as gradient descent to update the parameters of the natural language model, enabling it to generate higher-quality text that more closely resembles the reconstructed text in the next iteration. After iterative training with a large number of samples, a high-performance text generation model is finally obtained.

[0155] In the embodiments of this specification, during the natural language model training process, the natural language model is guided by the text topic information and constrained by the visual content description, so that the trained text generation model can generate text that is more in line with user habits and fits the visual content description.

[0156] Referring to Figure 3, which shows a flowchart of a visual text generation method provided in one embodiment of this specification, the method includes the following specific steps:

[0157] Step 302: Obtain the visual content description of the target visual data. Step 304: Generate the target text based on the visual content description using a text generation model, wherein the text generation model is trained according to the model training method.

[0158] The target visual data is any image or video data that is to be used to generate matching copy. It can be a user-uploaded personal video, content from the platform's media library, or a real-time captured image stream. For example, the target visual data could be a 15-second short video uploaded by a user, showing a Ragdoll cat playing with a ball of yarn on a sofa.

[0159] For example, referring to FIG4, FIG4 is a schematic diagram of generating target text based on visual content description according to an embodiment of this specification, as shown in FIG4:

[0160] After obtaining the target visual content description, "A long-haired white cat is playing with a red ball of yarn with its front paws...", this description is input into a pre-trained copywriting generation model. Based on the previously learned mapping relationship from "visual description" to "high-quality copywriting," the model generates the final target copy. The generated copy is: "The cat has discovered a new toy! Its focused look while playing with the yarn ball is so adorable, it's melting my heart~ #RagdollCat #CutePetsDailyLife #CatHappinessIsSimple"

[0161] In the embodiments of this specification, by acquiring the visual content description of the target visual data, and using a copywriting generation model to generate target copy based on the visual content description, it is possible to generate copy that is more in line with user habits and fits the visual content description, thereby improving user satisfaction.

[0162] In one optional embodiment of this specification, the visual content description of the target visual data is obtained, including:

[0163] Acquire target visual data; generate a visual content description of the target visual data based on the visual content of the target visual data using a visual content description model.

[0164] Referring, Figure 5 is an example of an embodiment of this specification illustrating the generation of a visual content description based on target visual data. As shown in Figure 5: First, a video file uploaded by a user about making latte art is received from a mobile device client as the target visual data (video). Then, the user clicks the "Generate visual content description based on target visual data" button (this step is optional), and the video is processed by a visual content description model (e.g., a video encoder-decoder model based on the Transformer architecture). The model analyzes the video content frame by frame or extracts keyframes, ultimately generating a comprehensive visual content description: "The video shows a person holding a milk pitcher pouring steamed milk into a cup of espresso, rhythmically shaking their wrist, eventually forming a white leaf-shaped pattern on the surface of the coffee."

[0165] In the embodiments of this specification, a visual content description model is used to generate a visual content description of the target visual data based on the visual content of the target visual data. This can improve the portability of generating visual content descriptions and enable users to generate text related to the target visual data simply by uploading the target visual data.

[0166] The following describes the application of the visual text generation method provided in the embodiments of this specification to the front-end interaction process of video generation. Please refer to Figure 6a. Figure 6a is a schematic diagram of a target visual data selection interface provided in an embodiment of this specification. As shown in Figure 6a, the target visual data selection interface can be displayed. The target visual data selection interface includes multiple media materials and a video generation control "One-Click Video Generation".

[0167] In response to the selection of media materials in the target visual data selection interface and the triggering of "one-click video creation", multiple target visual data in this embodiment can be obtained. Based on these multiple target visual data, the specific implementation process of the above-described visual text generation method can be executed to automatically generate corresponding visual content descriptions and target text. Taking Figure 6b as an example, Figure 6b is a schematic diagram of a video loading interface provided in an embodiment of this specification. The client can respond to the triggering operation of the video generation control "one-click video creation" in the target visual data selection interface to obtain the selected media materials. The server can generate a video inference process based on the selected media materials. The client can determine and display the video inference process 302a in the video loading interface. The inference process 302a includes at least one of the following inference information: at least one highlight segment 302b in the media material, at least one visual content description 302g of the media material, text theme information 302c, reconstructed text 302d, and target text 302e. The inference information is determined based on the selected at least one media material. The aforementioned reconstructed text 302d, target text 302e, and other text content are generated based on the visual text generation method provided in the embodiments of this specification.

[0168] Taking Figure 6b as an example, the reasoning process includes the target text 302e, namely, "An Unforgettable Trip, everyday narrative text, accompanied by relaxing music, and packaged in a simple, everyday style." Here, the video title is "An Unforgettable Trip"; the video text is "everyday narrative text," which can be understood as a video text type; the video background music is "accompanied by relaxing music," which can be understood as a video background music type; the video style is "simple, everyday style packaging," which can be understood as a video style type. The video voiceover can be understood as a video voiceover type, such as "funny voice."

[0169] In addition, the video loading interface also includes a command input field, through which update commands for at least one type of inference information are received. In response to the update command, the inference process updated based on the update command is displayed or dynamically displayed on the video loading interface. If the video inference process displayed on the video loading interface does not meet the video generation requirements, an update command can be input, and the updated inference process can then be displayed on the video loading interface. Continuing with Figure 6b as an example, the video loading interface includes a command input control 316a. Clicking the command input control 316a triggers the client to bring up the keyboard 318a in the video loading interface, as shown in Figure 6c. Figure 6c is a schematic diagram of an updated video loading interface provided in one embodiment of this specification. When the command input control 316a is clicked, the keyboard changes from a hidden state to a brought-up state. An update command can be input in the command input area 318b using the keyboard 318a. The input update command can be displayed in the command input area 318b. The associated location of the command input area 318b may also include an input confirmation control 318c. The client can respond to a trigger on the input confirmation control 318c to confirm the input update command and display the updated reasoning process in the video loading interface, as shown in Figure 6d. Figure 6d is a schematic diagram of an updated video loading interface provided in one embodiment of this specification.

[0170] Continuing with Figure 6b as an example, the video loading interface also includes a video viewing control 302f. The client can respond to a trigger operation on the video viewing control 302f to display the video browsing interface shown in Figure 6e. Figure 6e is a schematic diagram of a video browsing interface provided in one embodiment of this specification. This video browsing interface includes a video title, "An Unforgettable Trip," and a visual style, "Simple Style." As shown in Figure 6e, the video browsing interface also includes a video update control, "Regenerate." If the generated video is not satisfactory, or if the user wants to try generating another video, they can click the "Regenerate" video update control in the video browsing interface. The client can respond to the trigger operation on the "Regenerate" video update control to display a new video in the video browsing interface. The new video is different from the previously displayed video.

[0171] The following description, in conjunction with Figure 6f, uses the application of the model training method provided in this specification in the text generation module of an intelligent editing system as an example to further illustrate the model training method. Figure 6f shows a flowchart of the application of the model training method provided in this specification in the text generation module of an intelligent editing system, which may include the following specific steps:

[0172] Step 602: Obtain initial sample data, which includes visual content descriptions of the video data and initial text related to the video data. For example, initial sample data can be obtained from videos uploaded by users on video platforms and their corresponding text, such as a skiing video captioned "My first time trying snowboarding, although I fell a lot, it was really exciting!" The visual content description can be generated through a pre-trained visual-language model, such as generating "There are multiple skiers on the snow, one of whom is trying a snowboarding maneuver."

[0173] Step 604: Using the semantic reconstruction model, based on the semantic relevance between multiple entities in the visual content description and the initial copy, determine the target entity, and perform semantic rewriting on the target entity based on semantic transformation rules to obtain the reconstructed copy. The semantic transformation rules are learned by the semantic reconstruction model during the pre-training stage.

[0174] For example, semantic transformation rules learned by the semantic reconstruction model through pre-training may include: if the entity is a place noun and does not appear in the visual field, it is replaced with a generalized description (such as "a certain place"); if the entity is time information and is not visually relevant, it is deleted or replaced with "at this moment" or "this time"; if the entity is an emotional expression and has no visual support, it is adjusted to a neutral description.

[0175] Step 606: Using a topic extraction model, semantic aggregation is performed on the reconstructed text to obtain the text background. Based on the semantic relevance between the visual content description and the text background, the text theme information is extracted from the text background to obtain target sample data. For example, semantic aggregation is performed on the reconstructed text "Someone is skiing on the snow, their movements are a little clumsy but full of enthusiasm" to obtain the text background "The experience and challenges of skiing for beginners". Then, the semantic relevance between this text and the visual content description "snow, skier, skis" is calculated, and finally the text theme information "first skiing experience" is extracted.

[0176] Step 608: Using a natural language model, guided by the text's theme information, generate predicted copy based on visual content description. For example, the predicted copy could be: "Someone is skiing on the snow, their movements are a little clumsy but full of enthusiasm."

[0177] Step 610: Calculate the copy loss value based on the predicted copy and the reconstructed copy. For example, the copy loss value can be calculated using perplexity to assess the consistency between the predicted copy and the reconstructed copy.

[0178] Step 612: Based on the copywriting loss value, train a natural language model to obtain a copywriting generation model. For example, use gradient descent to iteratively update the parameters of the natural language model until the copywriting loss value converges, thus obtaining a trained copywriting generation model.

[0179] Step 614: Obtain the captured video data. For example, the video data captured by the user might be a short video of a family gathering, about 30 seconds long, showing the family making dumplings together.

[0180] Step 616: Using a visual content description model, generate a visual content description of the captured video data based on the visual content of the target visual data. For example, the visual content description model analyzes the video and outputs a visual content description: "Many people are sitting around in the kitchen, with dough and fillings on the table. They are making dumplings by hand, creating a warm atmosphere."

[0181] Step 618: Generate target text based on the visual content description using the text generation model, whereby the text generation model is trained according to the model training method described above. For example, the text generation model generates target text based on the above visual content description: "A family gathers together to make dumplings, enjoying a warm and harmonious family time."

[0182] In the embodiments of this specification, a semantic reconstruction model is used to semantically rewrite target entities in the initial text based on visual content description, resulting in reconstructed text. This allows for the semantic rewriting of entities in the initial text that do not conform to the visual content description, ensuring the accuracy of the training data. A topic extraction model is used to extract text topic information from the reconstructed text based on visual content description, obtaining target sample data. This target sample data includes visual content description, reconstructed text, and text topic information, enabling the retrieval of potential topic semantic information within the reconstructed text, providing data support for subsequent training of the natural language model. The collaborative mechanism of the semantic reconstruction model and the topic extraction model eliminates mismatched specific information while retaining reasonable background information, ensuring that the training data is both accurate and rich. This addresses the illusion problem from the training data source, rather than requiring post-processing corrections during inference, thereby fundamentally improving model quality and avoiding additional computational overhead during inference. Selective rewriting of the text based on visual content description avoids simplistic full replacement, maintaining the information integrity and naturalness of the training data, improving model training effectiveness and generalization ability, and enhancing user experience and acceptance.

[0183] Corresponding to the above method embodiments, this specification also provides a model training platform embodiment. Figure 7 shows a schematic diagram of the structure of a model training platform provided in one embodiment of this specification. As shown in Figure 7, the model training platform 700 includes a database interface 702 and a training module 704; the database interface 702 is used to obtain initial sample data from the database, wherein the initial sample data includes visual content descriptions of visual data and initial text related to the visual data;

[0184] Training module 704 is used to: semantically rewrite the target entities in the initial copy based on the visual content description using a semantic reconstruction model to obtain the reconstructed copy; extract the copy topic information from the reconstructed copy based on the visual content description using a topic extraction model to obtain target sample data, wherein the target sample data includes the visual content description, the reconstructed copy, and the copy topic information; and train a natural language model based on the target sample data to obtain a copy generation model.

[0185] Optionally, the training module 704 is specifically used to: determine the target entity based on the semantic relevance between multiple entities in the visual content description and the initial copy through the semantic reconstruction model, and perform semantic rewriting on the target entity based on semantic transformation rules to obtain the reconstructed copy, wherein the semantic transformation rules are learned by the semantic reconstruction model in the pre-training stage.

[0186] Optionally, the semantic transformation rules include at least one of the following: rewriting brand entities as general industry semantic tags; rewriting geographical entities as spatial feature semantic descriptions; rewriting time entities as time feature semantic descriptions; rewriting product model entities as functional attribute semantic representations; and rewriting personal name entities as role identity semantic identifiers.

[0187] Optionally, the training module 704 is specifically used for: acquiring first training data, wherein the first training data includes a first visual content description, a first initial text, and a first tag text of the first visual data, wherein the first tag text is obtained by semantically rewriting the first tag entity in the first initial text based on semantic transformation rules; using a first text generation model, based on the first visual content description, semantically rewriting the first predicted entity in the first initial text to obtain a predicted reconstructed text; and training the first text generation model based on the first predicted entity and the first tag entity, as well as the predicted reconstructed text and the first tag text, to obtain a semantic reconstruction model. Optionally, the training module 704 is specifically used for: determining a first predicted entity based on the semantic relevance between multiple entities in the first visual content description and the first initial copy through a first text generation model, and semantically rewriting the first predicted entity to obtain a predicted reconstructed copy; training the first text generation model based on the first predicted entity and the first labeled entity, as well as the predicted reconstructed copy and the first labeled copy, to obtain a semantic reconstruction model, including: determining an entity loss value based on the first predicted entity and the first labeled entity; determining a copy loss value based on the predicted reconstructed copy and the first labeled copy; determining a total loss value based on the entity loss value and the copy loss value; and training the first text generation model based on the total loss value to obtain a semantic reconstruction model.

[0188] Optionally, the training module 704 is specifically used to: perform semantic aggregation on the reconstructed text using a topic extraction model to obtain the text background, and extract text topic information from the text background based on the semantic relevance between the visual content description and the text background to obtain target sample data. Optionally, the training module 704 also includes a semantic aggregation unit, used to: perform semantic aggregation on the reconstructed text in at least one semantic dimension to obtain the text background, wherein the at least one semantic dimension includes an emotional attitude semantic dimension, an event background semantic dimension, a temporal movement semantic dimension, and a role experience semantic dimension.

[0189] Optionally, the training module 704 is specifically used for: acquiring second training data, wherein the second training data includes a second visual content description of the second visual data, a first reconstructed text, and tag text theme information, the tag text theme information being extracted from the first text background of the first reconstructed text based on the second visual content description; using a second text generation model, semantically aggregating the first reconstructed text based on the second visual content description to obtain a predicted text background, and extracting information from the predicted text background based on the semantic relevance between the second visual content description and the predicted text background to obtain predicted text theme information; and training the second text generation model based on the predicted text theme information, the tag text theme information, the predicted text background, and the first text background to obtain a theme extraction model.

[0190] Optionally, the training module 704 is specifically used for: determining the text background loss value based on the predicted text background and the first text background; determining the topic information loss value based on the predicted text topic information and the tagged text topic information; determining the total loss value based on the text background loss value and the topic information loss value; and training the second text generation model based on the total loss value to obtain the topic extraction model. Optionally, the text topic information includes multiple levels of text sub-topic information, and the semantic granularity between the multiple levels of text sub-topic information is different. Optionally, the training module 704 is specifically used for: generating predicted text based on visual content description through a natural language model under the guidance of the text topic information; calculating the text loss value based on the predicted text and the reconstructed text; and training the natural language model based on the text loss value to obtain the text generation model.

[0191] In the embodiments of this specification, the model training platform utilizes a semantic reconstruction model to semantically rewrite target entities in the initial text based on visual content descriptions, obtaining reconstructed text. This process semantically rewrites entities in the initial text that do not conform to the visual content descriptions, ensuring the accuracy of the training data. A topic extraction model, based on visual content descriptions, extracts text topic information from the reconstructed text, obtaining target sample data. This target sample data includes visual content descriptions, reconstructed text, and text topic information, revealing potential topic semantic information within the reconstructed text and providing data support for subsequent natural language model training. The collaborative mechanism of the semantic reconstruction model and the topic extraction model eliminates mismatched specific information while retaining reasonable background information, ensuring that the training data is both accurate and rich. This addresses the illusion problem from the training data source, rather than requiring post-processing corrections during inference, thus fundamentally improving model quality and avoiding additional computational overhead during inference. Selective rewriting of the text based on visual content descriptions avoids simplistic full replacement, maintaining the information integrity and naturalness of the training data, improving model training effectiveness and generalization ability, and enhancing user experience and acceptance.

[0192] The above is an illustrative scheme of a model training platform according to this embodiment. It should be noted that the technical solution of this model training platform and the technical solution of the model training method described above belong to the same concept. For details not described in detail in the technical solution of the model training platform, please refer to the description of the technical solution of the model training method described above.

[0193] Figure 8 shows a structural block diagram of a computing device according to an embodiment of this specification. The components of the computing device 800 include, but are not limited to, a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and a database 850 is used to store data.

[0194] The computing device 800 also includes an access device 840, which enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 840 may include one or more of any type of wired or wireless network interface (e.g., Network Interface Controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or Near Field Communication (NFC).

[0195] In one embodiment of this specification, the aforementioned components of the computing device 800, as well as other components not shown in FIG. 8, may be interconnected, for example, via a bus. It should be understood that the block diagram of the computing device shown in FIG. 8 is merely illustrative and not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0196] The computing device 800 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 800 can also be a mobile or stationary server.

[0197] The processor 820 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-described model training method.

[0198] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the model training method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the model training method described above.

[0199] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the model training method described above.

[0200] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solution of the model training method described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the model training method described above.

[0201] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described model training method.

[0202] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the above-described model training method belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the above-described model training method.

[0203] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0204] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content contained in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0205] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0206] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0207] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A model training method, characterized in that, include: Initial sample data is obtained, including a visual content description of the visual data and initial text related to the visual data. Using a semantic reconstruction model, target entities in the initial text are semantically rewritten based on the visual content description to obtain reconstructed text. Using a topic extraction model, text topic information is extracted from the reconstructed text based on the visual content description to obtain target sample data, including the visual content description, the reconstructed text, and the text topic information. Based on the target sample data, a natural language model is trained to obtain a text generation model.

2. The method according to claim 1, characterized in that, The step of semantically rewriting the target entity in the initial copy based on the visual content description using a semantic reconstruction model to obtain the reconstructed copy includes: determining the target entity based on the semantic relevance between the visual content description and multiple entities in the initial copy using a semantic reconstruction model, and performing semantic rewriting on the target entity based on semantic transformation rules to obtain the reconstructed copy, wherein the semantic transformation rules are learned by the semantic reconstruction model during the pre-training phase.

3. The method according to any one of claims 1-2, characterized in that, The pre-training phase of the semantic reconstruction model includes: acquiring first training data, wherein the first training data includes a first visual content description, a first initial text, and a first tag text of the first visual data, wherein the first tag text is obtained by semantically rewriting the first tag entity in the first initial text based on semantic transformation rules; using a first text generation model, based on the first visual content description, semantically rewriting the first predicted entity in the first initial text to obtain the predicted reconstructed text; and training the first text generation model based on the first predicted entity and the first tag entity, as well as the predicted reconstructed text and the first tag text, to obtain the semantic reconstruction model.

4. The method according to claim 3, characterized in that, The step of semantically rewriting the first predicted entity in the first initial text based on the first visual content description using a first text generation model to obtain a predicted reconstructed text includes: determining the first predicted entity based on the semantic relevance between multiple entities in the first visual content description and the first initial text using the first text generation model, and semantically rewriting the first predicted entity to obtain the predicted reconstructed text; the step of training the first text generation model based on the first predicted entity and the first tag entity, as well as the predicted reconstructed text and the first tag text, to obtain a semantic reconstruction model includes: determining the entity loss value based on the first predicted entity and the first tag entity; determining the text loss value based on the predicted reconstructed text and the first tag text; determining the total loss value based on the entity loss value and the text loss value; and training the first text generation model based on the total loss value to obtain the semantic reconstruction model.

5. The method according to claim 1, characterized in that, The step of extracting copy theme information from the reconstructed copy based on the visual content description using a topic extraction model to obtain target sample data includes: performing semantic aggregation on the reconstructed copy using a topic extraction model to obtain copy background, and extracting copy theme information from the copy background based on the semantic relevance between the visual content description and the copy background to obtain target sample data.

6. The method according to claim 5, characterized in that, The step of semantically aggregating the reconstructed text to obtain the text background includes: performing semantic aggregation on the reconstructed text in at least one semantic dimension to obtain the text background, wherein the at least one semantic dimension includes an emotional attitude semantic dimension, an event background semantic dimension, a temporal movement semantic dimension, and a role experience semantic dimension.

7. The method according to claim 1, 5, or 6, characterized in that, The pre-training phase of the topic extraction model includes: acquiring second training data, wherein the second training data includes a second visual content description of the second visual data, a first reconstructed text, and tag text topic information, wherein the tag text topic information is extracted from a first text background of the first reconstructed text based on the second visual content description; using a second text generation model, semantically aggregating the first reconstructed text based on the second visual content description to obtain a predicted text background, and extracting information from the predicted text background based on the semantic relevance between the second visual content description and the predicted text background to obtain predicted text topic information; and training the second text generation model based on the predicted text topic information, the tag text topic information, the predicted text background, and the first text background to obtain the topic extraction model.

8. The method according to claim 7, characterized in that, The step of training the second text generation model to obtain a topic extraction model based on the predicted text topic information, the tagged text topic information, the predicted text background, and the first text background includes: determining a text background loss value based on the predicted text background and the first text background; determining a topic information loss value based on the predicted text topic information and the tagged text topic information; determining a total loss value based on the text background loss value and the topic information loss value; and training the second text generation model to obtain a topic extraction model based on the total loss value.

9. The method according to claim 1, characterized in that, The copywriting theme information includes multiple levels of copywriting sub-theme information, and the semantic granularity of these multiple levels of copywriting sub-theme information is different.

10. The method according to claim 1, characterized in that, The step of training a natural language model based on the target sample data to obtain a copy generation model includes: generating predicted copy based on the visual content description using the natural language model under the guidance of the copy topic information; calculating a copy loss value based on the predicted copy and the reconstructed copy; and training the natural language model based on the copy loss value to obtain the copy generation model.

11. A method for generating visual text, characterized in that, include: Obtain a visual content description of the target visual data; The target text is generated based on the visual content description using a text generation model, wherein the text generation model is trained according to the method described in any one of claims 1-10.

12. The method according to claim 11, characterized in that, The process of obtaining the visual content description of the target visual data includes: obtaining the target visual data; and generating a visual content description of the target visual data based on the visual content of the target visual data using a visual content description model.

13. A model training platform, characterized in that, The system includes a database interface and a training module: the database interface is used to obtain initial sample data from a database, wherein the initial sample data includes a visual content description of the visual data and initial text related to the visual data; the training module is used to: semantically rewrite target entities in the initial text based on the visual content description using a semantic reconstruction model to obtain reconstructed text; extract text topic information from the reconstructed text based on the visual content description using a topic extraction model to obtain target sample data, wherein the target sample data includes the visual content description, the reconstructed text, and the text topic information; and train a natural language model based on the target sample data to obtain a text generation model.

14. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, It stores a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 12.

16. A computer program product, characterized in that, Includes a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 12.

Images