Method and system for generating travel short video description

Abstract

The invention discloses a method and a system for generating travel short video description. The method comprises the following steps: S1, collecting webpage information of user generated contents, wherein the webpage information of the user generated contents comprises scenic spot pictures and related characters; s2, collecting a short video, and extracting an image frame in the short video; s3, generating image-text pairs for the images and the image frames collected in the step S1 and the step S2 by using a knowledge graph; s4, generating and training a large language model; and S5, generating text description of the short video by using the big language model trained in the step S4 according to the image-text pairs generated in the step S3. By using the method and the device, the retrievability and the understandability of the tourism short video content can be effectively improved.

Description

Technical Field

[0001] The present invention relates to a method for generating a description of a tourism short video, and also relates to a system for generating a description of a tourism short video, belonging to the technical field of video annotation. Background Art

[0002] In the era of mobile Internet, short videos, as a form of user-generated content (UGC), have become an important carrier of information exchange. However, short videos usually only contain basic information such as titles and keywords, which makes video retrieval and content browsing inconvenient. In order to meet more sophisticated search needs, video content annotation technology based on artificial intelligence (AI) has become a key development direction to solve this problem. In addition, application scenarios such as video recommendation, video classification, and video tag generation also rely on such technology.

[0003] In the Chinese invention patent with patent number ZL 202310011629.5, a video integrated media content analysis and management system based on the Internet is disclosed. The system first uses the video extraction module for candidate materials to automatically obtain video materials from the integrated media platform, and extracts a set of key frame images through image processing techniques such as grayscale and binarization. Subsequently, the video content analysis module for candidate materials compares these images with the standard promotional images in the repository, calculates the content information conformity coefficient, and evaluates the consistency of the video content with the standard information for urban scenic spot promotion. At the same time, the system also analyzes the publicity benefit parameters of the video, such as the number of views, downloads, and forwardings, as well as the actual operating benefit parameters of urban scenic spots, such as traffic and per capita consumption, to determine the publicity value of the video and the popularity of the scenic spot. Finally, through the target city promotion video splicing module, the screened video materials are sorted and spliced ​​according to the benefit evaluation coefficient to generate a promotional video that is both in line with the characteristics of the urban scenic spot and attractive.

[0004] AI-based video content annotation technology can classify, label and annotate video data to provide training data for machine learning algorithms. This technology describes and accurately labels elements, scenes, actions, characters, objects, etc. in the video to improve the understanding and use of video content. The video annotation process includes video preprocessing, annotation task allocation, combination of manual annotation and automation, quality review and data output. In terms of technical implementation, video annotation relies on methods such as frame annotation, time sequence annotation, key frame annotation, semantic segmentation annotation and instance segmentation annotation to ensure the accuracy and consistency of annotation. Summary of the invention

[0005] The primary technical problem to be solved by the present invention is to provide a method for generating a description of a tourism short video.

[0006] Another technical problem to be solved by the present invention is to provide a system for generating descriptions of tourism short videos.

[0007] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0008] According to a first aspect of an embodiment of the present invention, a method for generating a description of a travel short video is provided, comprising the following steps:

[0009] S1: Collecting web page information of user-generated content, where the web page information of user-generated content includes pictures of tourist attractions and related texts;

[0010] S2: Collect short videos and extract image frames from them;

[0011] S3: Generate image-text pairs for the pictures and image frames collected in step S1 and step S2 using the knowledge graph;

[0012] S4: Generate and train a large language model;

[0013] S5: Using the large language model trained in step S4, a text description of the short video is generated according to the image-text pairs generated in step S3.

[0014] Preferably, in step S1, for each collected picture, K sentences before and after the picture are collected as corresponding texts, where K is a positive integer.

[0015] Preferably, step S3 includes the following sub-steps:

[0016] The knowledge graph is used to parse the text corresponding to the collected image, identify the named entities in the image by building a dependency tree, and extract keywords.

[0017] Preferably, the step S3 further includes the following sub-steps:

[0018] The named entity recognition keywords are stored as the attribute values ​​of the relationship, and the image and text information are encapsulated as image and text pairs.

[0019] Preferably, the image-text pair includes a collected image, text corresponding to the image, an image-text matching result, and a position of the text corresponding to the image in a paragraph.

[0020] Preferably, the image-text matching result is calculated through the following sub-steps:

[0021] S01: construct the context of the collected image;

[0022] S02: perform vector splicing;

[0023] S03: Associating the matched image with the text corresponding to the collected image.

[0024] Preferably, vector concatenation is performed through the following sub-steps:

[0025] S021: Collect corresponding context according to the image to be matched;

[0026] S022: Identify the category and content of the picture to be matched, and use the context of the picture to be matched to expand the dimension to form a first vector: [picture to be matched, context of the picture to be matched];

[0027] S023: forming a second vector according to the collected picture and the context of the collected picture: [collected picture, context of the collected picture];

[0028] S024: Setting a similarity threshold and calculating the cosine similarity between the first vector and the second vector as a picture-text matching result;

[0029] If the calculated similarity is greater than or equal to the similarity threshold, the image and text are considered to match, and the matching image is saved; if the calculated similarity is less than the similarity threshold, the image and text are considered to be unmatched, the image to be matched is eliminated, and the matching ends.

[0030] Preferably, in step S4, the large language model is trained by processing fine-tuning sample data, wherein the fine-tuning sample data includes image feature vectors, word vectors of text descriptions, and vectors after encoding matching information.

[0031] According to a second aspect of an embodiment of the present invention, there is provided a system for generating descriptions of tourism short videos, comprising a processor and a memory; wherein the memory is coupled to the processor and is used to store a computer program, and when the computer program is executed by the processor, the processor implements the above method.

[0032] Compared with the prior art, the present invention can effectively improve the retrievability and comprehensibility of tourism short video content by collecting user-generated tourist attraction pictures and related texts, extracting image frames from short videos, using knowledge graphs to generate image-text pairs, and using large language models such as BERT for training and generating descriptions. In addition, through precise image-text matching technology and detailed context analysis, the description generated by this method can more accurately reflect the video content, thereby improving the accuracy of video recommendations and user browsing experience. Test results show that compared with the prior art, the present invention has higher accuracy and coherence in generating descriptions of tourism short videos, and has significant technical effects and application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1A flow chart of a method for generating a description of a travel short video provided by an embodiment of the present invention;

[0034] Figure 2 A schematic diagram of a system for generating a description of a tourism short video provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The technical content of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] First embodiment

[0037] like Figure 1 As shown, a method for generating a travel short video description provided by the first embodiment of the present invention includes at least the following steps:

[0038] S1: Collecting web page information of user-generated content, where the web page information of user-generated content includes pictures of tourist attractions and related texts;

[0039] S2: Collect short videos and extract image frames from them;

[0040] S3: Generate image-text pairs for the pictures and image frames collected in step S1 and step S2 using the knowledge graph;

[0041] S4: Generate and train a large language model;

[0042] S5: Using the large language model trained in step S4, a text description of the short video is generated according to the image-text pairs generated in step S3.

[0043] In the process of collecting web page information for user-generated content (UGC), the embodiment of the present invention pays special attention to information containing photos of tourist attractions and their related texts. In order to collect and process this information, web page information of user-generated content is collected in step S1. Among them, web page information containing photos of tourist attractions and related texts is focused on. For each collected picture, it is recorded as P, and K sentences before and after the picture P are collected as the corresponding text, recorded as T, where K is a positive integer, preferably 2, that is, one sentence before and after the picture. In this way, a corresponding relationship between the picture P and its corresponding text T is formed.

[0044] In step S3, the knowledge graph is used to parse the text corresponding to the collected image. By building a dependency tree, the named entities in the image are identified and keywords are extracted. These keywords serve as the attribute values ​​of the relationship and provide a basis for further processing of the image and text information.

[0045] In an embodiment of the present invention, assume that we have collected a picture labeled as P. This picture is a photo of a tourist attraction, such as the Great Wall in China. At the same time, the text T related to this picture is collected, and the text content is as follows:

[0046] "The Great Wall is a symbol of China, and the Badaling Great Wall located on the outskirts of Beijing is the most famous section."

[0047] In sub-step S31, a knowledge graph will be used to parse the text T corresponding to the picture P. First, natural language processing (NLP) analysis needs to be performed on the text to construct a dependency tree to identify named entities in the text.

[0048] A dependency tree is a graph where nodes represent words in a sentence and edges represent the dependency relationships between words. In this example, the dependency tree analysis may produce the following results:

[0049] The Great Wall (nsubj: is a symbol of China)

[0050] China (attr: The Great Wall)

[0051] of (case: a symbol of China)

[0052] symbol (dobj: is China)

[0053] Located (ROOT: Located... Badaling Great Wall)

[0054] Beijing (nmod: Located on the outskirts of Beijing)

[0055] Outskirts (nmod: Located on the outskirts of Beijing)

[0056] Badaling Great Wall (dobj: Located... Badaling Great Wall)

[0057] is (cop: is the most famous section)

[0058] The most famous (amod: the most famous section)

[0059] section (dobj: is the most famous section)

[0060] Through dependency tree analysis, named entities in the text can be identified, such as "The Great Wall", "China", "Beijing", and "Badaling Great Wall". These named entities are keywords highly relevant to the picture content and can be used as nodes in the knowledge graph.

[0061] Next, these named entity keywords will be stored as attribute values of the relationship, and the picture-text information will be stored as a picture-text pair. In this example, the picture-text pairs include:

[0062] Picture P: A photo of the Great Wall

[0063] Text T: Sentences describing the Great Wall

[0064] Image-text matching results: the text description matches the image content

[0065] Position of text in paragraph: First sentence

[0066] Finally, all the image-text pairs are stored in the image-text library G. The image-text library G will contain multiple such image-text pairs, each of which is associated with the corresponding named entity keyword, forming a rich knowledge graph that can be used for subsequent image-text matching, information retrieval and other applications.

[0067] Through the above method, keywords related to the image content can be extracted from the text, and the knowledge graph can be used to enhance the semantic information of the image, thereby providing support for the generation of tourism Vlog short video descriptions.

[0068] In sub-step S32, the named entity recognition keywords are stored as attribute values ​​of the relationship, and the image-text information is encapsulated as an image-text pair. The image-text pair includes the collected picture P, the text T corresponding to the picture, the image-text matching result, and the position of the text corresponding to the picture in the paragraph. All image-text pairs are stored in an image-text library, which is denoted as G. It should be noted that the image frame extracted in step S2 is similar to the picture P, which will not be repeated here.

[0069] Through the above sub-steps, the tourist attraction pictures and related text information generated by users can be stored in a structured manner, which facilitates further analysis and application. This method not only improves the efficiency of information collection, but also provides rich data resources for the generation of tourism Vlog short video descriptions.

[0070] In one embodiment of the present invention, the image-text matching result is obtained through the following sub-steps.

[0071] S01: First, construct the context of the collected image P. The context of image P is denoted by C P , is composed of keywords in the text T corresponding to the image. These keywords are a set of keywords without duplicate words. For example, if the image P is a photo of the Great Wall, the corresponding text T may contain keywords such as "Great Wall", "China", "History", "Beijing", etc., then C P It is a collection of these keywords.

[0072] S02: Next, vector concatenation is performed. This step includes the following sub-steps:

[0073] First, in sub-step S021, according to the image M to be matched, the corresponding context C is collected. M. C M Including the short video title, the keywords described by the user, and the classification results of several pictures before and after the picture to be matched. In this example, suppose the picture M to be matched is another picture of the Great Wall, and its context C M It may include keywords such as "Great Wall", "tourism", "Beijing", and classification results of 15 pictures before and after.

[0074] In sub-step S022, the category and content of the image M to be matched are identified, and the context C of the image to be matched is used. M To expand the dimension, form a vector: [M, C M ], which is recorded as the first vector. For example, if the image M is identified as "The Great Wall", and C M Contains the keywords "Great Wall", "tourism", and "Beijing", so the first vector can be expressed as ["Great Wall", "Great Wall", "tourism", and "Beijing"].

[0075] In sub-step S023, according to the collected picture P and the context C of the collected picture P P , forming a vector: [P, C P ], which is recorded as the second vector. Continuing with the Great Wall example, if the context C of image P is P is ["Great Wall", "China", "History", "Beijing"], then the second vector can be expressed as ["Great Wall", "Great Wall", "China", "History", "Beijing"].

[0076] Finally, in sub-step S024, a similarity threshold (such as 0.5 or 0.6, etc.) is set, and the cosine similarity of the first vector and the second vector is calculated, and this cosine similarity is regarded as the image-text matching result. If the calculated similarity is greater than or equal to the preset similarity threshold, it is considered that the image M matches the image P, and the matched image M' is saved. If the similarity is less than the threshold, it is considered that there is no match, the image M to be matched is eliminated, and the matching process ends.

[0077] Through the above-mentioned vector concatenation process, the similarity between the image to be matched and the collected image can be systematically evaluated, thereby achieving efficient image-text matching.

[0078] In one embodiment of the present invention, the formula for calculating the cosine similarity between the first vector and the second vector is:

[0079]

[0080] Wherein, A is the first vector, B is the second vector, and n is a positive integer.

[0081] It should be noted that using the context in the above calculation process can effectively avoid too low a similarity score caused by too few keywords (low vector dimension).

[0082] S03: Match the image M ’ Associate the text corresponding to the collected image.

[0083] In step S2, a short video is collected and image frames in the short video are extracted. This step serves to generate image-text pairs using the knowledge graph (step S3). Specifically, after the short video is collected, a series of image frames representing the video content are extracted through some automated method. Then, the knowledge graph is used to analyze these image frames, identify key information in the image, such as objects, scenes, characters, etc., and extract text descriptions related to this information. In this way, each image frame and its corresponding text description form a picture-text pair.

[0084] Next, in the training model step S4 of the present invention, including the generation of large language models (LLMs) and the processing of fine-tuning sample data, specifically includes the following sub-steps:

[0085] Step S41: Generate Large Language Models (LLMs)

[0086] In this sub-step, large language models are generated based on the text descriptions. These models are able to understand and generate natural language text, providing a strong foundation for subsequent text generation tasks. Large language models usually require a large amount of data and complex neural network architectures to train in order to capture the complexity and diversity of language.

[0087] Step S42: Processing and fine-tuning sample data

[0088] Fine-tuning sample data is an important part of the large language model training process, which includes the following components:

[0089] Image feature vector: This is a feature vector extracted from an image using a graphics feature extraction algorithm. These feature vectors can represent the visual content of the image and provide basic information about the image to the large language model.

[0090] Word vector of text description: This is the vector obtained by segmenting the text T corresponding to the collected image. Word segmentation is the process of dividing text into meaningful words or phrases, while word vector converts these words or phrases into numerical representations so that large language models can process them.

[0091] Vector after matching information is encoded: This is the image-text matching result obtained in the previous step, which is encoded into a vector form so that the large language model can use this information to improve its prediction.

[0092] In one embodiment of the present invention, the preferred large language model is the BERT (Bidirectional Encoder Representations from Transformers) model. The BERT model is well known for its bidirectional training method, in which the output of each position is a prediction based on information outside the position. The input of the BERT model is usually divided into three parts: features of the video frame P, associated text T, and associated information (such as similarity, sentence position, etc.).

[0093] The word embedding part of the BERT model is the process of converting text into word vectors. You can use open source word vectors on the Internet, such as the word vector table of MetaAI's open source model L lama 2. Similarity information needs to be normalized, while position information needs to be effectively encoded. Partition embedding is used to distinguish the relationship between the three parts of the input, while position embedding follows the general rules of the large model, that is, the position order of the word vectors in the word embedding.

[0094] In the basic framework of the BERT model, the output Yi of the i-th position is a prediction based on three parts of information. The greater the probability that this output is correctly predicted, the smaller the corresponding loss. The output Yi of the i-th position involves two subtasks: one is a vector in the word space, corresponding to a symbol (Token) in the dictionary, and its distribution parameter is θ1; the other is the sentence-level output, which selects one from the [IsNext, NotNext] set to indicate whether a new sentence is generated, and its distribution parameter is θ2. These two subtasks are jointly learned and optimized through the cross-entropy loss function to achieve the best training effect.

[0095] The loss function for the joint learning of these two subtasks is as follows:

[0096]

[0097] Among them, m i ∈[1,2,....,|V|], V is the word vector space, |V| is the size of the word vector space. p(m=m i |θ,θ1) is the conditional probability of the ith output. i ∈[I sNext,NotNext]. p(n=n i |θ,θ2) is the conditional probability of whether the i-th position generates a new sentence.

[0098] Next, in step S5, the large language model trained in step S4 is used to generate a text description of the short video based on the image-text pairs generated in step S3. This means that the image-text pairs extracted from static images and dynamic videos will be integrated to form a detailed description of the entire video content. Such a description not only includes the visual information in the image and video frames, but also incorporates the related text information, making the generated text description richer and more comprehensive.

[0099] In one embodiment of the present invention, the Youku-mPLUG dataset is preferably used. The Youku-mPLUG dataset is a public Chinese dataset. The dataset is collected from the video sharing website Youku, and has the advantages of security, diversity, and high quality. The length of the video titles in this dataset is between 5 and 30 Chinese characters. There is less data for video description, only 186,081 pieces, and even less data for tourism, only 15,259 pieces (accounting for 8.2%). The corresponding tourism video subset is used as the test standard. The standard text of the dataset includes subtitles corresponding to the video and related background descriptions. The subtitles include the subject and object of the video and contain at least 15 Chinese characters.

[0100] The above steps S1 to S5 together complete the process of extracting image frames from short videos, generating image-text pairs through knowledge graphs, and finally generating text descriptions. This process enables the content of short videos to be recorded and expressed in text form, improving the searchability and comprehensibility of video content.

[0101] In order to verify the actual effect of the embodiment of the present invention, the inventor used the above-mentioned Chinese dataset Youku-mPLUG dataset for testing. Among them, assuming that the video is a foreign language (video-guided language), this task can be compared to translation work, so the three indicators BLEU-4, METEOR and ROUGE are used to measure the quality of the generated description. BLEU-4 focuses on the matching degree of the longest common 4-gram between the machine-generated text and the reference text; METEOR comprehensively considers the exact matching of words, synonym matching and word order penalties; ROUGE evaluates the proportion of the longest common subsequence between the machine-generated text and the reference text.

[0102] Table 1 Evaluation results of the present invention and mPLUG-2

[0103] BLEU-4 METEOR ROUGE mPLUG-2 47.4 27.9 53.2 The present invention 49.2 31.2 54.4

[0104] From the evaluation results shown in Table 1, the tourism short video description generated by the method proposed in the embodiment of the present invention reached 49.2 on the BLEU-4 index, 31.2 on the METEOR index, and 54.4 on the ROUGE index. In comparison, the baseline results of the mPLUG-2 test set (a public Chinese video description dataset used to evaluate the performance of video description generation models) were 47.4, 27.9, and 53.2, respectively. These evaluation results show that the method proposed in the embodiment of the present invention can more effectively capture the key information of the video content and generate more accurate and coherent descriptions in the automatic generation of tourism short video descriptions compared to the baseline model. The improvement of these evaluation indicators shows the effectiveness and superiority of this method in understanding and generating tourism short video content descriptions.

[0105] Second embodiment

[0106] Based on the above method, the second embodiment of the present invention provides a system for generating a description of a tourism short video. Figure 2 As shown, the system includes one or more processors 41 and a memory 42. The memory 42 is used to store one or more programs, and when the one or more programs are executed by the one or more processors 41, the method in the above embodiment is implemented.

[0107] Wherein, the processor is used to control the overall operation of the system to complete all or part of the steps of the above method. The processor can be a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a digital signal processing (DSP) chip, etc. The memory is used to store various types of data to support operations in the system, and these data may include, for example, instructions for any application or method for operating on the system, and application-related data. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, etc.

[0108] In an exemplary embodiment, the system can be implemented by a computer chip or entity, or by a product with a certain function, for executing the above method and achieving the same technical effect as the above method. Specifically, the computer can be, for example, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0109] In another exemplary embodiment, the present invention further provides a computer-readable storage medium including program instructions, which, when executed by a processor, implements the steps of the method in any of the above embodiments. For example, the computer-readable storage medium may be the above-mentioned memory including program instructions, which may be executed by a processor to complete the above-mentioned method and achieve the same technical effect as the above-mentioned method.

[0110] It should be noted that the above embodiments are only examples, and the technical solutions of the various embodiments can be combined, all within the protection scope of the present invention.

[0111] The above is a detailed description of the method and system for generating travel short video descriptions provided by the present invention. For those skilled in the art, any obvious changes made to it without departing from the essence of the present invention will constitute an infringement of the patent right of the present invention and will bear corresponding legal responsibilities.

Claims

1. A method for generating a description of a tourism short video, characterized in that The steps include: S1: Collecting web page information of user-generated content, where the web page information of user-generated content includes pictures of tourist attractions and related texts; S2: Collect short videos and extract image frames from them; S3: Generate image-text pairs for the pictures and image frames collected in step S1 and step S2 using the knowledge graph; S4: Generate and train a large language model; S5: Using the large language model trained in step S4, a text description of the short video is generated according to the image-text pairs generated in step S3.

2. The method according to claim 1, characterized in that In the step S1, for each collected picture, K sentences before and after the picture are collected as corresponding texts, where K is a positive integer.

3. The method according to claim 2, characterized in that The step S3 includes the following sub-steps: The knowledge graph is used to parse the text corresponding to the collected image, identify the named entities in the image by building a dependency tree, and extract keywords.

4. The method according to claim 3, characterized in that The step S3 further includes the following sub-steps: The named entity recognition keywords are stored as the attribute values ​​of the relationship, and the image and text information are encapsulated as image and text pairs.

5. The method according to claim 4, characterized in that: The image-text pair includes a collected image, text corresponding to the image, an image-text matching result, and a position of the text corresponding to the image in a paragraph.

6. The method according to claim 5, characterized in that The image-text matching results are calculated through the following sub-steps: S01: construct the context of the collected image; S02: perform vector splicing; S03: Associating the matched image with the text corresponding to the collected image.

7. The method according to claim 6, characterized in that Vector concatenation is performed through the following sub-steps: S021: Collect corresponding context according to the image to be matched; S022: Identify the category and content of the picture to be matched, and use the context of the picture to be matched to expand the dimension to form a first vector: [picture to be matched, context of the picture to be matched]; S023: forming a second vector according to the collected picture and the context of the collected picture: [collected picture, context of the collected picture]; S024: Setting a similarity threshold and calculating the cosine similarity between the first vector and the second vector as a picture-text matching result; If the calculated similarity is greater than or equal to the similarity threshold, the image and text are considered to match, and the matching image is saved; if the calculated similarity is less than the similarity threshold, the image and text are considered to be unmatched, the image to be matched is eliminated, and the matching ends.

8. The method according to claim 1, characterized in that In step S4, the large language model is trained by processing fine-tuning sample data, wherein the fine-tuning sample data includes image feature vectors, word vectors of text descriptions, and vectors after encoding matching information.

9. The method according to any one of claims 1 to 8, characterized in that: The large language model is a BERT model.

10. A system for generating travel short video descriptions, characterized in that It comprises a processor and a memory; wherein the memory is coupled to the processor and is used to store a computer program, and when the computer program is executed by the processor, the processor implements the method described in any one of claims 1 to 9.

Images