Short video powder expansion method, system and equipment based on multi-modal analysis and medium

Abstract

The invention provides a short video powder expansion method, system and device based on multi-modal analysis and a medium, and the method comprises the steps: obtaining a short video material and behavior information of a material related user, and generating a user tag of the material according to the behavior information; extracting multi-modal features of the short video material according to multi-modal analysis to obtain a material copywriting and a material key frame, and generating a cue word of the to-be-recommended material through the copywriting, the key frame and the user tag; and determining a user portrait of the powder expansion account, constructing a recommendation model, pushing a recommendation material to the powder expansion account according to the prompt word, carrying out secondary editing, and issuing a target powder expansion video.

Description

Technical Field

[0001] The present invention relates to the technical field of short video account maintenance, and in particular to a short video follower growth method, system, device and medium based on multimodal analysis. Background Art

[0003] In order for a short video account to attract enough fans to achieve the goal of quickly increasing fans, it is first necessary to determine the target user portrait that matches the content of the short video account, and then provide short video creators with popular materials that can attract target users for secondary creation and publication, thereby achieving the purpose of helping the account to maintain and increase fans. However, the existing short video recommendation technology mainly solves the viscosity problem between the platform and users through precise push. There is still a research gap in the scenario of account maintenance and fan growth, and it cannot meet the needs of creators to maintain their accounts.

[0004] Therefore, there is an urgent need for a short video follower growth method based on multimodal analysis to solve the problem of how to increase followers of short video accounts. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention proposes a short video follower growth method, system, device and medium based on multimodal analysis.

[0006] The first aspect of the present invention discloses a short video fan-increasing method based on multimodal analysis, comprising:

[0007] Obtain the behavior information of the short video material and the user associated with the material, and generate a user tag for the material based on the behavior information;

[0008] Extracting multimodal features of short video materials through multimodal analysis to obtain material text and material key frames, and generating prompt words for the material to be recommended through the text, key frames and user tags;

[0009] Determine the user portrait of the account that increases followers, build a recommendation model, push recommended materials to the account based on prompt words for secondary editing, and publish the target follower-increasing video.

[0010] In an optional embodiment, the obtaining of the short video material and the behavior information of the user associated with the material includes:

[0011] Use crawler technology to collect short video materials on short video platforms, where short video materials are graded according to the video browsing popularity value, and the collection weights are allocated to short video materials of different popularity grades;

[0012] Obtain the comment content of the short video material and the commentator's homepage information.

[0013] In an optional embodiment, generating a user tag of a material according to the behavior information includes:

[0014] A comment interest model is constructed based on user behavior information to filter active users in the material comment area. The comment interest model is a lightweight NLP algorithm model based on BERT. The comment interest model is pre-trained on the sensitivity of filtering active users through a short video comment training set.

[0015] Extract information from the homepage content of active users to determine the characteristic attributes of active users, wherein the characteristic attributes include age, gender, region, attention preference and historical preference;

[0016] Preset the statistical threshold of feature attributes, aggregate and count the feature attributes of active users, and filter the user labels of the output materials of active users according to the statistical threshold.

[0017] In an optional embodiment, the step of extracting multimodal features of short video materials according to multimodal analysis to obtain material text and material key frames includes:

[0018] Extracting a number of material key frames from the short video material through a key frame calculation model as a dynamic cover of the material;

[0019] The video and audio tracks of the short video material are extracted through the transcoding tool, and the background sound is removed and the human voice is retained by the separation model. The background sound is filtered out based on the audio characteristics to obtain the sound source data composed of the human voice and the human voice.

[0020] The sound source data is converted into the first text data based on automatic speech recognition, the text information in the material image is extracted through the OCR tool to obtain the second text data, and the first text data and the second text data are merged according to the timeline to remove the output material text.

[0021] In an optional embodiment, the generating of prompt words for the to-be-recommended material through the copy, keyframes and user tags includes:

[0022] The recommendation model is based on user tags as a training set for understanding user preferences for short video materials, where material text and material key frames are used as feature tags for short video materials, and prompt words are generated for the recommendation model to output a list of target user feature attributes for short video materials.

[0023] In an optional embodiment, determining the user portrait of the account with increased followers includes:

[0024] Obtain the homepage information of the account with increased followers on the platform, extract the characteristic attribute value interval of the target user of the account with increased followers, and determine the user portrait of the account with increased followers by matching the characteristic attribute list of the target user with the characteristic attribute value interval of the target user.

[0025] In an optional embodiment, the construction of the recommendation model pushes the recommended material to the follower-increasing account according to the prompt word for secondary editing and publishing the target follower-increasing video, including:

[0026] For the input recommended materials, the recommendation model generates video tags for the recommended materials based on the target user feature attribute list generated by the prompt words, and generates a recommended video list based on the similarity between the video tags and the user portrait of the account with increasing followers.

[0027] The second aspect of the present invention discloses a short video fan-increasing system based on multimodal analysis, the system comprising:

[0028] The material collection module is used to obtain short video materials and the behavioral information of users associated with the materials, and generate user tags for the materials based on the behavioral information;

[0029] The material processing module is used to extract the multimodal features of the short video material according to the multimodal analysis to obtain the material copy and material key frames, and generate prompt words for the material to be recommended through the copy, key frames and user tags;

[0030] The material recommendation module is used to determine the user portrait of the account that increases followers, build a recommendation model, push recommended materials to the account that increases followers based on prompt words, and then publish the target video that increases followers.

[0031] In an optional embodiment, the material collection module includes:

[0032] The data extraction module is used to collect short video materials on the short video platform using crawler technology, wherein the short video materials are graded according to the video browsing popularity value, and the collection weights are allocated to short video materials of different popularity grades; the comment content of the short video materials and the homepage information of the commentator are obtained.

[0033] In an optional embodiment, the material collection module includes:

[0034] The user tag extraction module is used to construct a comment interest model based on user behavior information to filter active users in the material comment area. The evaluation interest model is a lightweight NLP algorithm model based on BERT. The evaluation interest model is pre-trained on the sensitivity of filtering active users through a short video comment training set; information is extracted from the homepage content of active users to determine the characteristic attributes of active users, and the characteristic attributes include age, gender, region, attention preference and historical preference; a statistical threshold of the characteristic attribute is preset, the characteristic attributes of the active user are aggregated and counted, and the user tags of the output materials of the active users are filtered according to the statistical threshold.

[0035] In an optional embodiment, the material processing module includes:

[0036] The multimodal analysis module is used to extract a number of material key frames from the short video material as the dynamic cover of the material through a key frame calculation model; extract the video and audio tracks of the short video material through a transcoding tool, use a separation model to remove background sounds and retain human voices from the video and audio tracks, filter the background sounds according to audio features to remove artificial singing, and obtain sound source data composed of human voices and artificial singing; convert the sound source data into first text data based on automatic speech recognition, extract the text information in the material image through an OCR tool to obtain second text data, and merge the first text data and the second text data according to the time axis to remove the output material text.

[0037] In an optional embodiment, the material processing module includes:

[0038] The video feature processing module is used to use user tags as a training set for the recommendation model to understand the user's preference for short video materials, where the material copy and material key frames are used as feature tags of the short video materials, and prompt words are generated for the recommendation model to output a list of target user feature attributes of the short video materials.

[0039] In an optional embodiment, the material recommendation module includes:

[0040] The portrait confirmation module is used to obtain the homepage information of the account with increased followers on the platform, extract the characteristic attribute value interval of the target user of the account with increased followers, and determine the user portrait of the account with increased followers by matching the characteristic attribute list of the target user with the characteristic attribute value interval of the target user.

[0041] In an optional embodiment, the material recommendation module includes:

[0042] The recommendation submodule is used to generate video tags for the input recommendation materials based on the target user feature attribute list generated by the prompt words, and generate a recommended video list based on the similarity between the video tags and the user portrait of the account with increasing followers.

[0043] The third aspect of the present invention discloses a short video fan-increasing device based on multimodal analysis, comprising:

[0044] at least one processor, and

[0045] a memory communicatively connected to the at least one processor; wherein,

[0046] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the short video fan increase method based on multimodal analysis as described in any one of the first aspects of the present invention.

[0047] The fourth aspect of the present invention discloses a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the short video fan-increasing method based on multimodal analysis as described in any one of the items disclosed in the first aspect of the present invention.

[0048] The present invention proposes a short video fan-increasing method, system, device and medium based on multimodal analysis, the method comprising: obtaining the behavior information of the short video material and the user associated with the material, generating the user tag of the material according to the behavior information; extracting the multimodal features of the short video material according to the multimodal analysis to obtain the material copy and the material key frame, generating the prompt words of the material to be recommended through the copy, key frame and user tag; determining the user portrait of the account for increasing fans, constructing a recommendation model to push the recommended material to the account for increasing fans according to the prompt words for secondary editing and publishing the target video for increasing fans. Compared with the prior art, the advantage of the present invention is that the present invention completes the multi-label classification of short videos through multimodal video analysis technology, automatically adds the characteristic tags of the target user to the candidate material video and recommends it to the suitable creator as the creative material, thereby attracting the concerned users who fit the creator's target user portrait, and helping the creator to achieve the purpose of cultivating the account and increasing fans. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0050] Figure 1 This is a flow chart of a short video follower increasing method based on multimodal analysis of the present invention;

[0051] Figure 2 It is a schematic diagram of the short video follower growth system based on multimodal analysis of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0053] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0054] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0055] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0056] Example 1

[0057] See also Figure 1 The embodiment of the present invention discloses a short video fan-increasing method based on multimodal analysis, comprising:

[0058] 101. Obtaining the behavior information of the short video material and the user associated with the material, and generating a user tag for the material according to the behavior information;

[0059] It should be noted that the existing method of increasing fans of an account requires the creator of a short video account to publish high-quality creative videos or forward videos with high real-time popularity in order to increase the number of users (fans) following the account. However, creating videos requires high production costs. Popular videos cannot fully match the target audience preferences of the creator's account due to their real-time and preference characteristics. Screening popular videos also requires costs. Therefore, how to enable short video creators to quickly increase the number of fans of their accounts is a problem currently faced by platform creators. This application proposes the use of multimodal video analysis technology to provide users with popular materials suitable for user customer portraits for secondary editing and publishing, so as to help users maintain their accounts and increase fans. The specific plan is explained below.

[0060] In an optional embodiment, the obtaining of the short video material and the behavior information of the user associated with the material includes:

[0061] Use crawler technology to collect short video materials on short video platforms, where short video materials are graded according to the video browsing popularity value, and the collection weights are allocated to short video materials of different popularity grades;

[0062] Obtain the comment content of the short video material and the commentator's homepage information.

[0063] In an optional embodiment, generating a user tag of a material according to the behavior information includes:

[0064] A comment interest model is constructed based on user behavior information to filter active users in the material comment area. The comment interest model is a lightweight NLP algorithm model based on BERT. The comment interest model is pre-trained on the sensitivity of filtering active users through a short video comment training set.

[0065] Extract information from the homepage content of active users to determine the characteristic attributes of active users, wherein the characteristic attributes include age, gender, region, attention preference and historical preference;

[0066] Preset the statistical threshold of feature attributes, aggregate and count the feature attributes of active users, and filter the user labels of the output materials of active users according to the statistical threshold.

[0067] 102. Extracting multimodal features of short video materials according to multimodal analysis to obtain material text and material key frames, and generating prompt words for the material to be recommended through the text, key frames and user tags;

[0068] In an optional embodiment, the step of extracting multimodal features of short video materials according to multimodal analysis to obtain material text and material key frames includes:

[0069] Extracting a number of material key frames from the short video material through a key frame calculation model as a dynamic cover of the material;

[0070] The video and audio tracks of the short video material are extracted through the transcoding tool, and the background sound is removed and the human voice is retained by the separation model. The background sound is filtered out based on the audio characteristics to obtain the sound source data composed of the human voice and the human voice.

[0071] The sound source data is converted into the first text data based on automatic speech recognition, the text information in the material image is extracted through the OCR tool to obtain the second text data, and the first text data and the second text data are merged according to the timeline to remove the output material text.

[0072] Specifically, the underlying model of multimodal analysis can use a model such as the Qwen2-VL visual understanding model, which can perform image recognition and video analysis, and can call tools and interact with agents at the same time. Since the model architecture supports multimodal rotational position embedding, the original rotational embedding is decomposed into three parts: time, height, and width, so that it can simultaneously capture and integrate the position information of one-dimensional text sequences, two-dimensional visual images, and three-dimensional videos, so as to better process and reason about multimodal data in complex short video materials. It can not only understand long videos, but also be applied to video-based question and answer, dialogue, content creation and other scenarios, making it excellent in processing complex multimedia content.

[0073] Furthermore, the key frame calculation model can be calculated through linear interpolation and nonlinear interpolation to obtain the key frame picture that summarizes the content of the short video as the cover. Usually, 10 key frames in the short video material are taken. The video audio track extraction can use the audio processing function provided by FFmpeg to extract the audio stream. In order to reduce the interference of white noise such as short video background music on the material, the music separation model is used to separate the background music from the audio stream and only retain the human voice. In order to avoid the singing in the human voice being stripped as background sound, the singing recognition model is used to filter out the artificial singing data using MFCC features. The generated audio data can be used for ASR text recognition such as whisper, and the subtitles in the short video screen can be recognized by OCR of the video subtitles through paddle eocr to obtain the screen text. Finally, only the merging and de-duplication are performed on the timeline to generate a complete video text for the model to understand.

[0074] In an optional embodiment, the generating of prompt words for the to-be-recommended material through the copy, keyframes and user tags includes:

[0075] The recommendation model is based on user tags as a training set for understanding user preferences for short video materials, where material text and material key frames are used as feature tags for short video materials, and prompt words are generated for the recommendation model to output a list of target user feature attributes for short video materials.

[0076] Specifically, the recommendation model can understand the content of the video material through the copy, and then the user tags extracted from the user behavior allow the model to understand the user preferences of different short video users, and then set prompt words to let the model output the audience classification and specific attributes of the audience of the short video material. Specifically, prompt words are added to the copy. For example: the input content includes video key frames and complete copy, and then the inquiry prompt word "Please output the age and gender of the main target audience of the video" is added. The corresponding audience label of the short video material output by the model is: [('18-23 years old', 'male') or ('18-23 years old', 'female')], thereby solving the adaptation problem of the popular video material of short videos and the target customer portrait characteristics of users.

[0077] 103. Determine the user portrait of the account that is increasing followers, build a recommendation model, push recommended materials to the account that is increasing followers based on the prompt words, perform secondary editing, and publish the target follower-increasing video.

[0078] In an optional embodiment, determining the user portrait of the account with increased followers includes:

[0079] Obtain the homepage information of the account with increased followers on the platform, extract the characteristic attribute value interval of the target user of the account with increased followers, and determine the user portrait of the account with increased followers by matching the characteristic attribute list of the target user with the characteristic attribute value interval of the target user.

[0080] In an optional embodiment, the construction of the recommendation model pushes the recommended material to the follower-increasing account according to the prompt word for secondary editing and publishing the target follower-increasing video, including:

[0081] For the input recommended materials, the recommendation model generates video tags for the recommended materials based on the target user feature attribute list generated by the prompt words, and generates a recommended video list based on the similarity between the video tags and the user portrait of the account with increasing followers.

[0082] Specifically, account creators can fill in the characteristic attributes of their target users, such as age range and gender range, on the account homepage, and then identify the creator's user audience classification through the homepage information. For the collected short video materials, the audience can be classified according to the labels output by the model, and the short video materials that match the audience can be pushed to the creator. The creator can re-create the short video materials according to his or her own creative ideas and account characteristics and then publish them to other short video platforms, thereby attracting target customer portraits that fit the account, thereby helping the creator's account to maintain its number and increase its fans.

[0083] The present invention proposes a short video fan-increasing method, system, device and medium based on multimodal analysis, the method comprising: obtaining the behavior information of the short video material and the user associated with the material, generating the user tag of the material according to the behavior information; extracting the multimodal features of the short video material according to the multimodal analysis to obtain the material copy and the material key frame, generating the prompt words of the material to be recommended through the copy, key frame and user tag; determining the user portrait of the account for increasing fans, constructing a recommendation model to push the recommended material to the account for increasing fans according to the prompt words for secondary editing and publishing the target video for increasing fans. Compared with the prior art, the advantage of the present invention is that the present invention completes the multi-label classification of short videos through multimodal video analysis technology, automatically adds the characteristic tags of the target user to the candidate material video and recommends it to the suitable creator as the creative material, thereby attracting the concerned users who fit the creator's target user portrait, and helping the creator to achieve the purpose of cultivating the account and increasing fans.

[0084] like Figure 2 As shown, the second aspect of the present invention discloses a short video fan-increasing system based on multimodal analysis, and the system includes:

[0085] The material collection module is used to obtain short video materials and the behavioral information of users associated with the materials, and generate user tags for the materials based on the behavioral information;

[0086] The material processing module is used to extract the multimodal features of the short video material according to the multimodal analysis to obtain the material copy and material key frames, and generate prompt words for the material to be recommended through the copy, key frames and user tags;

[0087] The material recommendation module is used to determine the user portrait of the account that increases followers, build a recommendation model, push recommended materials to the account that increases followers based on prompt words, and then publish the target video that increases followers.

[0088] In an optional embodiment, the material collection module includes:

[0089] The data extraction module is used to collect short video materials on the short video platform using crawler technology, wherein the short video materials are graded according to the video browsing popularity value, and the collection weights are allocated to short video materials of different popularity grades; the comment content of the short video materials and the homepage information of the commentator are obtained.

[0090] In an optional embodiment, the material collection module includes:

[0091] The user tag extraction module is used to construct a comment interest model based on user behavior information to filter active users in the material comment area. The evaluation interest model is a lightweight NLP algorithm model based on BERT. The evaluation interest model is pre-trained on the sensitivity of filtering active users through a short video comment training set; information is extracted from the homepage content of active users to determine the characteristic attributes of active users, and the characteristic attributes include age, gender, region, attention preference and historical preference; a statistical threshold of the characteristic attribute is preset, the characteristic attributes of the active user are aggregated and counted, and the user tags of the output materials of the active users are filtered according to the statistical threshold.

[0092] In an optional embodiment, the material processing module includes:

[0093] The multimodal analysis module is used to extract a number of material key frames from the short video material as the dynamic cover of the material through a key frame calculation model; extract the video and audio tracks of the short video material through a transcoding tool, use a separation model to remove background sounds and retain human voices from the video and audio tracks, filter the background sounds according to audio features to remove artificial singing, and obtain sound source data composed of human voices and artificial singing; convert the sound source data into first text data based on automatic speech recognition, extract the text information in the material image through an OCR tool to obtain second text data, and merge the first text data and the second text data according to the time axis to remove the output material text.

[0094] In an optional embodiment, the material processing module includes:

[0095] The video feature processing module is used to use user tags as a training set for the recommendation model to understand the user's preference for short video materials, where the material copy and material key frames are used as feature tags of the short video materials, and prompt words are generated for the recommendation model to output a list of target user feature attributes of the short video materials.

[0096] In an optional embodiment, the material recommendation module includes:

[0097] The portrait confirmation module is used to obtain the homepage information of the account with increased followers on the platform, extract the characteristic attribute value interval of the target user of the account with increased followers, and determine the user portrait of the account with increased followers by matching the characteristic attribute list of the target user with the characteristic attribute value interval of the target user.

[0098] In an optional embodiment, the material recommendation module includes:

[0099] The recommendation submodule is used to generate video tags for the input recommendation materials based on the target user feature attribute list generated by the prompt words, and generate a recommended video list based on the similarity between the video tags and the user portrait of the account with increasing followers.

[0100] The third aspect of the present invention discloses a short video fan-increasing device based on multimodal analysis, comprising:

[0101] at least one processor, and

[0102] a memory communicatively connected to the at least one processor; wherein,

[0103] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the short video fan increase method based on multimodal analysis as described in any one of the first aspects of the present invention.

[0104] The computer device may be a terminal, and the computer device includes a processor, a memory, a network interface, a display screen, and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a short video fan increase method based on multimodal analysis is implemented. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a button, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0105] The fourth aspect of the present invention discloses a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the short video fan-increasing method based on multimodal analysis as described in any one of the items disclosed in the first aspect of the present invention.

[0106] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it may include the processes of the embodiments of the short video fan increase method based on multimodal analysis as described above. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0107] Alternatively, if the above modules of the present invention are implemented in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to execute all or part of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, RAM, ROM, magnetic disks or optical disks.

[0108] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A short video fan-increasing method based on multimodal analysis, characterized in that: The method comprises: Obtain the behavior information of the short video material and the user associated with the material, and generate a user tag for the material based on the behavior information; Extracting multimodal features of short video materials through multimodal analysis to obtain material text and material key frames, and generating prompt words for the material to be recommended through the text, key frames and user tags; Determine the user portrait of the account that increases followers, build a recommendation model, push recommended materials to the account based on prompt words for secondary editing, and publish the target follower-increasing video.

2. The short video fan-increasing method based on multimodal analysis according to claim 1 is characterized in that: The obtaining of the short video material and the behavior information of the user associated with the material includes: Use crawler technology to collect short video materials on short video platforms, where short video materials are graded according to the video browsing popularity value, and the collection weights are allocated to short video materials of different popularity grades; Obtain the comment content of the short video material and the commentator's homepage information.

3. The short video fan-increasing method based on multimodal analysis according to claim 1 is characterized in that: The step of generating a user tag of a material according to the behavior information includes: A comment interest model is constructed based on user behavior information to filter active users in the material comment area. The comment interest model is a lightweight NLP algorithm model based on BERT. The comment interest model is pre-trained on the sensitivity of filtering active users through a short video comment training set. Extract information from the homepage content of active users to determine the characteristic attributes of active users, wherein the characteristic attributes include age, gender, region, attention preference and historical preference; Preset the statistical threshold of feature attributes, aggregate and count the feature attributes of active users, and filter the user labels of the output materials of active users according to the statistical threshold.

4. The short video fan-increasing method based on multimodal analysis according to claim 1 is characterized in that: The method of extracting multimodal features of short video materials according to multimodal analysis to obtain material text and material key frames includes: Extracting a number of material key frames from the short video material through a key frame calculation model as a dynamic cover of the material; The video and audio tracks of the short video material are extracted through the transcoding tool, and the background sound is removed and the human voice is retained by the separation model. The background sound is filtered out based on the audio characteristics to obtain the sound source data composed of the human voice and the human voice. The sound source data is converted into the first text data based on automatic speech recognition, the text information in the material image is extracted through the OCR tool to obtain the second text data, and the first text data and the second text data are merged according to the timeline to remove the output material text.

5. The short video fan-increasing method based on multimodal analysis according to claim 1 is characterized in that: The generating of prompt words for the to-be-recommended material through the copy, keyframes and user tags includes: The recommendation model is based on user tags as a training set for understanding user preferences for short video materials, where material text and material key frames are used as feature tags for short video materials, and prompt words are generated for the recommendation model to output a list of target user feature attributes for short video materials.

6. The short video fan-increasing method based on multimodal analysis according to claim 5 is characterized in that: Determining the user profile of the account that increases followers includes: Obtain the homepage information of the account with increased followers on the platform, extract the characteristic attribute value interval of the target user of the account with increased followers, and determine the user portrait of the account with increased followers by matching the characteristic attribute list of the target user with the characteristic attribute value interval of the target user.

7. The short video fan-increasing method based on multimodal analysis according to claim 6 is characterized in that: The construction of the recommendation model pushes the recommended material to the follower-increasing account according to the prompt word for secondary editing and releases the target follower-increasing video, including: For the input recommended materials, the recommendation model generates video tags for the recommended materials based on the target user feature attribute list generated by the prompt words, and generates a recommended video list based on the similarity between the video tags and the user portrait of the account with increasing followers.

8. A short video fan-increasing system based on multimodal analysis, characterized in that: The system comprises: The material collection module is used to obtain short video materials and the behavioral information of users associated with the materials, and generate user tags for the materials based on the behavioral information; The material processing module is used to extract the multimodal features of the short video material according to the multimodal analysis to obtain the material copy and material key frames, and generate prompt words for the material to be recommended through the copy, key frames and user tags; The material recommendation module is used to determine the user portrait of the account that increases followers, build a recommendation model, push recommended materials to the account that increases followers based on prompt words, perform secondary editing, and publish the target video that increases followers.

9. A short video fan-increasing device based on multimodal analysis, characterized in that: include: at least one processor, and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the short video fan increase method based on multimodal analysis as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the short video fan-increasing method based on multimodal analysis as described in any one of claims 1 to 7.

Images