Video processing method and device, equipment and medium

Abstract

The embodiment of the invention relates to a video processing method and device, equipment and a medium. The method comprises the following steps: acquiring a target subtitle corresponding to a first video; performing text analysis on the target subtitle, and determining a target icon corresponding to the target subtitle based on a text analysis result of the target subtitle; obtaining video associated information corresponding to the target subtitle in the first video, and determining a recommended display mode of the target icon based on the video associated information; wherein the video associated information comprises video picture information and / or audio information; generating a target video based on the first video and the recommended display mode of the target icon; wherein the target video is a video in which the target subtitle and the target icon are displayed on a video picture of the first video. According to the embodiment of the invention, the proper target icon can be adaptively added, the information expression capability is further improved through the icon on the basis of the subtitles, the interestingness and infection of the video can be increased, and the video watching experience of the user is well improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a video processing method, device, equipment and medium. Background Art

[0002] Nowadays, more and more ordinary users, professional editors and other people need to use multimedia editing software to edit the videos they shoot to enrich the visual presentation effect. For example, in order to facilitate video viewers to clearly understand the content of audio and video, most video editors will add subtitles to the video, and some multimedia editing software has also provided users with functions such as audio to subtitle conversion. However, the inventor has found through research that the information expression ability provided by the video through subtitles is limited and the interest is poor, which needs further improvement. Summary of the invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a video processing method, device, equipment and medium.

[0004] An embodiment of the present disclosure provides a video processing method, which includes: obtaining target subtitles corresponding to a first video; performing text parsing on the target subtitles, and determining a target icon corresponding to the target subtitles based on the text parsing results of the target subtitles; obtaining video-related information corresponding to the target subtitles in the first video, and determining a recommended display method for the target icon based on the video-related information; wherein the video-related information includes video screen information and / or audio information; generating a target video based on the first video and the recommended display method for the target icon; wherein the target video is a video in which the target subtitles and the target icon are displayed on the video screen of the first video.

[0005] Optionally, the text parsing result of the target subtitle includes a word segmentation result of the target subtitle and target information corresponding to the word segmentation result, the target information including one or more of word position, word part of speech, word occurrence frequency, and associated words; determining the target icon corresponding to the target subtitle based on the text parsing result of the target subtitle includes: determining candidate icons corresponding to the target subtitle based on the word segmentation result of the target subtitle; and determining the target icon corresponding to the target subtitle from the candidate icons based on the target information corresponding to the word segmentation result.

[0006] Optionally, determining the candidate icons corresponding to the target subtitles based on the word segmentation results of the target subtitles includes: based on the word segmentation results of the target subtitles and a preset icon resource library, using a semantic matching model to determine an icon having a mapping relationship with a target word in the target subtitles; and determining the candidate icons corresponding to the target subtitles based on the icons corresponding to the target words.

[0007] Optionally, determining the target icon corresponding to the target subtitle from the candidate icons based on the target information corresponding to the word segmentation result includes: obtaining first reference information; the first reference information includes one or more of the following: stress information of the audio corresponding to the target subtitle, feature analysis information of the target subtitle, and quantity constraint information of the target icon; based on the first reference information and the target information corresponding to the word segmentation result, determining the target icon corresponding to the target subtitle from the candidate icons.

[0008] Optionally, obtaining the video-associated information corresponding to the target subtitle in the first video includes: obtaining motion information of the video screen corresponding to the target subtitle in the first video and target object information in the video screen, and obtaining the video screen information corresponding to the target subtitle based on the motion information and the target object information; and / or obtaining key point information of the audio corresponding to the target subtitle in the first video, and obtaining the audio information corresponding to the target subtitle based on the key point information.

[0009] Optionally, determining the recommended display method of the target icon based on the video-related information includes: determining a recommended display position and a recommended display animation of the target icon based on the video screen information; determining a recommended display time of the target icon based on the audio information; and obtaining a recommended display method of the target icon based on the recommended display position, the recommended display animation and the recommended display time.

[0010] Optionally, generating a target video based on the first video and the recommended display method of the target icon includes: obtaining second reference information; the second reference information includes one or more of the following: size information of the target icon, appearance timing information of the target icon, animation style information of the target icon; determining a target display method of the target icon based on the second reference information and the recommended display method of the target icon; generating a target video based on the first video and the target display method.

[0011] The disclosed embodiment also provides a video processing device, including: a subtitle acquisition module, used to acquire target subtitles corresponding to a first video; an icon determination module, used to perform text analysis on the target subtitles, and determine a target icon corresponding to the target subtitles based on the text analysis results of the target subtitles; a display information determination module, used to acquire video-related information corresponding to the target subtitles in the first video, and determine a recommended display method for the target icon based on the video-related information; wherein the video-related information includes video screen information and / or audio information; a video generation module, used to generate a target video based on the first video and the recommended display method for the target icon; wherein the target video is a video in which the target subtitles and the target icon are displayed on the video screen of the first video.

[0012] An embodiment of the present disclosure also provides an electronic device, which includes: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement the video processing method provided by the embodiment of the present disclosure.

[0013] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the video processing method provided by the embodiment of the present disclosure.

[0014] The embodiments of the present disclosure further provide a computer program product, including a computer program, which implements the video processing method provided in the embodiments of the present disclosure when executed by a processor.

[0015] The above technical solution provided by the embodiment of the present disclosure can obtain the target subtitles corresponding to the first video, and perform text analysis on the target subtitles, determine the target icon corresponding to the target subtitles based on the text analysis results of the target subtitles, and can further obtain the video-related information (video screen information and / or audio information) corresponding to the target subtitles in the first video, so as to determine the recommended display method of the target icon, thereby generating a target video with the target subtitles and the target icon displayed on the video screen of the first video. The above method can parse the subtitles, and adaptively add appropriate target icons in combination with the video screen information and / or audio information associated with the subtitles, further improve the information expression ability through icons on the basis of the subtitles, and also increase the fun and appeal of the video, thereby better improving the user's video viewing experience.

[0016] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 A schematic diagram of a flow chart of a video processing method provided by an embodiment of the present disclosure;

[0020] Figure 2 A schematic diagram of a video screen with a target icon added provided by an embodiment of the present disclosure;

[0021] Figure 3 A schematic diagram of a generation process of a recommendation display method provided in an embodiment of the present disclosure;

[0022] Figure 4 A schematic diagram of a video generation process provided by an embodiment of the present disclosure;

[0023] Figure 5 A schematic diagram of the structure of a video processing device provided by an embodiment of the present disclosure;

[0024] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0026] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0027] Figure 1 The flowchart of a video processing method provided by an embodiment of the present disclosure is shown in FIG. 1 , which can be executed by a video processing device, wherein the device can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 1 As shown, the method mainly includes the following steps S102 to S108:

[0028] Step S102: Obtain target subtitles corresponding to the first video.

[0029] The disclosed embodiments do not limit the content of the first video. The target subtitles corresponding to the first video may be the subtitles of the first video itself, or may be subtitles obtained by performing content recognition on the audio of the first video. In some implementation examples, each subtitle that appears in the first video may be used as a target subtitle. In other implementation examples, the target subtitles may also be user-specified subtitles or subtitles that meet preset features, such as: the number of subtitle characters is greater than the preset number of characters, the video screen where the subtitles are located is not a transition screen or a screen with a violent motion amplitude, etc. The conditions for obtaining the target subtitles can be flexibly set according to specific needs, and are not limited here.

[0030] Step S104: performing text analysis on the target subtitle, and determining a target icon corresponding to the target subtitle based on the text analysis result of the target subtitle.

[0031] Exemplarily, the text parsing result of the target subtitle includes the word segmentation result of the target subtitle and the target information corresponding to the word segmentation result. That is, the disclosed embodiment can perform word segmentation processing on the target subtitle, parse based on the word segmentation result, and obtain the target information corresponding to the word segmentation result. The word segmentation result can be used to indicate each word in the target subtitle, and further can also include a word group with a dependency relationship (that is, associated words). In addition, when parsing based on the word segmentation result, it is possible to determine the part of speech of each word in the word segmentation result (such as noun, verb, adjective, adverb, etc.), whether it is a stop word (such as a common word that appears frequently but usually does not carry an important meaning), whether it is a negative word (can be used to identify the negative meaning contained in the text, which is more convenient for accurately extracting text information and analyzing the meaning characteristics of the text), etc., and the position, frequency of occurrence, and words associated with each word in the word segmentation result can be further detected. The target information may include one or more information such as word position, word part of speech, word frequency of occurrence, and associated words. In addition, the above-mentioned text parsing may also include feature analysis, and the text parsing result also includes feature analysis information of the target subtitle. For example, by analyzing the content of the bullet text, using natural language processing technology to infer and identify the specified feature information carried by the bullet text, the feature analysis information of the target subtitle is obtained; wherein, the specified feature information can be a feature that affects the selection of the icon, such as a meaning feature, etc. In other words, the specified feature information can be used as a reference factor for the selection of the icon, and the specified features to be analyzed can be flexibly set, which is not limited here. In practical applications, when the target subtitle is parsed, the keyboard distance, edit distance, language model and context information can also be used to automatically detect and correct spelling errors in the text. For text in a specific language such as English, its grammatical and lexical context, lexical rules and morphological information can also be considered to restore the word to its basic form or dictionary form, such as converting the words represented by the past tense and other tenses into the present tense representation, thereby ensuring semantic consistency, helping to reduce the complexity of the vocabulary and improve the accuracy of text processing. The above are all examples, and the embodiments of the present disclosure do not limit the text parsing method of the subtitle. Through the above text parsing, it is helpful to select the target icon more reasonably and comprehensively based on the text parsing results of the target subtitle.

[0032] The icons referred to in the embodiments of the present disclosure include but are not limited to Emoji icons, and may also be other icons that can convey information or express specific meanings, which are not limited here. Based on the text parsing results of the target subtitles, the target icon that matches the target subtitles can be found from the multiple icons currently available. The target icon corresponding to the target subtitle may include icons corresponding to the key words in the target subtitles, and may also include icons corresponding to the semantics of the target subtitles. It should be noted that the number of target icons corresponding to the target subtitles may be one or more, or there may be no target icon. It is understandable that a video usually has multiple subtitles, and the key words or semantics in different subtitles may be different, and the corresponding target icons may also be different. In addition, some sentences may not have suitable icons because they contain many words without important meanings such as stop words or due to their own semantics.

[0033] Step S106, obtaining video association information corresponding to the target subtitle in the first video, and determining a recommended display method of the target icon based on the video association information; wherein the video association information includes video picture information and / or audio information.

[0034] The video screen information corresponding to the target subtitle is also the video screen that matches the audio corresponding to the target subtitle in the first video. In other words, the video-related information corresponding to the target subtitle has the same appearance time as the target subtitle in the first video. The disclosed embodiment can not only analyze the target subtitle, but also analyze the video-related information of the target subtitle to determine the recommended display method of the target icon. The recommended display method includes but is not limited to the recommended display position, recommended display time, and recommended display animation of the target icon, so as to ensure that the target icon can be more reasonably presented on the video screen, or to ensure that the target icon can be rhythmically displayed with the audio corresponding to the target subtitle. By analyzing the video-related information, the recommended display method of the target icon can be determined more reasonably to ensure the display effect of the target icon.

[0035] Step S108, generating a target video based on the first video and the recommended display method of the target icon; wherein the target video is a video with target subtitles and a target icon displayed on the video screen of the first video.

[0036] Based on the recommended display method of the target icon, the target icon can be rendered on the video screen of the first video corresponding to the target subtitle, and the target video can be obtained based on the rendered video screen. In addition, it should be noted that if the first video itself has a target subtitle, it is not necessary to render the target subtitle when rendering the target icon. If the first video itself does not have a target subtitle, and the target subtitle is obtained by performing voice recognition on the first video, it is also necessary to render the target subtitle when rendering the target icon. Regardless of the method, the target video finally presented will present both the target subtitle and the target icon on the basis of the video screen of the first video, and has a strong ability to express information.

[0037] For easier understanding, please refer to Figure 2 The schematic diagram of a video screen with a target icon added is shown, which illustrates a video screen in a target video, on which not only the target subtitle "I'm very happy today" is presented, but also a corresponding Emoji icon used to express the meaning of happiness is added. In this way, on the basis of the target subtitle, the meaning of happiness can be further strengthened through icons, and compared with subtitles, icons can also convey information more directly and interestingly.

[0038] In summary, the above method can parse the subtitles, and combine the video picture information and / or audio information associated with the subtitles to adaptively add appropriate target icons. On the basis of the subtitles, the information expression ability can be further improved through icons, which can also increase the fun and appeal of the video, and better improve the user's video viewing experience.

[0039] In some embodiments, the text parsing result of the target subtitle includes the word segmentation result of the target subtitle and the target information corresponding to the word segmentation result. The target information includes one or more of word position, word part of speech, word frequency of occurrence, and related words. In addition, the target information can also be used to indicate special words such as negative words and stop words in the word segmentation result or indicate key words with specified characteristics in the word segmentation result. Exemplarily, the step of determining the target icon corresponding to the target subtitle based on the text parsing result of the target subtitle can be performed with reference to the following steps A and B:

[0040] Step A, based on the word segmentation result of the target subtitle, determine the candidate icon corresponding to the target subtitle. In some implementation examples, the corresponding icon can be obtained based on each word obtained through the word segmentation process in the target subtitle, and the obtained icon is used as the candidate icon corresponding to the target subtitle; in other implementations, based on the word segmentation result of the target subtitle, the target word that meets the preset conditions can be selected from the words obtained through the word segmentation process in the target subtitle, and the icon corresponding to the target word can be used as the candidate icon corresponding to the target subtitle, such as, based on the word segmentation result of the target subtitle and the target information corresponding to the word segmentation result, the candidate icon corresponding to the target subtitle can be determined.

[0041] In some specific implementation examples, step A can be performed with reference to the following steps A1 to A2:

[0042] Step A1, based on the word segmentation results of the target subtitles and the preset icon resource library, use the semantic matching model to determine the icon that has a mapping relationship with the target word in the target subtitle. In specific implementation, label features can be constructed and extracted for the multiple existing icons, and the multiple icons and their corresponding icons' semantic features, expression meanings and other icon information are associated and stored in the icon resource library. Based on the word segmentation results, or based on the word segmentation results and the target information corresponding to the word segmentation results, use the semantic matching model to determine the icon that has a mapping relationship with the target word in the target subtitle. Among them, the semantic matching model is a neural network model obtained by pre-training, such as a language model, etc. For example, a training sample pair consisting of a word sample and a corresponding icon can be obtained, and the neural network model can be trained based on the training sample pair to obtain a semantic matching model, which has the ability to output matching icons for the input word segmentation results.

[0043] In some implementation examples, the target word may be each word in the target subtitle, and in other implementation examples, the target word may be a keyword with specified features in the target subtitle; the specified features include one or more of the following: the word position is at a specified position in the target subtitle (such as front, middle, back, etc.), has a specified word part of speech (such as noun, verb, adjective, etc.), the word frequency is in a preset frequency range (such as repeated twice or more in the target subtitle, or has appeared in the subtitle before the target subtitle, etc.), a stressed word, a word whose pronunciation duration exceeds a preset duration. In the case where the target word is a keyword with specified features, step A1 may specifically be based on the word segmentation result of the target subtitle, the target information corresponding to the word segmentation result, and a preset icon resource library, and use a semantic matching model to determine an icon that has a mapping relationship with the target word in the target subtitle.

[0044] Step A2, based on the icon corresponding to the target word, determine the candidate icon corresponding to the target subtitle. In some embodiments, if the target word is a keyword with specified characteristics, the icon corresponding to the target word can be directly used as the candidate icon corresponding to the target subtitle. In other embodiments, if the target word is all words with corresponding icons in the target subtitle, the keyword with specified characteristics can be further selected from the target word based on the target information corresponding to the word segmentation result, and the icon corresponding to the keyword word can be used as the candidate icon corresponding to the target subtitle. For example, the target subtitle is "After a whole day of work, she enjoyed a relaxing walk in the park under the golden sun's rays". Although there are multiple segmentations after the word segmentation process, many segmentations have corresponding icons, but the candidate icons corresponding to the target subtitles initially screened include: a briefcase icon corresponding to "work", a walking icon corresponding to "walk", a tree icon corresponding to "park", and a sun icon corresponding to "sun".

[0045] Step B, based on the target information corresponding to the word segmentation result, determine the target icon corresponding to the target subtitle from the candidate icons. In some implementations, the scoring dimensions of each candidate icon can be determined based on the target information corresponding to the word segmentation result, and the weights and scores corresponding to each scoring dimension can be obtained. The comprehensive scores of each candidate icon are obtained by weighted summation, and the candidate icons are sorted in descending order based on the comprehensive scores. Then, the top N target icons are selected according to the sorting results. N can be a preset default value, a user-set value, or a value based on the total number of candidate icons and a preset ratio, which is not limited here. In some specific implementation examples, step B can be performed with reference to the following steps B1 to B2:

[0046] Step B1, obtaining first reference information; the first reference information includes one or more of the following: stress information of the audio corresponding to the target subtitle (which may be called audio stress point), feature analysis information of the target subtitle, and quantity constraint information of the target icon.

[0047] In practical applications, the audio stress points corresponding to the target subtitles can be determined by methods such as stress recognition, the audio stress points corresponding to the target subtitles can be obtained, and the target subtitles can be feature analyzed. Specifically, natural language processing technology can be used to infer and identify the specified feature information carried by the text to obtain feature analysis information of the target subtitles. The above-mentioned stress information and feature analysis information can both be executed by background algorithms. The quantity constraint information of the above-mentioned target icons can be a background preset value, or it can be obtained through user setting information, which is not limited here. The quantity constraint information can be used to indicate the specific number or quantity range of the target icons corresponding to the target subtitles. In practical applications, the content contained in the first reference information can be flexibly selected according to needs, and other reference information can be added, which is not limited here.

[0048] Step B2, based on the first reference information and the target information corresponding to the word segmentation result, determine the target icon corresponding to the target subtitle from the candidate icons. For example, the target icon can be an icon with a specified word part of speech and a word frequency higher than a preset threshold and / or the word corresponding to the target icon is an audio stress point in the target subtitle; and the positions of the words corresponding to the multiple target icons are not adjacent positions, and the target icon also matches the feature analysis information of the target subtitle. In actual applications, the scoring dimensions corresponding to the candidate icons can be determined based on the target information corresponding to the word segmentation result, the information in the first reference information except the quantity constraint information of the target icon, and the words corresponding to the candidate icons, and the weights corresponding to the scoring dimensions and the scores corresponding to each candidate icon under the scoring dimensions are obtained. Based on the scores and weights corresponding to each candidate icon under each scoring dimension, the comprehensive scores of each candidate icon are obtained, and then based on the comprehensive scores of each candidate icon and the quantity constraint information of the target icon, the target icon corresponding to the target subtitle is determined from the candidate icons. In some implementation examples, the weight of the scoring dimension related to the frequency of word occurrence is affected by the word part of speech. For example, for words such as pronouns, adjectives, adverbs, common words, etc., even if their frequency of occurrence is high, the weight is low. The way to adjust the weight can be understood as performing a frequency reduction correction process on such words to reduce the positive impact of the frequency of such words on the score. For words such as verbs and nouns, the higher the frequency, the higher the corresponding weight and the higher the corresponding score. Through the above method, the most suitable one or more target icons can be reasonably selected. In addition, the repetition of the target icon can be constrained to avoid the repetition of the same target icon within and between sentences, thereby improving the richness and diversity of the icon display. In addition, the target information corresponding to the word segmentation result also includes the word position. Therefore, when selecting the target icon from the candidate icons, the icons corresponding to the adjacent words can also be excluded, that is, the adjacent word trigger suppression process is performed to avoid the appearance of multiple icons in succession and affect the perception.

[0049] The disclosed embodiment provides an implementation example of obtaining video association information corresponding to a target subtitle in a first video, which can be implemented by referring to the following (1) and / or (2):

[0050] (1) The motion information of the video screen corresponding to the target subtitle in the first video and the target object information in the video screen are obtained, and based on the motion information and the target object information, the video screen information corresponding to the target subtitle is obtained. In practical applications, visual feature detection can be performed on the first video, such as sensing the degree of motion of the video screen in the first video based on the motion optical flow of the first video, and providing basic screen features such as static, dynamic, and transition. In the above manner, the motion information of the video screen corresponding to the target subtitle in the first video can be obtained, which can specifically be the degree of motion information. In addition, the target object detection algorithm, the key point detection algorithm, etc. can also be used to obtain the target object information in the video screen, and the target object information includes the position of the target object, the key points of the target object, the size of the target object, the action information of the target object, etc. In the disclosed embodiment, the motion information and the target object information corresponding to the target subtitle in the first video can be used together as the video screen information corresponding to the target subtitle.

[0051] (2) Acquire the key point information of the audio corresponding to the target subtitle in the first video, and obtain the audio information corresponding to the target subtitle based on the key point information. The key point information of the audio includes but is not limited to highlights such as audio card points and audio stress points. In practical applications, the audio corresponding to the first video can be processed by beat detection, stress recognition, music category detection, etc., so as to obtain audio-related features and obtain the key point information of the audio corresponding to the first video.

[0052] In the case where the video-related information includes video picture information and audio information, determining a recommended display mode of the target icon based on the video-related information may be performed with reference to the following steps a to c:

[0053] Step a, based on the video screen information, determine the recommended display position and recommended display animation of the target icon. It is understandable that the video screen information may include the motion information of the video screen and the target object information in the video screen. Based on the motion information, it can be determined whether the target icon should be presented in the video screen and the display animation style of the presented target icon. For example, for a video screen with a large transition and a large motion amplitude, the target icon may not be presented clearly, and the display position may be set to be empty, while a video screen with a relatively small motion amplitude is more suitable for presenting the target icon. In addition, the recommended display position of the target icon can be determined based on the target object information, such as the position of the target icon does not overlap with the position of the target object, thereby avoiding the target icon from blocking the target object. In specific implementation, the target icon can be set in the background area of ​​the target object, and the target icon can be further arranged around the target object. In addition, the recommended display position of the target icon can be determined in combination with the subtitle position of the target subtitle in the video screen and the video screen information, such as setting the relative position between the target icon and the target subtitle, such as horizontal alignment, vertical alignment, center alignment of the icon and the text, icon overlapped at the bottom of the text, and random movement of the icon in the subtitle frame. Furthermore, the recommended display animation of the target icon can be determined based on the motion level of the video screen and / or the action of the target object, so as to achieve the effect of the target icon following the motion effect. The recommended display animation can be, for example, uniform rotation, random rotation, left and right movement, clockwise rotation of a specified angle in and out, zooming, shaking, etc. The above are only examples and should not be regarded as limitations.

[0054] Step b, determining the recommended display time of the target icon based on the audio information. For example, the recommended display time of the target icon can be determined based on the key point time such as the audio card point and the audio emphasis point in the audio corresponding to the target icon, such as making the target icon appear with the audio card point and the emphasis point, thereby enhancing the information expression effect.

[0055] Step c, based on the recommended display position, recommended display animation and recommended display time, obtain the recommended display mode of the target icon. In addition, the size of the target icon can also be intelligently matched based on the video screen information, that is, the recommended display mode can also include other display information such as the recommended display size, which is not limited here.

[0056] For easier understanding, please refer to Figure 3The schematic diagram of the generation process of a recommended display method shown in the figure shows that audio input data, video input data and text input data can all affect the target icon and its display method, wherein the content of the text input can be obtained based on the subtitles input by the user, or can be obtained by voice recognition and text conversion for audio or video. For the subtitle "After a whole day of work, she enjoys a relaxing walk in the park under the golden sun's rays", it can be processed through one or more text features such as text segmentation, part-of-speech tagging, part-of-speech restoration, spelling correction, text feature analysis, word frequency analysis, etc., and keyword detection and icon matching are performed based on the processing results to obtain multiple word-icon matching pairs, wherein the icons are candidate icons. Specifically, the candidate icons corresponding to the target subtitle include: briefcase icon corresponding to "work", walking icon corresponding to "walk", tree icon corresponding to "park", and sun icon corresponding to "sun". Then, each candidate icon is scored based on the aforementioned scoring dimensions, and based on the number of target icons selected (taking 2 as an example), the first two candidate icons with the highest scores are selected as the target icons. In addition, the display method of the target icon can be recommended in combination with data such as video input and text input, and further combined with the setting parameters of the target icon such as icon size and icon style input by the user to comprehensively determine the display method of the target icon. Through the above method, it is possible to reasonably and reliably obtain the target icon and the display method of the target icon that conforms to the current text subtitles and audio and video input information, including but not limited to the display size, display position, display animation and display time of the target icon, and better enhance the information expression ability and video appeal by displaying the target icon. It should be noted that Figure 3 This is only an exemplary description and should not be regarded as a limitation. In actual applications, more or less analysis and processing may be performed on the target barrage. For example, for Chinese, word form restoration is not required.

[0057] In some implementations, the step of generating the target video based on the first video and the recommended display mode of the target icon may be performed by referring to the following steps 1 to 3:

[0058] Step 1, obtaining second reference information; the second reference information includes one or more of the following: size information of the target icon, appearance timing information of the target icon, and animation style information of the target icon.

[0059] Exemplarily, a setting control for the second reference information may be provided to the user, and the second reference information set by the user may be obtained based on the setting control, so as to meet the needs of the user. For example, a slider control may be provided to allow the user to set the size of the target icon, a resource card control may be provided to allow the user to select the animation style of the target icon, and an option control may be provided to allow the user to set the appearance timing of the target icon by providing an option control for the appearance timing of the page / sentence / word icon, so as to determine whether the target icon appears with the video screen, with the corresponding target subtitle, or with the corresponding word in the target subtitle.

[0060] Step 2, based on the second reference information and the recommended display mode of the target icon, determine the target display mode of the target icon. It can be understood that the recommended display mode of the target icon can be obtained by the background algorithm through the analysis of the target subtitle and the video screen information corresponding to the target subtitle, and the second reference information can be obtained by the user setting. In some implementation examples, the target display mode is the union of the second reference information and the recommended display mode of the target icon, and when there is conflicting information between the second reference information and the recommended display mode of the target icon, the second reference information shall prevail. In other words, the user setting takes precedence. For example, if the animation style indicated by the second reference information is different from the recommended animation style of the target icon, the animation style indicated by the second reference information shall be adopted. In some implementations, a preview video can be presented to the user according to the recommended display mode of the target icon, and a control for setting the second reference information can be provided to the user. When the second reference information set by the user through the control is obtained, the target icon and the corresponding display mode are adjusted based on the second reference information to obtain the target display mode.

[0061] Step 3: Generate a target video based on the first video and the target display mode. Exemplarily, a target icon may be rendered on the video screen of the first video based on the target display mode, so as to generate the target video based on the rendered video screen.

[0062] Based on the foregoing, the present disclosure further provides Figure 4 The schematic diagram of a video generation process shown in the figure shows that the user can upload the audio and video through the client to the server for processing. The server can perform subtitle recognition and generate icon recommendation results corresponding to the subtitle recognition data using the icon algorithm recommendation service based on the first adjustment parameter, and return it to the client. The client can determine the target display method of the target icon in combination with the second adjustment parameter, thereby loading and rendering the video screen with the target icon added and displaying it on the terminal. For example, in Figure 4The main processing modules used in the icon algorithm recommendation service are symbolically illustrated in the figure. First, the multilingual text understanding module can be used to process text features, such as text classification and semantic feature extraction, intelligent word segmentation, part-of-speech tagging, keyword detection, text feature analysis, etc. After that, the semantic matching algorithm module can be processed based on the output results of the multilingual text understanding module. In specific implementation, the semantic matching algorithm module can construct and extract icon label features, and can also use the language model to map icons and word segmentation results. The mapping algorithm can be implemented based on the intelligent recommendation algorithm of multi-feature fusion. Finally, the target icon can be determined, and the recommended display method of the target icon can be further determined through the icon intelligent packaging module. It should be noted that Figure 4 This is just an example. In actual applications, the functional modules can be divided in other ways. Each functional module can contain more or fewer units. The units contained in different functional modules can also be adjusted. Units such as sensitive information shielding, common word frequency reduction, and word disambiguation can also be classified as multilingual text understanding modules or semantic matching modules, which are not limited here. In addition, Figure 4 Both the first adjustment parameter and the second adjustment parameter can be user-set parameters. For example, the first adjustment parameter can directly affect the icon algorithm recommendation service, while the second adjustment parameter can be used to change the recommendation result. Exemplarily, the first adjustment parameter can be the number constraint of target icons, the repetition constraint of target icons, etc., and the second adjustment parameter can be the size of the target icon, the timing of appearance, the animation style, etc., which can be flexibly set and are not limited here.

[0063] To sum up, the video processing method provided by the embodiments of the present disclosure can parse subtitles, and adaptively add appropriate target icons in combination with the video picture information and / or audio information associated with the subtitles. It can further enhance the information expression capability through icons on the basis of the subtitles, and can also increase the fun and appeal of the video, thereby better improving the user's video viewing experience.

[0064] Corresponding to the aforementioned video processing method, the embodiment of the present disclosure further provides a video processing device, Figure 5 FIG. 1 is a schematic diagram of a video processing device provided in an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated in an electronic device, such as Figure 5 As shown, the video processing device includes:

[0065] A subtitle acquisition module 502 is used to acquire target subtitles corresponding to the first video;

[0066] An icon determination module 504 is used to perform text analysis on a target subtitle and determine a target icon corresponding to the target subtitle based on the text analysis result of the target subtitle;

[0067] The display information determination module 506 is used to obtain the video association information corresponding to the target subtitle in the first video, and determine the recommended display mode of the target icon based on the video association information; wherein the video association information includes video picture information and / or audio information;

[0068] The video generation module 508 is used to generate a target video based on the first video and the recommended display method of the target icon; wherein the target video is a video with target subtitles and a target icon displayed on the video screen of the first video.

[0069] The above-mentioned device can parse the subtitles, and adaptively add appropriate target icons in combination with the video picture information and / or audio information associated with the subtitles. It can further enhance the information expression ability through icons on the basis of the subtitles, and can also increase the fun and appeal of the video, thereby better improving the user's video viewing experience.

[0070] In some embodiments, the text parsing result of the target subtitle includes a word segmentation result of the target subtitle and target information corresponding to the word segmentation result, wherein the target information includes one or more of word position, word part of speech, word occurrence frequency, and associated words; determining the target icon corresponding to the target subtitle based on the text parsing result of the target subtitle includes: determining a candidate icon corresponding to the target subtitle based on the word segmentation result of the target subtitle; and determining the target icon corresponding to the target subtitle from the candidate icons based on the target information corresponding to the word segmentation result.

[0071] In some embodiments, determining the candidate icons corresponding to the target subtitles based on the word segmentation results of the target subtitles includes: based on the word segmentation results of the target subtitles and a preset icon resource library, using a semantic matching model to determine an icon having a mapping relationship with a target word in the target subtitles; and determining the candidate icons corresponding to the target subtitles based on the icons corresponding to the target words.

[0072] In some embodiments, determining the target icon corresponding to the target subtitle from the candidate icons based on the target information corresponding to the word segmentation result includes: obtaining first reference information; the first reference information includes one or more of the following: stress information of the audio corresponding to the target subtitle, feature analysis information of the target subtitle, and quantity constraint information of the target icon; based on the first reference information and the target information corresponding to the word segmentation result, determining the target icon corresponding to the target subtitle from the candidate icons.

[0073] In some embodiments, obtaining the video-associated information corresponding to the target subtitle in the first video includes: obtaining motion information of the video screen corresponding to the target subtitle in the first video and target object information in the video screen, and obtaining the video screen information corresponding to the target subtitle based on the motion information and the target object information; and / or obtaining key point information of the audio corresponding to the target subtitle in the first video, and obtaining the audio information corresponding to the target subtitle based on the key point information.

[0074] In some embodiments, determining the recommended display method of the target icon based on the video-related information includes: determining the recommended display position and recommended display animation of the target icon based on the video screen information; determining the recommended display time of the target icon based on the audio information; and obtaining the recommended display method of the target icon based on the recommended display position, the recommended display animation and the recommended display time.

[0075] In some embodiments, generating a target video based on the first video and the recommended display method of the target icon includes: obtaining second reference information; the second reference information includes one or more of the following: size information of the target icon, appearance timing information of the target icon, animation style information of the target icon; determining a target display method of the target icon based on the second reference information and the recommended display method of the target icon; generating a target video based on the first video and the target display method.

[0076] The video processing device provided in the embodiments of the present disclosure can execute the video processing method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0077] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device embodiment can refer to the corresponding process in the method embodiment, and will not be repeated here.

[0078] An embodiment of the present disclosure provides an electronic device, which includes: a storage device on which a computer program is stored; and a processing device for executing the computer program in the storage device to implement the steps of any method in the present disclosure.

[0079] Reference below Figure 6, which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0080] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0081] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0082] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0083] In addition to the above-mentioned methods and devices, the embodiments of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, cause the processor to perform the method provided by the embodiments of the present disclosure. The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present disclosure, the programming languages ​​including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on a user computing device, partially on a user device, as an independent software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0084] In addition, the embodiment of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the video processing method provided by the embodiment of the present disclosure.

[0085] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0086] The embodiments of the present disclosure further provide a computer program product, including a computer program / instruction, which implements the video processing method in the embodiments of the present disclosure when executed by a processor.

[0087] It is understandable that before using the technical solutions disclosed in the various embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0088] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0089] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0090] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0091] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0092] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video processing method, characterized in that: include: Obtain target subtitles corresponding to the first video; Performing text parsing on the target subtitle, and determining a target icon corresponding to the target subtitle based on the text parsing result of the target subtitle; Acquire video-related information corresponding to the target subtitle in the first video, and determine a recommended display method of the target icon based on the video-related information; wherein the video-related information includes video picture information and / or audio information; A target video is generated based on the first video and the recommended display method of the target icon; wherein the target video is a video in which the target subtitles and the target icon are displayed on the video screen of the first video.

2. The method according to claim 1, characterized in that The text parsing result of the target subtitle includes the word segmentation result of the target subtitle and the target information corresponding to the word segmentation result, wherein the target information includes one or more of word position, word part of speech, word occurrence frequency, and related words; The determining the target icon corresponding to the target subtitle based on the text parsing result of the target subtitle includes: Based on the word segmentation result of the target subtitle, determining a candidate icon corresponding to the target subtitle; Based on the target information corresponding to the word segmentation result, a target icon corresponding to the target subtitle is determined from the candidate icons.

3. The method according to claim 2, characterized in that The determining, based on the word segmentation result of the target subtitle, a candidate icon corresponding to the target subtitle comprises: Based on the word segmentation result of the target subtitle and a preset icon resource library, a semantic matching model is used to determine an icon having a mapping relationship with a target word in the target subtitle; Based on the icon corresponding to the target word, a candidate icon corresponding to the target subtitle is determined.

4. The method according to claim 2, characterized in that: The determining, based on the target information corresponding to the word segmentation result, a target icon corresponding to the target subtitle from the candidate icons, comprises: Acquire first reference information; the first reference information includes one or more of the following: accent information of the audio corresponding to the target subtitle, feature analysis information of the target subtitle, and quantity constraint information of the target icon; Based on the first reference information and the target information corresponding to the word segmentation result, a target icon corresponding to the target subtitle is determined from the candidate icons.

5. The method according to claim 1, characterized in that The acquiring video association information corresponding to the target subtitle in the first video includes: Acquire motion information of a video screen corresponding to the target subtitle in the first video and target object information in the video screen, and obtain video screen information corresponding to the target subtitle based on the motion information and the target object information; and / or, Acquire key point information of the audio corresponding to the target subtitle in the first video, and obtain audio information corresponding to the target subtitle based on the key point information.

6. The method according to claim 1, characterized in that The determining of a recommended display method of the target icon based on the video association information includes: Determine a recommended display position and a recommended display animation of the target icon based on the video picture information; Determining a recommended display time of the target icon based on the audio information; Based on the recommended display position, the recommended display animation and the recommended display time, a recommended display mode of the target icon is obtained.

7. The method according to claim 1, characterized in that The generating a target video based on the first video and the recommended display mode of the target icon includes: Acquire second reference information; the second reference information includes one or more of the following: size information of the target icon, appearance timing information of the target icon, and animation style information of the target icon; Determining a target display mode of the target icon based on the second reference information and the recommended display mode of the target icon; A target video is generated based on the first video and the target display method.

8. A video processing device, characterized in that: include: A subtitle acquisition module, used to acquire target subtitles corresponding to the first video; An icon determination module, configured to perform text analysis on the target subtitle and determine a target icon corresponding to the target subtitle based on the text analysis result of the target subtitle; A display information determination module, used to obtain video-related information corresponding to the target subtitle in the first video, and determine a recommended display method for the target icon based on the video-related information; wherein the video-related information includes video picture information and / or audio information; A video generation module is used to generate a target video based on the first video and the recommended display method of the target icon; wherein the target video is a video with the target subtitles and the target icon displayed on the video screen of the first video.

9. An electronic device, characterized in that: The electronic device comprises: a storage device having a computer program stored thereon; A processing device, used to execute the computer program in the storage device to implement the steps of the video processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the video processing method described in any one of claims 1 to 7.

11. A computer program product, characterized in that The invention comprises a computer program, which implements the video processing method according to any one of claims 1 to 7 when being executed by a processor.

Images