Subtitle generation method and apparatus, electronic device, and medium

Abstract

The application discloses a subtitle generation method and device, electronic equipment and medium. Through the application of the technical solution, the existing image subtitle generation model can be assisted by a stylization control module to generate a subtitle content with a specific style category for each image. Specifically, on the one hand, the style factor of the style control module can be used to control the subtitle generation model to select a word with specific style characteristics for the image. The purpose of outputting the subtitle content with higher preference degree for different users is achieved. On the other hand, the similarity between the predicted word and the previously predicted word is calculated through the fluency factor, and the word with lower repetition degree is selected, so as to reduce the possibility of repeated words in the subtitle and improve the fluency of the sentence.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video data processing, and particularly relates to a subtitle generation method and device, electronic equipment and medium. BACKGROUND

[0002] Image captioning is a basic task in the intersection of natural language processing and computer vision. With the rise of generative models, people have begun to focus on generating image captions with specific styles. The essence of the stylized image captioning task is to give a picture so that the caption generation model generates a text with a specific style that can summarize the picture accordingly.

[0003] However, existing stylized image caption generation work is mainly completed by pre-strengthening learning and contrast learning on samples of a certain fixed style category, and multiple training of the model is required to generate captions of the target style. Understandably, this caption generation method is relatively single. SUMMARY

[0004] The embodiments of the present application provide a subtitle generation method and device, electronic equipment and medium, and the embodiments of the present application are used to solve the problem that the style category of the caption generation method is relatively single and affects the user viewing experience in the related art.

[0005] According to an aspect of the present application, a subtitle generation method is provided, which is applied to a caption generation model comprising an insertable style control module, and the method comprises:

[0006] obtaining a candidate word set predicted by the caption generation model for a to-be-processed image at a current time, and obtaining at least one prefix word corresponding to the candidate word set predicted by the caption generation model in a first historical period;

[0007] selecting a target word with a target style and a repetition rate lower than a preset standard from the candidate word set by using the insertable style control module;

[0008] combining the at least one prefix word and the target word as target caption content of the to-be-processed image.

[0009] Optionally, in another embodiment based on the above-mentioned method of the present application, the step of selecting a target word with a target style and a repetition rate lower than a preset standard from the candidate word set by using the insertable style control module comprises:

[0010] calculating a style factor of each candidate word in the candidate word set by using the insertable style control module, wherein the style factor is used to represent the closeness of the style of the candidate word to the target style; and

[0011] The insertion style control module is used to calculate fluency factors of each candidate word in the candidate word set, the fluency factors being used to represent repetition of the candidate word and historical words predicted by the subtitle generation model in a second historical period;

[0012] The target word is determined based on the style factor and the fluency factor of each candidate word.

[0013] Optionally, in another embodiment based on the above method of the present application, the insertion style control module is used to calculate the style factor of each candidate word in the candidate word set, comprising:

[0014] The prefix word and each candidate word are input into a text style classifier of the insertion style control module, so that the text style classifier identifies a style score corresponding to each candidate word;

[0015] The style score is taken as the style factor corresponding to the candidate word.

[0016] Optionally, in another embodiment based on the above method of the present application, the insertion style control module is used to calculate the fluency factor of each candidate word in the candidate word set, comprising:

[0017] The prefix word and any one of the candidate words in the candidate word set are combined to obtain a plurality of comparison combinations;

[0018] Each comparison combination is compared with historical words in similarity, to determine a similarity value of each comparison combination and each historical word;

[0019] Based on the similarity value, a fluency score of each candidate word is determined;

[0020] The fluency score is taken as the fluency factor corresponding to the candidate word.

[0021] Optionally, in another embodiment based on the above method of the present application, the target word is determined based on the style factor and the fluency factor of each candidate word, comprising:

[0022] A first weight coefficient corresponding to the style factor and a second weight coefficient corresponding to the fluency factor are obtained;

[0023] Based on the first weight coefficient and the second weight coefficient, a comprehensive score of each candidate word between the style factor and the fluency factor is calculated;

[0024] The candidate word with the highest comprehensive score is taken as the target word.

[0025] Optionally, in another embodiment based on the above method of the present application, the combination of the at least one prefix word and the target word as the target subtitle content comprises:

[0026] The prefix word and the target word are sequentially combined and input into the decoder of the subtitle generation model to obtain the target subtitle content with the target style.

[0027] According to yet another aspect of embodiments of the present application, a subtitle generation device is provided, which is applied to a subtitle generation model comprising an insertable style control module, and the device comprises:

[0028] The acquisition module is configured to acquire a candidate word set predicted by the subtitle generation model for a to-be-processed image at a current time, and acquire at least one prefix word corresponding to the candidate word set predicted by the subtitle generation model at a first historical period.

[0029] The selection module is configured to select, by using the insertable style control module, a target word with a target style and a repetition rate lower than a preset standard from the candidate word set.

[0030] The generation module is configured to combine the at least one prefix word and the target word as target subtitle content of the to-be-processed image.

[0031] According to yet another aspect of embodiments of the present application, an electronic device is provided, which comprises:

[0032] a memory configured to store executable instructions; and

[0033] a display configured to execute the executable instructions with the memory to complete the operations of any of the above-mentioned subtitle generation methods.

[0034] According to yet another aspect of embodiments of the present application, a computer device readable storage medium is provided, which is configured to store instructions readable by a computer device, and the instructions are executed to perform the operations of any of the above-mentioned subtitle generation methods.

[0035] In the present application, a candidate word set predicted by a subtitle generation model for a to-be-processed image at a current time can be acquired, and at least one prefix word corresponding to the candidate word set predicted by the subtitle generation model at a first historical period can be acquired; an insertable style control module is used to select a target word with a target style and a repetition rate lower than a preset standard from the candidate word set; and the combination of the at least one prefix word and the target word is used as target subtitle content of the to-be-processed image.

[0036] By applying the technical solution of the present application, the existing image caption generation model can be assisted by the stylization control module to generate a caption content with a specific style category for each image. Specifically, on the one hand, the stylization control module can control the caption generation model to select a word with specific style characteristics for the image. In order to achieve the purpose of outputting the caption content preferred by different users. On the other hand, the fluency factor is used to calculate the similarity between the predicted word and the previously predicted word, and the word with lower repetition degree is selected, so as to reduce the possibility of repeated words in the caption and improve the fluency of the sentence.

[0037] The technical solution of the present application will be described in detail below by using multiple embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0038] The accompanying drawings, which form a part of the specification, illustrate embodiments of the present application and, together with the description, serve to explain the principles of the present application.

[0039] The present application can be more clearly understood and appreciated from the following detailed description, taken in conjunction with the following drawings of which:

[0040] Figure 1 A method for generating a caption is shown in the schematic diagram of an embodiment of the present application;

[0041] Figure 2 A schematic diagram of the overall flow of a caption generation method is shown in an embodiment of the present application;

[0042] Figure 3 A system architecture diagram of a caption generation method is shown in an embodiment of the present application;

[0043] Figure 4 A schematic diagram of the structure of an electronic device is shown in an embodiment of the present application;

[0044] Figure 5 A schematic diagram of the structure of an electronic device is shown in an embodiment of the present application;

[0045] Figure 6 A schematic diagram of the structure of a storage medium is shown in an embodiment of the present application. DETAILED DESCRIPTION

[0046] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement, numerical expressions and values of the components and steps set forth in these embodiments do not limit the scope of the present application unless otherwise specifically stated.

[0047] It should be understood that the dimensions of the various elements shown in the figures are chosen for convenience only, and thus, are not necessarily drawn to scale.

[0048] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the scope of the application or its application or uses.

[0049] Techniques, methods, and apparatus known to those of ordinary skill in the relevant art can not be discussed in detail herein, but should be considered within the scope of the disclosure.

[0050] It should be noted that like reference numerals and letters refer to like items throughout the several views of the drawings, and thus, a discussion of the same will not be repeated in connection with the others of the drawings.

[0051] In addition, the technical solutions among the various embodiments of the present application can be combined with each other, but it must be based on the implementation of those of ordinary skill in the art, and when the combination of technical solutions appears contradictory or unimplementable, it should be considered that the combination of technical solutions does not exist, nor is it within the scope of protection required by the present application.

[0052] It should be noted that all directional indications, such as upper, lower, left, right, front, back, etc., are only used to explain the relative position relationship, movement condition, etc. between components in a certain posture (as shown in the drawings), and if the certain posture changes, the directional indications will also change accordingly.

[0053] The following describes a method for generating subtitles according to an exemplary embodiment of the present application in conjunction with Figures 1-3 It should be noted that the following application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the embodiments of the present application are not limited in this respect. On the contrary, the embodiments of the present application can be applied to any applicable scenario.

[0054] Further, the present application also proposes a method and device for generating subtitles, a driving vehicle and a medium.

[0055] Figure 1 A flowchart of a method for generating subtitles according to an embodiment of the present application is schematically shown. As Figure 1 shown, the method is applied to a subtitle generation model comprising an insert-style control module, comprising:

[0056] S101, obtaining a candidate word set predicted by a subtitle generation model for a to-be-processed image at a current time, and obtaining at least one prefix word corresponding to the candidate word set predicted by the subtitle generation model in a first historical period.

[0057] S102, selecting a target word with the target style and a repetition rate lower than a preset standard from the candidate word set by using the plug-in style control module.

[0058] S103, combining the at least one prefix word and the target word as target caption content of the image to be processed.

[0059] In the related art, image caption display is a basic task in the cross direction of natural language processing and computer vision. With the rise of generative models, users have begun to focus on generating image captions with specific styles. The essence of the stylized image caption task is to generate text caption content with a specific style that can summarize the image given a picture.

[0060] In one way, the generation method of the image caption can be: combining a pre-trained CLIP model with a large language model: inputting the image features extracted by CLIP into the large language model after adjustment, and generating the target style caption through the model fine-tuning of the style.

[0061] In one way, the existing stylized image caption generation work is mainly completed through reinforcement learning and contrastive learning, even with the assistance of large models such as CLIP and GPT, multiple training of the model is still needed to generate the target style caption. And previous work all need image-text pairs or general-specific style corpus pairs to help the model training.

[0062] However, under the caption generation method in the prior art, there are problems such as single type of stylized caption dataset, high cost of constructing image-text pairs and corpus pairs of target style. And this leads to the fact that most of the stylized image caption work can only generate captions of a single fixed style category.

[0063] Further, in actual application, business personnel usually need to generate captions of specific styles matching users or images according to different preferences of users or different styles of images, that is, cannot be limited to fixed style categories in the style caption dataset.

[0064] In addition, in the text generation part, most of the work uses decoding methods based on maximization (such as beam search, etc.) to guide the large language model to generate captions. But the text content generated by these decoding methods sometimes faces problems such as poor fluency of generating repetitive words, which affects the sentence quality of the image caption and further affects the viewing experience of users.

[0065] To solve the above problems, the application provides a subtitle generation method, which is characterized by designing a plugging stylized control module (PSCM) to assist an existing image subtitle generation model to generate an open style subtitle matching an image.

[0066] As an example, the PSCM is a plug-and-play module that can be directly plugged into any trained image subtitle generation model. In other words, the embodiment of the application can generate a target style category subtitle for a given image of any style without the need to train an additional stylized subtitle generation model.

[0067] Further, the PSCM proposed in the embodiment of the application can be composed of a style factor and a fluency factor. The style factor can be used to control the subtitle generation model to select a next word with target style characteristics. The fluency factor can select a candidate word with low similarity by calculating the similarity between the candidate word and the predicted historical word, thereby reducing the possibility of repeated words in the subtitle and improving the fluency of the sentence.

[0068] As shown in Figure 2 , the embodiment of the application further illustrates the scheme of the subtitle generation method based on the plugging stylized control module PSCM.

[0069] Step 1, obtaining a candidate word set predicted by the subtitle generation model for the to-be-processed image at the current time, and obtaining at least one prefix word corresponding to the candidate word set predicted by the subtitle generation model in the first historical period. Then go to step 2a and step 2b.

[0070] In one way, as shown in Figure 3 , the overall system architecture diagram of the subtitle generation method proposed in the application is shown.

[0071] As Figure 2 can be seen, the to-be-processed image is an image containing a dog. In one way, the embodiment of the application can first predict the prefix word (i.e. Image of a) of the subtitle content of the image according to the subtitle generation model.

[0072] As can be understood, the prefix word is the prefix part of the subtitle content of the image. That is, one or more prefix words corresponding to the candidate word set predicted by the subtitle generation model in the first historical period. As can be understood, the complete subtitle content of the image is composed of the prefix word and the determined target word.

[0073] Further, the subtitle generation model needs to continue to predict the next word at this time, which can be specifically selecting the target word from multiple candidate words (i.e., a candidate word set, including green, black, dog, grass, etc.). In this process, the style factor and the fluency factor need to participate in the decision together.

[0074] Step 2a, using the plug-in style control module, calculating the style factor of each candidate word in the candidate word set. Then go to step 3.

[0075] Wherein, the style factor is used to represent the closeness of the style of the candidate word to the target style.

[0076] In one way, the prefix word and each candidate word can be input into the text style classifier of the plug-in style control module, so that the text style classifier identifies the style score corresponding to each candidate word; and the style score is taken as the style factor corresponding to the candidate word.

[0077] Wherein, the style factor can be determined by the score of the text style classifier. In one way, the embodiment of the present application can pre-train a binary text style classifier (Classifier L ) by fine-tuning the GPT model according to the multiple sample images i and the corresponding multiple style categories (L) of the letter content, using the target text and the text part of the classic subtitle dataset MS-COCO.

[0078] Further, the embodiment of the present application obtains the multiple prefix words (x ＜t ) generated by the decoder of the subtitle generation model before the t time (i.e., the current time), and the multiple candidate words (v) predicted by the subtitle generation model at the current time, and combines them as the input of the text style classifier. Then the result of the last layer obtained through the network of the text style classifier is subjected to softmax calculation, so as to obtain the score of the style category corresponding to each candidate word (for example, including the humorous style category, the romantic style category, the formal style category, the horror style category, etc.).

[0079] As an example, the label score of the target style is the style factor. The embodiment of the present application can select the candidate word of a specific style category (i.e., the target style) as the target word according to the user's preference degree or the category of the image and other factors.

[0080] For example, the English of dog can be translated into different words according to different style categories, such as dog (formal style), Puppy (humorous style), Pooch (formal style), doggy (romantic style), etc.

[0081] It can be understood that, taking the formal style as an example of the target style, the candidate words belonging to the target style are dog and Pooch.

[0082] Step 2b, using the plug-in style control module, calculating the fluency factor of each candidate word in the candidate word set. Then go to step 3.

[0083] Wherein, the fluency factor is used to represent the repetition of the candidate word and the history word, and the history word set is the word predicted by the subtitle generation model in the second history period.

[0084] In one way, the embodiments of the present application can combine the prefix word with any one of the candidate words in the candidate word set to obtain a plurality of to-be-compared combinations;

[0085] Comparing each to-be-compared combination with the history word set respectively to determine the similarity value of each to-be-compared combination and each history word.

[0086] Based on the similarity value, the fluency score of each candidate word is determined, and the fluency score is taken as the fluency factor corresponding to the candidate word.

[0087] Wherein, the fluency factor can be determined by the calculation of cosine similarity. In one way, the embodiments of the present application can obtain a plurality of history words V (k) predicted by the subtitle generation model in the second history period (i.e. the set of the first k words predicted by the subtitle generation model before t time). And calculate the similarity s(h v ,h xj ) of each candidate word v and any history word generated before t time.

[0088] Wherein, h v and h xj represent the mapping of the candidate word v and any history word generated before t time in the model respectively.

[0089] In one way, the calculation of similarity can be realized by calculating the cosine similarity of two words.

[0090] In one way, the embodiments of the present application can also set the second weight coefficient β of the control penalty term in the fluency factor.

[0091] It can be understood that, the greater β is, the lower the score of the candidate word v causing the repetition of the sentence is. Therefore, the generation of repeated words can be reduced and the fluency of the sentence can be increased by selecting the candidate word with higher score.

[0092] Step 3, obtaining the first weight coefficient corresponding to the style factor and the second weight coefficient corresponding to the fluency factor.

[0093] Step 4, based on the first weight coefficient and the second weight coefficient, calculate the comprehensive score of each candidate word between the style factor and the fluency factor, and take the candidate word with the highest comprehensive score as the target word.

[0094] In one way, the embodiment of the present application can utilize the weight coefficients of the style factor and the fluency factor to perform weight summation on each candidate word, and take the word with the highest comprehensive score of the two factors as the target word.

[0095] Step 5, after sequentially combining the prefix word and the target word, input them into the decoder of the subtitle generation model to obtain the target subtitle content with the target style.

[0096] Specifically, the embodiment of the present application can comprehensively calculate the total score of the two factors in the text decoding process of the image subtitle model. By setting the first weight coefficient a and the second weight coefficient β, the word with the highest score in the candidate word set is selected and combined with the prefix, and then input into the decoder again, and finally the stylized and fluent subtitles with different degrees are obtained.

[0097] Step 6, display the target subtitle content on the image to be processed.

[0098] It can be understood that the embodiment of the present application can generate target style subtitles simply and effectively with the help of the existing trained image subtitle generation model. It no longer depends on the image-text pair and the general-special style corpus pair model for fine-tuning, and the inference process is simple and the effect is obvious.

[0099] By applying the technical solution of the present application, the existing image subtitle generation model can be assisted by the stylization control module to generate a subtitle content with a specific style category for each image. Specifically, on the one hand, the style factor control module can be used to control the subtitle generation model to select a word with more specific style characteristics for the image. In order to achieve the purpose of outputting the subtitle content preferred by different users to a higher degree. On the other hand, the fluency factor is used to calculate the similarity between the predicted word and the previously predicted word to select a word with a lower repetition rate, thereby reducing the possibility of repeated words in the subtitle and improving the fluency of the sentence.

[0100] Optionally, in another embodiment based on the above-mentioned method of the present application, the step of selecting a target word with a target style and a repetition rate lower than a preset standard from the candidate word set by using the insertable style control module comprises:

[0101] calculating the style factor of each candidate word in the candidate word set by using the insertable style control module, the style factor being used to represent the closeness of the style of the candidate word to the target style; and

[0102] The insertion style control module is used to calculate fluency factors of each candidate word in the candidate word set, the fluency factors being used to represent repetition of the candidate word and historical words predicted by the subtitle generation model in a second historical period.

[0103] The target word is determined based on the style factor and the fluency factor of each candidate word.

[0104] Optionally, in another embodiment based on the above method of the present application, the use of the insertion style control module to calculate the style factor of each candidate word in the candidate word set comprises:

[0105] The prefix word and each candidate word are input into a text style classifier of the insertion style control module, so that the text style classifier identifies a style score corresponding to each candidate word.

[0106] The style score is taken as the style factor corresponding to the candidate word.

[0107] Optionally, in another embodiment based on the above method of the present application, the use of the insertion style control module to calculate the fluency factor of each candidate word in the candidate word set comprises:

[0108] The prefix word is combined with any one of the candidate words in the candidate word set to obtain a plurality of comparison combinations.

[0109] Each comparison combination is compared with historical words in similarity to determine a similarity value of each comparison combination and each historical word.

[0110] Based on the similarity value, a fluency score of each candidate word is determined.

[0111] The fluency score is taken as the fluency factor corresponding to the candidate word.

[0112] Optionally, in another embodiment based on the above method of the present application, the determination of the target word based on the style factor and the fluency factor of each candidate word comprises:

[0113] A first weight coefficient corresponding to the style factor and a second weight coefficient corresponding to the fluency factor are obtained.

[0114] Based on the first weight coefficient and the second weight coefficient, a comprehensive score of each candidate word between the style factor and the fluency factor is calculated.

[0115] The candidate word with the highest comprehensive score is taken as the target word.

[0116] Optionally, in another embodiment based on the above method of the present application, the combination of the at least one prefix word and the target word as the target subtitle content comprises:

[0117] After the prefix word and the target word are sequentially combined, the combination is input into the decoder of the subtitle generation model to obtain the target subtitle content with the target style.

[0118] In another embodiment of the present application, as shown in Figure 4 The present application also provides a subtitle generation device. The device is applied to a subtitle generation model comprising an insertable style control module, and comprises:

[0119] The obtaining module 201 is configured to obtain a candidate word set predicted by the subtitle generation model for a to-be-processed image at a current time, and obtain at least one prefix word corresponding to the candidate word set predicted by the subtitle generation model at a first historical period.

[0120] The selecting module 202 is configured to select, by using the insertable style control module, a target word with a target style and a repetition rate lower than a preset standard from the candidate word set.

[0121] The generating module 03 is configured to combine the at least one prefix word and the target word as target subtitle content of the to-be-processed image.

[0122] By applying the technical solution of the present application, the existing image subtitle generation model can be assisted by the style control module to generate a subtitle content with a specific style category for each image. Specifically, on the one hand, the style control module can be used to control the subtitle generation model to select a word with more specific style characteristics for the image. In this way, the subtitle content with a higher preference degree for different users can be output. On the other hand, the fluency factor is used to calculate the similarity between the predicted word and the previously predicted word, so as to select a word with a lower repetition degree, thereby reducing the possibility of repeated words in the subtitle and improving the fluency of the sentence.

[0123] In another embodiment of the present application, the selecting module 202 is configured to:

[0124] The insertable style control module is used to calculate a style factor of each candidate word in the candidate word set, and the style factor is used to represent the closeness between the style of the candidate word and the target style; and

[0125] The fluency factor of each candidate word in the candidate word set is calculated by using the insertion style control module, and the fluency factor is used to represent the repetition degree of the candidate word and the historical word predicted by the subtitle generation model in a second historical period.

[0126] The target word is determined based on the style factor and the fluency factor of each candidate word.

[0127] In another embodiment of the present application, the selection module 202 is configured to:

[0128] The prefix word and each candidate word are input into the text style classifier of the insertion style control module, so that the text style classifier identifies the style score corresponding to each candidate word.

[0129] The style score is taken as the style factor corresponding to the candidate word.

[0130] In another embodiment of the present application, the selection module 202 is configured to:

[0131] The prefix word is combined with any one of the candidate words in the candidate word set to obtain a plurality of comparison combinations.

[0132] Each comparison combination is compared with the historical word in similarity, and the similarity value of each comparison combination and each historical word is determined.

[0133] The fluency score of each candidate word is determined based on the similarity value.

[0134] The fluency score is taken as the fluency factor corresponding to the candidate word.

[0135] In another embodiment of the present application, the selection module 202 is configured to:

[0136] A first weight coefficient corresponding to the style factor and a second weight coefficient corresponding to the fluency factor are obtained.

[0137] Based on the first weight coefficient and the second weight coefficient, the comprehensive score of each candidate word between the style factor and the fluency factor is calculated.

[0138] The candidate word with the highest comprehensive score is taken as the target word.

[0139] In another embodiment of the present application, the selection module 202 is configured to:

[0140] After the prefix word and the target word are sequentially combined, they are input into the decoder of the subtitle generation model to obtain the target subtitle content with the target style.

[0141] The embodiments of the present application also provide an electronic device for executing the above-mentioned subtitle generation method. Please refer to Figure 5 which shows a schematic diagram of an electronic device provided by some embodiments of the present application. As shown in Figure 5 , the electronic device 3 comprises a processor 300, a memory 301, a bus 302 and a communication interface 303, the processor 300, the communication interface 303 and the memory 301 are connected through the bus 302; the memory 301 stores a computer program which can be run on the processor 300, and the processor 300 executes the computer program to execute the subtitle generation method provided by any one of the preceding embodiments of the present application.

[0142] The memory 301 can include a high-speed random access memory (RAM: Random Access Memory) and can also include a non-volatile memory such as at least one disk memory. The communication connection between the device network element and at least one other network element is realized through at least one communication interface 303 (which can be wired or wireless), and the Internet, a wide area network, a local network, a metropolitan area network, etc. can be used.

[0143] The bus 302 can be an ISA bus, a PCI bus or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 301 is used to store programs, and the processor 300 executes the programs after receiving execution instructions. The video transmission method disclosed in any one of the preceding embodiments of the present application can be applied to the processor 300 or realized by the processor 300.

[0144] The processor 300 can be an integrated circuit chip with a processing capability of signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware in the processor 300 or the instruction in the form of software. The processor 300 described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a ready programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. Each method, step and logic block diagram disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read only memory, a programmable read only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the storage 301, and the processor 300 reads the information in the storage 301 and combines the hardware to complete the steps of the above method.

[0145] The electronic device provided by the embodiments of the present application and the subtitle generation method provided by the embodiments of the present application have the same beneficial effects as the method adopted, run or implemented by the electronic device.

[0146] The embodiments of the present application also provide a computer readable storage medium corresponding to the subtitle generation method provided by the preceding embodiments. Please refer to Figure 6 The computer readable storage medium shown in the figure is an optical disc 40, and a computer program (i.e. a program product) is stored on the optical disc 40. When the computer program is run by a processor, the video transmission method provided by any of the preceding embodiments is executed.

[0147] It should be noted that examples of the computer readable storage medium can also include, but are not limited to, a phase change memory (PRAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), other types of random access memory (RAM), a read only memory (ROM), an electrically erasable programmable read only memory (EEPROM), a flash memory or other optical, magnetic storage medium, which will not be described one by one here.

[0148] The computer readable storage medium provided by the above embodiments of the present application and the video transmission method provided by the embodiments of the present application have the same beneficial effects as the method adopted, run or implemented by the application program stored in the computer readable storage medium.

[0149] It is noted that

[0150] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail in order not to obscure the understanding of this description.

[0151] Similarly, it is to be understood that the mechanical details of the application that have been set forth in the description above of exemplary embodiments of the application are intended to be illustrative only. As such, changes in form and detail can be made without departing from the spirit, scope, and application of the application. It is therefore desired that what is claimed be supported by the applications described above.

[0152] In addition, those skilled in the art will appreciate that, unless otherwise indicated herein, the various embodiments described herein are not intended to be restricted or limited to any single particular type of implementation. Rather, many modifications, alternative arrangements, and equivalent function can be substituted for the specific structures and techniques described. Accordingly, it is to be understood that the application has been

[0153] The above description is only for the preferred specific embodiments of the present application, and the protection scope of the present application is not limited thereto, and any changes or substitutions within the scope of the present application disclosed herein can be easily conceived by those skilled in the art, and should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of generating subtitles, characterized by, Applied to a subtitle generation model comprising a plug-in style control module, the plug-in style control module is a plug-and-play module for direct insertion into the subtitle generation model that has been trained, the plug-and-play module is composed of a style factor and a fluency factor, the method comprises: Obtaining a candidate word set predicted by the subtitle generation model for a to-be-processed image at a current time, and obtaining at least one prefix word corresponding to the candidate word set predicted by the subtitle generation model in a first historical period; Using the plug-in style control module, selecting a target word with a target style and a repetition rate lower than a preset standard from the candidate word set, comprising: inputting the prefix word and each candidate word into the text style classifier of the plug-in style control module, so that the text style classifier identifies the style score corresponding to each candidate word; the style score is used as the style factor corresponding to the candidate word, and the style factor is used to control the subtitle generation model to select words with the target style for the to-be-processed image; based on the style factor and the fluency factor of each candidate word, the target word is determined; The text style classifier is a binary text style classifier trained in advance according to a plurality of sample images and a plurality of style category subtitle contents corresponding to the plurality of sample images, using target text and the text part of a classic subtitle data set; The fluency factor is used to represent the repetition degree of the candidate word and the historical word, the historical word is a word predicted by the subtitle generation model in a second historical period, the second weight coefficient of the control penalty term is set in the fluency factor, the second weight coefficient is inversely proportional to the score of the candidate word causing sentence repetition, and the increase of the second weight coefficient is used to control the generation of repeated words; The combination of the at least one prefix word and the target word is used as the target subtitle content of the to-be-processed image.

2. The method of claim 1, wherein, The use of the plug-in style control module to select a target word with a target style and a repetition rate lower than a preset standard from the candidate word set also includes: The style factor is used to represent the closeness of the style to which the candidate word belongs to the target style; the fluency factor of each candidate word in the candidate word set is calculated using the plug-in style control module.

3. The method of claim 2, wherein, The use of the plug-in style control module to calculate the fluency factor of each candidate word in the candidate word set comprises: Combining the prefix word with any one of the candidate words in the candidate word set to obtain a plurality of to-be-compared combinations; Comparing each to-be-compared combination with historical words respectively to determine the similarity values of each to-be-compared combination and each historical word; Based on the similarity values, the fluency score of each candidate word is determined; The fluency score is used as the fluency factor corresponding to the candidate word.

4. The method of claim 1, wherein, The determination of the target word based on the style factor and the fluency factor of each candidate word comprises: Obtaining a first weight coefficient corresponding to the style factor and a second weight coefficient corresponding to the fluency factor; based on the first weight coefficient and the second weight coefficient, calculate a comprehensive score of each candidate word between the style factor and the fluency factor; select the candidate word with the highest comprehensive score as the target word.

5. The method of claim 1, wherein, The combination of the at least one prefix word and the target word as the target subtitle content comprises: After the prefix word and the target word are combined in sequence, input the combination into a decoder of the subtitle generation model to obtain the target subtitle content with the target style.

6. A subtitle generation device, characterized in that: The application is applied to a subtitle generation model comprising a plug-in style control module, the plug-in style control module is a plug-and-play module, which is directly inserted into the subtitle generation model which has been trained, the plug-and-play module is composed of a style factor and a fluency factor, and the device comprises: The acquisition module is configured to acquire a candidate word set predicted by the subtitle generation model for a to-be-processed image at a current time, and acquire at least one prefix word corresponding to the candidate word set predicted by the subtitle generation model at a first historical period; The selection module is configured to select a target word with a target style and a repetition rate lower than a preset standard from the candidate word set by using the plug-in style control module; the selection module is specifically configured to input the prefix word and each candidate word into a text style classifier of the plug-in style control module, so that the text style classifier identifies a style score corresponding to each candidate word; the style score is used as a style factor corresponding to the candidate word, the style factor is used to control the subtitle generation model to select a word with the target style for the to-be-processed image; and the target word is determined based on the style factor and the fluency factor of each candidate word. The text style classifier is a binary text style classifier trained in advance according to a plurality of sample images and a plurality of style category subtitle contents corresponding to the plurality of sample images, using target text and a text part of a classic subtitle data set; The fluency factor is used to represent the repetition degree of the candidate word and a historical word, the historical word is a word predicted by the subtitle generation model within a second historical period, the second weight coefficient of a control penalty term is set in the fluency factor, the second weight coefficient is inversely proportional to the score of the candidate word causing sentence repetition, and the increase of the second weight coefficient is used to control the generation of repeated words. The generation module is configured to combine the at least one prefix word and the target word as target subtitle content of the to-be-processed image.

7. An electronic device, comprising: comprise: a memory for storing executable instructions; and a processor for executing the executable instructions with the memory to complete the operations of the subtitle generation method in any one of claims 1-5.

8. A computer readable storage medium storing computer readable instructions, characterized in that, The instructions are executed by the processor to perform the operations of the subtitle generation method in any one of claims 1-5. The instructions are executed by the processor to perform the operations of the subtitle generation method in any one of claims 1-5.

Images