Visual language model training method, video character recognition method, device and equipment

Abstract

The invention provides a visual language model training method, a video character recognition method, devices and equipment, and the visual language model training method comprises the steps: obtaining a training video data set, carrying out the separation processing of each training video, obtaining an audio stream and a training video stream corresponding to each training video, carrying out the voice recognition of the audio streams, and obtaining a training video stream corresponding to each training video; obtaining a training text sequence corresponding to each training video, constructing a guide prompt word sequence of each training video according to the training text sequence, and performing recognition by adopting a preset initial multi-mode visual language model according to each key frame in the training video stream and the guide prompt word sequence of each training video, and obtaining video picture positions of the plurality of recognized subtitle texts in the corresponding key frames, and training the initial multi-modal visual language model according to the video picture positions of the plurality of recognized subtitle texts in the corresponding key frames, the labeled subtitle texts and the picture positions of the labeled subtitle texts to obtain a target multi-modal visual language model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of character recognition, in particular to a visual language model training method, a video character recognition method, a device and equipment. BACKGROUND

[0002] The current mainstream video character detection and recognition technology combined with optical character recognition (OCR) is mainly based on the idea of image processing and deep learning fusion, and the core is divided into two technical paths of single frame processing and multi-modal fusion. In the single frame processing scheme, the technical process usually first locates the text area in the single frame of the video through a text detection algorithm, and then uses an optical character recognition model to extract the text content, and finally completes the recognition output of the single frame text. With the rise of multi-modal large models, general multi-modal models are gradually applied to this scene. Such models rely on the visual-language alignment ability learned in the pre-training process to directly perform global visual understanding and text extraction on video frames, trying to simplify the detection and recognition process through an end-to-end manner, and adapting to a wider range of video scenes with the powerful generalization ability of the model. In addition, some schemes will introduce a simple inter-frame information reuse mechanism, such as preliminarily screening the detection results of overlapping text areas in consecutive frames to reduce the cost of repeated calculation.

[0003] Although the existing VideoOCR technology has realized the leap from traditional algorithms to multi-modal large models, it still exposes many key shortcomings in actual application. First, the text recognition ability in complex scenes is insufficient. The visual encoder of the general multi-modal model is designed with global scene understanding as the core target, and has limited ability to perceive the details of low-resolution and small-size text in the video, which is prone to miss detection and misrecognition. SUMMARY

[0004] The purpose of the present application is to provide a visual language model training method, a video character recognition method, a device and equipment to guide the attention mechanism of the initial multi-modal visual language model by inputting the guide prompt word sequence of each training video as prior information, so as to comprehensively improve the recognition accuracy, processing efficiency, cross-modal understanding ability and scene generalization of the target multi-modal visual language model obtained by training.

[0005] To achieve the above purpose, the technical solutions adopted by the embodiments of the present application are as follows: In a first aspect, the embodiments of the present application provide a visual language model training method, comprising: obtaining a training video data set, the training video data set comprising: a plurality of training videos, each training video having labeled subtitle text and the position of the labeled subtitle text in the key frame; performing separate processing on each training video to obtain an audio stream and a training video stream corresponding to each training video; performing speech recognition on the audio stream corresponding to each training video to obtain a training text sequence corresponding to each training video, the training text sequence including a plurality of recognized subtitle texts and start and end times corresponding to the plurality of recognized subtitle texts; constructing a guide prompt word sequence of each training video according to the training text sequence; performing recognition on each key frame in the training video stream corresponding to each training video and the guide prompt word sequence of each training video using a preset initial multi-modal visual language model to obtain video picture positions of the plurality of recognized subtitle texts corresponding to the key frames; training the initial multi-modal visual language model according to the video picture positions of the plurality of recognized subtitle texts corresponding to the key frames, the annotated subtitle texts, and picture positions of the annotated subtitle texts to obtain a target multi-modal visual language model.

[0006] In an optional implementation, the initial multi-modal visual language model includes a visual encoder and a visual recognition module; the visual encoder includes a global encoder, a local encoder, and a fusion module. The performing recognition on each key frame in the training video stream corresponding to each training video and the guide prompt word sequence of each training video using a preset initial multi-modal visual language model to obtain video picture positions of the plurality of recognized subtitle texts corresponding to the key frames includes: performing visual feature coding on each key frame in the training video stream corresponding to each training video and the guide prompt word sequence of each training video using the global encoder to obtain a global visual feature map corresponding to each training video; performing visual feature coding on each key frame in the training video stream corresponding to each training video and the guide prompt word sequence of each training video using the local encoder to obtain a local visual feature map corresponding to each training video; performing feature fusion on the global visual feature map and the local visual feature map using the fusion module to obtain a visual feature map corresponding to each training video; performing recognition on the visual feature map corresponding to each training video using the visual recognition module to obtain video picture positions of the plurality of recognized subtitle texts corresponding to the key frames.

[0007] In an optional implementation, the step of training the initial multimodal visual language model to obtain a target multimodal visual language model based on the video frame positions of the multiple identified subtitle texts, the labeled subtitle texts, and the frame positions where the labeled subtitle texts are located includes: Based on the video frame positions of the multiple identified subtitle texts in the corresponding keyframes, the labeled subtitle texts, and the frame positions where the labeled subtitle texts are located, the fusion weights of the fusion module in the initial multimodal visual language model are adjusted to obtain the target multimodal visual language model.

[0008] In an optional implementation, the plurality of training videos includes: virtual training videos, and the method further includes: A pre-defined large oracle model is used to generate a text oracle library; The virtual training video is generated based on the text library; The initial multimodal visual language model is trained based on the virtual training video to obtain the target multimodal visual language model.

[0009] In an optional implementation, generating the virtual training video based on the text library includes: Get the preset video scene; Based on the preset video scene and the text library, a virtual training file is generated; The virtual training file is annotated with subtitles to obtain the virtual training video.

[0010] In an optional implementation, the plurality of training videos includes: real training videos, and the method further includes: Obtain the actual video file; Subtitles are annotated on the real video file to obtain the real training video; The initial multimodal visual language model is trained based on the real training video to obtain the target multimodal visual language model.

[0011] In an optional implementation, the method further includes: The visual encoder is trained based on the speech recognition database.

[0012] Secondly, embodiments of this application also provide a video text recognition method, including: Obtain the video to be recognized; The video to be identified is separated to obtain the audio stream and video stream corresponding to the video to be identified; Speech recognition is performed on the audio stream corresponding to the video to be identified to obtain the recognition text sequence corresponding to the video to be identified. The recognition text sequence includes: multiple subtitle texts and the start and end times of the multiple subtitle texts. Based on the identified text sequence, construct the guidance prompt word sequence for the video to be identified; Based on each keyframe in the video stream corresponding to the video to be identified and the sequence of guiding prompts in the video to be identified, a preset target multimodal visual language model is used for identification to obtain the video screen positions of the multiple subtitle texts in the corresponding keyframes; the target multimodal visual language model is a model trained using any of the above-mentioned first aspects; The subtitle file of the video to be identified is generated based on the multiple subtitle texts, the start and end times of the multiple subtitle texts, and the video frame positions of the multiple subtitle texts in the corresponding keyframes.

[0013] Thirdly, embodiments of this application also provide a visual language model training device, the device comprising: The first acquisition module is used to acquire a training video dataset, which includes: multiple training videos, and each training video has labeled subtitle text and the screen position where the labeled subtitle text is located in the keyframes of the video. The first separation module is used to separate each training video to obtain the audio stream and training video stream corresponding to each training video; The first recognition module is used to perform speech recognition on the audio stream corresponding to each training video to obtain a training text sequence corresponding to each training video. The training text sequence includes: multiple recognition subtitle texts and the start and end times corresponding to the multiple recognition subtitle texts. The first construction module is used to construct a sequence of guiding prompt words for each training video based on the training text sequence; The first recognition module is used to recognize the multiple recognized subtitle texts in the video frame of the corresponding key frame based on each key frame in the training video stream corresponding to each training video and the sequence of guiding prompt words in each training video, using a preset initial multimodal visual language model. The training module is used to train the initial multimodal visual language model based on the video frame positions of the multiple recognized subtitle texts in the corresponding keyframes, the labeled subtitle texts, and the screen positions where the labeled subtitle texts are located, to obtain the target multimodal visual language model.

[0014] Fourthly, embodiments of this application also provide a video text recognition device, including: The second acquisition module is used to acquire the video to be recognized; The second separation module is used to separate the video to be identified to obtain the audio stream and video stream corresponding to the video to be identified; The second recognition module is used to perform speech recognition on the audio stream corresponding to the video to be recognized, and obtain the recognition text sequence corresponding to the video to be recognized. The recognition text sequence includes: multiple subtitle texts and the start and end times of the multiple subtitle texts. The second construction module is used to construct a sequence of guiding prompt words for the video to be identified based on the identified text sequence; The second recognition module is further configured to recognize the multiple subtitle texts in the video frame corresponding to the video stream and the sequence of guiding prompts in the video stream according to each key frame, and to obtain the video frame positions of the multiple subtitle texts in the corresponding key frames; the target multimodal visual language model is a model trained using any one of the first aspects mentioned above. The second generation module is used to generate a subtitle file for the video to be identified based on the multiple subtitle texts, the start and end times of the multiple subtitle texts, and the video frame positions of the multiple subtitle texts in the corresponding keyframes.

[0015] Fifthly, embodiments of this application also provide a computer device, including: a processor, a storage medium, and a bus, wherein the storage medium stores program instructions executable by the processor, and when the computer device is running, the processor communicates with the storage medium via the bus, and the processor executes the program instructions to perform the steps of the visual language model training method as described in any of the first aspects, or the steps of the video text recognition method as described in the second aspect.

[0016] In a sixth aspect, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the visual language model training method as described in any of the first aspects, or the steps of the video text recognition method as described in the second aspect.

[0017] The beneficial effects of this application are: This application provides a visual language model training method, a video text recognition method, an apparatus, and a device. The visual language model training method includes: acquiring a training video dataset, which includes multiple training videos. Each training video has labeled subtitle text and the screen position of the labeled subtitle text in its keyframes. Each training video is separated to obtain an audio stream and a training video stream corresponding to each training video. Speech recognition is performed on the audio stream corresponding to each training video to obtain a training text sequence corresponding to each training video. The training text sequence includes multiple recognized subtitle texts and the start and end times corresponding to the multiple recognized subtitle texts. Based on the training text sequence, a sequence of guiding prompts for each training video is constructed. Based on each keyframe in the training video stream corresponding to each training video and the sequence of guiding prompts for each training video, a preset initial multimodal visual language model is used for recognition to obtain the video screen positions of the multiple recognized subtitle texts in the corresponding keyframes. Based on the video screen positions of the multiple recognized subtitle texts in the corresponding keyframes, the labeled subtitle texts, and the screen positions of the labeled subtitle texts, the initial multimodal visual language model is trained to obtain a target multimodal visual language model.

[0018] The method in this application guides the attention mechanism of the initial multimodal visual language model by inputting the guidance prompt word sequence of each training video as prior information. This causes the initial multimodal visual language model to prioritize areas in the image that may contain these keywords, thereby significantly narrowing the search range and accelerating the text localization process. Simultaneously, the contextual information provided by the guidance prompt word sequence helps the initial multimodal visual language model eliminate ambiguity among multiple candidate recognition results, allowing it to select the most semantically correct text guidance. Ultimately, this enables the trained target multimodal visual language model to achieve a comprehensive improvement in recognition accuracy, processing efficiency, cross-modal understanding ability, and scene generalization, laying a solid foundation for the leap from simple recognition to comprehensive understanding of images, text, and audio in video text detection and recognition. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is one of the flowcharts illustrating a visual language model training method provided in an embodiment of this application; Figure 2 A second schematic flowchart illustrating a visual language model training method provided in this application embodiment; Figure 3 The third schematic flowchart of a visual language model training method provided in this application embodiment; Figure 4 The fourth flowchart illustrates a visual language model training method provided in this application embodiment; Figure 5 Fifth of a flowchart illustrating a visual language model training method provided in this application embodiment; Figure 6 A flowchart illustrating a video text recognition method provided in an embodiment of this application; Figure 7 A schematic diagram of the functional modules of a visual language model training device provided in an embodiment of this application; Figure 8 This is a schematic diagram of the functional modules of a video text recognition device provided in an embodiment of this application; Figure 9 This is a schematic diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0022] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0023] In the description of this application, it should be noted that if the terms "upper", "lower", etc. appear to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship that the product of this application is usually placed in, it is only for the convenience of describing this application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0024] Furthermore, the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Additionally, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] It should be noted that, where there is no conflict, the features in the embodiments of this application can be combined with each other.

[0026] The visual language model training method provided in this application will be explained in detail below with reference to the accompanying drawings and specific examples. The visual language model training method provided in this application can be implemented by running an algorithm or software pre-installed on a computer device. The computer device can be, for example, a server or a terminal, and the terminal can be a user computer. Figure 1 This is one of the flowcharts illustrating a visual language model training method provided in an embodiment of this application; as shown below. Figure 1 As shown, the method includes: S101. Obtain the training video dataset.

[0027] The training video dataset includes multiple training videos, each with keyframes containing labeled subtitle text and the location of the labeled subtitle text within the frame.

[0028] In this embodiment, the training video dataset contains multiple training videos, covering various typical scenarios such as city streets, supermarket shelves, transportation hubs, office documents, movie clips, and short video advertisements, ensuring scene diversity to meet actual application needs. The duration, frame rate, and resolution of each training video can be uniformly set. For example, the duration of each training video can be 3-30 seconds, the frame rate can be 24-30fps, and the resolution can be uniformly set to 1920×1080.

[0029] Keyframes for each training video are extracted using the PySceneDetect intelligent sampling algorithm, with an average extraction rate of one frame every 1.5 seconds, resulting in 4-20 keyframes per training video. All keyframes undergo double annotation, including annotation of the subtitle text and annotation of the subtitle text's location within the video frame. The subtitle text annotation covers multiple languages, including Chinese, English, Japanese, Korean, French, German, and Arabic, and includes regular fonts, artistic fonts, and irregularly shaped text with curves, rotations, and perspective transformations. The location of the subtitle text within the video frame is annotated using pixel coordinates (x1, y1, x2, y2) to represent the bounding box, accurate to 1 pixel, where x1 and y1 are the coordinates of the top-left corner, and x2 and y2 are the coordinates of the bottom-right corner.

[0030] S102. Perform separation processing on each training video to obtain the audio stream and training video stream corresponding to each training video.

[0031] Specifically, the cross-platform audio and video processor FFmpeg is used to perform batch separation processing on all training videos. For the training video stream corresponding to each training video, FFmpeg is first used for hardware decoding to obtain continuous video frames. Then, PySceneDetect, a powerful video segmentation tool, is used to intelligently sample based on the inter-frame difference algorithm to remove obviously duplicate video frames, resulting in the processed training video stream. As for the audio stream, it is sent to the Whisper-Large-v3 multilingual speech recognition model of the Automatic Speech Recognition (ASR) module.

[0032] S103. Perform speech recognition on the audio stream corresponding to each training video to obtain the training text sequence corresponding to each training video.

[0033] The training text sequence includes: multiple recognized caption texts and the start and end times corresponding to the multiple recognized caption texts.

[0034] Specifically, the Whisper-Large-v3 model is used as the automatic speech recognition (ASR) tool. The extracted audio stream is preprocessed using FFmpeg for noise reduction and volume normalization to ensure stable audio quality. The model's language parameters are set to auto, supporting automatic detection of multiple target languages. Recognition accuracy is set to the highest level, prioritizing text accuracy. After recognition, the core information of each result is output: the recognized subtitle text, start time (accurate to 0.01 seconds), and end time (accurate to 0.01 seconds), forming a training text sequence. For example, the audio stream of a training video, after recognition, yields: [{"text":"New Product Launch, 50% Off","start":2.1,"end":3.82},{"text": "LimitedTimeOffer","start":4.05,"end":5.23}].

[0035] S104. Based on the training text sequence, construct the guidance prompt word sequence for each training video.

[0036] Specifically, based on the timestamps of the training text sequence and the timestamps of keyframes, a unique guiding prompt is constructed for each keyframe. If the timestamp t of a keyframe falls within the time interval of a recognized subtitle text in the training text sequence (start_time≤t≤end_time), a guiding prompt is constructed in the format "refer to ASRresult:[Recognized subtitle text], detect and recognize all text and its bounding boxes in the image", for example, "refer to ASRresult:[50% off], detect and recognize all text and its bounding boxes in the image". If the keyframe has no corresponding ASR recognized text (such as a silent period or only background music), the general prompt "detect and recognize all text and its bounding boxes in the image" is used. The prompts for all keyframes are arranged in chronological order to form a guiding prompt sequence corresponding to each training video, ensuring that the recognition of each keyframe receives targeted guidance.

[0037] S105. Based on each keyframe in the training video stream corresponding to each training video and the sequence of guiding prompts for each training video, a preset initial multimodal visual language model is used for recognition to obtain the video screen positions of multiple recognized subtitle texts in the corresponding keyframes.

[0038] The initial multimodal visual language model adopted was a customized and modified Qwen3-VL-2B model. Keyframes from the training video stream were preprocessed, including normalization and scaling to a model-fit size, and then synchronously input into the initial multimodal visual language model along with the corresponding guidance prompt sequence. The initial multimodal visual language model outputs the position of each recognized subtitle text within the keyframe, using (x1, y1, x2, y2) pixel coordinates to represent the bounding box, ultimately forming a mapping relationship between each recognized subtitle text and its corresponding position.

[0039] S106. Based on the video frame positions of multiple identified subtitle texts in the corresponding keyframes, the labeled subtitle texts, and the frame positions where the labeled subtitle texts are located, the initial multimodal visual language model is trained to obtain the target multimodal visual language model.

[0040] Specifically, by comparing the errors between the video frame positions of multiple identified subtitle texts and the frame positions of the labeled subtitle texts, the parameters in the initial multimodal visual language model are adjusted to obtain the target multimodal visual language model.

[0041] In summary, this application provides a visual language model training method, which includes: acquiring a training video dataset, the training video dataset including: multiple training videos, each training video having labeled subtitle text and the screen position of the labeled subtitle text in its keyframes; separating each training video to obtain an audio stream and a training video stream corresponding to each training video; performing speech recognition on the audio stream corresponding to each training video to obtain a training text sequence corresponding to each training video, the training text sequence including: multiple recognized subtitle texts and the start and end times corresponding to the multiple recognized subtitle texts; constructing a sequence of guiding prompts for each training video based on the training text sequence; using a preset initial multimodal visual language model for recognition based on each keyframe in the training video stream corresponding to each training video and the sequence of guiding prompts for each training video, obtaining the video screen positions of the multiple recognized subtitle texts in the corresponding keyframes; and training the initial multimodal visual language model based on the video screen positions of the multiple recognized subtitle texts in the corresponding keyframes, the labeled subtitle texts, and the screen position of the labeled subtitle texts, to obtain a target multimodal visual language model.

[0042] The method in this application guides the attention mechanism of the initial multimodal visual language model by inputting the guidance prompt word sequence of each training video as prior information. This causes the initial multimodal visual language model to prioritize areas in the image that may contain these keywords, thereby significantly narrowing the search range and accelerating the text localization process. Simultaneously, the contextual information provided by the guidance prompt word sequence helps the initial multimodal visual language model eliminate ambiguity among multiple candidate recognition results, allowing it to select the most semantically correct text guidance. Ultimately, this enables the trained target multimodal visual language model to achieve a comprehensive improvement in recognition accuracy, processing efficiency, cross-modal understanding ability, and scene generalization, laying a solid foundation for the leap from simple recognition to comprehensive understanding of images, text, and audio in video text detection and recognition.

[0043] Based on the visual language model training method provided in the above embodiments, this application also provides another possible implementation example of a visual language model training method. The initial multimodal visual language model includes: a visual encoder and a visual recognition module; the visual encoder includes: a global encoder, a local encoder, and a fusion module. Figure 2 This is a second schematic flowchart illustrating a visual language model training method provided in an embodiment of this application. Figure 2 As shown, based on the keyframes in the corresponding training video stream for each training video and the sequence of guiding prompts for each training video, a preset initial multimodal visual language model is used for recognition, resulting in the positions of multiple recognized subtitle texts in the video frame corresponding to the keyframes, including: S201. Based on each keyframe in the training video stream corresponding to each training video and the sequence of guiding prompts for each training video, a global encoder is used to encode visual features to obtain a global visual feature map corresponding to each training video.

[0044] In this embodiment, the global encoder adopts the original deep learning (Vision Transformer, ViT) architecture based on the self-attention mechanism of Qwen3-VL, and takes pre-processed (normalized and scaled) keyframe images and guidance prompt word encoding information as input. The encoder performs a global scan of the image through 12 Transformer layers, extracting macroscopic features such as the overall layout of the scene, scene type, and spatial relationships of objects. At the same time, it combines the semantic information of the guidance prompt words to enhance the global context awareness related to the text. During the encoding process, the global correlation of the feature maps is maintained, and the output is a global visual feature map with dimensions [B, C, H, W] (B is the batch size, C is the number of feature channels, and H / W is the feature map size), providing scene-level context support for subsequent text localization.

[0045] S202. Based on the keyframes in the training video stream corresponding to each training video and the sequence of guiding prompts for each training video, a local encoder is used to encode the visual features to obtain the local visual feature map corresponding to each training video.

[0046] The local encoder is a modified version of the ImageEncoder based on the Segment Anything Model (SAM). It removes the decoder portion of the original model while retaining high-resolution feature extraction capabilities. It takes the same preprocessed keyframes and prompt word encoding information as the global encoder as input. The encoder preserves high-resolution details of the image through fewer downsampling operations, focusing on capturing local features such as stroke outlines, edge textures, and subtle deformations of the text. For keywords mentioned in the prompt words, the local encoder enhances the extraction of local features in the corresponding regions, outputting dimensions similar to the global visual features. Figure 1 A detailed local visual feature map ensures that details of small-sized, blurred, and artistic fonts are not lost.

[0047] S203. Based on the global visual feature map and the local visual feature map, the fusion module is used to perform feature fusion to obtain the visual feature map corresponding to each training video.

[0048] The fusion module serves as the base class for all custom neural network modules (torch.nn.Module), containing a multi-head attention layer and six linear layers. First, the global and local visual feature maps are converted into feature sequences (Tokens). The multi-head attention layer calculates the correlation weights between the two types of features, and then the linear layers perform dimensional alignment and dynamic weighted fusion. For clear, large text regions in the image, the module automatically assigns higher global feature weights; for small, curved, occluded, or artistic text, it automatically increases the weights of local features. After fusion, a unified visual feature sequence is output, which is then converted into a visual feature map in [B,C,H,W] format.

[0049] S204. Based on the visual feature map corresponding to each training video, the visual recognition module is used to identify the multiple recognized subtitle texts in the video frame corresponding to the key frame.

[0050] Specifically, the visual recognition module is based on a modified version of the Qwen3-VL language model, taking the fused visual feature map and the sequence of guiding prompts as input. The module uses the Transformer's self-attention mechanism to cross-modal associate visual features with text prompts. It first locates regions in the image that may contain the target text, then extracts the text content within these regions using a text recognition head, and calculates the precise bounding box coordinates (x1, y1, x2, y2). During recognition, semantic constraints of the guiding prompts are incorporated to exclude irrelevant interference text, prioritizing the output of recognition results semantically consistent with the ASR results. Ultimately, this yields each recognized subtitle text and its image position mapping within keyframes.

[0051] The method provided in this application captures the overall scene and context information of the video image through a global encoder, and accurately extracts detailed features such as text strokes and edges through a local encoder. Then, the two types of features are dynamically weighted and fused by a fusion module, which effectively balances the needs of global scene understanding and local detail perception. This avoids the problem of insufficient capture of text details in complex scenes by a single encoder, and improves the accuracy of text localization and recognition through cross-modal association of guiding prompts. This enables the model to adaptively handle different types of recognition tasks such as clear large blocks of text, small-sized blurred text, and artistic fonts.

[0052] This application also provides another possible implementation example of a visual language model training method. Based on the video frame positions of multiple identified subtitle texts, the labeled subtitle texts, and the frame positions of the labeled subtitle texts, an initial multimodal visual language model is trained to obtain a target multimodal visual language model, including: Based on the video frame positions of multiple identified subtitle texts, the labeled subtitle texts, and the frame positions of the labeled subtitle texts, the fusion weights of the fusion module in the initial multimodal visual language model are adjusted to obtain the target multimodal visual language model.

[0053] In this embodiment, during model training, a dynamic adaptive strategy is adopted for tuning the fusion weights of the fusion module, with the matching degree between the recognition results and the annotation information in the keyframes as the core optimization criterion. First, two core evaluation metrics are defined: one is the intersection-union ratio (IUGR) between the screen position of the recognized subtitle text and the annotation position, used to measure the positioning accuracy; the other is the character accuracy of the recognized text and the annotation text, used to measure the recognition accuracy. For each training keyframe, the IUGR of the screen position of the annotation subtitle text (e.g., bbox: [320,450,580,520]) and the screen position of the recognized subtitle text output by the model is calculated, and the character accuracy of the recognized text and the annotation text (e.g., the annotation "50% off" and the recognition of "50% off") are compared.

[0054] If the text in a keyframe is in a small-sized artistic font, and the Intersection over Union (IoU) of the recognition result is <0.6 and the character accuracy (CER) is >0.1, it indicates that the local encoder is not capturing local details well enough. In this case, the weight parameters of the fusion module are adjusted through backpropagation to increase the weight of the local encoder features and decrease the weight of the global encoder features, thereby strengthening the model's attention to details such as character strokes and edges.

[0055] The method provided in this application adjusts the fusion weights of the fusion module in the initial multimodal visual language model based on the video frame positions of multiple identified subtitle texts, the labeled subtitle text, and the frame position of the labeled subtitle text. This allows the fusion module to form an adaptive adjustment mechanism. When dealing with clear, large blocks of regular text, it automatically increases the feature weights of the global encoder to fully utilize the scene context to assist in recognition. For small-sized, artistic fonts, blurry, or irregularly shaped text, it flexibly increases the feature weights of the local encoder to enhance text detail capture. This effectively compensates for the shortcomings of traditional single encoders in complex scene text recognition, which tend to prioritize global aspects over details or prioritize details over context. At the same time, by continuously calibrating the weight allocation ratio in conjunction with annotation information, the model's recognition accuracy and robustness are significantly improved in different types of text and different scenes. The resulting target multimodal visual language model can more accurately balance the needs of global scene understanding and local detail perception, greatly reducing the text false detection rate and false recognition rate in complex video scenes, providing core technical support for subsequent efficient video text detection and recognition.

[0056] This application also provides an example of another possible implementation of a visual language model training method. Multiple training videos include: virtual training videos, Figure 3 This is the third flowchart illustrating a visual language model training method provided in an embodiment of this application. Figure 3 As shown, the method also includes: S301. Use a pre-defined large oracle model to generate a text oracle library.

[0057] In this embodiment, the basic dataset is first obtained, including the hierarchical text dataset (HierText), the arbitrary shape text dataset (TotalText), and the synthetic text dataset (SynthText).

[0058] The pre-built large language model uses either the generative pre-trained transformer type 4 (GPT-4) or the large language model Qwen-Plus based on the Transformer architecture to process text from multiple base datasets, generating a multilingual text corpus covering various languages ​​(Chinese, English, Japanese, Korean, French, German, Russian, Thai, etc.). For example, each language contains 150,000 common words, 50,000 short sentences, and 20,000 dialogue texts, covering high-frequency video scenarios such as product promotions, road signs, advertising slogans, and office documents. During the generation process, it is ensured that the text contains different levels of difficulty, including both standard and artistic variations, abbreviations, and dialectal expressions, while maintaining a uniform text length distribution to meet the model's training needs for diverse texts.

[0059] S302. Generate virtual training videos based on the text prediction library.

[0060] Figure 4 This is the fourth flowchart illustrating a visual language model training method provided in this application. Figure 4 As shown, step S302 specifically includes: S401. Obtain the preset video scene.

[0061] The preset video scenes cover a variety of high-frequency application scenarios, including city streets, highway signs, supermarket shelves, convenience store signs, office documents, conference room whiteboards, film and television clips, short video advertisements, online education courseware, airport terminals, train station waiting halls, restaurant menus, outdoor billboards, electronic screens, and handheld shooting scenarios. Video clips are collected through web crawling or virtual scene videos are constructed using 3D modeling tools. For example, each scene collects no fewer than 50 video clips, each 2-5 seconds long, to ensure comprehensive and typical scene coverage.

[0062] S402. Generate virtual training files based on preset video scenes and text libraries.

[0063] Specifically, automated tools are used to match text from a text corpus with preset video scenes. For example, product names and price tags are rendered in a supermarket shelf scene, and place names and directional signs are rendered in a road sign scene. During text rendering, interference factors such as lighting changes (strong light, shadows, backlighting), motion blur (fast-moving scenes), and occlusion (partial text being obscured by objects) in the real environment are simulated to enhance the realism of the virtual training files. At the same time, a corresponding audio stream is synthesized for each virtual training file to ensure that the audio text is semantically consistent with the video text, generating a complete virtual training file containing both video and audio streams.

[0064] S403. Add subtitles to the virtual training files to obtain the virtual training video.

[0065] Specifically, since the text in the virtual training files is generated through manual rendering, the core annotation information for each piece of text can be directly recorded via the rendering script: the text content, its bounding box coordinates (x1, y1, x2, y2) in the frame, and the start and end frames. Simultaneously, the Whisper-Large-v3 model is used to perform speech recognition on the audio stream, generating a sequence of timestamped subtitle text. This annotation information is then associated with the virtual training files to form a complete virtual training video.

[0066] S303. Based on the virtual training video, train the initial multimodal visual language model to obtain the target multimodal visual language model.

[0067] Specifically, the generated virtual training videos are divided into training and validation sets in a 7:3 ratio and then mixed with public datasets and real training videos for model training. During training, the automatically labeled information from the virtual training videos is directly used as training labels without manual intervention. Training with virtual data focuses on optimizing the model's robustness to multilingual and irregularly shaped text, as well as its adaptability to text in different scenarios. Training is iterated alternately with real data to avoid overfitting the model to virtual scenes. Finally, parameters are adjusted based on the performance of the validation set to obtain a target multimodal visual language model with stronger generalization capabilities.

[0068] The method provided in this application uses a large language model to generate a diverse multilingual text corpus and combines it with real-world backgrounds to generate highly realistic virtual training videos. This not only solves the pain points of scarce and costly annotation of real multilingual and multi-form text training data, but also flexibly simulates various complex text scenarios and interference factors, providing the model with rich and diverse training samples. This effectively enhances the model's robustness to multilingual and irregular texts, while reducing the cost and cycle of training data construction, and helping the model to quickly improve its generalization ability.

[0069] This application also provides an example of another possible implementation of a visual language model training method. Multiple training videos include: real training videos, Figure 5 This is the fifth flowchart illustrating a visual language model training method provided in an embodiment of this application. Figure 5 As shown, the method also includes: S501, Obtain the actual video file.

[0070] S502. Add subtitles to real video files to obtain real training videos.

[0071] S503. Based on real training videos, train the initial multimodal visual language model to obtain the target multimodal visual language model.

[0072] In this embodiment, real video files were collected from publicly available internet channels and enterprise cooperation databases, covering various types such as short videos on social media, film and television clips, online education courses, outdoor surveillance videos, and news reports. For example, a total of 600 real videos were collected, ranging in length from 5 to 30 seconds, with resolutions ranging from 720P to 4K. The videos included a variety of complex scenes: shaky handheld shots, outdoor scenes with drastic lighting changes, fast-moving text (such as scrolling text), multilingual mixed text (such as cross-border e-commerce advertisements), and irregularly shaped logos and sign text, ensuring that the real videos could cover various challenges in the actual application of the model.

[0073] The Computer Vision Annotation Tool (CVAT) was used to annotate real video files. The annotation process consisted of three steps: First, keyframes were extracted using PySceneDetect. Then, text content (including multilingual text, artistic fonts, and blurred text) was identified frame by frame, and precise bounding box coordinates (x1, y1, x2, y2) were drawn. Finally, the annotation results were cross-validated. Simultaneously, the video audio track was extracted, and ASR text with timestamps was generated using Whisper-Large-v3. This ASR text was then associated with the manually annotated text content and bounding box information to form a complete real training video, which was used to train the model's generalization ability in real-world scenarios.

[0074] The generated real training videos are divided into training and validation sets in a 7:3 ratio, and then mixed with public datasets and virtual training videos for model training.

[0075] The method provided in this application, by collecting real and complex scene videos and performing precise manual annotation, compensates for the limitations of virtual training data in extreme scenarios (such as hand shaking, drastic changes in lighting) and real text distribution. It provides the model with high-value training samples that are close to actual applications, enabling the model to learn the diverse interference factors and distribution characteristics of text in real scenarios during the training process. This effectively improves the model's recognition reliability and generalization ability in real-world scenarios and avoids the overfitting problem caused by the model's excessive reliance on virtual data.

[0076] This application also provides an example of another possible implementation of a visual language model training method. The method further includes: The visual encoder is trained based on the speech recognition database.

[0077] In this embodiment, during the training phase, the real training video is mixed with a public OCR dataset and virtual training videos. A multi-stage fine-tuning strategy is used to train the initial multimodal visual language model: In the first stage, most language model parameters are frozen, and only the dual visual encoder and fusion module are trained to establish a basic mapping between real-world visual features and text; in the second stage, virtual training videos containing multilingual and irregular text are used to optimize the model's robustness to complex text forms; in the third stage, real video data with ASR timestamps are used to construct an input format that follows instructions, combining images and text prompts. The template format is: image:[video_frame_data], InputPrompt:"refer to ASR result:[Emergency Exit], detect and recognize all the texts and their bbox in the image", for example, Output:text:"Emergency Exit", bbox:[x1,y1,x2,y2]. This yields a target multimodal visual language model adapted to real complex scenes.

[0078] This application also provides a possible implementation example of a video text recognition method. Figure 6 This is a flowchart illustrating a video text recognition method provided in an embodiment of this application. Figure 6 As shown, the method includes: S601. Obtain the video to be recognized.

[0079] In this embodiment, the system receives videos uploaded by users via an API interface, with no limit on video length. Upon receiving the video, it first performs format verification and integrity checks, rejecting damaged, encrypted, or incompatible video files. Simultaneously, it records basic video information to provide parameter data for subsequent processing.

[0080] S602. Perform separation processing on the video to be identified to obtain the audio stream and video stream corresponding to the video to be identified.

[0081] Specifically, FFmpeg is used to perform real-time separation processing on the video to be recognized. For the video stream corresponding to the video to be recognized, FFmpeg is first used for hardware decoding to obtain continuous video frames. Then, PySceneDetect is used for intelligent sampling based on the inter-frame difference algorithm to remove obviously repeated video frames, thus obtaining the video stream. As for the audio stream, it is sent to the Automatic Speech Recognition (ASR) module (Whisper-Large-v3).

[0082] S603. Perform speech recognition on the audio stream corresponding to the video to be recognized to obtain the recognition text sequence corresponding to the video to be recognized.

[0083] The identified text sequence includes: multiple subtitle texts and the start and end times of the multiple subtitle texts.

[0084] Specifically, the Whisper-Large-v3 model is used as the automatic speech recognition (ASR) tool. The extracted audio stream is preprocessed using FFmpeg filters for noise reduction and volume normalization to ensure stable audio quality. The model's language parameters are set to auto, supporting automatic detection of multiple target languages. Recognition accuracy is set to the highest level, prioritizing text accuracy. After recognition, the core information of each result is output: subtitle text, start time (accurate to 0.01 seconds), and end time (accurate to 0.01 seconds), forming the recognized text sequence. For example, [{"text":"New Product Launch","start":2.1,"end":3.5},{"text":"50% Off","start":3.6,"end":5.0}].

[0085] S604. Based on the recognized text sequence, construct a sequence of guiding prompt words for the video to be recognized.

[0086] Specifically, based on the timestamps of the identified text sequence and the timestamps of keyframes, a unique guiding prompt is constructed for each keyframe. If the timestamp t of a keyframe falls within the time interval of a subtitle text in the identified text sequence (start_time≤t≤end_time), a guiding prompt is constructed in the format "refer to ASRresult:[identified subtitle text], detect and identify all text and its bounding boxes in the image", for example, "refer to ASR result:[50% off], detect and identify all text and its bounding boxes in the image". If the keyframe has no corresponding ASR identified text (such as a silent period or only background music), the general prompt "detect and identify all text and its bounding boxes in the image" is used. The prompts for all keyframes are arranged in chronological order to form a guiding prompt sequence corresponding to the video to be identified, ensuring that the identification of each keyframe receives targeted guidance.

[0087] S605. Based on the key frames in the video stream corresponding to the video to be identified and the sequence of guiding prompts in the video to be identified, a preset target multimodal visual language model is used for recognition to obtain the positions of multiple subtitle texts in the video frame corresponding to the key frames.

[0088] Specifically, after preprocessing the keyframes (normalizing and scaling to the model's fit size), they are synchronously input into the target multimodal visual language model, which has been trained using a visual language model training method, along with the corresponding guiding prompts.

[0089] The target multimodal visual language model extracts global and local features through dual visual encoders. After dynamic fusion by the fusion module, and combined with attention guidance from prompts, it accurately locates text regions in the image, identifies text content, and calculates bounding box coordinates (x1, y1, x2, y2). For repeated text in consecutive keyframes (content and position similarity ≥ 90%), the model temporarily stores the results to avoid redundant calculations, improving processing efficiency. Finally, it outputs the mapping relationship between each subtitle text and the corresponding keyframe image position.

[0090] S606. Generate a subtitle file for the video to be recognized based on multiple subtitle texts, the start and end times of the multiple subtitle texts, and the video frame positions of the multiple subtitle texts in the corresponding keyframes.

[0091] The output recognition results undergo post-processing: First, a caching mechanism is used to merge repeated text in consecutive keyframes, recording the earliest and latest timestamps of the text as the start and end times; then, a voting mechanism is used for temporal smoothing, for example, if 4 out of 5 consecutive frames are identified as "stop" and 1 as "stop dry", then it is corrected to "stop"; finally, low-confidence results with a confidence level below 0.9 are removed. The processed text content, start and end times, and screen position (bounding box coordinates) are integrated to generate subtitle files in multiple formats, supporting SRT, VTT, and JSON formats. The JSON format contains structured data, such as: {"texts":[{"content":"50% off", "bbox":[120,340,200,80], "start":3.6,"end":8.2}]}).

[0092] In summary, this application provides a video text recognition method, which includes: acquiring a video to be recognized; separating the video to be recognized to obtain an audio stream and a video stream corresponding to the video to be recognized; performing speech recognition on the audio stream corresponding to the video to be recognized to obtain a recognition text sequence corresponding to the video to be recognized, the recognition text sequence including: multiple subtitle texts and the start and end times of the multiple subtitle texts; constructing a sequence of guiding prompts for the video to be recognized based on the recognition text sequence; using a preset target multimodal visual language model to recognize the multiple subtitle texts at the video frame positions of the corresponding key frames based on each key frame in the video stream corresponding to the video to be recognized and the sequence of guiding prompts for the video to be recognized; and generating a subtitle file for the video to be recognized based on the multiple subtitle texts, the start and end times of the multiple subtitle texts, and the video frame positions of the multiple subtitle texts at the corresponding key frames.

[0093] The method in this application, through audio-video separation of the video to be identified, speech recognition to generate a sequence of recognized text, dynamic construction of guiding prompts, accurate recognition by the target model, and post-processing of the results, not only continues the advantages of high accuracy and high efficiency in the training phase, but also improves the real-time performance and reliability of video text recognition. It can quickly output structured subtitle files to meet the practical application needs of various scenarios such as social media short videos, film and television translation, and online education. At the same time, it supports multiple output formats, improving the flexibility and practicality of use.

[0094] The following will continue to explain the visual language model training device, video text recognition device, and computer equipment provided in any of the above embodiments of this application. The specific implementation process and the resulting technical effects are the same as those in the corresponding method embodiments. For the sake of brevity, parts not mentioned in this embodiment can be referred to the corresponding content in the method embodiments.

[0095] Figure 7 This is a schematic diagram of the functional modules of a visual language model training device provided in an embodiment of this application. Figure 7 As shown, the visual language model training device 100 includes: The first acquisition module 110 is used to acquire a training video dataset, which includes: multiple training videos, and each training video has labeled subtitle text and the screen position where the labeled subtitle text is located in the keyframes of the video. The first separation module 120 is used to separate each training video to obtain the audio stream and training video stream corresponding to each training video; The first recognition module 130 is used to perform speech recognition on the audio stream corresponding to each training video to obtain the training text sequence corresponding to each training video. The training text sequence includes: multiple recognition subtitle texts and the start and end times corresponding to the multiple recognition subtitle texts. The first construction module 140 is used to construct a sequence of guiding prompt words for each training video based on the training text sequence; The first recognition module 130 is used to recognize multiple subtitle texts in the video frame corresponding to each key frame in the training video stream and the sequence of guiding prompts in each training video using a preset initial multimodal visual language model, so as to obtain the video frame positions of the corresponding key frames. The training module 150 is used to train the initial multimodal visual language model based on the video frame positions of multiple recognized subtitle texts in corresponding keyframes, the labeled subtitle texts, and the frame positions where the labeled subtitle texts are located, to obtain the target multimodal visual language model.

[0096] Optionally, the initial multimodal visual language model includes: a visual encoder and a visual recognition module; the visual encoder includes: a global encoder, a local encoder, and a fusion module; The first recognition module 130 is further configured to: encode visual features using a global encoder based on each keyframe in the training video stream corresponding to each training video and the sequence of guiding prompts in each training video; encode visual features using a local encoder based on each keyframe in the training video stream corresponding to each training video and the sequence of guiding prompts in each training video; perform feature fusion using a fusion module based on the global visual feature map and the local visual feature map to obtain a visual feature map corresponding to each training video; and perform recognition using a visual recognition module based on the visual feature map corresponding to each training video to obtain the video frame positions of multiple recognized subtitle texts in the corresponding keyframes.

[0097] Optionally, the training module 150 is also used to adjust the fusion weights of the fusion module in the initial multimodal visual language model based on the video frame positions of multiple recognized subtitle texts in the corresponding keyframes, the labeled subtitle texts, and the frame positions where the labeled subtitle texts are located, so as to obtain the target multimodal visual language model.

[0098] Optionally, the multiple training videos include: virtual training videos, and the device also includes: The first generation module is used to generate a text prediction library using a pre-set large prediction model; and to generate virtual training videos based on the text prediction library. The training module 150 is also used to train the initial multimodal visual language model based on the virtual training video to obtain the target multimodal visual language model.

[0099] Optionally, the first acquisition module 110 is also used to acquire a preset video scene; The first generation module is used to generate virtual training files based on preset video scenes and a text library; and to annotate the virtual training files with subtitles to obtain virtual training videos.

[0100] Optionally, the multiple training videos include: real training videos; the first acquisition module 110 is also used to acquire real video files; and to annotate the real video files with subtitles to obtain real training videos. The training module 150 is also used to train the initial multimodal visual language model based on real training videos to obtain the target multimodal visual language model.

[0101] Optionally, the training module 150 is also used to train the visual encoder based on a speech recognition database.

[0102] Figure 8 This is a schematic diagram of the functional modules of a video text recognition device provided in an embodiment of this application. Figure 8 As shown, the video text recognition device 200 includes: The second acquisition module 210 is used to acquire the video to be recognized; The second separation module 220 is used to perform separation processing on the video to be identified, and obtain the audio stream and video stream corresponding to the video to be identified. The second recognition module 230 is used to perform speech recognition on the audio stream corresponding to the video to be recognized, and obtain the recognition text sequence corresponding to the video to be recognized. The recognition text sequence includes: multiple subtitle texts and the start and end times of the multiple subtitle texts. The second construction module 240 is used to construct a sequence of guiding prompt words for the video to be recognized based on the recognized text sequence; The second recognition module 230 is also used to recognize, based on each key frame in the video stream corresponding to the video to be recognized and the sequence of guiding prompt words in the video to be recognized, using a preset target multimodal visual language model, to obtain the video screen positions of multiple subtitle texts in the corresponding key frames; The second generation module 250 is used to generate a subtitle file for the video to be recognized based on multiple subtitle texts, the start and end times of the multiple subtitle texts, and the video frame positions of the multiple subtitle texts in the corresponding key frames.

[0103] The above-described device is used to execute the method provided in the foregoing embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0104] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more microprocessors, or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SoC).

[0105] Figure 9 This is a schematic diagram of a computer device provided in an embodiment of this application. This computer device can be used for visual language model training or video text recognition. Figure 9As shown, the computer device includes: a processor 310, a storage medium 320, and a bus 330.

[0106] Storage medium 320 stores machine-readable instructions executable by processor 310. When the computer device is running, processor 310 communicates with storage medium 320 via bus 330, and processor 310 executes the machine-readable instructions to perform the steps of the above method embodiment. The specific implementation and technical effects are similar, and will not be described again here.

[0107] Optionally, this application also provides a storage medium 320, on which a computer program is stored. When the computer program is run by a processor, it executes the steps of the above-described method embodiments. The specific implementation and technical effects are similar, and will not be repeated here.

[0108] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0109] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0110] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0111] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute certain steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0112] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A visual language model training method, characterized in that, include: Obtain a training video dataset, which includes: multiple training videos, each training video having labeled subtitle text and the screen position of the labeled subtitle text in its keyframes; Each training video is processed separately to obtain the audio stream and training video stream corresponding to each training video; Speech recognition is performed on the audio stream corresponding to each training video to obtain a training text sequence corresponding to each training video. The training text sequence includes: multiple recognized subtitle texts and the start and end times corresponding to the multiple recognized subtitle texts. Based on the training text sequence, construct a sequence of guiding prompt words for each training video; Based on each keyframe in the training video stream corresponding to each training video and the sequence of guiding prompts in each training video, a preset initial multimodal visual language model is used for recognition to obtain the video screen positions of the multiple recognized subtitle texts in the corresponding keyframes. Based on the video frame positions of the multiple identified subtitle texts in the corresponding keyframes, the labeled subtitle texts, and the screen positions where the labeled subtitle texts are located, the initial multimodal visual language model is trained to obtain the target multimodal visual language model.

2. The method according to claim 1, characterized in that, The initial multimodal visual language model includes a visual encoder and a visual recognition module; the visual encoder includes a global encoder, a local encoder, and a fusion module. The step involves using a preset initial multimodal visual language model to identify the multiple identified subtitle texts based on each keyframe in the corresponding training video stream and the sequence of guiding prompts for each training video, thereby obtaining the video frame positions of the corresponding keyframes. This includes: Based on each keyframe in the training video stream corresponding to each training video and the sequence of guiding prompts in each training video, the global encoder is used to encode visual features to obtain a global visual feature map corresponding to each training video. Based on each keyframe in the training video stream corresponding to each training video and the sequence of guiding prompts for each training video, the local encoder is used to encode visual features to obtain a local visual feature map corresponding to each training video. Based on the global visual feature map and the local visual feature map, the fusion module is used to perform feature fusion to obtain the visual feature map corresponding to each training video. Based on the visual feature map corresponding to each training video, the visual recognition module is used to identify the multiple recognized subtitle texts in the video frame corresponding to the key frame.

3. The method according to claim 2, characterized in that, The step of training the initial multimodal visual language model based on the video frame positions of the multiple identified subtitle texts, the labeled subtitle texts, and the frame positions of the labeled subtitle texts, to obtain the target multimodal visual language model, includes: Based on the video frame positions of the multiple identified subtitle texts in the corresponding keyframes, the labeled subtitle texts, and the frame positions where the labeled subtitle texts are located, the fusion weights of the fusion module in the initial multimodal visual language model are adjusted to obtain the target multimodal visual language model.

4. The method according to claim 1, characterized in that, The plurality of training videos includes: virtual training videos, and the method further includes: A pre-defined large oracle model is used to generate a text prediction library; The virtual training video is generated based on the text library; The initial multimodal visual language model is trained based on the virtual training video to obtain the target multimodal visual language model.

5. The method according to claim 4, characterized in that, The step of generating the virtual training video based on the text library includes: Get the preset video scene; Based on the preset video scene and the text library, a virtual training file is generated; The virtual training file is annotated with subtitles to obtain the virtual training video.

6. The method according to claim 4, characterized in that, The plurality of training videos includes: real training videos, and the method further includes: Obtain the actual video file; Subtitles are annotated on the real video file to obtain the real training video; The initial multimodal visual language model is trained based on the real training video to obtain the target multimodal visual language model.

7. The method according to claim 2, characterized in that, The method further includes: The visual encoder is trained based on the speech recognition database.

8. A video text recognition method, characterized in that, include: Obtain the video to be recognized; The video to be identified is separated to obtain the audio stream and video stream corresponding to the video to be identified; Speech recognition is performed on the audio stream corresponding to the video to be identified to obtain the recognition text sequence corresponding to the video to be identified. The recognition text sequence includes: multiple subtitle texts and the start and end times of the multiple subtitle texts. Based on the identified text sequence, construct the guidance prompt word sequence for the video to be identified; Based on each keyframe in the video stream corresponding to the video to be identified and the sequence of guiding prompts in the video to be identified, a preset target multimodal visual language model is used for identification to obtain the video screen positions of the multiple subtitle texts in the corresponding keyframes; the target multimodal visual language model is a model trained using any one of claims 1 to 7 above. The subtitle file of the video to be identified is generated based on the multiple subtitle texts, the start and end times of the multiple subtitle texts, and the video frame positions of the multiple subtitle texts in the corresponding keyframes.

9. A visual language model training device, characterized in that, The device includes: The first acquisition module is used to acquire a training video dataset, which includes: multiple training videos, and each training video has labeled subtitle text and the screen position where the labeled subtitle text is located in the keyframes of the video. The first separation module is used to separate each training video to obtain the audio stream and training video stream corresponding to each training video; The first recognition module is used to perform speech recognition on the audio stream corresponding to each training video to obtain a training text sequence corresponding to each training video. The training text sequence includes: multiple recognition subtitle texts and the start and end times corresponding to the multiple recognition subtitle texts. The first construction module is used to construct a sequence of guiding prompt words for each training video based on the training text sequence; The first recognition module is used to recognize the multiple recognized subtitle texts in the video frame of the corresponding key frame based on each key frame in the training video stream corresponding to each training video and the sequence of guiding prompt words in each training video, using a preset initial multimodal visual language model. The training module is used to train the initial multimodal visual language model based on the video frame positions of the multiple recognized subtitle texts in the corresponding keyframes, the labeled subtitle texts, and the screen positions where the labeled subtitle texts are located, to obtain the target multimodal visual language model.

10. A computer device, characterized in that, include: The computer device includes a processor, a storage medium, and a bus. The storage medium stores program instructions executable by the processor. When the computer device is running, the processor communicates with the storage medium via the bus. The processor executes the program instructions to perform the steps of the visual language model training method as described in any one of claims 1 to 7, or the steps of the video text recognition method as described in claim 8.

Images