Training data processing method, model training method and category identification method and device

Abstract

The invention provides a training data processing method, a model training method and a category recognition method and device, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like, and can be applied to multi-modal data processing. The specific implementation scheme is as follows: acquiring a to-be-processed video of a target category; based on the to-be-processed video, at least one group of image text pairs are generated, and the image text pairs comprise texts and images with matched contents; based on the texts and the images in the at least one group of image text pairs, generating image-text interlaced data with alternately arranged images and texts; and based on the image-text interlaced data, determining training data of the pre-training model, the training data being used for training the pre-training model to identify the target category.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to technical fields such as computer vision, deep learning, and large models, and can be applied to multimodal data processing, in particular to a training data processing method, a model training method, and a category recognition method and device. Background Art

[0002] In model training, training data is a key factor affecting model training performance. In order to train a model with better performance, more and higher-quality training data is often required. Based on this, how to obtain training data to improve model training performance has become an important technical issue in the field of model training. Summary of the invention

[0003] The present disclosure provides a training data processing method, a model training method, a category identification method and a device.

[0004] According to a first aspect of the present disclosure, a training data processing method is provided, comprising: obtaining a to-be-processed video of a target category; generating at least one set of image-text pairs based on the to-be-processed video, wherein the image-text pairs include text and images with matching contents; generating image-text interleaved data in which images and text are alternately arranged based on the text and images in the at least one set of image-text pairs; and determining training data of a pre-trained model based on the image-text interleaved data, wherein the training data is used to train the pre-trained model to recognize the target category.

[0005] According to a second aspect of the present disclosure, a model training method is provided, comprising: obtaining training data of a pre-trained model, wherein the training data comprises interlaced image-text data in which images and text are arranged alternately, the interlaced image-text data is generated based on at least one group of image-text pairs generated from a to-be-processed video of a target category, and is generated based on text and images in the at least one group of image-text pairs; based on the training data, training the pre-trained model for category recognition to obtain a category recognition model.

[0006] According to a third aspect of the present disclosure, a category recognition method is provided, comprising: obtaining data to be recognized; inputting the data to be recognized into a category recognition model to obtain a category output by the category recognition model for the data to be recognized; wherein the category recognition model is obtained by training a pre-trained model based on training data, the training data comprising interleaved image-text data in which images and texts are arranged alternately, the interleaved image-text data is generated based on at least one group of image-text pairs generated from a to-be-processed video of a target category, and is generated based on text and images in the at least one group of image-text pairs.

[0007] According to a fourth aspect of the present disclosure, a training data processing device is provided, comprising: an acquisition module for acquiring a to-be-processed video of a target category; a first generation module for generating at least one set of image-text pairs based on the to-be-processed video, wherein the image-text pairs include text and images with content matching; a second generation module for generating image-text interleaved data in which images and text are alternately arranged based on the text and images in the at least one set of image-text pairs; and a determination module for determining training data of a pre-trained model based on the image-text interleaved data, wherein the training data is used to train the pre-trained model to recognize the target category.

[0008] According to a fifth aspect of the present disclosure, a model training device is provided, comprising: an acquisition module for acquiring training data of a pre-trained model, wherein the training data comprises interlaced image-text data in which images and text are arranged alternately, and the interlaced image-text data is generated based on at least one group of image-text pairs generated from a to-be-processed video of a target category, and is generated based on text and images in the at least one group of image-text pairs; and a training module for training the pre-trained model for category recognition based on the training data to obtain a category recognition model.

[0009] According to a sixth aspect of the present disclosure, a category recognition device is provided, comprising: an acquisition module for acquiring data to be recognized; a processing module for inputting the data to be recognized into a category recognition model to obtain a category output by the category recognition model for the data to be recognized; wherein the category recognition model is obtained by training a pre-trained model based on training data, the training data comprising interleaved image-text data in which images and texts are arranged alternately, the interleaved image-text data is generated based on at least one group of image-text pairs generated from a video to be processed of a target category, and is generated based on text and images in the at least one group of image-text pairs.

[0010] According to a seventh aspect of the present disclosure, there is provided an electronic device, including:

[0011] at least one processor; and

[0012] a memory communicatively connected to the at least one processor; wherein,

[0013] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the training data processing method as described in the first aspect, or execute the model training method as described in the second aspect, or execute the category recognition method as described in the third aspect.

[0014] According to an eighth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the training data processing method as described in the first aspect, or to execute the model training method as described in the second aspect, or to execute the category identification method as described in the third aspect.

[0015] According to a ninth aspect of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the training data processing method as described in the first aspect, or implement the model training method as described in the second aspect, or implement the steps of the category identification method as described in the third aspect.

[0016] The present disclosure provides a training data processing method, a model training method, a category identification method and a device, which have the following beneficial effects:

[0017] Obtain a to-be-processed video of a target category; based on the to-be-processed video, generate at least one set of image-text pairs, wherein the image-text pairs include text and images with matching content; based on the text and images in the at least one set of image-text pairs, generate image-text interlaced data in which the images and texts are alternately arranged; based on the image-text interlaced data, determine training data for a pre-trained model, wherein the training data is used to train the pre-trained model to recognize the target category. The present disclosure can obtain image-text pairs with matching content based on a video, automatically generate image-text interlaced data with high correlation based on the image-text pairs, and then automatically generate training data for a pre-trained model based on the image-text interlaced data.

[0018] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0020] Figure 1 is a flowchart of a training data processing method provided according to an embodiment of the present disclosure;

[0021] Figure 2 is a flowchart of a training data processing method provided according to another embodiment of the present disclosure;

[0022] Figure 3 is a flowchart of a training data processing method provided according to another embodiment of the present disclosure;

[0023] Figure 4 is a flowchart of a training data processing method provided according to another embodiment of the present disclosure;

[0024] Figure 5 is a flowchart of a model training method provided according to an embodiment of the present disclosure;

[0025] Figure 6 is a flow chart of a category identification method provided according to an embodiment of the present disclosure;

[0026] Figure 7 A structural diagram of a training data processing device provided according to an embodiment of the present disclosure;

[0027] Figure 8 is a flow chart of a model training device provided according to an embodiment of the present disclosure;

[0028] Fig. 9 is a flow chart of a category identification device provided according to an embodiment of the present disclosure;

[0029] Fig.10 It is a block diagram of an electronic device used to implement the training data processing method, model training method or category recognition method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0031] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are all carried out with the user's consent, comply with the relevant laws and regulations, and do not violate public order and good morals.

[0032] When obtaining training data, for models with pure text input and output, such as large language models, it is relatively easy to obtain text training data because text is the main carrier of Internet information. However, for models with image and text input, such as multimodal large language models, it is more difficult to obtain training data such as interleaved image and text data.

[0033] In the related art, there are several solutions for obtaining interlaced image and text data:

[0034] (1) Obtain open source text-image data;

[0035] (2) Using crawler tools to capture graphic and text information on web pages, and after certain pre- and post-processing and quality filtering, generate graphic and text interlaced data;

[0036] (3) Use document parsing tools and optical character recognition (OCR) tools to parse documents and books and construct text-image data;

[0037] (4) Using the text and image data uploaded by users on the content application (Application, APP), construct text and image interleaving data through certain rules or schemes.

[0038] Among the above schemes, the amount of interleaved text and image data obtained by scheme (1) is limited, generally in the tens of millions, which is far less than the data scale required for training the model; scheme (2) can obtain a large amount of Internet text and image data, but since there are a large amount of irrelevant content such as advertisements, QR codes, logos, watermarks, etc. on the web page in addition to the main text, the interleaved text and image data obtained may have problems such as low text quality, low text and image relevance, and low information density; the data source of scheme (3) is books and documents, the information density is generally large, the text quality is also high, and the text and image relevance is generally high, but the content richness of books and documents is low, and the timeliness is also lagging, which affects the generalization of model learning; in addition, some books have fewer images, which limits the output of interleaved text and image data; scheme (4) is difficult to crawl data because the content APP has an anti-crawler mechanism.

[0039] Based on this, the present disclosure proposes a training data processing method, a model training method, a category identification method and a device. The training data processing method, the model training method, the category identification method and the device of the embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0040] It should be noted that the executor of the training data processing method of this embodiment is the training data processing device, the executor of the model training method is the model training device, and the executor of the category recognition method is the category recognition device. The training data processing device or the model training device or the category recognition device can be implemented by software and / or hardware and can be configured in an electronic device.

[0041] Figure 1 It is a flowchart of a training data processing method provided according to an embodiment of the present disclosure.

[0042] like Figure 1 As shown, the training data processing method includes:

[0043] Step 101, obtaining a to-be-processed video of a target category.

[0044] The target category may include one or more categories, the number of videos to be processed may be one or more, and the video to be processed may refer to at least one video among local videos and / or downloaded videos.

[0045] It should be noted that after obtaining local videos and / or downloaded videos, filtering / de-duplication can be performed by setting filtering / de-duplication rules to ensure the quality and knowledge density of the videos, and then obtain the interlaced text and image data based on the retained videos. Among them, the filtering rules are used to indicate the metadata of the video, such as the number of views, the number of likes, the author, the number of followers of the author, etc., to filter the videos and remove low-quality videos and videos containing sensitive data, such as filtering out videos with too low a number of views and filtering out videos with banned words in the text.

[0046] As an example, the target categories include technology, life and knowledge, etc.; as another example, the target categories may include animals, plants, vehicles and buildings, etc. It should be noted that the target category may also be a small category obtained by further subdividing a large category. For example, taking the large category of animals as an example, the target category may include dogs, cats and rabbits, etc.

[0047] It should be noted that the target category may be a predefined category, or may be a category obtained by classifying the video to be processed after obtaining the video to be processed.

[0048] Step 102: Generate at least one set of image-text pairs based on the video to be processed, wherein the image-text pairs include text and images with matching contents.

[0049] The text in the image-text pair is used to describe the content of the image in the image-text pair.

[0050] As an example, at least one frame of image is extracted from the video to be processed; for any target image in the at least one frame of image, the text of the target image is generated by a large model, or the text of the target image is given artificially; the target image and the corresponding text are combined to obtain a corresponding image-text pair.

[0051] Step 103: Generate text-image interleaved data in which the text and the image are alternately arranged based on the text and the image in at least one set of image-text pairs.

[0052] As an example, for any set of image-text pairs, the text and the image therein are arranged alternately to obtain corresponding image-text interleaved data.

[0053] As another example, for any set of image-text pairs, feature extraction is performed on the text and image respectively to obtain corresponding text features and image features; the text features and image features with matching contents are alternately arranged to obtain corresponding text-image interleaved data.

[0054] For example, the image in the image-text pair includes image element A and image element B, and the text in the image-text pair includes text describing image element A and image element B. The image features of image element A and the text features of image element A can be arranged alternately, and the image features of image element B and the text features of image element B can be arranged alternately to obtain corresponding image-text interleaved data.

[0055] Step 104, based on the interleaved text and image data, determine the training data of the pre-trained model, and the training data is used to train the pre-trained model to recognize the target category.

[0056] Among them, the interleaved image and text data obtained from the video to be processed based on the target category can be classified into the same data set to obtain the training data of the pre-trained model.

[0057] In the disclosed embodiment, the pre-trained model can be trained based on the training data to perform category recognition to obtain a category recognition model. It should be noted that the category recognition model can be embedded in the large model, that is, category recognition is one of the processes in the large model processing process, and the large model can also perform text search, image search and other processes according to the recognized categories.

[0058] For example, the big model can refer to a question-answering model that generates responses based on text and images input by users, a classification model for classifying images and / or videos, and an image generation model based on artificial intelligence (AI). Among them, the AI ​​image generation model can identify the category of the reference image input by the user or the reference image obtained by the query, and generate images according to the identified category.

[0059] Compared with the aforementioned solution (1), the present disclosure can obtain a large amount of videos, and thus can obtain a large amount of text-image interlaced data; compared with solution (2), the present disclosure can ensure the quality and knowledge density of the videos through reasonable coarse screening, and can obtain images and texts matching the content based on the videos, so the relevance of the text-image interlaced data is also guaranteed; compared with solution (3), the timeliness and richness of videos are much higher than books and documents, so the timeliness and richness of text-image interlaced data are also better; compared with solution (4), the crawling difficulty of videos is lower, and most Internet platforms have their own exclusive video ecosystems, so the present disclosure can obtain a large amount of videos at a relatively low cost.

[0060] In an embodiment of the present disclosure, a to-be-processed video of a target category is obtained; based on the to-be-processed video, at least one set of image-text pairs is generated, wherein the image-text pairs include text and images with matching content; based on the text and images in the at least one set of image-text pairs, image-text interlaced data with alternating image and text is generated; based on the image-text interlaced data, training data of a pre-trained model is determined, and the training data is used to train the pre-trained model to recognize the target category. The present disclosure can obtain image-text pairs with matching content based on a video, automatically generate image-text interlaced data with high correlation based on the image-text pairs, and then automatically generate training data for a pre-trained model based on the image-text interlaced data.

[0061] Figure 2 It is a flowchart of a training data processing method provided according to another embodiment of the present disclosure.

[0062] like Figure 2 As shown, the training data processing method includes:

[0063] Step 201, obtaining a to-be-processed video of a target category.

[0064] Step 202: segment the video to be processed to obtain at least one video segment.

[0065] As an example, the video to be processed may be segmented according to a set length to obtain at least one video segment, or the video to be processed may be segmented randomly to obtain at least one video segment.

[0066] Step 203: extract key frames from any target video segment in the at least one video segment to obtain at least one corresponding frame of image.

[0067] As an example, key frames are extracted from the target video clip through a key frame extraction model to obtain at least one corresponding frame of image; or, key frames selected when encoding the video to be processed are used as at least one corresponding frame of image; or, key frames are extracted from the target video clip at uniform and equal intervals to obtain at least one corresponding frame of image.

[0068] Step 204: Obtain the text segment corresponding to the target video segment.

[0069] As an example, a text segment corresponding to a target video segment is generated by a large model.

[0070] As another example, speech recognition is performed on the video to be processed, and / or text recognition is performed on the image in the video to be processed to obtain the text matching the video to be processed; based on the start time and end time of the target video segment, a text segment corresponding to the target video segment is extracted from the text matching the video to be processed.

[0071] Among them, an automatic speech recognition (Automatic Speech Recognition, ASR) algorithm can be used to perform speech recognition on the video to be processed, and the audio of the video to be processed can be converted into text.

[0072] Among them, an OCR algorithm can be used to perform text recognition on the image in the video to be processed, so as to extract the subtitles in the image and / or the text located in other positions in the image.

[0073] It should be noted that the text obtained by speech recognition and / or text recognition has timestamp information, that is, the time period corresponding to the text in the video. After obtaining the text matching the video to be processed, the text segment corresponding to the target video segment can be intercepted from the text matching the video to be processed according to the start time and end time of the target video segment in combination with the timestamp information corresponding to the text.

[0074] The present invention obtains the text matching the video to be processed through speech recognition technology and / or text recognition technology, and captures the corresponding text segment according to the time range of the target video segment, thereby ensuring that the image and text in the image-text pair are derived from video segments of the same time period and similar, thereby enhancing the correlation between the image and text in the image-text pair.

[0075] Step 205: obtaining an image-text pair corresponding to the target video segment based on at least one frame of image and a text segment corresponding to the target video segment.

[0076] In the case where the target video segment corresponds to multiple frames of images, each frame of image is combined with the text segment corresponding to the target video segment to obtain a corresponding image-text pair.

[0077] Step 206: Generate text-image interleaved data in which the text and the image are alternately arranged based on the text and the image in at least one set of image-text pairs.

[0078] Step 207, determining the training data of the pre-trained model based on the interleaved text and image data.

[0079] It should be noted that the explanation of step 201, step 206 and step 207 can be found in the relevant description in any embodiment of the present disclosure, and will not be repeated here.

[0080] In the disclosed embodiment, the video to be processed is segmented to obtain at least one video segment; key frames are extracted from any target video segment in the at least one video segment to obtain at least one corresponding frame of image; a text segment corresponding to the target video segment is obtained; and an image-text pair corresponding to the target video segment is obtained based on at least one frame of image and text segment corresponding to the target video segment. The disclosed segmentation of the video to be processed helps to reduce the amount of data processed at a single time and improve the efficiency of video processing, and the extraction of key frames and the acquisition of text segments based on video segments can ensure that the key frame images and text segments are closely associated in time, thereby improving the correlation between the image and the text in the image-text pair.

[0081] Figure 3 It is a flowchart of a training data processing method provided according to another embodiment of the present disclosure.

[0082] like Figure 3 As shown, the training data processing method includes:

[0083] Step 301, obtaining a to-be-processed video of a target category.

[0084] Step 302: for any frame of target image in the video to be processed, obtain adjacent frame images of the target image.

[0085] The target image may refer to any frame image in the images constituting the video to be processed, and the adjacent frame image of the target image may refer to the previous frame image of the target image or the next frame image of the target image.

[0086] Step 303: determining the degree of picture change between the target image and the adjacent frame image based on the pixel values ​​of the target image and the adjacent frame image at at least one pixel position.

[0087] The at least one pixel position may refer to all pixel positions or to a specified pixel position; the picture change degree is used to indicate whether the video to be processed enters the next scene picture from the target image.

[0088] As an example, for any pixel position, obtain the difference between the pixel value of the target image at the pixel position and the pixel value of the adjacent frame image at the pixel position; perform averaging processing based on the difference corresponding to at least one pixel position to obtain the degree of picture change.

[0089] The difference corresponding to at least one pixel position is averaged to obtain a difference mean, and the difference mean is used as the degree of picture change.

[0090] Calculating the difference in pixel position between the target image and the adjacent frame images can capture subtle pixel changes between video frames. Averaging the differences between pixel values ​​can eliminate the impact of fluctuations in individual pixels and more accurately reflect the overall changes in pixel values ​​in the corresponding image area, thereby achieving accurate quantification of the degree of picture change between video frames.

[0091] As another example, pixel values ​​of the target image at at least one pixel position are averaged to obtain a first mean value; pixel values ​​of an adjacent frame image are averaged at at least one pixel position to obtain a second mean value; and the difference between the first mean value and the second mean value is used as the degree of picture change.

[0092] The pixel position when calculating the first mean value and the pixel position when calculating the second mean value are the same pixel position.

[0093] Calculating the mean pixel value based on the target image and adjacent frame images can not only reduce the amount of calculation, but also reduce the impact of individual pixel outliers or noise on the quantification of the degree of picture change, and improve the stability and reliability of the quantification results. In addition, the mean difference can intuitively reflect the degree of change in overall brightness, color, etc. between the target image and the adjacent frame images. Based on the mean difference, the boundaries of the video segmentation can be determined more accurately, ensuring that each video clip contains complete and relatively independent content, and improving the quality of video segmentation.

[0094] Step 304 : when the degree of picture change is greater than a set threshold, the target image and adjacent frame images are divided into different video segments.

[0095] If the degree of picture change is greater than a set threshold, it means that the video enters the next scene. At this time, the target image and adjacent frame images can be divided into different video segments.

[0096] Step 305: extract key frames from any target video segment in the at least one video segment to obtain at least one corresponding frame of image.

[0097] Step 306: Obtain the text segment corresponding to the target video segment.

[0098] Step 307: obtaining an image-text pair corresponding to the target video segment based on at least one frame of image and a text segment corresponding to the target video segment.

[0099] Step 308: Generate text-image interleaved data in which the text and the image are alternately arranged based on the text and the image in at least one set of image-text pairs.

[0100] Step 309: Determine the training data of the pre-trained model based on the interleaved text and image data.

[0101] It should be noted that the explanation of step 301, step 305 and step 309 can be found in the relevant description in any embodiment of the present disclosure, and will not be repeated here.

[0102] In the disclosed embodiment, for any frame of target image in the video to be processed, the adjacent frame images of the target image are obtained; based on the pixel values ​​of the target image and the adjacent frame images at at least one pixel position, the degree of picture change between the target image and the adjacent frame images is determined; when the degree of picture change is greater than a set threshold, the target image and the adjacent frame images are divided into different video segments. Video segmentation is performed by using the difference between the target image and the adjacent frame images at the pixel level, which can accurately capture the dynamic changes of the video content and effectively distinguish different scenes or events.

[0103] Figure 4 It is a flowchart of a training data processing method provided according to another embodiment of the present disclosure.

[0104] like Figure 4 As shown, the training data processing method includes:

[0105] Step 401, obtaining a to-be-processed video of a target category.

[0106] Step 402: for any frame of target image in the video to be processed, obtain the adjacent frame images of the target image.

[0107] Step 403: determining the degree of picture change between the target image and the adjacent frame image based on the pixel value of the target image and the adjacent frame image at at least one pixel position.

[0108] Step 404 , when the degree of picture change is greater than a set threshold, the target image and adjacent frame images are divided into different video segments.

[0109] Step 405: extract key frames from any target video segment in at least one video segment to obtain at least one corresponding frame of image.

[0110] Step 406: Obtain the text segment corresponding to the target video segment.

[0111] Step 407 : determining the content similarity between the text segment corresponding to the target video segment and any one of the at least one image frame corresponding to the target video segment.

[0112] Among them, content similarity is used to indicate the matching degree of content between the image and the text segment.

[0113] When the target video segment corresponds to multiple frames of images, the content similarity between the text segment corresponding to the target video segment and each frame of image is calculated respectively.

[0114] As an example, the content similarity between an image and a text segment may be determined by an image-text matching model, such as a Contrastive Language-Image Pre-Training (CLIP) model.

[0115] Step 408 : Based on the content similarity, a target image matching the text segment corresponding to the target video segment is selected from at least one frame of image corresponding to the target video segment.

[0116] The image corresponding to the maximum value of the content similarity may be used as the target image that best matches the text segment corresponding to the target video segment.

[0117] As an example, based on the size relationship between content similarity and the content similarity threshold, candidate images with content similarity greater than or equal to the content similarity threshold are screened from at least one frame of image corresponding to the target video segment; and the target image is selected from the candidate images.

[0118] As an example, the method also includes: when the content similarity between any frame image corresponding to the target video segment and the text segment corresponding to the target video segment is less than the content similarity threshold, merging the text segment corresponding to the target video segment into the text segment corresponding to the associated video segment; wherein the associated video segment is an adjacent video segment of the target video segment, or the text similarity between the text segment corresponding to the target video segment and the text segment corresponding to the associated video segment is greater than the text similarity threshold.

[0119] Among them, the content similarities are all less than the content similarity threshold, indicating that the image does not match the text segment; the content similarities between any frame image corresponding to the target video segment and the text segment corresponding to the target video segment are all less than the content similarity threshold, indicating that the text segment does not match the target video segment.

[0120] When the content of the text segment does not match the image content of the target video segment, merging the text segment corresponding to the target video segment into the text segment corresponding to the associated video segment can effectively integrate the video and text information, avoid the generation of useless text segments, and improve the utilization rate of text information.

[0121] Step 409: Combine the target image and the text segment corresponding to the target video segment to obtain an image-text pair corresponding to the target video segment.

[0122] The target image and the corresponding text segment are combined into an image-text pair, which ensures the relevance between the image and the text in the image-text pair.

[0123] Step 410: Generate text-image interleaved data in which the text and the image are alternately arranged based on the text and the image in at least one set of image-text pairs.

[0124] Step 411, determining the training data of the pre-trained model based on the interleaved text and image data.

[0125] In the case where the target category includes multiple categories, as an example, the number of interleaved image-text data corresponding to any category in the target category is obtained; based on the number, the interleaved image-text data corresponding to any category in the target category is category balanced; and the interleaved image-text data after category balance is used as training data.

[0126] Among them, category balance is used to balance the proportion of interleaved image and text data in each category.

[0127] As an example, for a category whose quantity is greater than or equal to a set quantity threshold, the text-image interlaced data under the category is deleted according to the set deletion quantity and deletion rule.

[0128] As an example, for categories whose quantity is less than a set quantity threshold, the quantity is increased according to the setting, and the to-be-processed video corresponding to the category is re-acquired to obtain a certain amount of graphic and text interlaced data based on the to-be-processed video.

[0129] Category balancing is performed on the interleaved text and image data under each category to ensure the comprehensiveness and balance of the training data. This not only helps to improve the model's recognition accuracy for each category, but also avoids prediction bias caused by data skew.

[0130] In the disclosed embodiment, for any one of the at least one frame of images corresponding to the target video segment, the content similarity between the text segment corresponding to the target video segment is determined; based on the content similarity, a target image matching the text segment corresponding to the target video segment is selected from the at least one frame of images corresponding to the target video segment; the target image and the text segment corresponding to the target video segment are combined to obtain an image-text pair corresponding to the target video segment. Based on the content similarity, the target image that best matches the text segment corresponding to the target video segment can be screened out, ensuring a high correlation between the image and the text in the image-text pair, and providing an accurate and reliable data basis for subsequent model training.

[0131] Figure 5 It is a flowchart of a model training method provided according to an embodiment of the present disclosure.

[0132] like Figure 5 As shown, the model training method includes:

[0133] Step 501, obtaining training data for a pre-trained model, wherein the training data includes interleaved image-text data in which images and text are arranged alternately, and the interleaved image-text data is generated based on at least one set of image-text pairs generated from a to-be-processed video of a target category, and based on text and images in at least one set of image-text pairs.

[0134] Step 502: Based on the training data, the pre-trained model is trained to perform category recognition to obtain a category recognition model.

[0135] In an embodiment of the present disclosure, training data for a pre-trained model is obtained, wherein the training data includes interlaced image-text data in which images and texts are arranged alternately, and the interlaced image-text data is generated based on at least one set of image-text pairs generated from a to-be-processed video of a target category, and is generated based on texts and images in at least one set of image-text pairs; based on the training data, a pre-trained model is trained to perform category recognition, and a category recognition model is obtained. The present disclosure can obtain image-text pairs with matching content through a to-be-processed video, automatically generate image-text interlaced data with high correlation based on the image-text pairs, and then automatically generate training data for a pre-trained model based on the image-text interlaced data, with high efficiency in obtaining training data, high efficiency in model training, and good training effect.

[0136] Figure 6 It is a flowchart of a category identification method provided according to an embodiment of the present disclosure.

[0137] like Figure 6 As shown, the category identification method includes:

[0138] Step 601, obtaining data to be identified.

[0139] The data to be identified includes image and text data, or video and text data, or image data / video data.

[0140] Step 602: input the data to be identified into the category identification model to obtain the category output by the category identification model for the data to be identified.

[0141] Among them, the category recognition model is obtained by training a pre-trained model based on training data, and the training data includes image-text interlaced data in which images and text are arranged alternately. The image-text interlaced data is generated based on at least one set of image-text pairs generated from a to-be-processed video of the target category, and is generated based on text and images in at least one set of image-text pairs.

[0142] In the disclosed embodiment, data to be identified is obtained; the data to be identified is input into a category identification model to obtain a category output by the category identification model for the data to be identified; wherein the category identification model is obtained by training a pre-trained model based on training data, the training data includes interleaved image-text data in which images and texts are alternately arranged, the interleaved image-text data is generated based on at least one set of image-text pairs generated from a to-be-processed video of a target category, and is generated based on text and images in at least one set of image-text pairs. The interleaved image-text data is generated based on images and texts that match content, and therefore, the category identification model trained based on the interleaved image-text data can output accurate categories for the data to be identified.

[0143] Figure 7 It is a structural diagram of a training data processing device provided according to an embodiment of the present disclosure.

[0144] like Figure 7 As shown, the training data processing device includes:

[0145] An acquisition module 701 is used to acquire a to-be-processed video of a target category;

[0146] A first generating module 702, configured to generate at least one set of image-text pairs based on the video to be processed, wherein the image-text pairs include text and images with matching contents;

[0147] The second generating module 703 is used to generate image-text interleaved data in which images and texts are alternately arranged based on the text and image in at least one set of image-text pairs;

[0148] The determination module 704 is used to determine the training data of the pre-training model based on the image and text interlaced data, and the training data is used to train the pre-training model to identify the target category.

[0149] As a possible implementation of the embodiment of the present disclosure, the first generating module 702 is used to:

[0150] Segmenting the video to be processed to obtain at least one video segment;

[0151] Extracting key frames from any target video segment in at least one video segment to obtain at least one corresponding frame of image;

[0152] Get the text segment corresponding to the target video segment;

[0153] Based on at least one frame of image and a text segment corresponding to the target video segment, an image-text pair corresponding to the target video segment is obtained.

[0154] As a possible implementation of the embodiment of the present disclosure, the first generating module 702 is used to:

[0155] For any frame of target image in the video to be processed, obtain the adjacent frame images of the target image;

[0156] Determining a degree of picture change between the target image and the adjacent frame image based on pixel values ​​of the target image and the adjacent frame image at at least one pixel position;

[0157] When the degree of picture change is greater than a set threshold, the target image and adjacent frame images are divided into different video segments.

[0158] As a possible implementation of the embodiment of the present disclosure, the first generating module 702 is used to:

[0159] For any pixel position, obtaining the difference between the pixel value of the target image at the pixel position and the pixel value of the adjacent frame image at the pixel position;

[0160] Averaging is performed according to the difference corresponding to at least one pixel position to obtain the degree of picture change.

[0161] As a possible implementation of the embodiment of the present disclosure, the first generating module 702 is used to:

[0162] Performing averaging processing on pixel values ​​of at least one pixel position of the target image to obtain a first mean value;

[0163] Performing averaging processing on pixel values ​​at at least one pixel position of adjacent frame images to obtain a second mean value;

[0164] The difference between the first mean and the second mean is used as the degree of picture change.

[0165] As a possible implementation of the embodiment of the present disclosure, the first generating module 702 is used to:

[0166] For any frame image of at least one frame image corresponding to the target video segment, determining the content similarity between the text segment corresponding to the target video segment and the text segment corresponding to the target video segment;

[0167] Based on the content similarity, selecting a target image matching the text segment corresponding to the target video segment from at least one frame of image corresponding to the target video segment;

[0168] The target image and the text segment corresponding to the target video segment are combined to obtain an image-text pair corresponding to the target video segment.

[0169] As a possible implementation of the embodiment of the present disclosure, the device further includes:

[0170] a merging module, configured to merge the text segment corresponding to the target video segment into the text segment corresponding to the associated video segment when the content similarity between any frame image corresponding to the target video segment and the text segment corresponding to the target video segment is less than a content similarity threshold;

[0171] The associated video segment is an adjacent video segment of the target video segment, or the text similarity between the text segment corresponding to the target video segment and the text segment corresponding to the associated video segment is greater than a text similarity threshold.

[0172] As a possible implementation of the embodiment of the present disclosure, the first generating module 702 is used to:

[0173] Performing speech recognition on the video to be processed, and / or performing text recognition on the image in the video to be processed to obtain text matching the video to be processed;

[0174] According to the start time and end time of the target video segment, a text segment corresponding to the target video segment is intercepted from the text matched by the video to be processed.

[0175] As a possible implementation of the embodiment of the present disclosure, the determination module 704 is used to:

[0176] Get the number of interleaved text and image data corresponding to any category in the target category;

[0177] Based on the quantity, class balance is performed on the interleaved text and image data corresponding to any category in the target category;

[0178] The image-text interleaved data after category equalization is used as training data.

[0179] It should be noted that the aforementioned explanation of the training data processing method is also applicable to the training data processing device of this embodiment and will not be repeated here.

[0180] Figure 8 It is a structural diagram of a model training device provided according to an embodiment of the present disclosure.

[0181] like Figure 8 As shown, the model training device includes:

[0182] Acquisition module 801 is used to acquire training data for a pre-trained model, wherein the training data includes interleaved image-text data in which images and text are arranged alternately, and the interleaved image-text data is generated based on at least one set of image-text pairs generated from a to-be-processed video of a target category, and is generated based on text and images in at least one set of image-text pairs.

[0183] The training module 802 is used to train the pre-trained model for category recognition based on the training data to obtain a category recognition model.

[0184] It should be noted that the above explanation of the model training method is also applicable to the model training device of this embodiment and will not be repeated here.

[0185] Fig. 9 It is a structural diagram of a category identification device provided according to an embodiment of the present disclosure.

[0186] like Fig. 9 As shown, the category identification device includes:

[0187] The acquisition module 901 is used to acquire data to be identified.

[0188] The processing module 902 is used to input the data to be identified into the category identification model to obtain the category output by the category identification model for the data to be identified.

[0189] Among them, the category recognition model is obtained by training a pre-trained model based on training data, and the training data includes image-text interlaced data in which images and text are arranged alternately. The image-text interlaced data is generated based on at least one set of image-text pairs generated from a to-be-processed video of the target category, and is generated based on text and images in at least one set of image-text pairs.

[0190] It should be noted that the above explanation of the category identification method is also applicable to the category identification device of this embodiment and will not be repeated here.

[0191] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0192] Fig.10 A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device 1000 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0193] like Fig.10As shown, the electronic device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 1002 or a computer program loaded from a storage unit 1008 to a RAM (Random Access Memory) 1003. In the RAM 1003, various programs and data required for the operation of the electronic device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An I / O (Input / Output) interface 1005 is also connected to the bus 1004.

[0194] Multiple components in the electronic device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0195] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as training data processing methods, model training methods, and category identification methods. For example, in some embodiments, the training data processing methods, model training methods, and category identification methods may be implemented as computer software programs, which are tangibly included in machine-readable media, such as storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by computing unit 1001, one or more steps of the training data processing method, model training method, and category identification method described above may be performed. Alternatively, in other embodiments, computing unit 1001 may be configured to perform the training data processing method, model training method, or category identification method in any other appropriate manner (e.g., by means of firmware).

[0196] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor that may be a dedicated or general-purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0197] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0198] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0199] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0200] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0201] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0202] It should be noted that artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0203] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0204] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A training data processing method, comprising: Get the video to be processed of the target category; Based on the video to be processed, generating at least one set of image-text pairs, wherein the image-text pairs include text and images with matching contents; Based on the text and the image in the at least one set of image-text pairs, generating image-text interleaved data in which the images and the text are alternately arranged; Based on the image-text interlaced data, training data of a pre-trained model is determined, and the training data is used to train the pre-trained model to recognize the target category.

2. The method according to claim 1, wherein: The step of generating at least one set of image-text pairs based on the video to be processed comprises: Segmenting the video to be processed to obtain at least one video segment; Extracting key frames from any target video segment in the at least one video segment to obtain at least one corresponding frame of image; Obtaining a text segment corresponding to the target video segment; Based on the at least one frame of image corresponding to the target video segment and the text segment, an image-text pair corresponding to the target video segment is obtained.

3. The method according to claim 2, wherein the step of segmenting the video to be processed to obtain at least one video segment comprises: For any frame of target image in the video to be processed, obtaining adjacent frame images of the target image; Determining a degree of picture change between the target image and the adjacent frame image based on pixel values ​​of the target image and the adjacent frame image at at least one pixel position; When the degree of picture change is greater than a set threshold, the target image and the adjacent frame images are divided into different video segments.

4. The method according to claim 3, wherein: The determining, based on the pixel values ​​of the target image and the adjacent frame image at at least one pixel position, the degree of picture change between the target image and the adjacent frame image comprises: For any pixel position, obtaining a difference between a pixel value of the target image at the pixel position and a pixel value of the adjacent frame image at the pixel position; Averaging processing is performed on the difference corresponding to the at least one pixel position to obtain the degree of picture change.

5. The method according to claim 3, wherein: The determining, based on the pixel values ​​of the target image and the adjacent frame image at at least one pixel position, the degree of picture change between the target image and the adjacent frame image comprises: Performing averaging processing on pixel values ​​of the target image at the at least one pixel position to obtain a first mean value; Performing averaging processing on pixel values ​​of the adjacent frame images at the at least one pixel position to obtain a second mean value; The difference between the first mean value and the second mean value is used as the picture change degree.

6. The method according to claim 2, wherein: The obtaining, based on the at least one frame of image and the text segment corresponding to the target video segment, an image-text pair corresponding to the target video segment includes: For any one of the at least one frame of images corresponding to the target video segment, determining the content similarity between the text segment corresponding to the target video segment and the text segment corresponding to the target video segment; Based on the content similarity, selecting a target image matching the text segment corresponding to the target video segment from at least one frame of image corresponding to the target video segment; The target image and the text segment corresponding to the target video segment are combined to obtain an image-text pair corresponding to the target video segment.

7. The method according to claim 6, wherein: The method further comprises: When the content similarity between any frame image corresponding to the target video segment and the text segment corresponding to the target video segment is less than a content similarity threshold, merging the text segment corresponding to the target video segment into the text segment corresponding to the associated video segment; The associated video segment is an adjacent video segment of the target video segment, or the text similarity between the text segment corresponding to the target video segment and the text segment corresponding to the associated video segment is greater than a text similarity threshold.

8. The method according to claim 2, wherein: The obtaining of the text segment corresponding to the target video segment includes: Performing speech recognition on the video to be processed, and / or performing text recognition on the image in the video to be processed to obtain text matched by the video to be processed; According to the start time and the end time of the target video segment, a text segment corresponding to the target video segment is intercepted from the text matched by the video to be processed.

9. The method according to any one of claims 1 to 8, wherein: The target category includes a plurality of categories, and the determining of the training data of the pre-training model based on the image-text interlaced data includes: Obtain the number of graphic and text interlaced data corresponding to any category in the target category; Based on the quantity, class balancing is performed on the interleaved text and image data corresponding to any class in the target class; The image-text interleaved data after category equalization is used as the training data.

10. A model training method, comprising: Acquire training data of a pre-trained model, wherein the training data includes interlaced image-text data in which images and texts are arranged alternately, and the interlaced image-text data is generated based on at least one set of image-text pairs generated from a to-be-processed video of a target category, and based on texts and images in the at least one set of image-text pairs; Based on the training data, the pre-training model is trained to perform category recognition to obtain a category recognition model.

11. A category identification method, comprising: Obtaining data to be identified; Inputting the data to be identified into a category identification model to obtain a category output by the category identification model for the data to be identified; Among them, the category recognition model is obtained by training a pre-trained model based on training data, and the training data includes image-text interlaced data in which images and text are arranged alternately. The image-text interlaced data is generated based on at least one group of image-text pairs generated from a to-be-processed video of the target category, and is generated based on the text and image in the at least one group of image-text pairs.

12. A training data processing device, comprising: An acquisition module is used to acquire the video to be processed of the target category; A first generating module, configured to generate at least one set of image-text pairs based on the video to be processed, wherein the image-text pairs include text and images with matching contents; A second generating module, configured to generate image-text interleaved data in which images and texts are alternately arranged based on the texts and images in the at least one set of image-text pairs; A determination module is used to determine the training data of the pre-trained model based on the image and text interlaced data, and the training data is used to train the pre-trained model to identify the target category.

13. The device according to claim 12, wherein: The first generating module is used for: Segmenting the video to be processed to obtain at least one video segment; Extracting key frames from any target video segment in the at least one video segment to obtain at least one corresponding frame of image; Obtaining a text segment corresponding to the target video segment; Based on the at least one frame of image corresponding to the target video segment and the text segment, an image-text pair corresponding to the target video segment is obtained.

14. The device according to claim 13, wherein: The first generating module is used for: For any frame of target image in the video to be processed, obtaining adjacent frame images of the target image; Determining a degree of picture change between the target image and the adjacent frame image based on pixel values ​​of the target image and the adjacent frame image at at least one pixel position; When the degree of picture change is greater than a set threshold, the target image and the adjacent frame images are divided into different video segments.

15. The device according to claim 14, wherein: The first generating module is used for: For any pixel position, obtaining a difference between a pixel value of the target image at the pixel position and a pixel value of the adjacent frame image at the pixel position; Averaging processing is performed on the difference corresponding to the at least one pixel position to obtain the degree of picture change.

16. The device according to claim 14, wherein: The first generating module is used for: Performing averaging processing on pixel values ​​of the target image at the at least one pixel position to obtain a first mean value; Performing averaging processing on pixel values ​​of the adjacent frame images at the at least one pixel position to obtain a second mean value; The difference between the first mean value and the second mean value is used as the picture change degree.

17. The device according to claim 13, wherein: The first generating module is used for: For any one of the at least one frame of images corresponding to the target video segment, determining the content similarity between the text segment corresponding to the target video segment and the text segment corresponding to the target video segment; Based on the content similarity, selecting a target image matching the text segment corresponding to the target video segment from at least one frame of image corresponding to the target video segment; The target image and the text segment corresponding to the target video segment are combined to obtain an image-text pair corresponding to the target video segment.

18. The device according to claim 17, wherein: The device also includes: a merging module, configured to merge the text segment corresponding to the target video segment into the text segment corresponding to the associated video segment when the content similarity between any frame image corresponding to the target video segment and the text segment corresponding to the target video segment is less than a content similarity threshold; The associated video segment is an adjacent video segment of the target video segment, or the text similarity between the text segment corresponding to the target video segment and the text segment corresponding to the associated video segment is greater than a text similarity threshold.

19. The device according to claim 13, wherein: The first generating module is used for: Performing speech recognition on the video to be processed, and / or performing text recognition on the image in the video to be processed to obtain text matched by the video to be processed; According to the start time and the end time of the target video segment, a text segment corresponding to the target video segment is intercepted from the text matched by the video to be processed.

20. The device according to any one of claims 12 to 19, wherein: The determination module is used for: Obtain the number of graphic and text interlaced data corresponding to any category in the target category; Based on the quantity, class balancing is performed on the interleaved text and image data corresponding to any class in the target class; The image-text interleaved data after category equalization is used as the training data.

21. A model training device, comprising: An acquisition module, used to acquire training data of a pre-trained model, wherein the training data includes interlaced image-text data in which images and texts are arranged alternately, and the interlaced image-text data is generated based on at least one set of image-text pairs generated from a to-be-processed video of a target category, and based on texts and images in the at least one set of image-text pairs; The training module is used to train the pre-trained model to perform category recognition based on the training data to obtain a category recognition model.

22. A category identification device, comprising: An acquisition module, used for acquiring data to be identified; A processing module, used for inputting the data to be identified into a category identification model to obtain a category output by the category identification model for the data to be identified; Among them, the category recognition model is obtained by training a pre-trained model based on training data, and the training data includes image-text interlaced data in which images and text are arranged alternately. The image-text interlaced data is generated based on at least one group of image-text pairs generated from a to-be-processed video of the target category, and is generated based on the text and image in the at least one group of image-text pairs.

23. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any one of claims 1 to 9, or the method of claim 10, or the method of claim 11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the method of any one of claims 1 to 9, or the method of claim 10, or the method of claim 11.

25. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 9, or the method according to claim 10, or the method according to claim 11.

Images