Model training data sample acquisition method and apparatus, electronic device, and storage medium

Abstract

Embodiments of the present application provide a model training data sample acquisition method and device, electronic equipment and a storage medium, relating to the technical field of intelligent analysis, the method comprising: acquiring a source video and an explanatory video corresponding to the source video, the explanatory video containing images and text for explaining the source video; obtaining a matching result of text and a source video segment based on video segments matched with each other between the explanatory video and the source video and the text of the explanatory video; and generating a model training data sample corresponding to the matching result. Through the above method, the time-consuming for acquiring the model training data sample is shortened.

Description

Technical Field

[0001] This application relates to the field of intelligent analysis technology, and in particular to a method, apparatus, electronic device and storage medium for acquiring model training data samples. Background Technology

[0002] In the video domain, search, recommendation, and content creation typically require training multimodal models, which in turn necessitates a large amount of text and video matching data. Current technologies require significant human effort to annotate this data, resulting in lengthy acquisition times for training. Summary of the Invention

[0003] The purpose of this application is to provide a method, apparatus, electronic device, and storage medium for acquiring model training data samples, so as to shorten the time required to acquire model training data samples. The specific technical solution is as follows:

[0004] In a first aspect of this application, a method for obtaining model training data samples is provided, comprising:

[0005] Obtain the source video and the corresponding explanatory video, wherein the explanatory video contains images and text that explain the source video;

[0006] Based on the video segments that match the explanatory video and the source video, and the text of the explanatory video, the matching result between the text and the source video segments is obtained;

[0007] Generate model training data samples corresponding to the matching results.

[0008] Optionally, the step of obtaining the matching result between the text and the source video segment based on the video segment that matches the explanatory video and the text of the explanatory video includes:

[0009] Based on the similarity between frame images in the source video and frame images in the explanatory video, matching segments between the source video and the explanatory video are determined. The matching segments include a first segment and a second segment that match each other. The first segment is a video segment in the explanatory video, and the second segment is a video segment in the source video.

[0010] Perform text recognition on the frame images in the first segment to obtain the text corresponding to the first segment;

[0011] The text is matched with the second fragment to obtain the matching result.

[0012] Optionally, the step of obtaining the matching result between the text and the source video segment based on the video segment that matches the explanatory video and the text of the explanatory video includes:

[0013] Based on the text in the frame images of the explanatory video, determine the correspondence between the time interval of the explanatory video and the text in the explanatory video;

[0014] Based on the similarity between the frame images in the source video and the frame images in the explanatory video, the time intervals in which the explanatory video and the source video match are determined, and the correspondence between the time intervals of the explanatory video and the time intervals of the source video is obtained.

[0015] Based on the correspondence between the time intervals of the explanatory videos and the time intervals of the source videos, as well as the correspondence between the time intervals of the explanatory videos and the text in the explanatory videos, the correspondence between the time intervals of the source videos and the text in the explanatory videos is determined.

[0016] Obtain the source video segment corresponding to the time interval of the source video from the source video, and obtain the matching result between the source video segment and the text.

[0017] Optionally, the step of obtaining the matching result between the text and the source video segment based on the video segment that matches the explanatory video and the text of the explanatory video includes:

[0018] Based on the similarity between the frame images in the source video and the frame images in the explanatory video, the time intervals in which the explanatory video and the source video match are determined, and the correspondence between the time intervals of the explanatory video and the time intervals of the source video is obtained.

[0019] Based on the text in the frame images of the explanatory video, determine the correspondence between the time interval of the explanatory video and the text in the explanatory video;

[0020] Based on the correspondence between the time intervals of the explanatory videos and the time intervals of the source videos, as well as the correspondence between the time intervals of the explanatory videos and the text in the explanatory videos, the correspondence between the time intervals of the source videos and the text in the explanatory videos is determined.

[0021] Obtain the source video segment corresponding to the time interval of the source video from the source video, and obtain the matching result between the source video segment and the text.

[0022] Optionally, generating the model training data samples corresponding to the matching results includes:

[0023] Generate model training data corresponding to a frame of image in the source video segment and model training data corresponding to the text matched by the frame of image to form a model training data sample; or;

[0024] Based on a single frame from the source video segment, multiple frames from the source video segment are merged to generate model training data corresponding to the multiple frames from the source video segment and model training data corresponding to the text matched by the multiple frames, thus forming a model training data sample.

[0025] Optionally, determining the time intervals in which the explanatory video and the source video match based on the similarity of frame images in the source video and frame images in the explanatory video, and obtaining the correspondence between the time intervals of the explanatory video and the time intervals of the source video, includes:

[0026] For each frame in the explanatory video, retrieve similar frame images from the source video that have a similarity of not less than a preset similarity threshold.

[0027] If the similar frame images corresponding to adjacent frame images in the explanatory video meet the preset similarity conditions, then the time of the adjacent frame images is merged to obtain the first explanatory video time interval, and the time of the similar frame images is merged to obtain the source video time interval, wherein the adjacent frame images are the frame images that are temporally adjacent in the explanatory video.

[0028] Establish a correspondence between the time interval of the first explanatory video and the time interval of the source video.

[0029] Optionally, determining the correspondence between the time interval of the explanatory video and the text in the explanatory video based on the text of the frame images in the explanatory video includes:

[0030] For each frame in the explanatory video, a text recognition algorithm is used to perform text recognition on the frame to obtain the text recognition result corresponding to the frame.

[0031] If the text recognition results of adjacent frames in the explanatory video meet the preset matching conditions, then the text recognition results of adjacent frames will be unified as the target text.

[0032] The time intervals of the adjacent frames are merged to obtain the second explanatory video time interval;

[0033] Establish the correspondence between the time interval of the second explanatory video and the target text.

[0034] Optionally, determining the correspondence between the source video time interval and the text in the explanatory video based on the correspondence between the explanatory video time interval and the source video time interval, and the correspondence between the explanatory video time interval and the text in the explanatory video, includes:

[0035] For each time interval in at least one time interval included in the second explanatory video time interval, calculate the degree of overlap between the time interval and each time interval included in the first explanatory video time interval, wherein the degree of overlap is the ratio of the length of the intersection time interval to the length of the union time interval.

[0036] If the overlap is not less than a preset overlap threshold, then the correspondence between the source video time interval and the text is established based on the correspondence between the first explanatory video time interval and the source video time interval, and the correspondence between the second explanatory video time interval and the text.

[0037] Optionally, the similar frame images corresponding to adjacent frame images in the explanatory video satisfying the preset similarity conditions include: there are identical frame images among the similar frame images corresponding to adjacent frame images in the explanatory video, or the time difference between the similar frame images corresponding to adjacent frame images in the explanatory video is less than a preset time difference value.

[0038] Optionally, the text recognition results corresponding to adjacent frames in the explanatory video satisfying the preset matching conditions include: the text recognition results corresponding to adjacent frames in the explanatory video are the same, or the matching degree between the text recognition results corresponding to adjacent frames in the explanatory video is not greater than the preset matching degree.

[0039] Optionally, unifying the text recognition results corresponding to adjacent frame images into the target text includes:

[0040] If the strings in the text recognition results corresponding to adjacent frames are different, the frequency of occurrence of the different strings will be counted, and the text recognition result corresponding to the string with the most occurrences will be taken as the text corresponding to the adjacent frame.

[0041] In a second aspect of this application, a model training data sample acquisition device is also provided, comprising:

[0042] The first acquisition module is used to acquire a source video and an explanatory video corresponding to the source video, wherein the explanatory video includes images and text that explain the source video.

[0043] The matching module is used to obtain the matching result between the text and the source video segment based on the video segments that match each other between the explanatory video and the source video, and the text of the explanatory video;

[0044] The generation module is used to generate model training data samples corresponding to the matching results.

[0045] Optionally, the matching module includes:

[0046] The first determining submodule is used to determine matching segments between the source video and the explanatory video based on the similarity between frame images in the source video and frame images in the explanatory video. The matching segments include a first segment and a second segment that match each other. The first segment is a video segment in the explanatory video, and the second segment is a video segment in the source video.

[0047] The first acquisition submodule is used to perform text recognition on the frame image in the first segment to obtain the text corresponding to the first segment;

[0048] The second acquisition submodule is used to match the text with the second fragment to obtain a matching result.

[0049] Optionally, the matching module includes:

[0050] The second determining submodule is used to determine the correspondence between the time interval of the explanatory video and the text in the explanatory video based on the text of the frame images in the explanatory video.

[0051] The third determining submodule is used to determine the time interval between the explanatory video and the source video that match each other based on the similarity between the frame images in the source video and the frame images in the explanatory video, and to obtain the correspondence between the time interval of the explanatory video and the time interval of the source video.

[0052] The fourth determining submodule is used to determine the correspondence between the time interval of the source video and the text in the explanatory video based on the correspondence between the time interval of the explanatory video and the time interval of the source video, as well as the correspondence between the time interval of the explanatory video and the text in the explanatory video.

[0053] The third acquisition submodule is used to acquire the source video segment corresponding to the time interval of the source video from the source video, and obtain the matching result between the source video segment and the text.

[0054] Optionally, the matching module includes:

[0055] The fifth determining submodule is used to determine the time interval between the explanatory video and the source video that match each other based on the similarity between the frame images in the source video and the frame images in the explanatory video, and to obtain the correspondence between the time interval of the explanatory video and the time interval of the source video.

[0056] The sixth determining submodule is used to determine the correspondence between the time interval of the explanatory video and the text in the explanatory video based on the text of the frame images in the explanatory video.

[0057] The seventh determination submodule is used to determine the correspondence between the time interval of the source video and the text in the explanatory video based on the correspondence between the time interval of the explanatory video and the time interval of the source video, as well as the correspondence between the time interval of the explanatory video and the text in the explanatory video.

[0058] The fourth acquisition submodule is used to acquire the source video segment corresponding to the time interval of the source video from the source video, and obtain the matching result of the source video segment and the text.

[0059] Optionally, the generation module includes:

[0060] The first generation submodule is used to generate model training data corresponding to a frame of image in the source video segment and model training data corresponding to the text matched by the frame of image, to form a model training data sample; or;

[0061] The second generation submodule is used to merge multiple frames of images in the source video segment based on one frame of the image in the source video segment, and generate model training data corresponding to the multiple frames of the image in the source video segment and model training data corresponding to the text matched by the multiple frames of the image, thereby forming a model training data sample.

[0062] Optionally, the fifth determining submodule is specifically used for: for each frame image in the explanatory video, retrieving from the source video similar frame images with a similarity of not less than a preset similarity threshold; if the similar frame images corresponding to adjacent frame images in the explanatory video meet a preset similarity condition, merging the times of the adjacent frame images to obtain a first explanatory video time interval, and merging the times of the similar frame images to obtain a source video time interval, wherein the adjacent frame images are time-adjacent frame images in the explanatory video; and establishing a correspondence between the first explanatory video time interval and the source video time interval.

[0063] Optionally, the sixth determining submodule is specifically used for: for each frame image in the explanatory video, performing text recognition on the frame image using a text recognition algorithm to obtain the text recognition result corresponding to the frame image; if the text recognition results corresponding to adjacent frame images in the explanatory video meet a preset matching condition, then unifying the text recognition results corresponding to adjacent frame images into the target text; merging the time of the adjacent frame images to obtain a second explanatory video time interval; and establishing a correspondence between the second explanatory video time interval and the target text.

[0064] Optionally, the seventh determining submodule is specifically used to: calculate the overlap degree between each time interval included in at least one time interval of the second explanatory video time interval and each time interval included in the first explanatory video time interval, wherein the overlap degree is the ratio of the intersection time interval length to the union time interval length; if the overlap degree is not less than a preset overlap degree threshold, then establish the correspondence between the source video time interval and the text based on the correspondence between the first explanatory video time interval and the source video time interval and the correspondence between the second explanatory video time interval and the text.

[0065] Optionally, the similar frame images corresponding to adjacent frame images in the explanatory video satisfying the preset similarity conditions include: there are identical frame images among the similar frame images corresponding to adjacent frame images in the explanatory video, or the time difference between the similar frame images corresponding to adjacent frame images in the explanatory video is less than a preset time difference value.

[0066] Optionally, the text recognition results corresponding to adjacent frames in the explanatory video satisfying the preset matching conditions include: the text recognition results corresponding to adjacent frames in the explanatory video are the same, or the matching degree between the text recognition results corresponding to adjacent frames in the explanatory video is not greater than the preset matching degree.

[0067] Optionally, the sixth determining submodule is specifically used for:

[0068] If the strings in the text recognition results corresponding to adjacent frames are different, the frequency of occurrence of the different strings will be counted, and the text recognition result corresponding to the string with the most occurrences will be taken as the text corresponding to the adjacent frame.

[0069] In a third aspect of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0070] Memory, used to store computer programs;

[0071] When a processor executes a program stored in memory, it implements the model training data sample acquisition method described in any of the first aspects above.

[0072] In a fourth aspect of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when executed by a processor, the computer program implements the model training data sample acquisition method described in any of the first aspects above.

[0073] In another aspect of this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the model training data sample acquisition method described in any of the first aspects above.

[0074] This application provides a method, apparatus, electronic device, and storage medium for acquiring model training data samples, comprising: acquiring a source video and a corresponding explanatory video, wherein the explanatory video includes images and text explaining the source video; obtaining a matching result between text and source video segments based on video segments that match the explanatory video and the source video, and the text of the explanatory video; and generating model training data samples corresponding to the matching result. By matching the text of the explanatory video with the corresponding source video segments to form text-video matching pairs, text and video matching pair data samples are provided for the training of multimodal models; and the source video segments and corresponding text can be automatically extracted, shortening the time required to acquire model training data samples. Attached Figure Description

[0075] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0076] Figure 1 A schematic diagram of the first method for obtaining model training data samples provided in an embodiment of this application;

[0077] Figure 2 This is a schematic diagram of a second process for obtaining model training data samples provided in an embodiment of this application;

[0078] Figure 3a A schematic diagram of a third method for obtaining model training data samples provided in an embodiment of this application;

[0079] Figure 3b A schematic diagram illustrating a practical application of the model training data sample acquisition method provided in the embodiments of this application;

[0080] Figure 4 This is a schematic diagram of the fourth process for obtaining model training data samples provided in the embodiments of this application;

[0081] Figure 5 A schematic diagram of the fifth process for obtaining model training data samples provided in the embodiments of this application;

[0082] Figure 6 A sixth flowchart illustrating the method for obtaining model training data samples provided in this application embodiment;

[0083] Figure 7A schematic diagram of the seventh method for obtaining model training data samples provided in the embodiments of this application;

[0084] Figure 8 A schematic diagram of the eighth method for obtaining model training data samples provided in the embodiments of this application;

[0085] Figure 9 A schematic diagram of a model training data sample acquisition device provided in an embodiment of this application;

[0086] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0087] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0088] To shorten the time required to acquire model training data samples, embodiments of this application provide a method, apparatus, electronic device, and storage medium for acquiring model training data samples.

[0089] By matching the text of explanatory videos with the corresponding source video segments to form text-video matching pairs, text and video matching pair data samples are provided for the training of multimodal models; and the source video segments and corresponding text can be automatically extracted, shortening the time spent obtaining model training data samples.

[0090] First, a method for obtaining model training data samples provided in the embodiments of this application will be described in detail, see [link to relevant documentation]. Figure 1 This includes the following steps:

[0091] Step S101: Obtain the source video and the corresponding explanatory video. The explanatory video includes images and text that explain the source video.

[0092] The source video can be a movie video, variety show video, TV series video, etc., and the explanatory video can be a commentary video, documentary video, etc., and this application does not make specific limitations in this regard. The explanatory video contains images and text that explain the source video. For example, there are many movie commentary videos with subtitles on the Internet. When the subtitles of the commentary video say "Wang Xiaoming suddenly picked up a gun and pointed it at Wang Jian's head", the video content usually describes the same content, that is, it describes the corresponding content in the movie video.

[0093] Step S102: Based on the video segments that match the explanatory video and the source video, and the text of the explanatory video, obtain the matching result between the text and the source video segments.

[0094] In one example, when the subtitle of a narration video says "Wang Xiaoming suddenly picked up a gun and pointed it at Wang Jian's head," the corresponding segment of the narration video is matched. Based on the matching segments of the narration video and the movie video, the subtitle is matched with the matching segment of the movie video to form a text-video matching pair.

[0095] Step S103: Generate model training data samples corresponding to the matching results.

[0096] In the above embodiments, by matching the text of the explanatory video with the corresponding source video segment to form a text-video matching pair, text and video matching pair data samples are provided for the training of the multimodal model; and the source video segment and the corresponding text can be automatically extracted, shortening the time spent obtaining model training data samples.

[0097] See Figure 2 This is a schematic diagram of the second process of the model training data sample acquisition method provided in the embodiments of this application, based on Figure 1 Step S102 has been refined to include the following steps:

[0098] Step S201: Based on the similarity between frame images in the source video and frame images in the explanatory video, determine the matching segments between the source video and the explanatory video. The matching segments include a first segment and a second segment that match each other. The first segment is a video segment in the explanatory video; the second segment is a video segment in the source video.

[0099] When calculating the similarity between each frame of an explanatory video and its corresponding source video, the similarity can be calculated for all frames of both videos, or it can be calculated for frames extracted from the source video. There are various ways to calculate the similarity. For example, a frame image similarity model can be used to calculate the similarity between each frame of the explanatory video and its corresponding source video. Frames exceeding a similarity threshold are considered identical to frames in the source video, and then the matching segments between the explanatory video and the source video are determined. The similarity threshold can be determined based on actual needs or experience. To ensure the accuracy of the matching segments, the similarity threshold can be set relatively high.

[0100] Step S202: Perform text recognition on the frame image in the first segment to obtain the text corresponding to the first segment.

[0101] The first segment is a matching video clip from an explanatory video. OCR (optical character recognition) is used to identify the text in each frame of the first segment, thus obtaining the corresponding text. OCR refers to the use of optical and computer technologies to detect text in an image and then recognize the text content. In one example, the first segment could be a matching clip from an explanatory video; that is, OCR text recognition technology can be used to identify the subtitles in each frame of the matching clip from the explanatory video.

[0102] Step S203: Match the text with the second segment to obtain the matching result.

[0103] In step S201, the matching segments of the source video and the explanatory video have been determined, namely the first segment and the second segment that match each other. The first segment is a video segment in the explanatory video, and the second segment is a video segment in the source video. In step S202, the text corresponding to the first segment is obtained. Based on steps S201 and S202, the text is matched with the second segment to obtain the matching result of the second segment and the corresponding text.

[0104] In the above embodiments, the matching segments of the source video and the explanatory video are first obtained, and then the text corresponding to the matching segments of the explanatory video is obtained. Based on the above matching and correspondence, the matching segments of the source video are matched with the text to form text-video matching pairs. This provides text-video matching pair data samples for the training of multimodal models, which can avoid the need for a lot of manpower for annotation and shorten the time spent obtaining model training data samples.

[0105] See Figure 3a This is a schematic diagram of the third process of obtaining model training data samples provided in the embodiments of this application, based on Figure 1 Step S102 has been refined to include the following steps:

[0106] Step S301: Based on the text in the frame images of the explanatory video, determine the correspondence between the time interval of the explanatory video and the text in the explanatory video.

[0107] By using OCR text recognition technology to identify the text in each frame of an explanatory video clip, and merging the results with identical text, the text in the explanatory video clip can be obtained, thus revealing the time interval of the explanatory video clip and the correspondence between the text in the explanatory video clip.

[0108] Step S302: Based on the similarity between frame images in the source video and frame images in the explanatory video, determine the time intervals in which the explanatory video and the source video match each other, and obtain the correspondence between the time intervals of the explanatory video and the time intervals of the source video.

[0109] It should be noted that in this embodiment, the correspondence between the time interval of the explanatory video and the text in the explanatory video is performed first, that is, step S301 is executed first, and then step S302 is executed.

[0110] The similarity between each frame of the explanatory video and the corresponding source video can be calculated using a frame image similarity model. Alternatively, based on the correspondence between the time interval of the explanatory video and the text in the explanatory video in step S301, a portion of the explanatory video's time interval can be selected, and the similarity between the frames in that portion of the explanatory video's time interval and the frames in the source video can be calculated using the frame image similarity model. Frames exceeding a threshold are considered identical to the frames in the source video. Then, frames with similar times in the frame image matching results are merged to form the matching results for video segments, determining the time intervals of the mutually matching segments between the explanatory video and the source video, thus obtaining the time interval correspondence between the explanatory video segments and the source video segments.

[0111] When calculating the similarity between each frame in the source video and the explanatory video, the frame can be any frame in the explanatory video and the source video, or it can be multiple frame images obtained by extracting frames from the explanatory video and the source video.

[0112] Frames are extracted from both the explanatory video and its corresponding source video at a preset frame rate, resulting in frame images of the explanatory video and the source video. The time each frame appears in its corresponding video is then calculated based on the preset frame rate for subsequent time interval confirmation. The preset frame rate can be 24 fps (frames per second), 25 fps, 30 fps, 48 ​​fps, etc. In one example, to ensure a sufficient number of extracted frame images, the frame rate can be set to 48 fps.

[0113] Step S303: Based on the correspondence between the time interval of the explanatory video and the time interval of the source video, and the correspondence between the time interval of the explanatory video and the text in the explanatory video, determine the correspondence between the time interval of the source video and the text in the explanatory video.

[0114] In step S301, the correspondence between the time interval of the explanatory video and the text in the explanatory video is obtained. In step S302, the correspondence between the time interval of the explanatory video and the time interval of the source video is obtained. Based on the above two correspondences, the correspondence between the time interval of the source video and the text in the explanatory video is determined.

[0115] Step S304: Obtain the source video segments corresponding to the time interval of the source video from the source video, and obtain the matching results between the source video segments and the text.

[0116] After obtaining the correspondence between the time interval of the source video and the text in the explanatory video, the corresponding source video segments are extracted according to the time interval of the source video, thus obtaining the text-video matching pair.

[0117] The following are examples, such as Figure 3b As shown, the explanatory videos are narration videos, and the corresponding source videos are movie videos. Based on the subtitles of the frames in the narration video's time interval, text recognition is performed using OCR to determine the correspondence between the narration video's time interval and its corresponding text. Figure 3b As indicated by the labels in the second rectangle 1 and the second rectangle 2, the text content is "The people are discussing matters at the Prince's Mansion" and "He questioned the Princess about why she did this." Based on the similarity between frame images in the film video and frame images in the narration video, the time intervals in which the narration video and the film video match are determined, i.e. Figure 3b As indicated by the first rectangle 1 and the first rectangle 2, the correspondence between the narration video time interval and the movie video time interval is obtained. Based on the correspondence between the narration video time interval and the movie video time interval, as well as the correspondence between the narration video time interval and its text, the correspondence between the movie video time interval and the text is determined. The movie video segments corresponding to the movie video time intervals are then extracted from the movie video, resulting in the matching results between the movie video segments and the text. Figure 3b The third rectangle in the middle is labeled as shown.

[0118] In the above embodiments, by integrating the correspondence between the time intervals of the explanatory videos and the text within them, as well as the correspondence between the time intervals of the explanatory videos and the time intervals of the source videos, the correspondence between the time intervals of the source videos and the text within them is determined. Then, corresponding source video segments are extracted according to the time intervals of the source videos, thereby automatically extracting the source video segments and their corresponding text, obtaining text-video matching pairs, and providing text-video matching pair data samples for the training of multimodal models. This automated approach quickly obtains data suitable for training multimodal models, enabling rapid development of multimodal models that can be used in business areas such as retrieval, recognition, and creation. On the other hand, based on the correspondence between the time intervals of the explanatory videos and the text within them in step S301, a portion of the explanatory video time intervals can be selected, and the similarity between the frame images in that portion of the explanatory video time interval and the frame images of the source video can be calculated. Therefore, executing step S301 first avoids performing a full matching of all frame images of the explanatory videos, saving computational resources.

[0119] See Figure 4 This is a schematic diagram of the fourth process of the model training data sample acquisition method provided in the embodiments of this application, based on Figure 1 Step S102 has been refined to include the following steps:

[0120] Step S401: Based on the similarity between frame images in the source video and frame images in the explanatory video, determine the time intervals in which the explanatory video and the source video match each other, and obtain the correspondence between the time intervals of the explanatory video and the time intervals of the source video.

[0121] The similarity between each frame of the explanatory video and the corresponding source video is calculated using a frame image similarity model. If the similarity exceeds a threshold, the frame image of the explanatory video is considered to be consistent with the frame image of the source video. Then, frames with similar times in the frame image matching results are merged to form the matching results of video segments. The time interval of the segments that match each other between the explanatory video and the source video is determined, thus obtaining the time interval correspondence between the explanatory video segments and the source video segments.

[0122] Step S402: Based on the text in the frame images of the explanatory video, determine the correspondence between the time interval of the explanatory video and the text in the explanatory video.

[0123] The specific analysis is the same as above, and will not be repeated here.

[0124] It should be noted that in this embodiment, the execution order of steps S401 and S402 is the reverse of the execution order of steps S301 and S302 in the above embodiment. In this embodiment, the correspondence between the explanatory video time interval and the text in the explanatory video is performed later; that is, step S302 in the above embodiment is executed first, followed by step S301 in the above embodiment.

[0125] Step S403: Based on the correspondence between the time interval of the explanatory video and the time interval of the source video, and the correspondence between the time interval of the explanatory video and the text in the explanatory video, determine the correspondence between the time interval of the source video and the text in the explanatory video.

[0126] The specific analysis is the same as above, and will not be repeated here.

[0127] Step S404: Obtain the source video segments corresponding to the time interval of the source video from the source video, and obtain the matching results between the source video segments and the text.

[0128] The specific analysis is the same as above, and will not be repeated here.

[0129] In the above embodiments, by integrating the correspondence between the time intervals of the explanatory video and the time intervals of the source video, as well as the correspondence between the time intervals of the explanatory video and the text in the explanatory video, the correspondence between the time intervals of the source video and the text in the explanatory video is determined. Then, the corresponding source video segments are extracted according to the time intervals of the source video, thereby automatically extracting the source video segments and the corresponding text, obtaining text-video matching pairs, and providing text-video matching pair data samples for the training of multimodal models. Through automation, data that can be used for training multimodal models can be obtained quickly, thereby rapidly developing multimodal models that can be used in business areas such as retrieval, recognition, and creation. On the other hand, first determining the correspondence between the time intervals of the explanatory video and the time intervals of the source video does not predetermine the correspondence between the text and the explanatory video. Then, selecting the portion of the explanatory video containing text and calculating the similarity between the frame images in its time interval and the frame images of the source video, it is necessary to calculate the similarity between each frame of the explanatory video and the corresponding source video, which can improve the recall rate of the text-video matching pair data samples.

[0130] See Figure 5 This is a schematic diagram of the fifth process of the model training data sample acquisition method provided in the embodiments of this application, based on Figure 1 Step S103 has been refined and includes the following steps:

[0131] Step S501: Generate model training data corresponding to a frame of image in the source video segment and model training data corresponding to the text matched by a frame of image, forming a model training data sample.

[0132] Step S502: Based on a single frame image from the source video segment, merge multiple frames from the source video segment to generate model training data corresponding to the multiple frames from the source video segment and model training data corresponding to the text matched by the multiple frames, thus forming model training data samples.

[0133] In the above embodiments, the training data samples for the multimodal model can be composed of model training data corresponding to a single frame of an image in the source video segment and model training data corresponding to text matched by that single frame of an image, or they can be composed of model training data corresponding to multiple frames of images in the source video segment and model training data corresponding to text matched by multiple frames of images. The number of model training data samples is not limited.

[0134] See Figure 6 This is a schematic diagram of the sixth process of the model training data sample acquisition method provided in the embodiments of this application, based on Figure 4 Step S401 has been refined to include the following steps:

[0135] Step S601: For each frame image in the explanatory video, retrieve similar frame images from the source video whose similarity to the frame image is not less than a preset similarity threshold.

[0136] Using any frame image retrieval method, for each frame image in an explanatory video, source video frames with a similarity exceeding a preset similarity threshold are retrieved, and the top K retrieval results are retained. To ensure the accuracy of the matching segments, the value of k can be set to 2 to 5; in one example, k can be set to 3. Frame image retrieval methods include perceptual hashing, color distribution, and content feature methods. In one example, the content feature method can be used to retrieve data from the source video. Furthermore, to ensure high similarity between frame images in the explanatory video and frame images in the source video, the preset similarity threshold can be set relatively high.

[0137] For each frame in a multi-frame image in an explanatory video, the multi-frame image can be all the frame images in the explanatory video, or it can be a multi-frame image obtained by extracting frames from the explanatory video.

[0138] Frames are extracted from both the explanatory video and its corresponding source video at a preset frame rate, resulting in frame images of the explanatory video and the source video. The time each frame appears in its corresponding video is then calculated based on the preset frame rate for subsequent time interval confirmation. The preset frame rate can be 24 fps (frames per second), 25 fps, 30 fps, 48 ​​fps, etc. In one example, to ensure a sufficient number of extracted frame images, the frame rate can be set to 48 fps.

[0139] Step S602: If the similar frame images corresponding to adjacent frame images in the explanatory video meet the preset similarity conditions, then the time of the adjacent frame images is merged to obtain the first explanatory video time interval, and the time of the similar frame images is merged to obtain the source video time interval, wherein the adjacent frame images are the frame images that are temporally adjacent in the explanatory video.

[0140] In one possible implementation, the similar frame images corresponding to adjacent frame images in the explanatory video satisfying the preset similarity conditions include: there are identical frame images among the similar frame images corresponding to adjacent frame images in the explanatory video, or the time difference between the similar frame images corresponding to adjacent frame images in the explanatory video is less than a preset time difference value.

[0141] If the search results for adjacent frames in the explanatory video contain the same frame image, or if the time difference between adjacent frames in the explanatory video is less than a preset time difference, then the times when these frames appear in the explanatory video are merged to form the time interval of the first explanatory video. At the same time, the times when the frames (similar frames) of the source video that meet the aforementioned conditions appear in the source video are merged to form the time interval of the source video. Thus, the time intervals of the first explanatory video and the source video are obtained respectively.

[0142] The preset time difference can be set to a range of 1 / 48 to 1 / 24 of a second. In one example, the preset time difference can be set to 1 / 48 of a second.

[0143] Step S603: Establish the correspondence between the time interval of the first explanatory video and the time interval of the source video.

[0144] Based on the obtained time intervals of the first explanatory video and the source video, establish the correspondence between the time intervals of the first explanatory video and the time intervals of the source video.

[0145] In the above embodiments, if the similar frame images corresponding to adjacent frame images in the explanatory video meet the preset similarity conditions, the time of the adjacent frame images is merged to obtain the first explanatory video time interval, and the time of the similar frame images is merged to obtain the source video time interval; through the obtained time intervals of the first explanatory video and the source video, the correspondence between the first explanatory video time interval and the source video time interval is obtained.

[0146] See Figure 7 This is a schematic diagram of the seventh process of the model training data sample acquisition method provided in the embodiments of this application, based on Figure 4 Step S402 has been refined to include the following steps:

[0147] Step S701: For each frame image in the explanatory video, a text recognition algorithm is used to perform text recognition on the frame image to obtain the text recognition result corresponding to the frame image.

[0148] Multiple frame images can be all frame images from the explanatory video, or multiple frame images obtained by extracting frames from the explanatory video. Frames are extracted from the explanatory video at a preset frame rate to obtain the explanatory video's frame images. The appearance time of each frame image in the explanatory video is calculated based on the preset frame rate for subsequent time interval confirmation. The preset frame rate can be 24fps, 25fps, 30fps, 48fps, etc. In one example, to ensure a sufficient number of frame images are obtained through extraction, the frame rate can be set to 48fps.

[0149] OCR text recognition algorithms include CNN+RNN+CTC (Convolutional Neural Networks; Recurrent Neural Networks; Connectionist Temporal Classification), CNN+RNN based on Attention (Convolutional Neural Networks; Recurrent Neural Networks; Attention-based mechanism), etc. Using any one of these OCR algorithms, the OCR result, i.e., the corresponding text recognition result, can be identified from each frame of a multi-frame explanatory video. In one example, CNN+RNN+CTC can be used to identify the text in each frame of a multi-frame explanatory video.

[0150] Step S702: If the text recognition results corresponding to adjacent frame images in the explanatory video meet the preset matching conditions, then the text recognition results corresponding to adjacent frame images are unified as the target text.

[0151] In one possible implementation, the text recognition results corresponding to adjacent frames in the explanatory video satisfying a preset matching condition includes: the text recognition results corresponding to adjacent frames in the explanatory video are the same, or the matching degree between the text recognition results corresponding to adjacent frames in the explanatory video is not greater than a preset matching degree.

[0152] If the matching degree (edit distance between two texts / string length of the shorter text, i.e., the ratio of the edit distance between the text recognition results of any two adjacent frames in the explanatory video to the string length of the shorter text itself, and this ratio is used as a parameter to judge the matching degree between the text recognition results of adjacent frames in the explanatory video) is not greater than a preset matching degree, then these OCRs are unified into a single identical text (if they are all identical, then they are already identical text; if they are different, a vote is held on the frequency of different strings, and the one with the most frequency is taken as the final OCR text). The edit distance is mainly used to compare the similarity between two strings; it is a string metric that measures the difference between two strings. The edit distance between two strings is the minimum number of single-character edits (insertion, deletion, or replacement) required to convert one string into another. Generally speaking, the smaller the edit distance, the greater the similarity between the two strings; the larger the edit distance, the lower the similarity between the two strings.

[0153] Alternatively, the string length of the longer of the two texts can be used as the denominator of the matching degree parameter mentioned above, and this application does not limit this.

[0154] In one example, to ensure high similarity between text recognition results of adjacent frames, the preset matching degree can be set relatively low, which also facilitates the correspondence of subsequent time intervals.

[0155] Step S703: Merge the times of adjacent frame images to obtain the second explanatory video time interval.

[0156] The time intervals of adjacent frame images corresponding to the text recognition results that meet the preset matching conditions in the explanatory video are combined to form the time interval of the second explanatory video.

[0157] Step S704: Establish the correspondence between the time interval of the second explanatory video and the target text.

[0158] Based on the obtained time intervals of the second type of explanatory videos and the corresponding target text, the correspondence between the time intervals of the second type of explanatory videos and the target text is obtained.

[0159] In the above embodiments, the text recognition results corresponding to adjacent frames in the explanatory video are unified into the target text according to preset matching conditions. That is, if the strings in the text recognition results corresponding to adjacent frames are different, the occurrence frequency of the different strings is counted, and the text recognition result corresponding to the string with the most occurrences is taken as the text (target text) corresponding to the adjacent frame. Then, the time of adjacent frames is merged, that is, the occurrence time of adjacent frames corresponding to the text recognition results that meet the preset matching conditions in the explanatory video is merged to obtain the second explanatory video time interval. Based on the obtained second explanatory video time interval and the corresponding target text, the correspondence between the second explanatory video time interval and the target text is obtained.

[0160] See Figure 8 This is a schematic diagram of the eighth process of the model training data sample acquisition method provided in the embodiments of this application, based on Figure 4 Step S403 has been refined to include the following steps:

[0161] Step S801: For each time interval in at least one time interval included in the second explanatory video time interval, calculate the overlap degree between each time interval and each time interval included in the first explanatory video time interval, wherein the overlap degree is the ratio of the length of the intersection time interval to the length of the union time interval.

[0162] The first type of explanatory video time interval includes at least one time interval, and the second type of explanatory video time interval includes at least one time interval.

[0163] For each time interval within the first and second explanatory video time intervals, which may not be entirely identical, a specific method is needed to correlate them: For each second explanatory video time interval, iterate through each first explanatory video time interval and calculate their time interval overlap ratio (IoU) (IoU = intersection time interval length / union time interval length). The intersection time interval length is the length of the intersection between the first and second explanatory video time intervals, and the union time interval length is the length of the union between the first and second explanatory video time intervals.

[0164] Step S802: If the overlap is not less than the preset overlap threshold, then establish the correspondence between the source video time interval and the text based on the correspondence between the first explanatory video time interval and the source video time interval and the correspondence between the second explanatory video time interval and the text.

[0165] For each time interval in the first and second explanatory video time intervals, if the IoU is not less than a preset overlap threshold, the first and second explanatory video time intervals are considered to be basically corresponding. Therefore, their corresponding source video time intervals are mapped to the target text, establishing a correspondence between the source video time intervals and the target text. To ensure the establishment of this correspondence, the preset overlap threshold can be set relatively high.

[0166] In the above embodiments, by calculating the overlap between the first explanatory video time interval and the second explanatory video time interval, if the overlap is not less than a preset overlap threshold, it is considered that the first explanatory video time interval and the second explanatory video time interval are basically corresponding. The corresponding source video time intervals and target texts are then matched, thereby realizing the acquisition of the correspondence between the source video time intervals and the target texts.

[0167] Based on the same inventive concept as the model training data sample acquisition method provided in the above embodiments, this application provides a model training data sample acquisition device, see [link to previous document]. Figure 9 The device includes:

[0168] The first acquisition module 910 is used to acquire a source video and an explanatory video corresponding to the source video, wherein the explanatory video includes images and text that explain the source video.

[0169] The matching module 920 is used to obtain the matching result between the text and the source video segment based on the video segments that match each other between the explanatory video and the source video, and the text of the explanatory video;

[0170] The generation module 930 is used to generate model training data samples corresponding to the matching results.

[0171] In the above embodiments, by matching the text of the explanatory video with the corresponding source video segment to form a text-video matching pair, text and video matching pair data samples are provided for the training of the multimodal model; and the source video segment and the corresponding text can be automatically extracted, shortening the time spent obtaining model training data samples.

[0172] In one possible implementation, the matching module 920 includes:

[0173] The first determining submodule is used to determine matching segments between the source video and the explanatory video based on the similarity between frame images in the source video and frame images in the explanatory video. The matching segments include a first segment and a second segment that match each other. The first segment is a video segment in the explanatory video, and the second segment is a video segment in the source video.

[0174] The first acquisition submodule is used to perform text recognition on the frame image in the first segment to obtain the text corresponding to the first segment;

[0175] The second acquisition submodule is used to match the text with the second fragment to obtain a matching result.

[0176] In the above embodiments, the matching segments of the source video and the explanatory video are first obtained, and then the text corresponding to the matching segments of the explanatory video is obtained. Based on the above matching and correspondence, the matching segments of the source video are matched with the text to form text-video matching pairs. This provides text-video matching pair data samples for the training of multimodal models, which can avoid the need for a lot of manpower for annotation and shorten the time spent obtaining model training data samples.

[0177] In one possible implementation, the matching module 920 includes:

[0178] The second determining submodule is used to determine the correspondence between the time interval of the explanatory video and the text in the explanatory video based on the text of the frame images in the explanatory video.

[0179] The third determining submodule is used to determine the time interval between the explanatory video and the source video that match each other based on the similarity between the frame images in the source video and the frame images in the explanatory video, and to obtain the correspondence between the time interval of the explanatory video and the time interval of the source video.

[0180] The fourth determining submodule is used to determine the correspondence between the time interval of the source video and the text in the explanatory video based on the correspondence between the time interval of the explanatory video and the time interval of the source video, as well as the correspondence between the time interval of the explanatory video and the text in the explanatory video.

[0181] The third acquisition submodule is used to acquire the source video segment corresponding to the time interval of the source video from the source video, and obtain the matching result between the source video segment and the text.

[0182] In the above embodiments, by integrating the correspondence between the time intervals of explanatory videos and the text within those videos, as well as the correspondence between the time intervals of explanatory videos and the time intervals of source videos, the correspondence between the time intervals of source videos and the text within the explanatory videos is determined. Then, corresponding source video segments are extracted according to the time intervals of the source videos, thereby automatically extracting the source video segments and their corresponding text, obtaining text-video matching pairs. This provides text-video matching pair data samples for training the multimodal model. Through automation, data suitable for training the multimodal model is quickly obtained, enabling rapid development of the multimodal model for use in business areas such as retrieval, recognition, and creation. On the other hand, the similarity between a portion of the time intervals of the explanatory videos and the corresponding frames of the source videos can be calculated using a frame image similarity model, eliminating the need for full matching of all explanatory video frames and saving computational resources.

[0183] In one possible implementation, the matching module 920 includes:

[0184] The fifth determining submodule is used to determine the time interval between the explanatory video and the source video that match each other based on the similarity between the frame images in the source video and the frame images in the explanatory video, and to obtain the correspondence between the time interval of the explanatory video and the time interval of the source video.

[0185] The sixth determining submodule is used to determine the correspondence between the time interval of the explanatory video and the text in the explanatory video based on the text of the frame images in the explanatory video.

[0186] The seventh determination submodule is used to determine the correspondence between the time interval of the source video and the text in the explanatory video based on the correspondence between the time interval of the explanatory video and the time interval of the source video, as well as the correspondence between the time interval of the explanatory video and the text in the explanatory video.

[0187] The fourth acquisition submodule is used to acquire the source video segment corresponding to the time interval of the source video from the source video, and obtain the matching result of the source video segment and the text.

[0188] In the above embodiments, by integrating the correspondence between the time intervals of the explanatory video and the time intervals of the source video, as well as the correspondence between the time intervals of the explanatory video and the text within the explanatory video, the correspondence between the time intervals of the source video and the text within the explanatory video is determined. Then, corresponding source video segments are extracted according to the source video time intervals, thereby automatically extracting the source video segments and their corresponding text, obtaining text-video matching pairs, and providing text-video matching pair data samples for training the multimodal model. This automated approach quickly obtains data suitable for training the multimodal model, enabling rapid development of the model for applications in retrieval, recognition, and creation. On the other hand, by calculating the similarity between each frame of the explanatory video and the corresponding source video using a frame image similarity model, and obtaining the correspondence between the time intervals of the explanatory video and the source video, text recognition can be performed, improving the recall rate of the text-video matching pair data samples.

[0189] In one possible implementation, the generation module 930 includes:

[0190] The first generation submodule is used to generate model training data corresponding to a frame of image in the source video segment and model training data corresponding to the text matched by the frame of image, to form a model training data sample; or;

[0191] The second generation submodule is used to merge multiple frames of images in the source video segment based on one frame of the image in the source video segment, and generate model training data corresponding to the multiple frames of the image in the source video segment and model training data corresponding to the text matched by the multiple frames of the image, thereby forming a model training data sample.

[0192] In the above embodiments, the training data samples for the multimodal model can be composed of model training data corresponding to a single frame of an image in the source video segment and model training data corresponding to text matched by that single frame of an image, or they can be composed of model training data corresponding to multiple frames of images in the source video segment and model training data corresponding to text matched by multiple frames of images. The number of model training data samples is not limited.

[0193] In one possible implementation, the fifth determining submodule is specifically used for: for each frame image in the explanatory video, retrieving from the source video similar frame images whose similarity to the frame image is not less than a preset similarity threshold; if the similar frame images corresponding to adjacent frame images in the explanatory video meet a preset similarity condition, merging the times of the adjacent frame images to obtain a first explanatory video time interval, and merging the times of the similar frame images to obtain a source video time interval, wherein the adjacent frame images are time-adjacent frame images in the explanatory video; and establishing a correspondence between the first explanatory video time interval and the source video time interval.

[0194] In the above embodiments, if the similar frame images corresponding to adjacent frame images in the explanatory video meet the preset similarity conditions, the time of the adjacent frame images is merged to obtain the first explanatory video time interval, and the time of the similar frame images is merged to obtain the source video time interval; through the obtained time intervals of the first explanatory video and the source video, the correspondence between the first explanatory video time interval and the source video time interval is obtained.

[0195] In one possible implementation, the sixth determining submodule is specifically used for: performing text recognition on each frame of the explanatory video using a text recognition algorithm to obtain the text recognition result corresponding to the frame; if the text recognition results corresponding to adjacent frames in the explanatory video meet a preset matching condition, then unifying the text recognition results corresponding to adjacent frames into the target text; merging the time of the adjacent frames to obtain a second explanatory video time interval; and establishing a correspondence between the second explanatory video time interval and the target text.

[0196] In the above embodiments, the text recognition results corresponding to adjacent frames in the explanatory video are unified into the target text according to preset matching conditions. That is, if the strings in the text recognition results corresponding to adjacent frames are different, the occurrence frequency of the different strings is counted, and the text recognition result corresponding to the string with the most occurrences is taken as the text (target text) corresponding to the adjacent frame. Then, the time of adjacent frames is merged, that is, the occurrence time of adjacent frames corresponding to the text recognition results that meet the preset matching conditions in the explanatory video is merged to obtain the second explanatory video time interval. Based on the obtained second explanatory video time interval and the corresponding target text, the correspondence between the second explanatory video time interval and the target text is obtained.

[0197] In one possible implementation, the seventh determining submodule is specifically used to: calculate the overlap degree between each time interval included in at least one time interval of the second explanatory video time interval and each time interval included in the first explanatory video time interval, wherein the overlap degree is the ratio of the intersection time interval length to the union time interval length; if the overlap degree is not less than a preset overlap degree threshold, then establish the correspondence between the source video time interval and the text based on the correspondence between the first explanatory video time interval and the source video time interval and the correspondence between the second explanatory video time interval and the text.

[0198] In the above embodiments, by calculating the overlap between the first explanatory video time interval and the second explanatory video time interval, if the overlap is not less than a preset overlap threshold, it is considered that the first explanatory video time interval and the second explanatory video time interval are basically corresponding. The corresponding source video time intervals and target texts are then matched, thereby realizing the acquisition of the correspondence between the source video time intervals and the target texts.

[0199] In one possible implementation, the similar frame images corresponding to adjacent frame images in the explanatory video satisfying the preset similarity conditions include: there are identical frame images among the similar frame images corresponding to adjacent frame images in the explanatory video, or the time difference between the similar frame images corresponding to adjacent frame images in the explanatory video is less than a preset time difference value.

[0200] In one possible implementation, the text recognition results corresponding to adjacent frames in the explanatory video satisfying the preset matching conditions include: the text recognition results corresponding to adjacent frames in the explanatory video are the same, or the matching degree between the text recognition results corresponding to adjacent frames in the explanatory video is not greater than the preset matching degree.

[0201] In one possible implementation, the sixth determining submodule is specifically used for:

[0202] If the strings in the text recognition results corresponding to adjacent frames are different, the frequency of occurrence of the different strings will be counted, and the text recognition result corresponding to the string with the most occurrences will be taken as the text corresponding to the adjacent frame.

[0203] In the above embodiments, by counting the number of occurrences of different strings, the text recognition results corresponding to adjacent frames in the explanatory video are unified, and the text corresponding to adjacent frames is obtained.

[0204] This application also provides an electronic device, such as... Figure 10As shown, it includes a processor 1001, a communication interface 1002, a memory 1003, and a communication bus 1004, wherein the processor 1001, the communication interface 1002, and the memory 1003 communicate with each other through the communication bus 1004.

[0205] Memory 1003 is used to store computer programs;

[0206] When the processor 1001 executes the program stored in the memory 1003, it implements the method steps of the model training data sample acquisition method provided in any of the above embodiments.

[0207] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0208] The communication interface is used for communication between the aforementioned terminal and other devices.

[0209] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0210] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0211] In another embodiment provided in this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the model training data sample acquisition method described in any of the above embodiments.

[0212] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the model training data sample acquisition methods described in the above embodiments.

[0213] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0214] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0215] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0216] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for obtaining model training data samples, characterized in that, The method includes: Obtain the source video and the corresponding explanatory video, wherein the explanatory video contains images and text that explain the source video; Based on the video segments that match the explanatory video and the source video, and the text of the explanatory video, the matching result between the text and the source video segments is obtained; The model training data sample is formed by generating model training data corresponding to a frame of image in the source video segment and model training data corresponding to the text matched by the frame of image; or; Based on a single frame from the source video segment, multiple frames from the source video segment are merged to generate model training data corresponding to the multiple frames from the source video segment and model training data corresponding to the text matched by the multiple frames, thus forming the model training data sample.

2. The method according to claim 1, characterized in that, The process of obtaining a matching result between the text and the source video segment based on the video segment that matches the explanatory video and the text of the explanatory video includes: Based on the similarity between frame images in the source video and frame images in the explanatory video, matching segments between the source video and the explanatory video are determined. The matching segments include a first segment and a second segment that match each other. The first segment is a video segment in the explanatory video, and the second segment is a video segment in the source video. Perform text recognition on the frame images in the first segment to obtain the text corresponding to the first segment; The text is matched with the second fragment to obtain the matching result.

3. The method according to claim 1, characterized in that, The process of obtaining a matching result between the text and the source video segment based on the video segment that matches the explanatory video and the text of the explanatory video includes: Based on the text in the frame images of the explanatory video, determine the correspondence between the time interval of the explanatory video and the text in the explanatory video; Based on the similarity between the frame images in the source video and the frame images in the explanatory video, the time intervals in which the explanatory video and the source video match are determined, and the correspondence between the time intervals of the explanatory video and the time intervals of the source video is obtained. Based on the correspondence between the time intervals of the explanatory videos and the time intervals of the source videos, as well as the correspondence between the time intervals of the explanatory videos and the text in the explanatory videos, the correspondence between the time intervals of the source videos and the text in the explanatory videos is determined. Obtain the source video segment corresponding to the time interval of the source video from the source video, and obtain the matching result between the source video segment and the text.

4. The method according to claim 1, characterized in that, The process of obtaining a matching result between the text and the source video segment based on the video segment that matches the explanatory video and the text of the explanatory video includes: Based on the similarity between the frame images in the source video and the frame images in the explanatory video, the time intervals in which the explanatory video and the source video match are determined, and the correspondence between the time intervals of the explanatory video and the time intervals of the source video is obtained. Based on the text in the frame images of the explanatory video, determine the correspondence between the time interval of the explanatory video and the text in the explanatory video; Based on the correspondence between the time intervals of the explanatory videos and the time intervals of the source videos, as well as the correspondence between the time intervals of the explanatory videos and the text in the explanatory videos, the correspondence between the time intervals of the source videos and the text in the explanatory videos is determined. Obtain the source video segment corresponding to the time interval of the source video from the source video, and obtain the matching result between the source video segment and the text.

5. The method according to claim 4, characterized in that, The step of determining the time intervals in which the explanatory video and the source video match based on the similarity of frame images in the source video and frame images in the explanatory video, and obtaining the correspondence between the time intervals of the explanatory video and the time intervals of the source video, includes: For each frame in the explanatory video, retrieve similar frame images from the source video that have a similarity of not less than a preset similarity threshold. If the similar frame images corresponding to adjacent frame images in the explanatory video meet the preset similarity conditions, then the time of the adjacent frame images is merged to obtain the first explanatory video time interval, and the time of the similar frame images is merged to obtain the source video time interval, wherein the adjacent frame images are the frame images that are temporally adjacent in the explanatory video. Establish a correspondence between the time interval of the first explanatory video and the time interval of the source video.

6. The method according to claim 5, characterized in that, The step of determining the correspondence between the time interval of the explanatory video and the text in the explanatory video based on the text in the frame images of the explanatory video includes: For each frame in the explanatory video, a text recognition algorithm is used to perform text recognition on the frame to obtain the text recognition result corresponding to the frame. If the text recognition results of adjacent frames in the explanatory video meet the preset matching conditions, then the text recognition results of adjacent frames will be unified as the target text. The time intervals of the adjacent frames are merged to obtain the second explanatory video time interval; Establish the correspondence between the time interval of the second explanatory video and the target text.

7. The method according to claim 6, characterized in that, The step of determining the correspondence between the source video time interval and the text in the explanatory video based on the correspondence between the explanatory video time interval and the source video time interval, and the correspondence between the explanatory video time interval and the text in the explanatory video, includes: For each time interval in at least one time interval included in the second explanatory video time interval, calculate the degree of overlap between the time interval and each time interval included in the first explanatory video time interval, wherein the degree of overlap is the ratio of the length of the intersection time interval to the length of the union time interval. If the overlap is not less than a preset overlap threshold, then the correspondence between the source video time interval and the text is established based on the correspondence between the first explanatory video time interval and the source video time interval, and the correspondence between the second explanatory video time interval and the text.

8. The method according to claim 5, characterized in that, The similarity conditions for adjacent frames in the explanatory video to satisfy the preset similarity conditions include: the presence of identical frames among the similar frames corresponding to adjacent frames in the explanatory video, or the time difference between the similar frames corresponding to adjacent frames in the explanatory video is less than a preset time difference value.

9. The method according to claim 6, characterized in that, The text recognition results corresponding to adjacent frames in the explanatory video satisfy the preset matching conditions, including: the text recognition results corresponding to adjacent frames in the explanatory video are the same, or the matching degree between the text recognition results corresponding to adjacent frames in the explanatory video is not greater than the preset matching degree.

10. The method according to claim 6, characterized in that, The step of unifying the text recognition results corresponding to adjacent frame images into the target text includes: If the strings in the text recognition results corresponding to adjacent frames are different, the frequency of occurrence of the different strings will be counted, and the text recognition result corresponding to the string with the most occurrences will be taken as the text corresponding to the adjacent frame.

11. A device for acquiring model training data samples, characterized in that, The device includes: The first acquisition module is used to acquire a source video and an explanatory video corresponding to the source video, wherein the explanatory video includes images and text that explain the source video. The matching module is used to obtain the matching result between the text and the source video segment based on the video segments that match each other between the explanatory video and the source video, and the text of the explanatory video; The generation module includes: The first generation submodule is used to generate model training data corresponding to a frame of image in the source video segment and model training data corresponding to the text matched by the frame of image, to form a model training data sample; or; The second generation submodule is used to merge multiple frames of images in the source video segment based on one frame of the image in the source video segment, and generate model training data corresponding to the multiple frames of the image in the source video segment and model training data corresponding to the text matched by the multiple frames of the image, thereby forming a model training data sample.

12. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-10.

Images