A training method and device of a cross-modal retrieval model

Abstract

The application provides a training method and device of a cross-modal retrieval model. The method comprises: mining video of an existing training data set to select each type of video clip, wherein each type of video clip only includes a person, an object or a picture scene of the same type; identifying attribute description text of the video clip from text of the existing training data set; constructing a new training data set according to the attribute description text and the video clip; supplementing the new training data set to the existing training data set to obtain a target training data set, and training a cross-modal retrieval model by using the target training data set. The application improves the description ability and retrieval effect of the cross-modal model on fine-grained attribute information.

Description

A training method and apparatus for a cross-modal retrieval model Technical Field

[0001] This application relates to the field of neural network technology, and in particular to a training method and apparatus for a cross-modal retrieval model. Background Technology

[0002] Cross-modal retrieval is a novel retrieval method that returns related search results in other different modalities for a query term in one modality. It is an emerging technology in cross-media retrieval. With the increasing intelligence and portability of mobile devices and the booming development of online video platforms, a large number of internet users choose to share and disseminate information through video media. Video-text cross-modal retrieval aims to minimize the differences between the video modal representation and the text modal representation of the same video, determining the similarity of representations in different modalities within the feature space, thereby achieving the effect of cross-modal retrieval.

[0003] Existing cross-modal retrieval models typically focus on the overall description of a video, but such retrieval often overlooks details and lacks expression of the video's detailed features, resulting in poor cross-modal retrieval performance. For example, searching for "a man in a gray shirt talking to a woman holding a dog" might return a video with the content "a woman in a gray shirt talking to a man in a white shirt, with a dog sitting on the ground," indicating misplaced clothing information or incorrect relationships between people or animals.

[0004] There is currently no good solution to the problem that training cross-modal retrieval models in existing technologies lacks detailed descriptions of video features. Summary of the Invention

[0005] To address the aforementioned technical problems, or at least partially address them, this application provides a training method and apparatus for a cross-modal retrieval model.

[0006] Firstly, this application provides a method for training a cross-modal retrieval model, the method comprising:

[0007] By mining videos from the existing training dataset, video segments of each category are selected, wherein each video segment category includes only people, objects or scenes of the same category.

[0008] Identify the attribute description text of the video segment from the text in the existing training dataset;

[0009] A new training dataset is constructed based on the attribute description text and the video clip;

[0010] The newly added training dataset is added to the existing training dataset to obtain the target training dataset, and the cross-modal retrieval model is trained using the target training dataset.

[0011] Optionally, by mining the videos in the existing training dataset, the video segments of each category can be selected, including:

[0012] By performing object detection on the videos in the existing training dataset, video frames containing only the objects are obtained, wherein the objects are people, objects, or scene scenes;

[0013] The timing information of the video frame is determined using positioning technology;

[0014] After cutting the video according to the time point information, the multiple video frames obtained from the cutting are combined to obtain a video segment.

[0015] Optionally, the step of performing object detection on the videos in the existing training dataset to obtain video frames containing only the objects includes:

[0016] Perform person detection on the videos in the existing training dataset to obtain person detection boxes;

[0017] By tracking the person detection bounding boxes in the video, video frames containing only the person are obtained.

[0018] Optionally, the step of performing object detection on the videos in the existing training dataset to obtain video frames containing only the objects includes:

[0019] Perform general object detection on the videos in the existing training dataset to obtain object detection boxes and object category labels;

[0020] By tracking the object detection boxes that have the same object category label, video frames containing only objects of the same type are obtained.

[0021] Optionally, the step of performing object detection on the videos in the existing training dataset to obtain video frames containing only the objects includes:

[0022] Each video frame in the existing training dataset is analyzed to obtain video frames containing scene information, wherein the video frames carry scene category labels.

[0023] Optionally, identifying the attribute description text of the video segment from the text of the existing training dataset includes:

[0024] A text analysis scheme is used to identify the description data of each attribute from the text in the existing training dataset;

[0025] From the attribute description data, find the attribute description information that matches the video segment.

[0026] Optionally, a text analysis scheme is employed to identify the description data of each attribute from the text of the existing training dataset, including:

[0027] Part-of-speech tagging (POS) is used to identify nouns and adjectives from the text in the existing training dataset.

[0028] Using syntactic analysis techniques, we can identify word combinations of nouns and adjectives that have a modifying relationship, as well as standalone nouns that do not have an adjective modifying them;

[0029] The word combination or the individual noun is treated as a single attribute description data.

[0030] Optionally, if the video clip is a video clip of a person, retrieving attribute description information matching the video clip from the attribute description data includes:

[0031] If the similarity between the frame features in the video clip of the person and the attribute description data is greater than a first similarity threshold, then the attribute description data is used as the attribute description information of the video clip of the person.

[0032] Optionally, if the video clip is an object video clip, retrieving attribute description information matching the video clip from the attribute description data includes:

[0033] If a noun in the attribute description data matches the object category label, or if the similarity between the frame features in the object video clip and the attribute description data is greater than the second similarity threshold, then the attribute description data will be used as attribute description information.

[0034] Optionally, if the video clip is a scene video clip, retrieving attribute description information matching the video clip from the attribute description data includes:

[0035] If a noun in the attribute description data matches the scene category label, or if the similarity between the frame features in the scene video clip and the attribute description data is greater than the third similarity threshold, then the attribute description data will be used as attribute description information.

[0036] Secondly, a training device for a cross-modal retrieval model is provided, the device comprising:

[0037] The selection module is used to select video segments of each category by mining the videos in the existing training dataset, wherein each video segment category includes only people, objects or scenes of the same category.

[0038] The recognition module is used to identify the attribute description text of the video segment from the text in the existing training dataset;

[0039] The building module is used to construct a new training dataset based on the attribute description text and the video clip;

[0040] The supplementary module is used to supplement the existing training dataset with the newly added training dataset to obtain the target training dataset, and to train the cross-modal retrieval model using the target training dataset.

[0041] Thirdly, an electronic device is provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0042] Memory, used to store computer programs;

[0043] The processor, when executing a program stored in memory, implements the training method steps of any of the cross-modal retrieval models described above.

[0044] Fourthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the training method steps of any of the cross-modal retrieval models described above.

[0045] The technical solutions provided in this application have the following advantages compared with the prior art:

[0046] The method provided in this application selects video clips containing only one type of object from the training dataset, and selects attribute description information matching the video clips. Thus, the attribute description information and video clips are data describing only the object, and can be used as new training datasets. These new training datasets are added to the existing training dataset to obtain the target training dataset, which is then used to train a cross-modal retrieval model. This target training dataset contains both an overall description of the video and detailed feature descriptions, improving the cross-modal model's ability to describe fine-grained attribute information and its retrieval performance. Attached Figure Description

[0047] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 is a flowchart of a training method for a cross-modal retrieval model provided in an embodiment of this application;

[0050] Figure 2 is a training block diagram of a cross-modal retrieval model provided in an embodiment of this application;

[0051] Figure 3 is a schematic diagram of the structure of a training device for a cross-modal retrieval model provided in an embodiment of this application;

[0052] Figure 4 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0054] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustration and has no specific meaning in itself. Therefore, "module" and "part" may be used interchangeably.

[0055] To address the problems mentioned in the background art, according to one aspect of the embodiments of this application, an embodiment of a training method for a cross-modal retrieval model is provided.

[0056] This application provides a method for training a cross-modal retrieval model, which can be applied to a server to improve cross-modal retrieval performance.

[0057] The following will describe in detail a training method for a cross-modal retrieval model provided in this application embodiment, with reference to specific implementation methods. As shown in Figure 1, the specific steps are as follows:

[0058] Step 101: Select video segments of each category by mining the videos in the existing training dataset.

[0059] Each video clip category includes only people, objects of the same category, or scenes.

[0060] In this embodiment, the existing training dataset for the cross-modal retrieval model includes videos and text, with the text providing a comprehensive description of the videos. The videos include people, objects, and scenes, which can appear individually or simultaneously. Objects include items such as tables, sofas, and lamps, while scenes can be architectural scenes, living room scenes, sunset scenes, etc.

[0061] The server uses detection technology to determine the object detection box of a certain type of object, and determines the time point information of the object detection box in the video. The time point information includes the start time and the end time. The server cuts the video according to the time point information to obtain a video segment containing only one type of object.

[0062] Step 102: Identify the attribute description text of video clips from the text in the existing training dataset.

[0063] The server uses text analysis technology to analyze the text in the existing training dataset and identify attribute description data. Attribute description data consists of adjectives and nouns with modifying relationships, as well as nouns without adjective modifications. Since the attribute description data is used to describe the video, some attribute description data may describe object attributes; this portion of the attribute description data is considered attribute description information.

[0064] Step 103: Construct a new training dataset based on the attribute description text and video clips.

[0065] Since the attribute description information matches the object attribute information, the attribute description information and video clips are used as new training datasets.

[0066] Step 104: Add the new training dataset to the existing training dataset to obtain the target training dataset.

[0067] The server adds new training datasets to the existing training dataset to obtain the target training dataset, and then uses the target training dataset to train the cross-modal retrieval model.

[0068] This application selects video clips containing only one type of object from the training dataset and extracts attribute descriptions matching those video clips. This means that both the attribute descriptions and the video clips are data describing only that object, and can be used as new training data. These new training datasets are then added to the existing training dataset to obtain the target training dataset, which is used to train a cross-modal retrieval model. This target training dataset contains both an overall description of the video and detailed feature descriptions, improving the cross-modal model's ability to describe fine-grained attribute information and its retrieval performance.

[0069] As an optional implementation method, the video segments of each category are selected by mining the videos in the existing training dataset, including: performing object detection on the videos in the existing training dataset to obtain video frames containing only objects, where the objects are people, objects, or scene scenes; using localization technology to determine the time point information of consecutive video frames; cutting the video according to the time point information, and combining the multiple cut video frames to obtain video segments.

[0070] The server uses detection technology to determine the object detection box of a certain type of object. By tracking and locating the object detection box, the server obtains the time point information of the object detection box in the video. The time point information includes the start time and the end time. The server cuts the video according to the time point information to obtain multiple video sub-segments containing only one type of object. Then, the multiple video sub-segments are combined to obtain the video segment.

[0071] This application utilizes spatial and temporal ranges to cut videos, obtaining video clips containing only the objects, further reducing the influence of other video content and improving the expression of object detail features.

[0072] Exemplary, this application includes three embodiments.

[0073] If the object is a person, person detection technology is used to detect people in the videos of the existing training dataset to obtain person detection boxes. Then, tracking technology is used to track the person detection boxes to obtain the screen area and time point where a single person appears in the video. The time point information of the person detection box in the video is obtained, that is, the start time and end time of the video containing the person. The original video is then cut using the spatial and temporal ranges to generate a video clip containing only the person.

[0074] If the object is a real object, object detection technology is used to perform general object detection on the videos in the existing training dataset to determine the object detection box, obtain the position information of the object detection box in the frame and the object category label, and then track the object detection box to obtain the frame area and time point where the individual object appears in the video frame; a general object tracking method is used to track the object box to obtain the time point information of the object box appearing in the video, that is, the start time and end time of the video containing the object frame; the original video is then cut using spatial and temporal ranges to generate video content segments containing the object frame and the object category label.

[0075] If the object is a scene, this method analyzes each video frame in the existing training dataset to obtain video frames containing scene images, identifies the scene category label corresponding to the image, and uses shot detection technology to locate the shot intervals of video frames containing the same scene category label, identifying the time point of the shot where the scene image is located, i.e., the start time and end time of the shot; the original video is cut using the time range to generate video content segments containing scene images and scene category labels.

[0076] As an optional implementation method, a text analysis scheme is adopted to identify each attribute description data from the text of the existing training dataset, including: using part-of-speech analysis technology to identify nouns and adjectives from the text of the existing training dataset; using syntactic analysis technology to identify word combinations of nouns and adjectives with modifying relationships, as well as single nouns without adjective modification; and treating word combinations or single nouns as one attribute description data.

[0077] This application uses text part-of-speech analysis and (dependency) syntactic analysis techniques to identify name components in text descriptions and uses the corresponding modifying components as attribute information descriptions of the nouns. Part-of-speech analysis can obtain noun and adjective elements in the text description; syntactic analysis can obtain lexical combinations of nouns and adjectives with modification dependency relationships; adjectives and nouns with modification relationships are combined as one attribute description data; nouns without adjective modification are separated into independent attribute description data.

[0078] This application extracts nouns from text as attribute description data. On this basis, it adds adjectives that modify the nouns to enrich the features of the attribute description data, which can also improve the training effect in subsequent model training.

[0079] Finding attribute description information that matches video clips from attribute description data includes three implementation methods:

[0080] In one embodiment, the server compares the object category label with each attribute description data. If there are nouns in the attribute description data, and a noun in a certain attribute description data matches the object category label, then the attribute description data is used as attribute description information.

[0081] In another embodiment, the server determines the text similarity between the object category label and the nouns in each attribute description data. If the text similarity between a noun in the attribute description data and the object category label is greater than a set feature threshold, it indicates that the attribute description data is likely to describe the object, and the attribute description data is then used as attribute description information.

[0082] In another embodiment, the server uses a cross-modal image retrieval model to compare the similarity between the video frame features and the attribute description data. If the similarity is greater than a preset feature threshold, the attribute description data is used as attribute description information.

[0083] The server can also expand the attribute description information to make it a complete sentence, making it more coherent as training data.

[0084] For example, feature similarity is used to match video clips of people with attribute description data to generate video-text attribute description data pairs of people's attributes:

[0085] A cross-modal image retrieval model is used to compare the similarity between keyframe features in a person's video clip and a certain attribute description data. If the similarity is greater than the first similarity threshold th1 (range 0-1, typical value 0.3), the person's video clip is considered to match the text description and can be used as a pair of training data.

[0086] For example, label similarity is used to filter object video clips and attribute description data to generate video-text attribute description data pairs for object attributes:

[0087] 1. Compare the object category label with the attribute description data. If a noun in the attribute description data matches the label, the object video clip is considered to match the attribute description data and can be used as a pair of training data.

[0088] 2. Compare the object category label with each attribute description data. If there is a noun in the attribute description data whose text similarity to the label is greater than th2 (0-1 range, typical value 0.8), then the object video clip is considered to match the attribute description data and can be used as a pair of training data.

[0089] 3. Use a cross-modal image retrieval model to compare the similarity between the key frame features in the video clip of an object and the description data of a certain attribute. If the similarity is greater than the second similarity threshold th3 (range 0-1, typical value 0.5), the video clip of the object is considered to match the description data of that attribute and can be used as a pair of training data.

[0090] For example, tag similarity is used to filter scene video clips and attribute description data to generate video-text attribute description data pairs for scene attributes:

[0091] 1. Compare the scene category label with the attribute description data. If a noun in the attribute description data matches the label, the scene video clip is considered to match the text description.

[0092] 2. Compare the scene category label with each attribute description data. If a noun in the attribute description data has a text similarity greater than th4 (0-1 range, typical value 0.7) with the label, then the scene video clip is considered to match the attribute description data.

[0093] 3. Use a cross-modal image retrieval model to compare the similarity between the keyframe image features in the scene video clip and the attribute description data. If the similarity is greater than the third similarity threshold th5 (range 0-1, typical value 0.5), then the person video clip is considered to match the attribute description data.

[0094] Based on the same technical concept, this application also provides a training block diagram of a cross-modal retrieval model, as shown in Figure 2.

[0095] Based on the same technical concept, this application embodiment also provides a training device for a cross-modal retrieval model, as shown in Figure 3. The device includes:

[0096] The selection module 301 is used to select video segments of each category by mining the videos in the existing training dataset. Each video segment includes only people, objects or scenes of the same category.

[0097] The recognition module 302 is used to recognize the attribute description text of video segments from the text in the existing training dataset;

[0098] Module 303 is used to build new training datasets based on attribute description text and video clips;

[0099] The supplementary module 304 is used to supplement the existing training dataset with the new training dataset to obtain the target training dataset, and to train the cross-modal retrieval model using the target training dataset.

[0100] Optionally, module 301 is selected for:

[0101] By performing object detection on videos in the existing training dataset, video frames containing only objects are obtained, where objects can be people, objects, or scenes.

[0102] Positioning technology is used to determine the time point information of video frames;

[0103] After cutting the video according to the time point information, the cut video frames are combined to obtain a video segment.

[0104] Optionally, module 301 is selected for:

[0105] Perform person detection on videos in the existing training dataset to obtain person detection boxes;

[0106] By tracking the bounding boxes of people in the video, video frames containing only people are obtained.

[0107] Optionally, module 301 is used to: perform general object detection on videos in the existing training dataset to obtain object detection boxes and object category labels;

[0108] By tracking object detection boxes with the same object category label, video frames containing only objects of the same type are obtained.

[0109] Optionally, the selection module 301 is used to: analyze each video frame of the existing training dataset to obtain video frames containing scene images, wherein the video frames carry scene category labels.

[0110] Optionally, the identification module 302 is used for:

[0111] A text analysis approach is used to identify the description data of each attribute from the text in the existing training dataset;

[0112] Find the attribute description information that matches the video clip from the attribute description data.

[0113] Optionally, the identification module 302 is used for:

[0114] Part-of-speech tagging (POS) is used to identify nouns and adjectives from the text in the existing training dataset.

[0115] Using syntactic analysis techniques, we can identify word combinations of nouns and adjectives that have a modifying relationship, as well as standalone nouns that do not have an adjective modifying them;

[0116] Treat word combinations or individual nouns as a single attribute description data.

[0117] Optionally, if the video clip is a video clip of a person, the recognition module 302 is used for:

[0118] If the similarity between the frame features in a person's video clip and the attribute description data is greater than the first similarity threshold, then the attribute description data will be used as the attribute description information of the person's video clip.

[0119] Optionally, if the video clip is an object video clip, the recognition module 302 is used for:

[0120] If a noun in the attribute description data matches the object category label, or if the similarity between the frame features in the object video clip and the attribute description data is greater than the second similarity threshold, then the attribute description data will be used as attribute description information.

[0121] Optionally, if the video clip is a scene video clip, the recognition module 302 is used for:

[0122] If a noun in the attribute description data matches the scene category label, or if the frame features in the scene video clip have a similarity greater than the third similarity threshold with the attribute description data, then the attribute description data will be used as attribute description information.

[0123] Based on the same technical concept, this embodiment of the invention also provides an electronic device, as shown in FIG4, including a processor 401, a communication interface 402, a memory 403, and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404.

[0124] Memory 403 is used to store computer programs;

[0125] The processor 401 performs the above steps when executing the program stored in the memory 403.

[0126] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0127] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0128] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0129] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0130] In another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and the computer program, when executed by a processor, implements the steps of any of the above methods.

[0131] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the methods described above.

[0132] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0133] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0134] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A training method for a cross-modal retrieval model, characterized in that, The method includes: mining videos from an existing training dataset to select video segments of each category, wherein each video segment category includes only people, objects of the same category, or scene images; identifying attribute description text of the video segments from the text of the existing training dataset; constructing a new training dataset based on the attribute description text and the video segments; supplementing the existing training dataset with the new training dataset to obtain a target training dataset, and training a cross-modal retrieval model using the target training dataset, wherein the target training dataset contains an overall description of the video and a detailed feature description of the video; wherein identifying attribute description text of the video segments from the text of the existing training dataset includes: using part-of-speech tagging to identify nouns and adjectives from the text of the existing training dataset; using syntactic analysis to identify word combinations of nouns and adjectives with modifying relationships, as well as single nouns without adjective modification; using the word combination or the single noun as an attribute description data entry; and finding attribute description information matching the video segment from the attribute description data.

2. The method according to claim 1, characterized in that, By mining videos from the existing training dataset, the selection of video segments for each category includes: performing object detection on the videos in the existing training dataset to obtain video frames containing only the objects, where the objects are people, objects, or scene scenes; using localization technology to determine the time point information of the video frames; cutting the video according to the time point information, and combining the cut video frames to obtain video segments.

3. The method according to claim 2, characterized in that, The step of obtaining video frames containing only the objects by performing object detection on the videos in the existing training dataset includes: performing human detection on the videos in the existing training dataset to obtain human detection boxes; and tracking the human detection boxes in the videos to obtain video frames containing only the human.

4. The method according to claim 2, characterized in that, The step of obtaining video frames containing only the objects by performing object detection on the videos in the existing training dataset includes: performing general object detection on the videos in the existing training dataset to obtain object detection boxes and object category labels; and tracking the object detection boxes with the same object category label to obtain video frames containing only the same type of object.

5. The method according to claim 2, characterized in that, The step of performing object detection on the videos in the existing training dataset to obtain video frames containing only the scene includes: analyzing each video frame in the existing training dataset to obtain video frames containing the scene, wherein the video frames carry scene category labels.

6. The method according to claim 1, characterized in that, If the video clip is a person video clip, finding the attribute description information that matches the video clip from the attribute description data includes: if the similarity between the frame features in the person video clip and the attribute description data is greater than a first similarity threshold, then the attribute description data is used as the attribute description information of the person video clip.

7. The method according to claim 1, characterized in that, If the video clip is an object video clip, finding attribute description information that matches the video clip from the attribute description data includes: if there are nouns in the attribute description data that match the object category label, or if the similarity between the frame features in the object video clip and the attribute description data is greater than a second similarity threshold, then the attribute description data is used as the attribute description information.

8. The method according to claim 1, characterized in that, If the video clip is a scene video clip, finding attribute description information that matches the video clip from the attribute description data includes: if there are nouns in the attribute description data that match the scene category label, or if the similarity between the frame features in the scene video clip and the attribute description data is greater than the third similarity threshold, then the attribute description data is used as the attribute description information.

9. A training device for a cross-modal retrieval model, characterized in that, The apparatus includes: a selection module for mining videos in an existing training dataset to select video segments of each category, wherein each video segment category includes only people, objects of the same category, or scene elements; a recognition module for recognizing attribute description text of the video segments from the text in the existing training dataset; a construction module for constructing a new training dataset based on the attribute description text and the video segments; and a supplementation module for supplementing the new training dataset to the existing training dataset to obtain a target training dataset, and using the target training dataset to train a cross-modal retrieval model, wherein the target training dataset includes an overall description of the video and a detailed feature description of the video; wherein the recognition module is used to: use part-of-speech tagging to identify nouns and adjectives from the text in the existing training dataset; use syntactic analysis to identify word combinations of nouns and adjectives with modifying relationships, as well as single nouns without adjective modification; treat the word combinations or the single nouns as attribute description data; and find attribute description information matching the video segments from the attribute description data.

10. An electronic device, characterized in that, The system includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; the memory is used to store computer programs; and the processor, when executing the program stored in the memory, implements the steps of the method described in any one of claims 1-9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-9.

Images