Method and electronic device for constructing a retrieval dataset based on a video source dataset

Abstract

The present disclosure provides a method and an electronic device for constructing a retrieval dataset based on a video source dataset, the method comprising: obtaining first video data and second video data; determining a plurality of reference images from a plurality of first images of the first video data; for each reference image, performing the following operations: determining similarity information between the reference image and each first image, and determining a first image satisfying a preset similarity condition as a candidate target image of the reference image; generating relative description information between the reference image and the candidate target image as candidate relative description information; constructing a positive sample set and a negative sample set of the reference image based on the candidate target image according to the candidate relative description information; and constructing the retrieval dataset based on the positive sample set and the relative description information of each reference image, the negative sample set, and a plurality of second images of the second video data.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to the field of multimedia technology, and more particularly, to a method for constructing a retrieval dataset based on a video source dataset and an electronic device. BACKGROUND

[0002] In the field of image retrieval, the task of Composed Image Retrieval (CIR) based on text description has made significant progress, especially in the task of Zero-Shot Composed Image Retrieval (ZS-CIR). Zero-Shot Composed Image Retrieval aims to solve an important challenge, that is, to retrieve relevant images (called target images) from an image library through a query composed of a reference image and a relative description text without explicit training samples. Although a large number of researches in recent years have improved the performance of CIR, especially by using deep learning and cross-modal learning methods, there are still some serious problems in existing zero-shot composed image retrieval datasets, which affect the performance and credibility of the model in practical applications.

[0003] One of the prominent problems is that there is often a lack of consistency between the image pairs (i.e., reference images and target images) in existing zero-shot composed image retrieval datasets. Specifically, there may be visual or semantic mismatches between the image pairs, for example, some images may be taken under different backgrounds, angles, or lighting conditions, resulting in large visual differences between them; and in terms of semantics, the objects or scenes in the images may overlap but lack strict correlation. This makes it difficult for the model to capture effective visual-textual correlation information during training, and even worse, the semantic correlation between the image pairs is loose and the visual difference is large, which cannot represent the natural consistency between images in the real world. This inconsistency causes interference in the model training process, reducing the accuracy and reliability of the retrieval model. SUMMARY

[0004] The present disclosure provides a method for constructing a retrieval dataset based on a video source dataset and an electronic device, which is used to solve at least one of the above problems.

[0005] According to a first aspect of the embodiments of the present disclosure, a method for constructing a retrieval dataset based on a video source dataset is provided, the method comprising: obtaining first video data and second video data; determining a plurality of reference images from a plurality of first images of the first video data; for each reference image, performing the following operations: determining similarity information between the reference image and each first image, and determining a first image whose similarity information meets a preset similarity condition as a candidate target image of the reference image; generating relative description information between the reference image and the candidate target image as candidate relative description information, wherein the relative description information is used to describe the content association between two images; constructing a positive sample set and a negative sample set of the reference image based on the candidate target image according to the candidate relative description information; and constructing a retrieval dataset based on the positive sample set of each reference image and its relative description information, the negative sample set, and a plurality of second images of the second video data.

[0006] Optionally, the determining a plurality of reference images from a plurality of first images of the first video data comprises: dividing the plurality of first images into at least one image subset according to a set number of frames; extracting a plurality of candidate reference images from the at least one image subset; determining visual similarity and semantic similarity between the plurality of candidate reference images; and determining images with visual similarity less than a first visual threshold and semantic similarity less than a semantic threshold from the plurality of candidate reference images to obtain the plurality of reference images.

[0007] Optionally, the similarity information comprises at least one of visual similarity and semantic similarity, and the determining a first image whose similarity information meets a preset similarity condition as a candidate target image of the reference image comprises: determining a first image whose each similarity is within a preset value range corresponding to the similarity as a preliminary target image of the reference image; and determining a preliminary target image with visual similarity less than or equal to a second visual threshold with other preliminary target images as the candidate target image of the reference image.

[0008] Optionally, the constructing a positive sample set and a negative sample set of the reference image based on the candidate target image according to the candidate relative description information comprises: determining candidate relative description information meeting a preset condition in each candidate relative description information as reference relative description information; determining text similarity between each candidate relative description information and the reference relative description information as candidate similarity of the candidate relative description information; and constructing the positive sample set and the negative sample set of the reference image based on the candidate target image according to the candidate similarity.

[0009] Optionally, the determining the candidate relative description information satisfying the preset condition as the reference relative description information comprises: determining, for each candidate relative description information, a text similarity between the candidate relative description information and other candidate relative description information as a first similarity of the candidate relative description information; determining a maximum value in each first similarity of the candidate relative description information, and determining a larger one between the maximum value and a preset bottom similarity as a second similarity of the candidate relative description information; determining a maximum value in second similarities of all candidate relative description information as a reference similarity, and determining a candidate relative description information corresponding to the reference similarity as the reference relative description information.

[0010] Optionally, the constructing the positive sample set and the negative sample set of the reference image based on the candidate target image according to the candidate similarity comprises: determining a final relative description information based on candidate relative description information corresponding to a candidate similarity greater than or equal to the reference similarity; and selecting a candidate target image corresponding to the final relative description information to obtain a positive sample set of the reference image; and determining a set of images in the candidate target image that are not included in the positive sample set of the reference image as a negative sample set of the reference image.

[0011] Optionally, the publishing time of the first video data and the second video data is later than a preset time, and the preset time is the latest time at which a pre-training of an image processing model disclosed when the method of constructing a retrieval data set based on a video source data set is performed is completed.

[0012] According to a second aspect of the embodiments of the present disclosure, a device for constructing a retrieval data set based on a video source data set is provided, and the device comprises: an acquisition unit configured to acquire first video data and second video data; a determination unit configured to determine a plurality of reference images from a plurality of first images of the first video data; an execution unit configured to, for each reference image, perform the following operations: determining similarity information between the reference image and each first image, and determining a first image satisfying a preset similarity condition as a candidate target image of the reference image; generating relative description information between the reference image and the candidate target image as candidate relative description information, wherein the relative description information is used to describe the content association between two images; and constructing a positive sample set and a negative sample set of the reference image based on the candidate target image according to the candidate relative description information; and a construction unit configured to construct a retrieval data set based on the positive sample set and the relative description information of each reference image, the negative sample set, and a plurality of second images of the second video data.

[0013] Optionally, the determining unit is further configured to: divide the multiple frames of first images into at least one image subset according to a set number of frames; extract multiple candidate reference images from the at least one image subset; determine visual similarity and semantic similarity between the multiple candidate reference images; and determine, from the multiple candidate reference images, images with visual similarity less than a first visual threshold and semantic similarity less than a semantic threshold, to obtain the multiple reference images.

[0014] Optionally, the similarity information includes at least one of visual similarity and semantic similarity, and the executing unit is further configured to: determine, as a preliminary target image of the reference image, a first image in which each similarity in the similarity information is within a preset value range corresponding to the similarity; and determine, as a candidate target image of the reference image, a preliminary target image with visual similarity less than or equal to a second visual threshold between the preliminary target image and other preliminary target images.

[0015] Optionally, the executing unit is further configured to: determine, as reference relative description information, candidate relative description information that meets a preset condition in each candidate relative description information; determine, as a candidate similarity of the candidate relative description information, a text similarity between each candidate relative description information and the reference relative description information; and construct, based on the candidate target image, a positive sample set and a negative sample set of the reference image according to the candidate similarity.

[0016] Optionally, the executing unit is further configured to: for each candidate relative description information, determine, as a first similarity of the candidate relative description information, a text similarity between the candidate relative description information and other candidate relative description information; determine, as a second similarity of the candidate relative description information, a maximum value in the first similarities of the candidate relative description information and a larger one between a preset bottom guarantee similarity; determine, as a reference similarity, a maximum value in the second similarities of all candidate relative description information; and determine, as the reference relative description information, candidate relative description information corresponding to the reference similarity.

[0017] Optionally, the executing unit is further configured to: determine, based on candidate relative description information corresponding to a candidate similarity greater than or equal to the reference similarity in each candidate similarity, a final relative description information; select a candidate target image corresponding to the final relative description information to obtain a positive sample set of the reference image; and determine, as a negative sample set of the reference image, a set of images in the candidate target image that are not included in the positive sample set of the reference image.

[0018] Optionally, the publishing time of the first video data and the second video data are both later than a preset time, wherein the preset time is the latest time when the image processing model disclosed in the method of constructing the retrieval dataset based on the video source dataset is completed for pre-training.

[0019] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor; at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, cause the at least one processor to perform the method of constructing a retrieval dataset based on a video source dataset according to the exemplary embodiments of the present disclosure.

[0020] According to a fourth aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, wherein the instructions in the computer readable storage medium, when executed by at least one processor, cause the at least one processor to perform the method of constructing a retrieval dataset based on a video source dataset according to the exemplary embodiments of the present disclosure.

[0021] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, comprising computer instructions, wherein the computer instructions, when executed by at least one processor, cause the at least one processor to perform the method of constructing a retrieval dataset based on a video source dataset according to the exemplary embodiments of the present disclosure.

[0022] The technical solutions provided by the embodiments of the present disclosure at least bring the following beneficial effects: the method and the electronic device for constructing a retrieval dataset based on a video source dataset according to the present disclosure are based on the video source dataset to construct the retrieval dataset. Since each frame in the video carries continuous image information, the scene, the person and the activity expressed by the video are often more coherent in vision. By determining the reference image and the corresponding target image from the same video data (i.e. the first video data), the image in the same video data naturally has strong consistency in vision and semantics, which can effectively construct image pairs with high consistency, thereby reducing the common problem of inconsistency in vision and semantics in the traditional retrieval dataset. In addition, by constructing the target image corresponding to the reference image as a positive sample set and constructing other images in the candidate target image as a negative sample set of the reference image, the retrieval dataset can be distinguished from the existing retrieval dataset, and examples of error queries can be provided, which lays a foundation for constructing a more real and accurate retrieval model and can effectively improve the retrieval performance of the model.

[0023] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure, and do not constitute an improper limitation on the present disclosure.

[0025] Figure 1 is a flowchart of a method of constructing a retrieval dataset based on a video source dataset according to an exemplary embodiment of the present disclosure.

[0026] Figure 2 is a flowchart of constructing a positive sample set and a negative sample set of reference images by a method of constructing a retrieval dataset based on a video source dataset according to an exemplary embodiment of the present disclosure.

[0027] Figure 3 is a schematic diagram of a framework of a method of constructing a retrieval dataset based on a video source dataset according to a specific embodiment of the present disclosure.

[0028] Figure 4 is a schematic diagram of a retrieval dataset according to a specific embodiment of the present disclosure.

[0029] Figure 5 is a block diagram of an apparatus for constructing a retrieval dataset based on a video source dataset according to an exemplary embodiment of the present disclosure.

[0030] Figure 6 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] In order to make the ordinary person in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the drawings.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following embodiments do not represent all embodiments consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0033] It is to be noted that "at least one of a plurality" appearing in the present disclosure represents three types of alternatives, i.e., "any one of the plurality", "a combination of any multiple of the plurality", and "all of the plurality". For example, "including at least one of A and B" includes three alternatives, i.e., (1) including A, (2) including B, and (3) including A and B. For another example, "performing at least one of step one and step two" represents three alternatives, i.e., (1) performing step one, (2) performing step two, and (3) performing step one and step two.

[0034] Hereinafter, a method of constructing a retrieval dataset based on a video source dataset and an electronic device according to exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0035] Figure 1 is a flowchart of a method of constructing a retrieval dataset based on a video source dataset according to exemplary embodiments of the present disclosure. The method can be performed on an electronic device having sufficient computing power. The constructed retrieval dataset, for example, includes but is not limited to a combined image retrieval dataset, and specifically, for example, can be a zero-shot combined image retrieval dataset.

[0036] Referring to Figure 1 In step S101, first video data and second video data are acquired.

[0037] It should be understood that the naming of the first video data and the second video data is intended to represent that they are two video data with different functions, and as will be known from the following description, the first video data serves as the source of the reference image and the images of the positive sample set and the negative sample set of the reference image, and the second video data serves as the source of other noise images in the complete retrieval pool, which can simulate the actual retrieval scenario. In other words, in addition to the query structure for the reference image constructed based on the first video data, the retrieval pool also contains images that do not belong to the query structure, which are constructed based on the second video data.

[0038] As an example, a plurality of video source datasets of different types can be selected, and they can be divided into a plurality of categories and subcategories as needed, and then the first video data and the second video data can be selected from them as needed.

[0039] In step S102, a plurality of reference images are determined from a plurality of first images of the first video data.

[0040] The step extracts multiple images from the first video data to obtain the reference images as the basis. It should be noted that the first images are images in the first video data as the source of the reference images, which means that all images in the first video data can be used as the first images, or only part of the images in the first video data can be used as the first images, which is equivalent to preliminarily screening the images in the first video data, for example, but not limited to, extracting frames in the first video data as the first images according to a fixed period (for example, 1s), extracting key frames in the first video data as the first images, and further determining multiple reference images from the multiple first images obtained through the preliminary screening. As an example, different reference images representing different contents in the first video data can be extracted to enrich the data.

[0041] In step S103, for each reference image, a positive sample set and a negative sample set of the reference image are constructed based on the multiple first images of the first video data.

[0042] Figure 2 The execution flow of the step is shown.

[0043] Referring to Figure 2 In step S201, the similarity information between the reference image and each first image is determined, and the first image whose similarity information meets a preset similarity condition is determined as a candidate target image of the reference image.

[0044] By selecting each first image according to the relative consistency, it can be ensured that each candidate target image selected has high correlation in vision and semantics between the corresponding reference image. The candidate target image obtained at this time can be used as the final basis for determining the positive sample set and the negative sample set.

[0045] In step S202, relative description information between the reference image and the candidate target image is generated as candidate relative description information.

[0046] The relative description information is used to describe the content correlation between two images. Specifically, the relative description information for each pair of reference image and candidate target image can be generated through a multi-stage large language model (LLM) assisted generation process. Each generated relative description information can accurately reflect the relationship between the reference image and the candidate target image, ensure the semantic consistency and accuracy of the description content, eliminate ambiguity in the image description, and ensure the adaptability of the generated data to image retrieval tasks (for example, but not limited to, zero-shot combined image retrieval tasks).

[0047] In step S203, according to the candidate relative description information, the positive sample set and the negative sample set of the reference image are constructed based on the candidate target image.

[0048] As an example, the reference image and the candidate target image can be screened by a Vision Transformer (ViT) model and a CLIP (Contrastive Language-Image Pre-training) model, images irrelevant to the content of the reference image or visually redundant are excluded, and finally the remaining images constitute a positive sample set, ensuring the consistency of each pair of reference image and target image in vision and semantics. The excluded candidate target images can be used to construct a negative sample set, so that the images in the negative sample set are very similar to the reference image but do not meet the retrieval requirements, thereby providing high-quality error query examples.

[0049] Referring back to Figure 1 In step S104, based on the positive sample set of each reference image and its relative description information, the negative sample set, and the multiple frames of second images of the second video data, a retrieval data set is constructed.

[0050] This step combines different video source data and generated relative description information to construct a complete image retrieval data set. The data set can contain multiple queries and multiple images, and each query is provided with multiple positive target images (i.e. images in the positive sample set) and negative target images (i.e. images in the negative sample set), ensuring that the data set can comprehensively evaluate the ability of the retrieval model (e.g. including but not limited to zero-shot combined image retrieval model) and effectively support the corresponding image retrieval task.

[0051] As an example, after the final data set is constructed, the data set can be evaluated in multiple dimensions to ensure the quality of the data set, avoid the bias of the data source, and provide real and high-quality test data for subsequent image retrieval tasks.

[0052] The method for constructing a retrieval data set based on a video source data set according to an example embodiment of the present disclosure is based on a video source data set to construct a retrieval data set. Since each frame in a video carries continuous image information, the scenes, characters and activities expressed by the video are often more coherent in vision. By determining the reference image and its corresponding target image from the same video data (i.e. the first video data), the image pairs with high consistency can be effectively constructed by taking advantage of the strong consistency in vision and semantics of the images in the same video data, thereby reducing the common problem of inconsistency in vision and semantics in traditional retrieval data sets. In addition, by constructing the target image corresponding to the reference image as a positive sample set and constructing other images in the candidate target image as a negative sample set of the reference image, the error query examples can be provided, which is different from the existing retrieval data set, and lays a foundation for constructing a more real and accurate retrieval model, and improves the accuracy of performing an image retrieval task by using such a retrieval model.

[0053] Next, the method of constructing a retrieval dataset based on a video source dataset according to an example embodiment of the present disclosure is further introduced.

[0054] Regarding step S102, i.e., how to determine the plurality of reference images, optionally, the step comprises: dividing the plurality of first images into at least one image subset according to a set number of frames; extracting a plurality of candidate reference images from the at least one image subset; determining visual similarity and semantic similarity between the plurality of candidate reference images; and determining, from the plurality of candidate reference images, images with visual similarity less than a first visual threshold and semantic similarity less than a semantic threshold, to obtain the plurality of reference images. By dividing the plurality of first images into subsets, first images that are close in time and have strong content correlation can be preliminarily divided into an image subset as much as possible, which helps to extract candidate reference images of different content. As an example, LVLM (Large Vision Language Model) can be used in combination with a prompt word to preliminarily screen these image subsets and select candidate reference images at this time. The prompt word can be used to avoid selecting similar images when screening candidate reference images, for example. In addition, by further calculating the visual and semantic similarity between the plurality of candidate reference images obtained and selecting images with smaller similarity, candidate reference images that are too consistent in visual and semantic similarity can be removed, further ensuring the relative independence of each selected reference image.

[0055] As an example, when determining the visual similarity and semantic similarity between the plurality of candidate reference images, each candidate reference image can be traversed, for example, to calculate the visual similarity and semantic similarity between the candidate reference image and each other candidate reference image, on the basis of which it is determined whether to retain the candidate reference image. For example, the average of the calculated visual similarities can be taken as the visual similarity of the candidate reference image, and the average of the calculated semantic similarities can be taken as the semantic similarity of the candidate reference image, and then the first visual threshold and the semantic threshold are combined to compare and determine whether to retain the candidate reference image; as another example, it can be determined whether each calculated visual similarity is less than the first visual threshold, and if so, the candidate reference image is retained, otherwise it is excluded, and the same applies to the judgment of semantic similarity. For the latter, in order to reduce the amount of calculation, when traversing subsequent candidate reference images, similarity calculation can only be performed on the remaining candidate reference images, and no longer on the excluded candidate reference images. Of course, other reasonable ways can also be used, which are not limited by the present disclosure.

[0056] As an example, the first visual threshold is 0.4, 0.5, 0.6, etc., and the semantic threshold is 0.7, 0.8, 0.9, etc.

[0057] As to the step S201 in the step S103, specifically, how to determine the first image satisfying the preset similarity condition in the similarity information as the candidate target image of the reference image. Optionally, the similarity information comprises at least one of visual similarity and semantic similarity. The operation comprises: determining the first image in which each similarity in the similarity information is in the preset value range corresponding to the similarity as the preliminary target image of the reference image; and determining the preliminary target image in which the visual similarity between the preliminary target image and other preliminary target images is less than or equal to the second visual threshold value as the candidate target image of the reference image.

[0058] By means of the similarity information of the reference image and each first image, the first image in which the similarity is in the preset value range is first screened as the preliminary target image. The preliminary target image having a certain similarity in vision and / or semantics with the reference image but not completely the same can be obtained by means of the moderate preset value range, which helps to improve the quality of the search data set. For the case that the similarity information comprises two or more similarities, for example, the visual similarity and the semantic similarity, as an example, the complete similarity information of all the first images can be determined first, and compared with the corresponding preset value range respectively; or one of the similarities of all the first images can be determined first, and the first image in which the similarity is in the corresponding preset value range is screened, and then the next similarity of the screened first image is calculated, and further screened in combination with the corresponding preset value range until the calculation and screening of all the similarities are completed, which can reduce the calculation amount of the comparison between the similarity and the value range. Of course, other reasonable manners can also be adopted, which are not limited in the present disclosure.

[0059] On this basis, by continuing to calculate and screen the visual similarity of the screened multiple preliminary target images, the risk of including the image which is too redundant in vision in each target image corresponding to each reference image can be reduced, and the quality of the obtained candidate target image is improved.

[0060] Optionally, the operation of constructing the positive sample set and the negative sample set of the reference image based on the candidate target image according to the candidate relative description information in step S203 includes: determining candidate relative description information in each candidate relative description information that meets a preset condition as reference relative description information; determining a text similarity between each candidate relative description information and the reference relative description information as a candidate similarity of the candidate relative description information; and constructing the positive sample set and the negative sample set of the reference image based on the candidate target image according to the candidate similarity. The candidate relative description information between each candidate target image and the reference image obtained in step S202 can reflect the content association of the candidate target image and the reference image. By determining information that meets the preset condition, specifically, by using the preset condition to find one of the candidate relative description information that is most representative of the content, the candidate relative description information can be used as a reference to evaluate other candidate relative information, and the candidate target image corresponding to the candidate relative description information with high text similarity can be classified into the positive sample set, thereby obtaining the positive sample set with close association with the reference image.

[0061] Further optionally, the operation of determining candidate relative description information in each candidate relative description information that meets a preset condition as reference relative description information includes: determining, for each candidate relative description information, a text similarity between the candidate relative description information and other candidate relative description information as a first similarity of the candidate relative description information; determining a maximum value in each first similarity of the candidate relative description information, and determining a larger one between the maximum value and a preset bottom similarity as a second similarity of the candidate relative description information; determining a maximum value in the second similarity of all candidate relative description information as a reference similarity, and determining candidate relative description information corresponding to the reference similarity as the reference relative description information. By calculating the first similarity and further obtaining the second similarity, the content representativeness of each information in all candidate relative description information can be understood, that is, whether the information can represent other information. The maximum value in the second similarity can reflect the closest similarity, and the candidate relative description information corresponding to the maximum value is the reference relative description information that is most representative of the content.

[0062] Optionally, the operation of constructing the positive sample set and the negative sample set of the reference image based on the candidate target image according to the candidate similarity includes: determining the final relative description information based on the candidate relative description information corresponding to the candidate similarity greater than or equal to the reference similarity in each candidate similarity; selecting the candidate target image corresponding to the final relative description information to obtain the positive sample set of the reference image; and determining the set of images in the candidate target image that are not included in the positive sample set of the reference image as the negative sample set of the reference image. By taking the reference similarity as the reference basis for judging whether each candidate relative description information has a high text similarity with the reference relative description information, the judgment standard can be improved, the final relative description information highly matched with the reference image and the corresponding positive target image are obtained, and the quality of the positive sample set is improved.

[0063] In addition to the problems of the prior art described above, another problem of the existing combined image retrieval dataset is that many existing zero-sample combined image retrieval datasets often contain a large amount of dataset that has been used for pre-training (e.g., pre-training of deep learning models such as CLIP). Due to the inclusion of extensive image data, these pre-trained datasets cause the model to have been trained on certain images in the zero-sample combined image retrieval task, thereby not truly being in a zero-sample environment. This "data leakage" phenomenon causes the model not to truly face a zero-sample situation in the test phase, thereby losing the true test of the zero-sample learning ability, resulting in the retrieval task being unable to truly test the generalization ability and zero-sample ability of the model. As an example, the candidate relative description information corresponding to the candidate similarity greater than or equal to the reference similarity can be directly determined as the final relative description information, or the candidate relative description information corresponding to the candidate similarity greater than or equal to the reference similarity can be further processed in quality, such as using a large language model to convert it according to a preset description requirement to obtain the final relative description information, which is not limited in the present disclosure.

[0064] To solve this problem, optionally, the publication time of the first video data and the second video data of the exemplary embodiments of the present disclosure is later than a preset time, wherein the preset time is the latest time when the disclosed image processing model completes pre-training in the method of constructing a retrieval dataset based on a video source dataset.

[0065] After the latest time of completing pre-training by using the currently disclosed image processing model, the video data published after the time can be conveniently ensured to be used to construct the retrieval dataset, ensuring that the created retrieval dataset is irrelevant to the existing pre-training model, guaranteeing that the model is in a real zero-shot environment during the training and testing process. This means that the model needs to rely on the direct matching between the relative description information and the image content, rather than relying on previously seen data. This feature avoids the influence of data leakage, making the method more consistent with the real application scenario of zero-shot learning, effectively testing the generalization ability of the model, and promoting the innovation of image retrieval technology in various practical applications. Taking the pre-training model such as CLIP as an example, the preset time can be March 31, 2022, which can ensure that the video content used is not included in the training data of the pre-training model such as CLIP, that is, to avoid the used video dataset has been trained by the pre-training model such as CLIP, which can reduce the risk of data bias problem.

[0066] Next, combined with Figure 3 The method for constructing a retrieval dataset based on a video source dataset according to a specific embodiment of the present disclosure is introduced. The retrieval dataset constructed is named ZeroSight. It should be noted that, Figure 3 In the specific embodiments, a white rectangle is used to represent the processed image.

[0067] Referring to Figure 3 The specific embodiment includes the following steps.

[0068] Step 1, video dataset screening.

[0069] This step corresponds to step S101 in Figure 1 , from the high-quality video dataset published after March 31, 2022, the video data that meets the conditions is selected to ensure that the selected video has high quality and representativeness. In order to provide high-quality video sources, videos with clear image quality and rich scene changes can be selected to ensure that the images extracted later have sufficient visual information and diversity. Multiple different types of video source datasets can be selected and divided into multiple categories and subcategories as needed.

[0070] Step 2, video frame extraction.

[0071] This step performs frame extraction on the video data selected from the video data according to the method of extracting 1 frame per second, and the extracted video frame is used as the first image. The set of these first images is denoted as F (i.e. Figure 3 The row of images located below in the part of step 2 in

[0072] Step 3, reference image generation.

[0073] This step corresponds to step S102 in Figure 1 Further reference images are extracted from the same video (i.e., the first video data) based on the first images extracted in step 2. To ensure the diversity of ZeroSight, the reference images are made non-redundant and significantly different. To this end, the extracted first image set F is first divided into equidistant subsets , which can be represented by the following formula.

[0074]

[0075] In the above formula, denotes the number of elements in the set F , i.e., the number of first images extracted in step 2. denotes the set of equidistant subsets , each subset contains at most 10 frames of images.

[0076] Then, existing LVLMs (e.g., GPT-4, Generative Pre-trained Transformer 4) are used to preliminarily screen these subsets and select candidate reference images , which can be represented by the following formula.

[0077]

[0078] The meaning of this formula is that if the first candidate reference image is selected, only the LVLM is used to screen each subset in turn; if a new candidate reference image is selected under the condition that there are already several selected candidate reference images, the previously selected candidate reference image is also input into the LVLM to continue screening the subsets, so that each selected candidate reference image is different from the previous one, thereby ensuring that similar images are avoided when preliminarily selecting candidate reference images. Among them, is the prompt for using the LVLM when selecting the first candidate reference image, is the prompt for using the LVLM when selecting a new candidate reference image under the condition that there are already several selected candidate reference images. At most one candidate reference image is determined for each subset, and the candidate reference images selected in this process form the candidate reference image set , where denotes the number of elements in the set C , i.e., the number of subsets the number of selected images. To further ensure the relative independence of each selected image, an additional step can also be performed.

[0079] First, a visual transformer (ViT) model is used to remove visually too consistent candidate reference images, expressed in the following equation.

[0080]

[0081] In the above equation, represents a set of candidate reference images that are relatively independent in the visual aspect, each element of which is a candidate reference image. is the upper limit of the filtered visual similarity threshold, i.e., the first visual threshold. The visual similarity of each candidate reference image with other candidate reference images is checked in turn, and all candidate reference images with a visual similarity exceeding the upper limit of the visual similarity threshold are deleted for each candidate reference image.

[0082] Then, a CLIP model is used to remove semantically too consistent candidate reference images. The final result is a set of reference images that are relatively independent in both visual and semantic aspects, each element of which is a reference image, expressed in the following equation.

[0083]

[0084] In the above equation, is the upper limit of the filtered semantic similarity threshold, i.e., the semantic threshold. The semantic similarity of each candidate reference image with other candidate reference images is checked in turn, and all candidate reference images with a semantic similarity exceeding the upper limit of the semantic similarity threshold are deleted for each candidate reference image. Corresponding to the part of step 3 in Figure 3 , the multiple images on the left represent candidate reference images, and after the first screening criterion (i.e., visual similarity < 50%, semantic similarity < 80%) is screened, the multiple reference images on the right are obtained. It should be understood that the ellipsis at the bottom of the three reference images on the right represents other reference images, Figure 3 , the ellipsis in other parts also represents the omitted content, which will not be described one by one in the following.

[0085] Step 4, candidate target image generation.

[0086] This step corresponds to step S201 in Figure 2 , a plurality of target images similar to the reference images but not completely identical in vision are selected to construct consistent image pairs, ensuring the coherence of the image pairs in both visual and semantic aspects.

[0087] After generating a reference image set for a video, the In the frame images (i.e., the plurality of first images), a plurality of target images are constructed for each reference image. The goal is to select images that are similar but not identical to the reference image. To this end, one can first use to filter out images that are not visually similar to the first reference image . The following equation is used.

[0088]

[0089] In the above equation, and are the lower and upper bounds of the visual similarity threshold, respectively, which constitute the preset value range of the visual similarity.

[0090] Next, to further ensure that the filtered similar images have a semantic connection with the reference image, one can consider the semantic similarity of these images with the reference image and use CLIP to further filter out , i.e., the set of images that have a certain semantic similarity with the first reference image. The following equation is used.

[0091]

[0092] In the above equation, and are the lower and upper bounds of the semantic similarity threshold, respectively, which constitute the preset value range of the semantic similarity.

[0093] Finally, to ensure that the target image set corresponding to each reference image does not contain images that are too visually redundant, one can use to filter out images that are too visually similar in . The following equation is used.

[0094]

[0095] This represents the final candidate target image set for the first reference image, where each element is a candidate target image for the first reference image. Here, is the upper bound of the visual similarity threshold, i.e., the second visual threshold. Corresponding to the part of step 4 in Figure 3 , each dashed box on the left represents a set of images , which, after being filtered by the second filtering criterion (i.e., visual similarity < 85%), results in the candidate target image set represented by the corresponding dashed box on the right .

[0096] Subsequently, for the images that are not used to generate ​video (i.e., second video data), every several frames (e.g., including but not limited to 5 frames) extract a frame image, and these images are associated with combined to form a complete retrieval pool (i.e., retrieval dataset) for the model.

[0097] Step 5, candidate relative description generation.

[0098] Based on the image pairs generated in step 4, the relative description of each image pair is generated using LVLM (such as GPT-4), obtaining candidate relative description information, ensuring semantic consistency between each image pair, while avoiding unnecessary noise and redundant information, enhancing the accuracy and relevance of the text description.

[0099] Step 6, generating relative description information, positive sample set and negative sample set.

[0100] This step realizes the construction and labeling of the retrieval dataset. Based on the image pairs generated in step 4 and the candidate relative description information generated in step 5, a dataset containing multiple positive and negative sample images and relative descriptions is constructed, ensuring the diversity and complexity of the dataset, avoiding dataset bias, and further improving the generalization ability and robustness of the model. In order to provide effective samples for the construction of the dataset, it is necessary to ensure that each combination of image pairs and relative description information is strictly screened and labeled, and the distinction between positive and negative samples is clear, to ensure the high quality and comprehensiveness of the dataset.

[0101] Specifically, after generating the reference image set and the candidate target image set, in step 5, LVLMs (such as GPT-4) are first used to generate candidate relative description information for each reference image and each candidate target image in its corresponding candidate target image set. Then, for each reference image, a candidate relative description set is constructed. , the candidate relative description set of the th reference image is represented by the following formula.

[0102]

[0103] In the above formula, is the candidate target image set corresponding to the th reference image , is the th candidate target image in , represents the candidate relative description information generated for the th reference image and the th candidate target image. Subsequently, BERT is used to calculate the text similarity between all candidate relative description pairs in , which is represented by the following formula.

[0104]

[0105] This means calculating the text similarity between the first candidate relative description information and all other candidate relative description information in the same set, i.e., the first similarity. In order to obtain the final relative description, the following function is defined.

[0106]

[0107] In the above formula, is a variable parameter, representing a preset bottom guarantee similarity, representing the second similarity. Then, the set is constructed, which is represented by the following formula.

[0108]

[0109] In the above formula, represents the reference similarity, represents the reference relative description information, and the candidate relative description information whose text similarity (specifically, the BERT similarity) with the reference relative description information is greater than or equal to the reference similarity is screened out, which is equivalent to screening out the candidate relative description information whose proportion reaches from , and these candidate relative description information are similar to each other (the text similarity is greater than or equal to ), obtaining , which is a subset of used to generate the final relative description. Corresponding to the part of step 5 in Figure 3 , each dashed box on the left side represents a candidate relative description set , which is screened out by the third screening criterion (i.e., the text similarity of the candidate relative description information whose proportion reaches k >= k), and the candidate relative description information marked with “X” is obtained, which is represented by the corresponding dashed box on the right side.

[0110] Step 6 specifically includes two parts, step 6.1 and step 6.2.

[0111] Step 6.1 is multi-target image generation, which can generate the final positive sample set based on the first reference image , which is represented by the following formula. The positive sample is a positive target image.

[0112]

[0113] the ​a reference image a negative sample set The positive sample is a negative target image.

[0114]

[0115] The negative sample set is unique in the retrieval data set of the present disclosure, and is distinguished from other data sets.

[0116] Step 6.2 is the generation of a relative description. The constructed data set can be checked for quality to remove potential errors and inconsistent samples, and the data set can be optimized to ensure that each sample can provide an effective learning signal for model training. Specifically, the LLM (such as GPT-4) can be used to adjust a semantic summary description of to generate the first reference image unified final relative description of which is expressed by the following formula.

[0117]

[0118] In the above formula, is a prompt for generating a summary text of the meaning.

[0119] Step 7, query generation.

[0120] Finally, a triple is obtained as a query structure in the retrieval data set. Figure 4 is a schematic diagram of a query structure in the retrieval data set according to a specific embodiment of the present disclosure, and with reference to Figure 4 each column represents a query, and the text at the top represents the type of relative description information. Of course, the types here are only examples and can include other reasonable types, which are not listed one by one. The image in the first row is a reference image, the text in the second row is a relative description text, i.e., relative description information, the image in the third row is a positive target image, and the image in the fourth row is a negative target image.

[0121] The generated retrieval data set can be used for model training and evaluation of its performance in real zero-shot tasks to ensure that the proposed method can effectively retrieve on unseen image and text pairs and improve the zero-shot generalization ability of the model. In order to provide reliable evaluation basis, multi-dimensional evaluation indicators can be used, including retrieval accuracy, recall rate and F1 score, etc., to comprehensively reflect the performance of the model in actual zero-shot tasks.

[0122] ​To evaluate this particular embodiment, training and testing were performed on the ZeroSight dataset, which contains 5746 training queries and 1437 testing queries. The ZeroSight dataset is split into two categories of methods evaluation, ZS-CIR and CIR. The ZS-CIR method performs retrieval of target images by reference images and corresponding relative description texts, while the CIR method is trained on the training set and only the reference images and corresponding description texts are provided at testing time.

[0123] The following Table 1 and Table 2 are comparisons of different methods using three evaluation indicators: mAP@k (Mean Average Precision at k, average precision in the first k results), PNR-mAP@k (Positive-Negative Ratio Mean Average Precision at k, average precision of positive-negative ratio in the first k results), and average. Among them, mAP@k is used to measure the accuracy of retrieval, and PNR-mAP@k is used to evaluate the precision of retrieval. In Table 1, we compare the following methods (where untrained means that the corresponding method does not require additional training process, i.e., it can be directly used): CIReVL (Compositional Image Retrieval through Vision-by-Language, image retrieval based on language vision), published in 2024 ICLR (International Conference on Learning Representations, International Conference on Learning Representations) conference; LDRE (LLM-based Divergent Reasoning and Ensemble, LLM-based divergent reasoning and ensemble), published in 2024 SIGIR (ACM Special Interest Group on Information Retrieval, International Information Retrieval Conference, where ACM is Association for Computing Machinery, International Computer Association) conference; SEIZE (Semantic Editing Increment for ZS-CIR, semantic editing increment for zero-shot combined image retrieval), published in 2024 MM (ACM International Conference on Multimedia, International Multimedia Conference) conference; PALAVRA (Personalizing Language Vision Representations, personalizing language vision representations), published in 2022 ECCV (European Conference on Computer Vision, European Conference on Computer Vision) conference;SEARLE-OTI (Zero-shot Composed Image Retrieval With Optimization-based Textual Inversion), published in 2023 ICCV (International Conference on Computer Vision); SEARLE (Zero-shot Composed Image Retrieval With Textual Inversion), published in 2023 ICCV; Captioning (captioning method), a method that generates captions for reference images using a pre-trained captioning model and extracts text features of the captions through a CLIP text encoder for retrieval; Text-only (text-only method), using only the relative description features extracted by the CLIP text encoder as retrieval features to calculate the similarity of retrieval; Image-only (image-only method), using only the features of the reference image extracted by the CLIP image encoder to calculate the similarity of retrieval; Pic2Word (a method that relies on training, which uses a pre-trained text inversion network to capture pseudo-word labels through a contrastive loss optimization and combines with relative descriptions for retrieval), published in 2023 CVPR (IEEE / CVF Conference on Computer Vision and Pattern Recognition); LinCIR (Language-only training for CIR, pure language training for CIR), published in 2024 CVPR. In Table 2, we compare the following methods: APTEMIS (Attention-based Retrieval with Text-Explicit Matching and Implicit Similarity), published in 2022 ICLR (International Conference on Learning Representations); AMC (Adaptive Multi-expert Collaborative Network), published in 2023 TOMM (ACM Transactions on Multimedia Computing, Communications, and Applications).VAL (Visiolinguistic Attention Learning), published in 2020 CVPR conference; TIRG (Text Image Residual Gating), published in 2019 CVPR conference; MAAF (Modality-Agnostic Attention Fusion), published in 2020; DCNet (Dual Composition Network), published in 2021 AAAI conference (AAAI Conference on Artificial Intelligence); DWC (Dynamic Weighted Combiner), published in 2024 AAAI conference; CLIP4CIR (a two-stage method that combines task-oriented fine-tuning with the training of a combination network that can perform fine-grained merging of multi-modal features), published in 2023 TOMM journal. According to Tables 1 and 2, it can be seen that the specific embodiment significantly outperforms other methods on the ZeroSight dataset, showing excellent performance.

[0124] Table 1

[0125]

[0126] Table 2

[0127]

[0128] The specific embodiment proposes a new method for constructing a zero-shot combined image retrieval dataset based on a video source dataset. This method effectively avoids the inconsistency problem of image pairs in existing methods by extracting consistent image pairs from video data and generating accurate text descriptions in combination with large language models. By filtering video data based on release time, the problem of pre-training data leakage and dependence on pre-training datasets in existing datasets can be solved, ensuring that the retrieval task is performed in a real zero-shot environment, making the zero-shot combined image retrieval task more reliable and challenging, and improving the performance and generalization ability of the retrieval model. This provides a new solution for the zero-shot combined image retrieval field and lays the foundation for constructing more accurate and accurate retrieval models.

[0129] Figure 5 is a block diagram of an apparatus for constructing a retrieval dataset based on a video source dataset according to an example embodiment of the present disclosure. Referring to Figure 5The apparatus 500 for constructing a retrieval dataset based on a video source dataset comprises an acquisition unit 501, a determination unit 502, an execution unit 503, and a construction unit 504.

[0130] The acquisition unit 501 can acquire first video data and second video data.

[0131] The determination unit 502 can determine a plurality of reference images from a plurality of first images of the first video data.

[0132] The execution unit 503 can, for each reference image, perform the following operations: determine similarity information between the reference image and each first image, determine a first image whose similarity information meets a preset similarity condition as a candidate target image of the reference image; generate relative description information between the reference image and the candidate target image as candidate relative description information, wherein the relative description information is used to describe the content association between two images; and construct a positive sample set and a negative sample set of the reference image based on the candidate target image according to the candidate relative description information.

[0133] The construction unit 504 can construct the retrieval dataset based on the positive sample set and its relative description information, the negative sample set of each reference image, and a plurality of second images of the second video data.

[0134] Optionally, the determination unit 502 can further: divide the plurality of first images into at least one image subset according to a set number of frames; extract a plurality of candidate reference images from the at least one image subset; determine visual similarity and semantic similarity between the plurality of candidate reference images; and determine images whose visual similarity is less than a first visual threshold and whose semantic similarity is less than a semantic threshold from the plurality of candidate reference images to obtain the plurality of reference images.

[0135] Optionally, the similarity information comprises at least one of the following: visual similarity, semantic similarity, and the execution unit 503 can further: determine a first image whose each similarity in the similarity information is within a preset value range corresponding to the similarity as a preliminary selected target image of the reference image; and determine a preliminary selected target image whose visual similarity with other preliminary selected target images is less than or equal to a second visual threshold as the candidate target image of the reference image.

[0136] Optionally, the execution unit 503 can further: determine candidate relative description information that meets a preset condition in each candidate relative description information as reference relative description information; determine a text similarity between each candidate relative description information and the reference relative description information as a candidate similarity of the candidate relative description information; and construct the positive sample set and the negative sample set of the reference image based on the candidate target image according to the candidate similarity.

[0137] Optionally, the execution unit 503 can further: determine, for each candidate relative description information, a text similarity between the candidate relative description information and other candidate relative description information as a first similarity of the candidate relative description information; determine a maximum value in the first similarities of the candidate relative description information, and determine a larger one between the maximum value and a preset bottom similarity as a second similarity of the candidate relative description information; determine a maximum value in the second similarities of all candidate relative description information as a reference similarity, and determine a candidate relative description information corresponding to the reference similarity as a reference relative description information.

[0138] Optionally, the execution unit 503 can further: determine a final relative description information based on candidate relative description information corresponding to a candidate similarity greater than or equal to the reference similarity in the candidate similarities; select a candidate target image corresponding to the final relative description information to obtain a positive sample set of the reference image; and determine a set of images in the candidate target image that are not included in the positive sample set of the reference image as a negative sample set of the reference image.

[0139] Optionally, the publishing time of the first video data and the second video data is later than a preset time, wherein the preset time is the latest time when the image processing model disclosed in the method of constructing a retrieval data set based on a video source data set completes pre-training.

[0140] As to the apparatus in the above embodiments, the specific manner in which each unit performs operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

[0141] Figure 6 A structural block diagram of an electronic device 600 according to an example embodiment of the present disclosure is shown.

[0142] Referring to Figure 6 The electronic device 600 includes at least one memory 601 and at least one processor 602, the at least one memory 601 having computer-executable instructions stored therein, which, when executed by the at least one processor 602, cause the at least one processor to perform the method of constructing a retrieval data set based on a video source data set as described in the above example embodiments.

[0143] As an example, the electronic device 600 can be a PC computer, a tablet device, a personal digital assistant, a smart phone, or other devices capable of executing the above instruction set. Here, the electronic device 600 does not necessarily have to be a single electronic device 600, but can also be any device or collection of circuits capable of executing the above instructions (or instruction set) individually or jointly. The electronic device 600 can also be part of an integrated control system or system manager, or can be configured to interface with a local or remote (e.g., via wireless transmission) portable electronic device 600.

[0144] In the electronic device 600, the processor 602 can include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, the processor 602 can also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0145] The processor 602 can execute instructions or code stored in the memory 601, where the memory 601 can also store data. The instructions and data can also be transmitted and received over a network via a network interface device, which can employ any known transmission protocol.

[0146] The memory 601 can be integrated with the processor 602, such as by being disposed within an integrated circuit microprocessor, etc. Further, the memory 601 can include a separate device, such as an external disk drive, a storage array, or other storage device usable by any database system. The memory 601 and the processor 602 can be operatively coupled, or can communicate with each other, such as through I / O ports, network connections, etc., so that the processor 602 can read files stored in the memory.

[0147] Further, the electronic device 600 can also include a video display (such as a liquid crystal display) and a user interface (such as a keyboard, a mouse, a touch input device, etc.). All components of the electronic device 600 can be connected to each other via a bus and / or a network.

[0148] According to an exemplary embodiment of the present disclosure, there can also be provided a computer-readable storage medium storing instructions, wherein the instructions, when executed by at least one processor, cause the at least one processor to perform the method of constructing a search dataset based on a video source dataset as described in the above exemplary embodiment. Examples of the computer-readable storage medium herein include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, a hard disk drive (HDD), a solid state drive (SSD), a card-type memory (such as a multimedia card, a secure digital (SD) card, or an extreme digital (XD) card), a magnetic tape, a floppy disk, a magneto-optical data storage device, an optical data storage device, a hard disk, a solid state disk, and any other device configured to store computer programs and any associated data, data files, and data structures in a non-transitory manner and provide the computer programs and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer programs. The computer programs in the above computer-readable storage medium can be executed in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc., and, in addition, in one example, the computer programs and any associated data, data files, and data structures are distributed over a networked computer system so that the computer programs and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0149] According to an exemplary embodiment of the present disclosure, there can also be provided a computer program product including computer instructions, which, when executed by at least one processor, perform the method of constructing a search dataset based on a video source dataset as described in the above exemplary embodiment.

[0150] Other embodiments of the present disclosure will be readily apparent to those skilled in the art upon considering the specification and practice of the disclosed application. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure following the general principles thereof and including such departures from the present disclosure that come within known, accepted, or customary practice in the art to which the present disclosure pertains. The specification and examples are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

[0151] It should be understood that the present disclosure is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be made by those of ordinary skill in the art without departing from the scope of this disclosure. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for constructing a retrieval dataset based on a video source dataset, characterized in that, The method includes: Acquire the first video data and the second video data; Multiple reference images are determined from multiple frames of the first image in the first video data; For each reference image, perform the following operations: Determine the similarity information between the reference image and the first image in each frame, wherein the similarity information includes at least one of the following: visual similarity and semantic similarity; The first image in which each similarity in the similarity information falls within the preset value range corresponding to that similarity is determined as the initial target image of the reference image; The preliminary target images whose visual similarity to each of the other preliminary target images is less than or equal to the second visual threshold are determined as candidate target images of the reference image; Generate relative description information between the reference image and the candidate target image as candidate relative description information, wherein the relative description information is used to describe the content association between the two images; Among the candidate relative description information, the candidate relative description information that meets the preset conditions is determined as the reference relative description information; The text similarity between each candidate relative description information and the reference relative description information is determined as the candidate similarity information of the candidate relative description information; Based on the candidate similarity, a positive sample set and a negative sample set of the reference image are constructed based on the candidate target image; A retrieval dataset is constructed based on the positive sample set and its relative descriptive information of each reference image, the negative sample set, and multiple frames of the second image from the second video data.

2. The method as described in claim 1, characterized in that, Determining multiple reference images from multiple frames of the first image in the first video data includes: According to a set number of frames, the multiple first images are divided into at least one subset of images; Extract multiple candidate reference images from the at least one image subset; Determine the visual similarity and semantic similarity among the plurality of candidate reference images; The plurality of reference images are obtained by determining images from the plurality of candidate reference images whose visual similarity is less than a first visual threshold and whose semantic similarity is less than a semantic threshold.

3. The method as described in claim 1 or 2, characterized in that, The step of determining the candidate relative description information that meets the preset conditions from among the candidate relative description information as the reference relative description information includes: For each candidate relative description information, the text similarity between the candidate relative description information and other candidate relative description information is determined, which is used as the first similarity of the candidate relative description information; Determine the maximum value among the first similarities of the candidate relative description information, and determine the larger of the maximum value and the preset minimum similarity as the second similarity of the candidate relative description information; The maximum value among the second similarities of all candidate relative description information is determined as the reference similarity, and the candidate relative description information corresponding to the reference similarity is determined as the reference relative description information.

4. The method as described in claim 3, characterized in that, The step of constructing a positive sample set and a negative sample set of the reference image based on the candidate target image according to the candidate similarity includes: Based on the candidate relative description information corresponding to the candidate similarity that is greater than or equal to the reference similarity among the candidate similarities, the final relative description information is determined; Select the candidate target image corresponding to the final relative description information to obtain the positive sample set of the reference image; The set of images in the candidate target images that are not included in the positive sample set of the reference image is determined as the negative sample set of the reference image.

5. The method as described in claim 1 or 2, characterized in that, The release times of both the first video data and the second video data are later than a preset time, wherein the preset time is the latest time when the publicly disclosed image processing model has completed pre-training when the method for constructing a retrieval dataset based on the video source dataset is executed.

6. An electronic device, characterized in that, include: At least one processor; At least one memory that stores computer-executable instructions. Wherein, when the computer-executable instructions are executed by the at least one processor, the at least one processor causes the at least one processor to perform the method for constructing a retrieval dataset based on a video source dataset as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor causes the at least one processor to perform the method of constructing a retrieval dataset based on a video source dataset as described in any one of claims 1 to 5.

8. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by at least one processor, the at least one processor causes the processor to perform the method of constructing a retrieval dataset based on a video source dataset as described in any one of claims 1 to 5.

Images