Image-text data set construction method, electronic device, storage medium and program product

Abstract

The present application discloses a method for constructing a picture-text pair dataset, an electronic device, a storage medium, and a program product, relating to the technical field of dataset construction. The method comprises: performing instance segmentation on a picture-text pair image to obtain an instance segmentation result of the image, wherein the instance segmentation result includes a target instance and instance segmentation information of the target instance; matching the target instance with the text of the picture-text pair based on the instance segmentation information of the target instance to obtain an instance matching result of the picture-text pair; if the instance matching result is a match, determining the picture-text pair as a target picture-text pair, and constructing a picture-text pair dataset based on at least one target picture-text pair. The present application can construct a high-quality picture-text pair dataset.

Description

Technical Field

[0001] The present application relates to the technical field of data set construction, and in particular to a method for constructing a text-image data set, an electronic device, a storage medium, and a program product. Background Art

[0002] With the rapid development of artificial intelligence technology, the industry's demand for high-quality image and text datasets is increasing.

[0003] A high-quality image-text pair dataset cannot be separated from high-quality image-text pairs, and the quality of an image-text pair is positively correlated with the degree of matching between the image and the text in the pair. Therefore, the traditional image-text pair dataset construction method uses feature extraction technology and vector similarity algorithm to calculate the vector similarity between the feature vector of the image in the image-text pair and the feature vector of the text as the degree of matching between the image and the text in the image-text pair, thereby screening out image-text pairs with a high degree of matching for the construction of the image-text pair dataset.

[0004] However, this method is difficult to accurately screen out image-text pairs with high matching degrees when the content of the images in the image-text pairs is relatively complex and there are many elements, resulting in low quality of the constructed image-text pair dataset. Summary of the Invention

[0005] The main purpose of this application is to provide a method for constructing a picture-text pair dataset, an electronic device, a storage medium and a program product, aiming to solve the technical problem of low quality of the picture-text pair dataset constructed in the related art.

[0006] To achieve the above objectives, the present application provides a method for constructing an image-text pair dataset, the method comprising:

[0007] Performing instance segmentation on the image of the image-text pair to obtain an instance segmentation result of the image, wherein the instance segmentation result includes a target instance and instance segmentation information of the target instance;

[0008] Matching the target instance with the text of the image-text pair according to the instance segmentation information of the target instance to obtain an instance matching result of the image-text pair;

[0009] If the instance matching result is a match, the image-text pair is determined as a target image-text pair, and an image-text pair dataset is constructed based on at least one target image-text pair.

[0010] In one example, the step of performing instance segmentation on the image of the image-text pair to obtain the instance segmentation result of the image includes:

[0011] Performing instance segmentation on the image of the image-text pair to obtain at least one instance of the image and instance segmentation information of the at least one instance;

[0012] determining a target instance from the at least one instance according to instance segmentation information of the at least one instance;

[0013] The target instance and the instance segmentation information of the target instance are used as the instance segmentation result of the image.

[0014] In one example, the instance segmentation information includes a category label, and the step of determining a target instance from the at least one instance based on the instance segmentation information of the at least one instance includes:

[0015] Determining a target category label from the category label of the at least one instance;

[0016] From the at least one instance, an instance whose class label is a target class label is determined as a target instance.

[0017] In one example, the step of determining a target category label from the category label of the at least one instance includes:

[0018] According to the category label of the at least one instance, obtaining by counting the number of repetitions of different category labels in the at least one instance;

[0019] The category label with the highest number of repetitions is determined as the target category label.

[0020] In one example, the instance segmentation information includes a bounding box mask, and the step of determining the target category label from the category label of the at least one instance includes:

[0021] determining an instance area of ​​the at least one instance based on the bounding box mask of the at least one instance;

[0022] Obtaining, based on the instance area and the category label of the at least one instance, instance areas corresponding to different category labels in the at least one instance;

[0023] The category label with the largest corresponding instance area is determined as the target category label.

[0024] In one example, the instance segmentation information includes a bounding box mask, and the step of determining the target instance from the at least one instance based on the instance segmentation information of the at least one instance includes:

[0025] Determining a weight factor of the at least one instance according to the bounding box mask of the at least one instance, and calculating a weight of the at least one instance according to the weight factor of the at least one instance, wherein the weight factor includes at least one of an instance area, an instance position, and an instance depth of field;

[0026] From the at least one instance, an instance whose weight satisfies a preset weight condition is determined as a target instance.

[0027] In one example, the instance segmentation information includes a category label, and the step of matching the target instance with the text of the image-text pair based on the instance segmentation information of the target instance to obtain an instance matching result of the image-text pair includes:

[0028] Performing text classification on the text of the image-text pair to obtain a text classification result of the text;

[0029] If the category label of the target instance matches the text classification result, then the instance matching result of the image-text pair is determined to be a match.

[0030] In one example, the instance segmentation information includes a bounding box mask, and the step of matching the target instance with the text of the image-text pair based on the instance segmentation information of the target instance to obtain an instance matching result of the image-text pair includes:

[0031] Performing image feature extraction on the target instance to obtain an image feature vector of the target instance, and performing text feature extraction on the text of the image-text pair to obtain a text feature vector of the text;

[0032] Calculating the vector similarity between the image feature vector and the text feature vector based on a preset vector similarity algorithm;

[0033] performing a weighted calculation on the vector similarity according to the weight of the target instance to obtain the instance text similarity between the target instance and the text, wherein the weight of the target instance is determined based on the bounding box mask of the target instance;

[0034] When the instance text similarity is greater than a preset similarity threshold, the instance matching result of the image-text pair is determined to be a match.

[0035] In addition, to achieve the above-mentioned purpose, the present application also provides an electronic device, which includes: a memory, a processor, and a computer program stored on the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the above-mentioned method for constructing a picture-text pair data set.

[0036] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, which is a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is executed by the processor, the method for constructing a text-image data set as described above is implemented.

[0037] In addition, to achieve the above-mentioned purpose, the present application also provides a program product, which is a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, it implements the steps of the above-mentioned method for constructing a picture-text pair dataset.

[0038] The embodiments of the present application provide a method for constructing a picture-text pair dataset, an electronic device, a storage medium, and a program product. The technical solution of the embodiments of the present application is to perform instance segmentation on the image of the picture-text pair to obtain an instance segmentation result of the image, wherein the instance segmentation result includes a target instance and instance segmentation information of the target instance; based on the instance segmentation information of the target instance, the target instance is matched with the text of the picture-text pair to obtain an instance matching result of the picture-text pair; if the instance matching result is a match, the picture-text pair is determined to be a target picture-text pair, and a picture-text pair dataset is constructed based on at least one target picture-text pair, so that the embodiments of the present application use instance segmentation technology to accurately identify and separate target instances that can represent the core content of the image from images with relatively complex content and a large number of elements, thereby matching the target instance with the text of the picture-text pair based on the instance segmentation information of the target instance, effectively reducing the interference of background and other irrelevant elements in the image, screening out high-quality target picture-text pairs, and improving the accuracy of picture-text matching, and then constructing a high-quality picture-text pair dataset based on the high-quality target picture-text pairs, ultimately solving the technical problem of low quality of the picture-text pair dataset constructed in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0041] Figure 1 A flowchart of the first embodiment of the method for constructing a text-image data set in this application is provided;

[0042] Figure 2 A flowchart of the second embodiment of the method for constructing a text-image data set of this application is provided;

[0043] Figure 3 A flowchart of the third embodiment of the method for constructing a text-image data set of this application is provided;

[0044] Figure 4 A flowchart of the fourth embodiment of the method for constructing a text-image pairing dataset of this application is provided;

[0045] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the method for constructing an image-text pair dataset in an embodiment of the present application.

[0046] The purpose, features and advantages of this application will be further explained with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0047] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0048] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0049] Traditional image-text pair dataset construction methods can accurately screen out image-text pairs with high matching degrees for image-text pair dataset construction when faced with image-text pairs with relatively simple image content and few elements. However, when faced with image-text pairs with relatively complex image content and many elements, it is difficult to extract effective image feature vectors from complex images. It is easily disturbed by the complex and diverse element content in the image, which in turn affects the quality of the screened image-text pairs, and ultimately leads to low quality of the constructed image-text pair dataset.

[0050] In this regard, the main solution of the embodiment of the present application is: perform instance segmentation on the image of the image-text pair to obtain the instance segmentation result of the image, wherein the instance segmentation result includes the target instance and the instance segmentation information of the target instance; according to the instance segmentation information of the target instance, match the target instance with the text of the image-text pair to obtain the instance matching result of the image-text pair; if the instance matching result is a match, determine the image-text pair as a target image-text pair, and construct an image-text pair dataset based on at least one of the target image-text pairs.

[0051] The embodiment of the present application uses instance segmentation technology to accurately identify and separate target instances that can represent the core content of an image from an image with relatively complex content and a large number of elements, and then matches the target instance with the text of the image-text pair based on the instance segmentation information of the target instance, effectively reducing the interference of the background and other irrelevant elements in the image, screening out high-quality target image-text pairs, and improving the accuracy of image-text matching. Based on the high-quality target image-text pairs, a high-quality image-text pair dataset is constructed, ultimately solving the technical problem of low quality of the image-text pair dataset constructed in the related art.

[0052] It should be noted that the execution entities of the embodiments of the present application are electronic devices, which may include but are not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs (Televisions) and desktop computers, or any other electronic device capable of implementing the above functions. The embodiments of the present application are not specifically limited to this. The following embodiments of the present application are described using electronic devices as the execution entities as examples.

[0053] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0054] This application proposes a method for constructing an image-text pair dataset in a first embodiment.

[0055] Please refer to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the method for constructing a text-image data set in this application.

[0056] In this embodiment, the method for constructing an image-text pair dataset may include steps S100 to S300:

[0057] Step S100, performing instance segmentation on the image of the image-text pair to obtain an instance segmentation result of the image, wherein the instance segmentation result includes a target instance and instance segmentation information of the target instance;

[0058] Those skilled in the art will recognize that an image-text pair refers to paired data consisting of an image and its corresponding textual description (i.e., text), typically used to train multimodal models (e.g., image-text retrieval and image description generation). Instance segmentation is a technique that combines object detection and semantic segmentation to identify the precise boundaries (masks), category labels, and confidence levels of each individual object in an image.

[0059] It should be noted that the target instance refers to the instance that can represent the core content of the image. The instance segmentation information generally includes the instance mask, category label, confidence and spatial position (bounding box coordinates or center point coordinates). Among them, the mask and bounding box coordinates can be collectively referred to as the bounding box mask.

[0060] In this embodiment, a target instance that can represent the core content of the image can be directly segmented from the image using a specific instance segmentation algorithm (for example, a specific instance segmentation algorithm obtained by combining instance segmentation technology with saliency detection technology). Alternatively, at least one instance of the image can be segmented first, and then a target instance that can represent the core content of the image can be determined therefrom.

[0061] For example, in a feasible implementation, step S100 may include steps S110 to S130:

[0062] Step S110, performing instance segmentation on the image of the image-text pair to obtain at least one instance of the image and instance segmentation information of the at least one instance;

[0063] Step S120, determining a target instance from the at least one instance based on instance segmentation information of the at least one instance;

[0064] In step S130 , the target instance and the instance segmentation information of the target instance are used as the instance segmentation result of the image.

[0065] This embodiment uses a common instance segmentation algorithm to segment at least one instance of the image in the image-text pair and the instance segmentation information of the at least one instance. Then, based on the instance segmentation information of the at least one instance, a template instance that can represent the core content of the image is determined from the at least one instance, so that the target instance and its corresponding instance segmentation information are used as the instance segmentation result of the image for subsequent screening of high-quality image-text pairs.

[0066] It should be noted that, when determining the target instance based on instance segmentation information, this embodiment can determine the instance with the highest number of category label repetitions as the target instance, or the instance with the center point coordinates closest to the image center as the target instance, or the instance with the largest area corresponding to the bounding box coordinates as the target instance. This embodiment does not impose specific restrictions on this. Users can flexibly set the selection criteria for the target instance according to actual needs, and only need to ensure that the target instance can represent the core content of the image.

[0067] Step S200 , matching the target instance with the text of the image-text pair according to the instance segmentation information of the target instance, to obtain an instance matching result of the image-text pair;

[0068] It should be noted that the instance matching result refers to the matching evaluation result of the degree of semantic association between the target instance and the text description of the image-text pair.

[0069] In this embodiment, when the instance matching result is a match, it indicates that there is a significant semantic association between the target instance and the text description; when the instance matching result is a mismatch, it indicates that there is insufficient semantic association between the target instance and the text description.

[0070] For example, in a feasible implementation, the instance segmentation information includes a category label, and step S200 may include steps S210 to S220:

[0071] Step S210, performing text classification on the text of the image-text pair to obtain a text classification result of the text;

[0072] Those skilled in the art will recognize that text classification is the process of assigning a given text to one or more predefined categories using natural language processing techniques. It identifies and distinguishes different topics or categories based on the text's content, semantic features, and other factors. The result of text classification is one or a set of category labels determined based on the text's content. These labels represent the text's subject matter or the primary object being described. For example, if a text primarily describes "cats," the result of the text classification might be "animals" or, more specifically, "cats."

[0073] In step S220 , if the category label of the target instance matches the text classification result, the instance matching result of the image-text pair is determined to be a match.

[0074] This example compares the target instance's category label with the text classification result to determine whether there is a significant semantic association between the target instance and the text. If the target instance's category label matches the text classification result, it means that the core content of the image is consistent with the theme of the text description. This indicates that the target instance and the text description have a high degree of match, and therefore it can be considered that there is a significant semantic association between them.

[0075] It should be noted that in this embodiment, whether the category label of the target instance matches the text classification result can be determined by judging whether the category label of the target instance is the same as the text classification result. If the category label of the target instance is the same as the text classification result, it is determined that the category label of the target instance matches the text classification result. Alternatively, whether the category label of the target instance matches the text classification result can be determined by calculating the semantic vector similarity between the category label of the target instance and the text classification result. If the semantic vector similarity between the category label of the target instance and the text classification result is greater than a preset threshold, it is determined that the category label of the target instance matches the text classification result. This embodiment does not impose specific restrictions on this.

[0076] It is worth mentioning that when the text in the image-text pair is a text description of the image type, there is no need to perform text classification on the text. It is possible to directly determine whether the category label of the target instance matches the text, thereby quickly determining the instance matching result of the image-text pair.

[0077] Step S300: If the instance matching result is a match, the image-text pair is determined as a target image-text pair, and an image-text pair dataset is constructed based on at least one target image-text pair.

[0078] It should be noted that the target image-text pair refers to a high-quality image-text pair with a high degree of matching between the image and the text.

[0079] In this embodiment, since the target instance can represent the core content of the image, when the instance matching result is a match, indicating that there is a significant semantic association between the target instance and the text description, the degree of match between the image and text of the image-text pair is high, and the image-text pair can be considered a high-quality image-text pair, that is, the target image-text pair, for constructing a high-quality image-text pair dataset. Correspondingly, when the instance matching result is a mismatch, indicating that there is insufficient semantic association between the target instance and the text description, the degree of match between the image and text of the image-text pair is low, and the image-text pair can be considered a low-quality image-text pair.

[0080] When faced with complex images, traditional methods are easily interfered by the background or other irrelevant elements, making it difficult to screen out high-quality image-text pairs, resulting in low-quality constructed image-text pair datasets. However, this embodiment uses instance segmentation technology to accurately identify and separate target instances that can represent the core content of the image from images with relatively complex content and more elements, and then match the target instance with the text of the image-text pair based on the instance segmentation information of the target instance, effectively reducing the interference of the background and other irrelevant elements in the image, screening out high-quality target image-text pairs, and improving the accuracy of image-text matching. Based on the high-quality target image-text pairs, a high-quality image-text pair dataset is constructed, ultimately solving the technical problem of low-quality image-text pair datasets constructed in related technologies.

[0081] It is worth mentioning that after constructing an image-text pair dataset based on at least one target image-text pair, the image-text pair dataset can be used to train or fine-tune the model, thereby improving the model's capabilities.

[0082] For example, the Chinese CLIP (Contrastive Language–Image Pretraining) model is a visual language model that supports the understanding and processing of the Chinese language. Because the image-text pair dataset used in the training process of the Chinese CLIP model is not constructed from image-text pairs in a Chinese environment, some defects are inevitable when the Chinese CLIP model processes image-text pair data in a Chinese environment. Therefore, the method provided in this embodiment can be used to screen high-quality target image-text pairs from image-text pairs in a Chinese environment, construct a high-quality image-text pair dataset consisting of target image-text pairs in a Chinese environment, and then use this image-text pair dataset to fine-tune the Chinese CLIP model (referring to the process of further training the model using a smaller dataset based on an existing pretrained model for a specific task or domain), resulting in even better performance of the fine-tuned Chinese CLIP model in Chinese environments.

[0083] Based on the above first embodiment, a method for constructing an image-text pair dataset according to a second embodiment of the present application is proposed.

[0084] In the second embodiment of the present application, for the same or similar contents as those in the above embodiments, please refer to the above introduction and will not be repeated hereafter.

[0085] Please refer to Figure 2 , Figure 2 A flowchart of the second embodiment of the method for constructing a text-image data set is provided in this application.

[0086] In this embodiment, step S120 determines a target instance from at least one instance based on instance segmentation information of at least one instance, and may include steps S121 to S122:

[0087] Step S121, determining a target category label from the category label of at least one instance;

[0088] It should be noted that the target category label refers to the category label that can represent the core content of the image.

[0089] In this embodiment, an image includes at least one instance, each of which has a corresponding category label (e.g., "cat," "dog," "car," etc.). However, not all instance category labels are relevant to the core content of the image. Therefore, it is necessary to filter out the target category label that best reflects the core content of the image from these category labels.

[0090] In one example, step S121 may include steps A10 to A20:

[0091] Step A10: according to the category label of at least one instance, obtaining the number of repetitions of different category labels in at least one instance by counting;

[0092] In step A20, the category label with the highest number of repetitions is determined as the target category label.

[0093] This embodiment counts the number of repetitions of different category labels across all instances in an image and selects the category label with the highest number of repetitions as the target category label. This cleverly focuses on the most common object types in the image, effectively highlighting the core content of the image and reducing interference from background and other non-critical elements. This frequency-based (i.e., repetition-based) filtering mechanism is not only simple to implement and computationally efficient, but also particularly well-suited for image scenes dominated by a single type of object, ensuring that the target instance accurately represents the core content of the image.

[0094] In another example, the instance segmentation information includes a bounding box mask, and step S121 may further include steps B10 to B30:

[0095] Step B10, determining the instance area of ​​at least one instance based on the bounding box mask of at least one instance;

[0096] It should be noted that the instance area can be expressed in pixels. This can be the number of pixels occupied by the instance's bounding box or the number of pixels occupied by the instance's mask. The number of pixels occupied by the bounding box can be calculated from the bounding box coordinates in the bounding box mask, and the number of pixels occupied by the mask can be calculated from the mask in the bounding box mask.

[0097] Step B20: Calculating instance areas corresponding to different category labels in the at least one instance based on the instance area and category label of the at least one instance;

[0098] In step B30, the category label corresponding to the largest instance area is determined as the target category label.

[0099] This implementation method is based on the bounding box mask in the instance segmentation information. By calculating the number of pixels covered by the instance's bounding box or mask, it quantifies the area occupied by each instance in the image (i.e., the instance area). Subsequently, the total area corresponding to different categories is aggregated and counted based on the category labels. Finally, the category label with the largest total area is determined as the target category label. This introduces instance area as a core screening indicator, providing a more objective and stable decision-making basis for determining the target category label. Furthermore, the physical dimension of instance area is used to directly reflect the visual salience of the instance in the image, effectively capturing the core objects that dominate the image and determining the target instance that best represents the core content of the image.

[0100] Through the above implementation, this embodiment can accurately identify the target category label that best represents the core content of the image from multiple category labels, providing a reliable basis for the subsequent determination of target instances.

[0101] Step S122: From the at least one instance, determine an instance whose category label is the target category label as a target instance.

[0102] After determining the target category label that can represent the core content of the image, this embodiment needs to filter out instances whose category labels are consistent with the target category label from all instances of the image and determine them as target instances. It ensures that the filtered target instances can represent the core content of the image and eliminates interference from the background and other irrelevant elements in the image. In this way, even when the image content is relatively complex and there are many elements, high-quality target image-text pairs can be accurately filtered out, thereby improving the quality of the constructed image-text pair dataset.

[0103] It is worth mentioning that after screening out instances whose category labels are consistent with the target category labels, these instances can be directly used as the final target instances, or they can be used as candidate target instances first, and the confidence of each candidate target instance can be further compared. The instance with the highest confidence is selected as the final target instance, thereby effectively eliminating false detections or low-quality instances and ensuring that the target instance can represent the core content of the image.

[0104] It is not difficult to understand that in addition to determining the final target instance from multiple candidate target instances through confidence, we can also prioritize the instance closer to the image center as the final target instance based on the center point coordinates of each candidate target instance, or prioritize the instance with a larger area as the final target instance based on the bounding box coordinates of each candidate target instance, or comprehensively consider the segmentation information of multiple instances, calculate the weight of each candidate target instance, and prioritize the instance with the largest weight as the target instance.

[0105] Based on the above first embodiment, a method for constructing an image-text pair dataset according to a third embodiment of the present application is proposed.

[0106] In the third embodiment of the present application, for the same or similar contents as those in the above embodiments, please refer to the above introduction and will not be repeated hereafter.

[0107] Please refer to Figure 3 , Figure 3 This is a flow chart of the third embodiment of the method for constructing a text-image data set in this application.

[0108] In this embodiment, step S120 determines a target instance from at least one instance based on instance segmentation information of at least one instance, and may further include steps S123 and S124:

[0109] Step S123: determining a weight factor of the at least one instance based on the bounding box mask of the at least one instance, and calculating a weight of the at least one instance based on the weight factor of the at least one instance, wherein the weight factor includes at least one of the instance area, the instance position, and the instance depth of field;

[0110] It should be noted that the instance position refers to the coordinates of the center point of the instance, and the instance depth of field is used to characterize whether the instance is located in the foreground or background of the image.

[0111] It should also be noted that weights are used to represent the importance or contribution of an instance in an image. The larger the weight, the more important or contributing the instance is to the image, and the more it represents the core content of the image.

[0112] It is worth mentioning that in addition to instance area, instance position, and instance depth of field, the weight factor can also include the number of repetitions of category labels.

[0113] It's easy to understand that the closer the instance is to the image center (i.e., the smaller the distance between the instance and the image center), the larger the instance area, and the more repeated the category label, the greater the corresponding calculated weight. Regarding instance depth, instances in the foreground have a greater weight than instances in the background.

[0114] Step S124 : From the at least one instance, determine an instance whose weight satisfies a preset weight condition as a target instance.

[0115] It should be noted that the preset weight condition is a preset judgment criterion used to screen target instances that represent the core content of the image from all instances of the image. For example, the preset weight condition can be the largest weight, a weight greater than a preset value, or the top three weight values ​​in descending order.

[0116] This embodiment determines the weight factor of each instance and comprehensively calculates the weight of each instance, so that the weight reflects the importance of the instance in the image. Then, through preset weight conditions, it accurately screens out target instances that can represent the core content of the image, reduces the interference of other elements such as complex background in the image, and ensures that the image-text pairs used to construct the image-text pair dataset are all high-quality target image-text pairs.

[0117] Through the combined application of instance segmentation technology and weighting factors, this embodiment can more meticulously identify each instance in an image and its importance, reducing interference from complex backgrounds and other elements. Furthermore, multi-dimensional weight calculation enables this embodiment to assess the importance of instances from multiple perspectives, avoiding the bias introduced by a single feature. This allows for a better understanding of image content, providing strong support for subsequent image-text matching, and ultimately achieving higher confidence in image-text matching in complex scenarios, ensuring more accurate and reliable final image-text matching results.

[0118] Based on the above embodiments, a method for constructing an image-text pair dataset according to a fourth embodiment of the present application is proposed.

[0119] In the fourth embodiment of the present application, for the same or similar contents as those in the above embodiments, please refer to the above introduction and will not be repeated hereafter.

[0120] Please refer to Figure 4 , Figure 4 This is a flowchart of the fourth embodiment of the method for constructing a text-image data set in this application.

[0121] In this embodiment, the instance segmentation information includes a bounding box mask. Step S200 matches the target instance with the text of the image-text pair based on the instance segmentation information of the target instance to obtain an instance matching result of the image-text pair. Steps S230 to S260 may be included:

[0122] Step S230: performing image feature extraction on the target instance to obtain an image feature vector of the target instance, and performing text feature extraction on the text of the image-text pair to obtain a text feature vector of the text;

[0123] This embodiment can use a deep learning model to extract image feature vectors that represent the visual characteristics of the target instance. At the same time, natural language processing techniques can be used to extract text feature vectors that represent semantic information from the text description. These feature vectors can effectively capture the core content of the target instance and the text, providing a basis for subsequent similarity calculations.

[0124] Step S240: Calculate the vector similarity between the image feature vector and the text feature vector based on a preset vector similarity algorithm;

[0125] Those skilled in the art will appreciate that a vector similarity algorithm is an algorithm used to calculate the degree of similarity between vectors. Commonly used vector similarity algorithms include the cosine similarity algorithm and the Euclidean distance algorithm.

[0126] This implementation can quantify the degree of semantic association between the target instance and the text description by calculating the vector similarity between the image feature vector and the text feature vector, thereby evaluating whether they match.

[0127] Step S250: performing weighted calculation on the vector similarity based on the weight of the target instance to obtain the instance text similarity between the target instance and the text, wherein the weight of the target instance is determined based on the bounding box mask of the target instance;

[0128] It should be noted that instance text similarity refers to the overall degree of semantic connection between the target instance and the text description. Since there may be more than one target instance, and different target instances have different weights in the image, it is necessary to perform a weighted calculation on the vector similarity between the image feature vector and the text feature vector of each target instance using its weight. This yields an instance text similarity that accurately reflects the overall degree of semantic connection between the target instance and the text description. This instance text similarity is then used to determine whether the target instance matches the text of the image-text pair, resulting in an instance matching result for that image-text pair.

[0129] Step S260 : When the instance text similarity is greater than a preset similarity threshold, determining that the instance matching result of the image-text pair is a match.

[0130] It should be noted that the preset similarity threshold is a preset threshold used to determine whether the instance matching result of the image-text pair matches.

[0131] In this embodiment, when the instance text similarity is greater than the preset similarity threshold, it can be determined that the instance matching result of the image-text pair is a match, that is, it is determined that the degree of matching between the image and the text in the image-text pair is high, and the image-text pair is a high-quality image-text pair, which can be used as a target image-text pair for constructing a high-quality image-text pair dataset.

[0132] This embodiment provides a more accurate image-text matching strategy by combining the similarity calculation of the image feature vector and the text feature vector, and introducing the weight factor of the target instance for weighted adjustment on this basis. This strategy not only takes into account the direct correlation between the target instance and the text content, but also fully considers the relative importance of the target instance in the image and its visual significance. Therefore, compared with the traditional method that relies only on category labels or simple frequency statistics, this embodiment can more effectively screen out high-quality image-text pairs, reduce the interference caused by complex backgrounds and other irrelevant elements, and further improve the overall quality and practicality of the image-text pair data set. In addition, by flexibly setting the preset similarity conditions, it can also adapt to the needs of different application scenarios to ensure that the selected target image-text pairs are all high-quality image-text pairs.

[0133] In addition, please refer to Figure 5 , Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the method for constructing an image-text pair dataset in an embodiment of the present application.

[0134] The present application also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the steps of the image-text data set construction method in the above embodiment.

[0135] Reference below Figure 5 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic devices in the embodiments of the present application may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), vehicle-mounted terminals, etc., and fixed terminals such as desktop computers, etc., or any other electronic device capable of implementing the above functions. Figure 5 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0136] like Figure 5As shown, the electronic device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory 1002 or programs loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the electronic device. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems may be connected to the input / output interface 1006: an input device 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; an output device 1008 including, for example, an LCD, speaker, or vibrator; a storage device 1003 including, for example, a magnetic tape or hard disk; and a communication device 1009. The communication device 1009 may allow the electronic device to communicate with other devices wirelessly or wired to exchange data. Although the figures show electronic devices with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0137] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are performed.

[0138] The electronic device provided in this application utilizes the image-text pair dataset construction method described in the aforementioned embodiments, resolving the technical issue of low-quality image-text pair datasets constructed in related technologies. Compared to the prior art, the electronic device provided in this application achieves the same beneficial effects as the image-text pair dataset construction method described in the aforementioned embodiments. Other technical features of this electronic device are the same as those disclosed in the aforementioned embodiments and are not further elaborated upon here.

[0139] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0140] The above are merely specific embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the above claims.

[0141] In addition, the present application also provides a storage medium, which is a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, and the computer-readable program instructions are used to execute the steps of the image-text pair dataset construction method in the above embodiment.

[0142] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical fiber, a portable compact disk read-only memory, an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0143] The computer-readable storage medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0144] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by an electronic device, the electronic device is enabled to: perform instance segmentation on the image of the image-text pair to obtain an instance segmentation result of the image, wherein the instance segmentation result includes a target instance and instance segmentation information of the target instance; match the target instance with the text of the image-text pair according to the instance segmentation information of the target instance to obtain an instance matching result of the image-text pair; if the instance matching result is a match, the image-text pair is determined as a target image-text pair, and an image-text pair dataset is constructed based on at least one target image-text pair.

[0145] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network or a wide area network, or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0146] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0147] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0148] The computer-readable storage medium provided in this application stores computer-readable program instructions (i.e., a computer program) for executing the steps of the aforementioned method for constructing an image-text pair dataset. This addresses the technical issue of low-quality image-text pair datasets constructed in related art. Compared to existing technologies, the beneficial effects of the computer-readable storage medium provided in this application are similar to those of the method for constructing an image-text pair dataset provided in the aforementioned embodiments, and are not further elaborated here.

[0149] In addition, an embodiment of the present application further provides a program product, which is a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the method for constructing a picture-text pair data set in the above embodiment are implemented.

[0150] The computer program product provided in this application can address the technical issue of low-quality image-text pair datasets constructed in related technologies. Compared to the prior art, the beneficial effects of the computer program product provided in this embodiment are the same as those of the image-text pair dataset construction method provided in the aforementioned embodiment, and will not be elaborated upon here.

[0151] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for constructing a picture-text pair dataset, characterized in that: The method comprises: Performing instance segmentation on the image of the image-text pair to obtain at least one instance of the image and instance segmentation information of the at least one instance; Determining a target instance from the at least one instance according to instance segmentation information of the at least one instance, wherein the target instance is an instance representing core content of the image; Matching the target instance with the text of the image-text pair according to the instance segmentation information of the target instance to obtain an instance matching result of the image-text pair; If the instance matching result is a match, the image-text pair is determined as a target image-text pair, and an image-text pair dataset is constructed based on at least one target image-text pair, wherein the target image-text pair is an image-text pair whose core content is consistent with the text description theme.

2. The method for constructing an image-text pair dataset as claimed in claim 1, wherein: The instance segmentation information includes a category label, and the step of determining a target instance from the at least one instance based on the instance segmentation information of the at least one instance includes: Determining a target category label from the category label of the at least one instance; From the at least one instance, an instance whose class label is a target class label is determined as a target instance.

3. The method for constructing an image-text pair dataset as claimed in claim 2, wherein: The step of determining a target category label from the category label of the at least one instance includes: According to the category label of the at least one instance, obtaining by counting the number of repetitions of different category labels in the at least one instance; The category label with the highest number of repetitions is determined as the target category label.

4. The method for constructing an image-text pair dataset as claimed in claim 2, wherein: The instance segmentation information includes a bounding box mask, and the step of determining a target category label from the category label of the at least one instance includes: determining an instance area of ​​the at least one instance based on the bounding box mask of the at least one instance; Obtaining, based on the instance area and the category label of the at least one instance, instance areas corresponding to different category labels in the at least one instance; The category label with the largest corresponding instance area is determined as the target category label.

5. The method for constructing an image-text pair dataset as claimed in claim 1, wherein: The instance segmentation information includes a bounding box mask, and the step of determining a target instance from the at least one instance based on the instance segmentation information of the at least one instance includes: Determining a weight factor of the at least one instance according to the bounding box mask of the at least one instance, and calculating a weight of the at least one instance according to the weight factor of the at least one instance, wherein the weight factor includes at least one of an instance area, an instance position, and an instance depth of field; From the at least one instance, an instance whose weight satisfies a preset weight condition is determined as a target instance.

6. The method for constructing an image-text pair dataset according to any one of claims 1 to 5, wherein: The instance segmentation information includes a category label. The step of matching the target instance with the text of the image-text pair based on the instance segmentation information of the target instance to obtain an instance matching result of the image-text pair includes: Performing text classification on the text of the image-text pair to obtain a text classification result of the text; If the category label of the target instance matches the text classification result, then the instance matching result of the image-text pair is determined to be a match.

7. The method for constructing an image-text pair dataset according to any one of claims 1 to 5, wherein: The instance segmentation information includes a bounding box mask. The step of matching the target instance with the text of the image-text pair based on the instance segmentation information of the target instance to obtain an instance matching result of the image-text pair includes: Performing image feature extraction on the target instance to obtain an image feature vector of the target instance, and performing text feature extraction on the text of the image-text pair to obtain a text feature vector of the text; Calculating the vector similarity between the image feature vector and the text feature vector based on a preset vector similarity algorithm; performing a weighted calculation on the vector similarity according to the weight of the target instance to obtain the instance text similarity between the target instance and the text, wherein the weight of the target instance is determined based on the bounding box mask of the target instance; When the instance text similarity is greater than a preset similarity threshold, the instance matching result of the image-text pair is determined to be a match.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the method for constructing a picture-text pair dataset according to any one of claims 1 to 7 is implemented.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the method for constructing an image-text pair dataset according to any one of claims 1 to 7 is implemented.

10. A program product, characterized in that The program product is a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of the method for constructing an image-text pair dataset according to any one of claims 1 to 7 are implemented.

Images