Dataset generation, semantic matching, and semantic matching model training method and device

Abstract

The present application discloses a method and apparatus for dataset generation, semantic matching, and semantic matching model training, which are applied to a client, wherein the dataset generation method comprises: determining a first dataset, the first dataset comprising a plurality of first image pairs, each first image pair having corresponding category information; inputting the first dataset into a dataset processing model, the dataset processing model performing dataset generation processing based on the first dataset to obtain a second dataset, the second dataset comprising a plurality of second image pairs, each second image pair having a corresponding pixel-level label, the pixel-level label being obtained based on the category information of the first image pair, and the pixel-level label being used to represent the pixel-level matching relationship of the second image pair. Thus, the processing process of labeling a large number of datasets to obtain a dataset that meets the requirements is effectively simplified, labor costs are saved, and the semantic matching model obtained by training based on the dataset has stronger generalization ability and accuracy.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method and apparatus for data set generation, semantic matching, and semantic matching model training. Background Art

[0002] In the field of computer vision, learning semantic matching relationships between instances of the same category has become a fundamental problem. For example, semantic matching algorithms have been widely used in tasks such as small-shot learning, multi-target tracking, and image editing. With the development of deep learning technology, the semantic matching models corresponding to the most advanced semantic matching algorithms can now produce high-quality semantic matching results. However, currently popular semantic matching datasets, such as FG3DCar, PF-PASCAL, SPair71k, and Caltech-UCSD Birds, contain only small amounts of manual annotations. Therefore, the semantic matching models derived from these semantic matching datasets suffer from weak generalization and poor accuracy.

[0003] In order to improve the generalization ability and performance of the semantic matching model, it is necessary to construct a semantic matching dataset with dense annotations. However, for semantic matching tasks, the process of constructing a semantic matching dataset with dense annotations requires labeling important object components or key points containing significant and unique semantic information. This processing is very complicated and requires higher labor costs.

[0004] In summary, existing technologies suffer from complex processes and high labor costs associated with labeling semantic matching datasets. Furthermore, semantic matching models trained using small, manually labeled semantic matching datasets suffer from weak generalization and accuracy. Summary of the Invention

[0005] This application provides a method and apparatus for dataset generation, semantic matching, and semantic matching model training. These methods address the complex and labor-intensive processes associated with semantic matching dataset annotation in existing technologies. These methods effectively simplify the semantic matching dataset annotation process, saving labor costs. Furthermore, they improve the generalization and accuracy of semantic matching models.

[0006] To solve the above technical problems, in the first aspect, an embodiment of the present application provides a data set generation method, which is applied to a client, and the method includes: determining a first data set, the first data set includes multiple first image pairs, and each first image pair has corresponding category information; inputting the first data set into a data set processing model, and the data set processing model performs data set generation processing based on the first data set to obtain a second data set, the second data set includes multiple second image pairs, and each second image pair has a corresponding pixel-level label, the pixel-level label is obtained based on the category information of the first image pair, and the pixel-level label is used to represent the pixel-level matching relationship of the second image pair.

[0007] In the implementation of the present application, a first data set containing only a first image pair and the category information corresponding to the first image pair is input into a data set processing model. The data set processing model performs data set generation processing on the first data set, thereby obtaining a second data set with a pixel-level matching relationship. The second data set with a pixel-level matching relationship can contain dense image annotations. In this process, there is no need to manually perform a large amount of key point annotation processing on the first data set to obtain a second data set with dense annotations (such as a semantic matching data set), which effectively simplifies the process of performing a large amount of annotation on the data set to obtain a second data set that meets the requirements, saving labor costs.

[0008] Furthermore, since the second dataset has dense pixel-level image annotations, using the second dataset to train the semantic matching model can make the obtained semantic matching model have stronger generalization ability and accuracy, thereby improving the accuracy of the semantic matching results.

[0009] In a possible implementation of the first aspect above, a first data set is input into a data set processing model, and the data set processing model performs data set generation processing on the first data set to obtain a second data set, including: inputting the first data set into the data set processing model, and the data set processing model performs multi-level data set generation processing on the first data set to obtain a second data set, wherein the data set obtained by each level of data set generation processing includes multiple image pairs, each image pair has a corresponding category label, and the density of category labels of the image pairs included in the data set obtained by the latter level of data set generation processing is greater than the density of category labels of the image pairs included in the data set obtained by the previous level of data set generation processing.

[0010] In the implementation of the present application, the second dataset is obtained by performing multi-level dataset generation processing on the first dataset by the dataset processing model, and the density of category labels for the image pairs included in the dataset obtained by the latter dataset generation processing is greater than the density of category labels for the image pairs included in the dataset obtained by the former dataset generation processing. By generating the datasets and the category labels for the image pairs included in the datasets multiple times through multi-level dataset generation, the second image pairs included in the final second dataset, as well as the pixel-level labels for each second image pair, can be made more accurate and dense, which is more conducive to model training.

[0011] In a possible implementation of the first aspect above, the dataset processing model includes at least two dataset processing modules cascaded in sequence, the dataset processing module is used to perform dataset generation processing, a first dataset is input into the dataset processing model, and the dataset processing model performs multi-level dataset generation processing on the first dataset to obtain a second dataset, including: sequentially inputting the dataset output by the previous-level dataset processing module into the next-level dataset processing module for dataset generation processing, until the last-level dataset processing module outputs the corresponding dataset as the second dataset, wherein the input of the first-level dataset processing module is the first dataset.

[0012] In the implementation of the present application, a multi-level dataset generation process is performed on the first dataset by cascading at least two dataset processing modules in sequence to obtain a second dataset. This can make the second image pairs included in the final second dataset and the pixel-level labels of each second image pair more accurate and dense, which is more conducive to model training.

[0013] In a possible implementation of the first aspect above, the dataset processing module includes an aggregation unit and a label generation unit, and the dataset processing module performs dataset generation processing, including: the aggregation unit obtains hypercube data corresponding to the image pairs included in the dataset based on the received dataset, and inputs the hypercube data into the label generation unit, where the hypercube data is four-dimensional data; and the label generation unit obtains the dataset output by the dataset processing module based on the hypercube data.

[0014] In the implementation of the present application, during the dataset generation process performed by the dataset processing module, the aggregation unit first generates hypercube data corresponding to each image pair, and then the label generation unit generates an output dataset based on the hypercube data. This can increase the accuracy of the image pairs and the category labels of the image pairs in the dataset output by each dataset processing module, thereby making the second image pairs included in the resulting second dataset and the pixel-level labels of each second image pair more accurate and dense, which is more conducive to model training.

[0015] In a possible implementation of the first aspect above, an aggregation unit obtains hypercube data corresponding to the image pairs included in the data set based on the received data set, including: performing feature aggregation processing on the image pairs included in the data set to obtain initial feature pairs; performing optimization processing on the initial feature pairs to obtain target feature pairs; and obtaining hypercube data based on the target feature pairs.

[0016] In the implementation of this application, feature aggregation is performed on the image pairs, the initial feature pairs are then optimized, and finally, hypercube data is obtained based on the target feature pairs. This increases the accuracy of the obtained hypercube data, thereby increasing the accuracy and density of the second image pairs included in the second dataset and the pixel-level labels of each second image pair.

[0017] In a possible implementation of the first aspect, obtaining hypercube data according to the target feature pair includes: determining the cosine similarity of the target feature pair; and obtaining the hypercube data according to the cosine similarity.

[0018] In the implementation of the present application, the hypercube data obtained according to the cosine similarity of the target feature pair is more accurate.

[0019] In a possible implementation of the first aspect above, the label generation unit obtains the dataset output by the dataset processing module based on the hypercube data, including: converting the hypercube data into Hough space to obtain converted hypercube data; calculating the domain matching consistency score of the converted hypercube data; determining the target image pair based on the domain matching consistency score; and obtaining the dataset output by the dataset processing module based on the target image pair.

[0020] In the implementation of the present application, the image pairs and category labels of the data set output by the data set processing module obtained by the above method are more accurate, which increases the accuracy and density of the second image pairs included in the final second data set and the pixel-level labels of each second image pair, which is more conducive to model training.

[0021] In a possible implementation of the first aspect above, the dataset processing model includes a first-level dataset processing module, a second-level dataset processing module and a third-level dataset processing module, and the first dataset is input into the dataset processing model, and the dataset processing model performs dataset generation processing based on the first dataset to obtain a second dataset, including: inputting the first dataset into the first-level dataset processing module for dataset generation processing to obtain a third dataset, the third dataset including multiple third image pairs, each third image pair having a corresponding image-level label, and the image-level label is used to represent the image-level matching relationship of the third image pairs; inputting the third dataset into the second-level dataset processing module for dataset generation processing to obtain a fourth dataset, the fourth dataset including multiple fourth image pairs, each fourth image pair having a corresponding region-level label, and the region-level label is used to represent the region-level matching relationship of the fourth image pairs; inputting the fourth dataset into the third-level dataset processing module for dataset generation processing to obtain the second dataset.

[0022] In the implementation method of the present application, through the first-level data set processing module, the second-level data set processing module and the third-level data set processing module, the third data set corresponding to the image pairs with image-level labels is first obtained, then the fourth data set corresponding to the image pairs with region-level labels is obtained, and finally the second data set corresponding to the image pairs with pixel-level labels is obtained. The image pairs and the category labels corresponding to the image pairs in the data set output by each level of data set processing module are optimized step by step, so that the second image pairs included in the final second data set and the pixel-level labels of each second image pair are more accurate and dense, which is more conducive to model training.

[0023] In a possible implementation of the first aspect above, the secondary data set processing module includes a label generation unit that obtains a data set output by the secondary data set processing module based on hypercube data, including: obtaining a fourth initial data set based on the hypercube data, the fourth initial data set including multiple fifth image pairs, each fifth image pair having a corresponding region-level label; selecting some region-level labels whose label similarity is greater than a label threshold in the region-level labels of the fifth image pairs, and determining the position of each partial region-level label on the image included in the corresponding first image pair; converting the position of the partial region-level label on the image in the corresponding fifth image pair to the position of the partial region-level label on the image included in the corresponding first image pair; and performing a non-maximum suppression algorithm on the partial region-level labels after the conversion position to obtain a fourth data set.

[0024] In the implementation of the present application, the fourth data set obtained by the above method includes multiple fourth image pairs, and the region-level labels of each fourth image pair are more accurate.

[0025] In a possible implementation of the first aspect above, the fourth data set is input into the third-level data set processing module for data set generation processing to obtain a second data set, including: inputting the fourth data set into the third-level data set processing module for data set generation processing to obtain a second initial data set, the second initial data set including multiple sixth image pairs, each sixth image pair having a corresponding pixel-level label; determining the first image pair corresponding to the sixth image pair, and determining the third image based on the first image pair; determining the first feature point from the image in the first image pair; obtaining the second feature point according to the image matching path, the image matching path being obtained based on the first image pair and the third image; calculating the Euclidean distance between the first feature point and the second feature point, and when the Euclidean distance is less than a preset distance threshold, selecting the sixth image pair as the final second image pair; and obtaining the second data set based on the multiple second image pairs.

[0026] In the implementation of this application, in the process of obtaining the second dataset, a second initial dataset including a sixth image pair with pixel-level labels is first obtained. Then, a closed-loop consistency check is performed on the sixth image pair to determine a final second image pair with pixel-level labels, and the second dataset is obtained based on the second image pair. This can make the second image pairs included in the final second dataset, as well as the pixel-level labels of each second image pair, more accurate and dense, which is more conducive to model training.

[0027] In a possible implementation of the first aspect, the third image pair includes an image corresponding to a region of the foreground image of the first image pair whose area is the target window value.

[0028] In the implementation of the present application, the foreground image of the first image pair and the image corresponding to the area with an area of ​​a preset window value are selected as the third image pair, so that the images in the third image pair contain similar semantic information, which is more convenient for the generation of image pairs included in the data set output at the next level and the region-level labels corresponding to the image pairs, and increases the accuracy of the multiple fourth image pairs included in the fourth data set and the region-level labels of each fourth image pair.

[0029] In a possible implementation of the first aspect above, the first image pair includes a first positive image pair and a first negative image pair, and the third image pair includes a second positive image pair and a second negative image pair, wherein the images in the first positive image pair and the second positive image pair belong to the same category, and the images in the first negative image pair and the second negative image pair belong to different categories.

[0030] In the implementation of the present application, the first image pair and the third image pair are divided into a positive image pair and a negative image pair, which is more convenient for subsequent data set generation and processing.

[0031] In a possible implementation of the first aspect above, determining the first data set includes: determining an initial data set; removing samples that cannot perform single-instance semantic matching tasks from the initial data set to obtain an intermediate data set; performing an object detection algorithm on the samples in the intermediate data set, and selecting intermediate samples that only contain one confidence instance greater than a confidence threshold; calculating a consistency score for each intermediate sample, and selecting a preset proportion of sample data from the intermediate samples as target samples based on the consistency score; and determining the first data set based on multiple target samples.

[0032] In the implementation of this application, a preset proportion of sample data is selected from the initial dataset among intermediate samples containing only one instance with a confidence score greater than a confidence threshold, thereby determining the first dataset. This improves the quality of the first dataset and makes it more suitable for training a semantic matching model.

[0033] In the second aspect, an embodiment of the present application provides a semantic matching method, which includes: determining the data to be matched; inputting the data to be matched into a target semantic matching model for semantic matching processing to obtain a semantic matching result of the data to be matched, wherein the target semantic matching model is obtained by training a second data set, and the second data set is generated by the aforementioned data set generation method.

[0034] In the implementation of this application, the second dataset generated by the aforementioned dataset generation method contains dense pixel-level image annotations, and the target semantic matching model trained using the second dataset has stronger generalization ability and accuracy. Therefore, the semantic matching results for the data to be matched are also more accurate.

[0035] In a third aspect, an embodiment of the present application provides a semantic matching model training method, which includes: determining a second data set, where the second training data set is generated by the aforementioned data set generation method; inputting the second data set into the initial semantic matching model for model training to obtain a target semantic matching model.

[0036] In the implementation of the present application, the second dataset generated by the aforementioned dataset generation method contains dense pixel-level image annotations. Therefore, the target semantic matching model obtained by training the second dataset has stronger generalization ability and accuracy.

[0037] In a fourth aspect, an embodiment of the present application provides a data set generation device, comprising: a first input module, for determining a first data set, the first data set including multiple first image pairs, each first image pair having corresponding category information; a data set generation module, for inputting the first data set into a data set processing model, and the data set processing model performing data set generation processing based on the first data set to obtain a second data set, the second data set including multiple second image pairs, each second image pair having a corresponding pixel-level label, the pixel-level label being obtained based on the category information of the first image pair, and the pixel-level label being used to represent the pixel-level matching relationship of the second image pair.

[0038] In the fifth aspect, an embodiment of the present application provides a semantic matching device, including: a second input module for determining the data to be matched; a semantic matching module for inputting the data to be matched into a target semantic matching model for semantic matching processing to obtain a semantic matching result of the data to be matched, wherein the target semantic matching model is obtained by training a second data set, and the second data set is generated by the aforementioned data set generation method.

[0039] In the sixth aspect, an embodiment of the present application provides a semantic matching model training device, including: a fourth input module, used to determine a second data set, the second training data set is generated by the aforementioned data set generation method; a model training module, used to input the second data set into the initial semantic matching model for model training to obtain a target semantic matching model.

[0040] In the seventh aspect, an embodiment of the present application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster executes the aforementioned data set generation method, or so that the computing device cluster executes the aforementioned semantic matching method, or so that the computing device cluster executes the aforementioned semantic matching model training method.

[0041] In an eighth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed by a computing device cluster, enables the computing device cluster to execute the aforementioned data set generation method, or enables the computing device cluster to execute the aforementioned semantic matching method, or enables the computing device cluster to execute the aforementioned semantic matching model training method.

[0042] In the ninth aspect, an embodiment of the present application provides a computer-readable storage medium, including computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the aforementioned data set generation method, or enables the computing device cluster to execute the aforementioned semantic matching method, or enables the computing device cluster to execute the aforementioned semantic matching model training method.

[0043] The relevant beneficial effects of the fourth to ninth aspects mentioned above can be found in the relevant descriptions of the first to third aspects mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings used in the description of the implementation methods.

[0045] Figures 1A-1D The figure shows a commonly used semantic matching dataset and a schematic diagram of the annotation of SC-ImageNet in the prior art;

[0046] Figure 2 is a schematic diagram showing a flow chart of a method for generating a data set according to some embodiments of the present application;

[0047] Figure 3 is a schematic diagram showing the overall operation flow of a three-stage multi-instance learning pipeline according to some embodiments of the present application;

[0048] Figure 4 is a schematic diagram showing a process flow of a data set processing module performing data set generation processing according to some embodiments of the present application;

[0049] Figure 5 is a schematic diagram showing a process of obtaining hypercube data corresponding to image pairs included in a data set according to some embodiments of the present application;

[0050] Figure 6 is a schematic diagram showing a process of obtaining hypercube data based on target feature pairs according to some embodiments of the present application;

[0051] Figure 7 is a schematic diagram showing a process of obtaining a data set output by a data set processing module based on hypercube data by a label generation unit according to some embodiments of the present application;

[0052] Figure 8 is a schematic diagram showing a specific working process of a pseudo-label generator according to some embodiments of the present application;

[0053] Figure 9 is a schematic diagram showing a process of obtaining a second data set according to some embodiments of the present application;

[0054] Figure 10 According to some embodiments of the present application, a schematic diagram of a process in which a secondary data set processing module obtains a data set output by the data set processing module based on hypercube data by a label generation unit is shown;

[0055] Figure 11 is a schematic diagram showing another process of obtaining a second data set according to some embodiments of the present application;

[0056] Figure 12 1 is a schematic diagram showing a construction process of SC-ImageNet according to some embodiments of the present application;

[0057] Figure 13 is a schematic diagram showing a process of determining a first data set according to some embodiments of the present application;

[0058] Figure 14 is a flowchart illustrating a semantic matching method according to some embodiments of the present application;

[0059] Figure 15 is a flowchart illustrating a method for training a semantic matching model according to some embodiments of the present application;

[0060] Figure 16 According to some embodiments of the present application, a schematic diagram of a matching relationship generated based on semantic matching targets of different granularities is shown;

[0061] Figure 17 is a schematic diagram showing a visualization result on the SPair-71k dataset according to some embodiments of the present application;

[0062] Figure 18 is a schematic diagram of a data generating device according to some embodiments of the present application;

[0063] Figure 19 1 is a schematic diagram of a semantic matching device according to some embodiments of the present application;

[0064] Figure 20 is a schematic diagram showing a semantic matching model training device according to some embodiments of the present application;

[0065] Figure 21 is a schematic structural diagram of an electronic device according to some embodiments of the present application;

[0066] Figure 22 1 is a schematic diagram showing some structural diagrams of computing devices according to some embodiments of the present application;

[0067] Figure 23 1 is a schematic diagram showing some structures of a computing device cluster according to some embodiments of the present application. DETAILED DESCRIPTION

[0068] The technical solution of this application will be described in further detail below with reference to the accompanying drawings.

[0069] As mentioned earlier, currently popular semantic matching datasets, such as FG3DCar, PF-PASCAL, SPair71k, and Caltech-UCSD Birds, contain only small amounts of manual annotations. Consequently, semantic matching models derived from these datasets suffer from weak generalization and accuracy. Furthermore, constructing densely annotated semantic matching datasets requires labeling important object components or key points containing significant and unique semantic information. This process is complex and requires significant labor costs.

[0070] Therefore, existing technologies suffer from complex processes and high labor costs associated with labeling semantic matching datasets. Furthermore, semantic matching models trained using small-scale manually labeled semantic matching datasets suffer from weak generalization and accuracy.

[0071] The following is a brief description of the current research content related to semantic matching in the field of computer vision at home and abroad.

[0072] As mentioned earlier, learning semantic matching relationships between instances of the same category has become a fundamental problem in computer vision. For example, semantic matching algorithms have been widely used in tasks such as few-shot learning, multi-object tracking, and image editing. With the development of deep learning technology, state-of-the-art algorithms are now able to produce high-quality semantic matching results. However, currently popular semantic matching datasets, such as FG3DCar (for details of the dataset, please refer to TANIAI T, SINHA SN, SATOY. Joint recovery of dense correspondence and cosegmentation in two images [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 4246-4255), PF-PASCAL (for details of the dataset, please refer to HAM B, CHO M, SCHMID C, et al. Proposal flow: Semantic correspondences from object proposals [J]. IEEE transactions on pattern analysis and machine in-telligence, 2017, 40 (7): 1711-1725.), SPair71k (for details of the dataset, please refer to MIN J, LEE J, PONCE J, et al. Spair-71k: A large-scale benchmark for semanticcorrespondence[A].2019.arXiv:1908.10543) and Caltech-UCSD Birds (for details about the dataset, please see WELINDER P, BRANSON S, MITA T, et al. Caltech-ucsd birds 200[M]. California Institute of Technology, 2010), both contain only small-scale manual annotations, which weakens the generalization ability of the algorithm (i.e., the semantic matching model).

[0073] Unlike other visual recognition tasks, building densely annotated semantic matching datasets is more complex and requires higher labor costs because important object components or key points containing significant and unique semantic information need to be paid special attention to during the process. Figures 1A-1CThe annotation of three popular semantic matching datasets is shown. Figure 1A As shown in the figure, the Caltech-UCSD Birds dataset (for details about the dataset, please refer to WELINDER P, BRANSONS, MITAT, et al. Caltech-ucsdbirds 200[M]. California Institute of Technology, 2010) describes fine-grained bird species (CUB describes fine-grained bird species), annotates 5794 images, 10000 image pairs, 200 target object categories (i.e., different bird species), and 15 key points per image. Figure 1B As shown in the figure, the PF-PASCAL dataset describes 20 classes of common objects, annotates 1,345 images, and has 20 target object classes. The number of key points in each image ranges from 4 to 17. In addition, the PF-WILLOW dataset (for details about the dataset, please refer to HAM B, CHO M, SCHMID C, et al. Proposal flow: Semantic correspondences from object proposals [J]. IEEE transactions on pattern analysis and machine intelligence, 2017, 40 (7): 1711-1725) also includes 100 images and 900 image pairs for further evaluation. These images cover 10 categories. The SPair71k dataset describes 18 categories of common objects with challenging contexts, annotates 1,800 images and 70,958 image pairs from 18 different object categories (i.e., target object types), and in each image, the number of key points ranges from 3 to 30, with an average of 7. Compared with other large-scale datasets, such as Figure 1DCompared with the SC-ImageNet dataset in

[15] (for details of the dataset, please refer to DENG J, DONG W, SOCHER R, et al. Imagenet: A large-scale hierarchical image database [C] / / 2009 IEEE conference on computer vision and pattern recognition. 2009: 248-255) and others such as MSCOCO (for details of the dataset, please refer to LIN TY, MAIRE M, BELONGIE S, et al. Microsoft coco: Common objects in context [C] / / Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V13. Springer, 2014: 740-755), the image scale in the semantic matching dataset is significantly smaller. The SC-ImageNet dataset contains a large number of general object or scene categories, with 113,516 annotated images and 794,612 image pairs from 679 different object categories (i.e., target object categories). In each image, the number of key points is 32.

[0074] Semantic matching aims to find pixel-level semantic correspondences between different instances of the same object. Unlike previous image matching tasks, semantic matching methods focus on finding semantic associations between instances. Based on how deep features are used, semantic matching methods can be categorized as those based on handcrafted features and those based on deep neural networks capable of end-to-end learning. Handcrafted feature-based methods typically use a pretrained backbone network with fixed weights for feature extraction. Subsequently, before calculating the similarity matrix and generating matching relationships, selection-based or computation-based preprocessing algorithms are often employed to improve the discriminability of semantic information between patches. Deep learning-based methods, on the other hand, enhance deep features in a dynamic and implicit manner. Deep semantic information descriptors can be dynamically updated during training based on task requirements. Most current state-of-the-art methods fall into this category. However, commonly used semantic matching datasets, such as Spair-71k and PF-PASCAL, only provide sparse keypoint annotations, which deviates from the goal of semantic matching. This limitation limits the performance of semantic matching algorithms.

[0075] Furthermore, current fully supervised semantic matching methods are already able to provide high-quality semantic matching results. However, considering that it is quite difficult to determine the exact matching relationship between pixels and the size of the dataset is usually limited, in order to expand this research field, weakly supervised algorithms that only require instance-level or image-level annotations are particularly important. Among them, some works based on image-level annotations attempt to obtain pixel-level matching relationships by comparing positive and negative samples. At the same time, DISCOBOX proposed a collaborative training method that optimizes the implicit matching similarity matrix by supervising the bounding box of the instance. Some other works have attempted to enhance the existing manual annotations and further train the semantic matching model on this basis to optimize the model's prediction accuracy.

[0076] In summary, existing semantic matching technologies have problems such as complex processing procedures such as dataset annotation, high labor costs, and weak generalization and accuracy of semantic matching models trained using semantic matching datasets containing small-scale manual annotations.

[0077] Based on this, this application provides a data set generation method, which is applied to the client, such as Figure 2 As shown, the method includes the following steps:

[0078] S100: Determine a first data set, where the first data set includes a plurality of first image pairs, each first image pair having corresponding category information.

[0079] The category information of the first image pair may be the category information of an image roughly annotated manually, or the category information of an image obtained by annotating an image using an existing automatic annotation method.

[0080] S200: Input the first data set into the data set processing model, and the data set processing model performs data set generation processing based on the first data set to obtain a second data set, where the second data set includes multiple second image pairs, each second image pair has a corresponding pixel-level label, and the pixel-level label is obtained based on the category information of the first image pair, and the pixel-level label is used to represent the pixel-level matching relationship of the second image pair.

[0081] The dataset generation method provided in this application inputs a first dataset containing only a first image pair and the category information corresponding to the first image pair into a dataset processing model. The dataset processing model performs dataset generation processing on the first dataset to obtain a second dataset with a pixel-level matching relationship. The second dataset with a pixel-level matching relationship can contain dense image annotations. In this process, there is no need to manually perform a large amount of key point annotation processing on the first dataset to obtain a second dataset with dense annotations (such as a semantic matching dataset), which effectively simplifies the process of performing a large amount of annotation on the dataset to obtain a second dataset that meets the requirements, saving labor costs.

[0082] Furthermore, since the second dataset has dense pixel-level image annotations, using the second dataset to train the semantic matching model can make the obtained semantic matching model have stronger generalization ability and accuracy, thereby improving the accuracy of the semantic matching results.

[0083] Among them, the dataset processing model builds a pipeline from coarse to fine, such as Figure 3 As shown in the figure, the pipeline adopts a three-stage optimization process and can find accurate pixel-level matching relationships between image pairs only through image-level annotation (i.e., image category information). Throughout the process, following the idea of ​​Multiple Instance Learning (MIL), in the first stage, image-level matching is performed. The backbone network of the dataset processing model is used to obtain the pixel-level matching relationship between the given image pairs (I s , I t )(Positive ImagePair; Negative ImagePair) extracts a rough matching area (R s , R t), which contains possible pixel-level matching pairs. Then, region-level matching is performed. In the region-level matching stage, each region-level matching pair (R s , R t ), there are possible pixel-level matching pairs (P s , P t Finally, pixel-level matching is performed. s , P t ) is used to supervise the generation of pixel-level matching relationships, that is, to generate the image pairs and pixel-level labels corresponding to each image pair included in the final dataset for training the semantic matching model, and finally perform optimized correspondence refinement to generate optimized matching pairs. In this process, the parameters of the backbone network are iteratively updated through forward learning and reverse learning calculations. The solid arrows in the figure represent the forward learning computing direction (Curriculum Learning Computing Direction), and the dotted arrows represent the backpropagation computing direction (Backpropagation Computing Direction). In summary, the present application has trained a pseudo-label generator (as an example of a dataset processing model) that can generate dense pixel-level matching pairs, and during the training process, only image-level manual annotations are needed. Next, the multi-instance learning pipeline of the present application is introduced from the perspectives of image level, region level, and pixel level. Next, the overall generation process of pseudo-labels (i.e., category labels of image pairs) will be analyzed, and finally the specific process of constructing the first dataset (such as the SC-ImageNet dataset) will be given.

[0084] Therefore, in one implementation of the present application, a first data set is input into a data set processing model, and the data set processing model performs data set generation processing on the first data set to obtain a second data set, including: inputting the first data set into the data set processing model, and the data set processing model performs multi-level data set generation processing on the first data set to obtain a second data set, wherein the data set obtained by each level of data set generation processing includes multiple image pairs, each image pair has a corresponding category label, and the density of category labels of the image pairs included in the data set obtained by the latter level of data set generation processing is greater than the density of category labels of the image pairs included in the data set obtained by the previous level of data set generation processing.

[0085] Among them, the density of category labels of image pairs included in the dataset obtained by the latter level dataset generation processing is greater than the density of category labels of image pairs included in the dataset obtained by the previous level dataset generation processing, thereby realizing the rough to fine pipeline of the present application, generating category labels of image pairs step by step, and enabling the image pairs and pixel-level labels of the image pairs finally obtained in the second dataset to be more accurate.

[0086] In one implementation of the present application, a data set processing model includes at least two data set processing modules cascaded in sequence, and the data set processing module is used to perform data set generation processing. A first data set is input into the data set processing model, and the data set processing model performs multi-level data set generation processing on the first data set to obtain a second data set, including: sequentially inputting the data set output by the previous-level data set processing module into the next-level data set processing module for data set generation processing, until the last-level data set processing module outputs the corresponding data set as the second data set, wherein the input of the first-level data set processing module is the first data set.

[0087] The data set processing model may include two data set processing modules, or may include more data set processing modules, such as three or four.

[0088] The following introduces the data set processing module provided by the present application (ie, the core module of the coarse-to-fine pipeline provided by the present application).

[0089] In one implementation of the present application, the data set processing module includes an aggregation unit and a label generation unit, such as Figure 4 As shown, the data set processing module performs data set generation processing, including the following steps:

[0090] S210: The aggregation unit obtains hypercube data corresponding to the image pairs included in the data set based on the received data set, and inputs the hypercube data into the label generation unit. The hypercube data is four-dimensional data.

[0091] S220: The label generation unit obtains the data set output by the data set processing module based on the hypercube data.

[0092] Among them, the aggregation unit can also be called the Matching Volume Aggregation Module (MVAM), and the label generation unit can also be called the Pseudo Matching Pair Selection (PMPS) module. The aggregation unit, also known as the Matching Hypercube Aggregation Module, is used to generate an accurate similarity hypercube (as an example of hypercube data), and the label generation unit, also known as the Pseudo Matching Pair Selection Module, is used to select potential fine matching pairs (i.e., image pairs) for the next stage (i.e., the next-level dataset processing module) to be used as supervision signals.

[0093] In one implementation of this application, Figure 5 As shown, the aggregation unit obtains the hypercube data corresponding to the image pairs included in the data set based on the received data set, including the following steps:

[0094] S211: Perform feature aggregation processing on the image pairs included in the data set to obtain initial feature pairs.

[0095] S212: Optimize the initial feature pair to obtain the target feature pair.

[0096] S213: Obtain hypercube data based on the target feature pair.

[0097] Among them, in one implementation of this application, Figure 6 As shown in Figure 2, according to the target feature pair, the hypercube data is obtained, which includes the following steps:

[0098] S2131: Determine the cosine similarity of the target feature pair.

[0099] S2132: Obtain hypercube data based on cosine similarity.

[0100] After obtaining the hypercube data, a second data set is obtained based on the hypercube data. In one implementation of the present application, Figure 7 As shown, the label generation unit obtains the data set output by the data set processing module based on the hypercube data, including the following steps:

[0101] S221: Convert the hypercube data into Hough space to obtain converted hypercube data.

[0102] S222: Calculate the domain matching consistency score of the transformed hypercube data.

[0103] S223: Determine the target image pair based on the domain matching consistency score.

[0104] S224: Obtaining a data set output by the data set processing module based on the target image pair.

[0105] Specific as Figure 8 As shown, in one implementation of the present application, in the matching hypercube aggregation module, the image pair (block) (I s , I t ) or (R s , R t ), that is, by Figure 8 The image pair consisting of the original image (Source) and the target image (Target) in the image (as an example of the first image pair) will first be sent to the iBOT backbone network (Backbone) to extract features and obtain feature pairs (F s , F t ) (as an example of an initial feature pair), the construction process of the feature pair introduces the gated attention mechanism (Gated CrossAttention) and the first 4D correlation module (4D Correlation), and then the optimized similarity hypercube v is constructed through the 4D Correlation Aggregation module f (As an example of hypercube data). Figure 8 As shown in the figure, the matching hypercube aggregation module can be divided into three components, namely feature aggregation Ψ(F s , F t ) (i.e., feature pairs are obtained through the backbone network), matching hypercubes are calculated Φ (i.e., the 4D correlation of image pairs obtained through the gated cross-attention mechanism) and hypercube aggregation ρ(v r ) (i.e., 4D correlation aggregation is performed to obtain a similarity hypercube). The feature aggregation Ψ(·) in the s→t direction can be formally expressed as the following formula:

[0106]

[0107]

[0108]

[0109] Where σ(·) represents the Sigmoid activation function, T t , T s Represents the [CLS] token used to express global semantic information. represents the optimized features, represents the inner product of vectors, ⊙ represents the bitwise multiplication of elements (tokens) in the matrix, and MHA(·) represents the multi-head attention mechanism.

[0110] After calculating the features and (excluding [CLS] token) and calculate its cosine similarity Figure 8 After being resized to 4 dimensions by the second 4D Correlation module, the similarity hypercube is obtained. Then, referring to the VAT algorithm, a 4-dimensional window attention structure is introduced to design the hypercube aggregation module, namely ρ(·). In order to avoid excessive computational cost, a 4-dimensional spatial downsampling adaptive pooling operation is performed on the input similarity hypercube (e.g. Figure 8 The data after the communication voting is processed by smoothed correlation to obtain the output data, and the data used for the supervision of the next stage is used for communication generation using the saliency mask. The data after the adaptive pooling operation is reallocated to the feature communication channel to restore the initial shape of the similarity hypercube. The similarity hypercube with significant similarity score differences is obtained for the matched / unmatched images (blocks) through MVAM. For subsequent PMPS use.

[0111] Next, in the pseudo-matching pair selection module, based on the obtained similarity hypercube, possible next-level matching relationships are extracted to achieve fine-grained supervision (i.e., pseudo-supervision learning is performed through the learning objective to obtain matching relationships). For example, the next-level matching relationship of image-level matching is region-level matching. To achieve this goal, extreme values ​​with neighborhood consistency are selected on the similarity hypercube, and a matching voting mechanism on the Hough space is introduced. In this mechanism, the similarity hypercube is converted into the Hough space. In this space, the neighborhood matching consistency score is used to re-determine the similarity of the matching pairs. Subsequently, a one-to-one selection method based on a greedy strategy is used to generate the final pseudo-matching pairs.

[0112] Next, the specific process of the coarse-to-fine multi-instance learning pipeline provided by this application is described in detail.

[0113] In one implementation of the present application, the data set processing model includes a primary data set processing module, a secondary data set processing module and a tertiary data set processing module, such as Figure 9 As shown, the first data set is input into the data set processing model, and the data set processing model performs data set generation processing based on the first data set to obtain the second data set, including the following steps:

[0114] S201: Input the first data set into the first-level data set processing module for data set generation processing to obtain a third data set, where the third data set includes multiple third image pairs, each third image pair has a corresponding image-level label, and the image-level label is used to represent the image-level matching relationship of the third image pairs.

[0115] Step S201 is the image-level matching optimization pipeline provided by this application. For image-level matching optimization, first, construct a positive sample pair (i.e., a first positive image pair) (input image pairs with images belonging to the same category) and a negative sample pair (i.e., a first negative image pair) (input image pairs with images belonging to different categories) for further processing. That is, in one implementation of this application, the first image pair includes a first positive image pair and a first negative image pair.

[0116] After the cascaded MVAM and PMPS modules (i.e., the first-level dataset processing module), a rough pixel-level matching relationship (i.e., image-level label) is obtained for each positive sample pair. However, these matching relationships are not accurate and can only indicate that there may be a certain correlation between the areas where the two key points are located. Therefore, a saliency detector is used to ensure that the selected key points are all from the foreground area of ​​the image (i.e., the first image). Then, a neighborhood window with a range of 5×5 (i.e., the target window value, unit: token) is used to represent each pair of possible key points with a matching relationship, representing the area with similar semantic information. Finally, these windows will become the input of the region-level matching optimization pipeline (i.e., the third image pair).

[0117] That is, in one implementation of the present application, the third image pair includes an image corresponding to a region of the foreground image from the first image pair and having an area equal to the target window value.

[0118] S202: Input the third data set into the secondary data set processing module for data set generation processing to obtain a fourth data set, where the fourth data set includes multiple fourth image pairs, each fourth image pair has a corresponding region-level label, and the region-level label is used to represent the region-level matching relationship of the fourth image pairs.

[0119] Step S202, that is, the region-level matching optimization pipeline provided by the present application, for region-level matching optimization, the goal is to find the pixel-level matching pair with the highest confidence. In addition, it is worth noting that only one pixel-level matching pair will be selected from the above-mentioned region pair. Similar to image-level matching optimization, it is necessary to generate positive sample pairs (from the same key point pair, i.e., the second positive image pair) and negative sample pairs (from different key point pairs, i.e., the second negative image pair) at the regional granularity. That is, in one implementation of the present application, the third image pair includes the second positive image pair and the second negative image pair.

[0120] After processing by the cascaded MVAM and PMPS modules (i.e., the secondary dataset processing module) and the foreground constraint module, possible pixel-level matching pairs are obtained. The keypoint pairs with the highest similarity are then extracted from the 4D similarity hypercube. Based on the positions of each region in the original image, the keypoint positions obtained in this step are converted to their positions in the original image (i.e., the first image pair). Finally, through further non-maximum suppression, 16 pairs of matches with high confidence are selected (i.e., the fourth image pair).

[0121] That is, in one implementation of the present application, Figure 10 As shown, the label generation unit included in the secondary data set processing module obtains the data set output by the secondary data set processing module based on the hypercube data, including the following steps:

[0122] S2021: Based on the hypercube data, a fourth initial data set is obtained, where the fourth initial data set includes multiple fifth image pairs (i.e., possible pixel-level matching pairs), and each fifth image pair has a corresponding region-level label.

[0123] S2022: Select some region-level labels (i.e., key point pairs with the highest similarity) from the region-level labels of the fifth image pair whose label similarity is greater than the label threshold, and determine the position of each partial region-level label on the image (i.e., the original image) included in the corresponding first image pair.

[0124] S2023: Convert the position of the partial region-level label on the image in the corresponding fifth image pair to the position of the partial region-level label on the image included in the corresponding first image pair.

[0125] S2024: Perform a non-maximum suppression algorithm on the partial region-level labels after the position conversion to obtain a fourth data set.

[0126] S203: Input the fourth data set into the third-level data set processing module for data set generation processing to obtain a second data set.

[0127] Step S203, that is, the pixel-level matching optimization pipeline provided by this application, generates a pixel-level pseudo-label generator at this stage. Based on the 16 pairs of matching pairs with high confidence obtained from the region-level matching optimization pipeline, a pixel-level pseudo-label generator is trained. This pseudo-label generator inherits the basic structure of the previous matching optimization pipeline. In other words, the pixel-level pseudo-label generator takes the image pair as input, outputs a high-quality similarity hypercube, and then screens based on the similarity hypercube to obtain the final matching pair (i.e., the second image pair).

[0128] In the coarse-to-fine multi-instance learning pipeline provided in this application, only the image category information is used as a supervisory signal. During the training of the image-level and region-level pipelines, a contrastive learning task can be designed to maximize the difference between the similarity hypercubes of matched and unmatched objects to distinguish them. This task can be formally expressed as the following formula:

[0129]

[0130] Among them, AP(·) represents the similarity hypercube v f It should be noted that in order to generate negative sample pairs, this application refers to the NCNet (for details, please refer to ROCCO I, CIMPOI M, ARANDJELOVIC R, et al. Ncnet: Neighbourhood con-sensus networks for estimating image correspondences [J]. IEEE Transactions on PatternAnalysis and MachineIntelligence, 2020) approach and directly performs a sliding operation on the positive sample pairs. In the pixel-level pseudo-label generator, the supervisory signal used is the matching point pair from the region-level matching optimization pipeline, and with reference to the general practice of the semantic matching algorithm, the binary cross entropy loss is used as the pixel-level loss function. Finally, the loss function of the entire pipeline is the image-level loss. Regional-level losses Pixel-level loss The weighted average of is as follows:

[0131]

[0132] in, is the loss of the entire pipeline, and λ1, λ2, and λ3 are weights.

[0133] After training is completed, a pixel-level pseudo-label generator is used to generate pseudo-labels for pre-training semantic matching models on the first dataset (such as the public ImageNet dataset). In this process, it is necessary to further screen image pairs with accurate pixel-level matching relationships. Therefore, a closed-loop check method is introduced to select the final image pairs and their pseudo-labels. In the inference phase, for the image pairs (I s , I t ) introduces a third image I m For I s A specific point P is selected s (You can also select a specific point P from It s ), through I s →I m →I t →I s The matching path obtains P′ S Using P s and P′ S The Euclidean distance between them is used as the consistency score to evaluate whether the image pair can be selected (using the average score of all matching pairs in the image) and a single matching pair.

[0134] That is, in one implementation of the present application, Figure 11 As shown, the fourth data set is input into the third-level data set processing module for data set generation processing to obtain the second data set, including the following steps:

[0135] S2031: Input the fourth data set into the third-level data set processing module for data set generation processing to obtain a second initial data set, where the second initial data set includes multiple sixth image pairs, each sixth image pair having a corresponding pixel-level label.

[0136] S2032: Determine the first image pair (i.e., I s , I t ), and determine the third image (i.e., I m ).

[0137] S2033: Determine the first feature point (i.e., P) on the image in the first image pair. s ).

[0138] S2034: According to the image matching path (i.e., I s →I m →I t →I s ) Get the second feature point (ie P′ S ), the image matching path is obtained based on the first image pair and the third image.

[0139] S2035: Calculate the Euclidean distance between the first feature point and the second feature point, and if the Euclidean distance is less than a preset distance threshold, select the sixth image pair as the final second image pair.

[0140] S2036: Obtain a second data set based on the plurality of second image pairs.

[0141] Furthermore, in order to make the first data set more conducive to the generation of a data set processing model, the present application provides a solution for screening and processing a large-scale classification data set to obtain a data set of relatively high quality.

[0142] Here, SC-ImageNet (as an example of the first dataset) is constructed based on the large-scale classification dataset ImageNet (which includes 1,281,167 images from 1,000 categories). In order to make the ImageNet dataset suitable for semantic matching tasks, three levels of screening are required. Figure 12As shown, some categories are not suitable for the single-instance semantic matching task. For example, images in the 'conch' category often contain too many instances. Meanwhile, it's difficult to find salient keypoints with clear semantic information in images labeled 'lake-side'. Based on the above logic, 321 categories were removed (i.e., filtered categories) from the ImageNet raw categories using filtering rules. Subsequently, an object detector trained on MSCOCO was used to select images from the raw images that contained only one high-confidence instance (i.e., removed and selected), resulting in filtered images. After filtering, 379,458 single-instance images from 679 categories remained in ImageNet. During the training process, images are selected from the screened images (SelectedImages), and images of the same category in the selected images are randomly paired, and each image can only be used once to obtain image pairs (One-To-One Pairing). The image pairs are divided into training divisions (Training Division), and pixel-level labels are generated for samples in the SC-ImageNet dataset through the C2F training pipeline (C2F Training Pipeline), that is, the dataset processing model. In the label generation (Label Generate Division) stage, the closed-loop matching score (i.e., closed-loop inspection) is calculated 100 times for each image through the closed-loop label filter (Close-loop Label Filtering), and based on the average score obtained, the top 30% of images with the consistency score are selected. Finally, SC-ImageNet contains 113,516 highly consistent single-instance images from 679 categories, forming 794,612 image pairs (i.e., the first image pairs).

[0143] That is, in one implementation of the present application, Figure 13 As shown, determining the first data set includes the following steps:

[0144] S110: Determine the initial data set.

[0145] S120: Remove samples that cannot perform the single-instance semantic matching task from the initial dataset to obtain an intermediate dataset.

[0146] S130: Performing object detection algorithm processing on the samples in the intermediate data set, and selecting intermediate samples that only contain one instance with a confidence level greater than a confidence threshold.

[0147] S140: Calculate the consistency score of each intermediate sample, and select a preset proportion of sample data from the intermediate samples as target samples based on the consistency score.

[0148] S150: Determine a first data set based on multiple target samples.

[0149] Further, if Figure 14 As shown, this application provides a semantic matching method, comprising the following steps:

[0150] S100': Determine the data to be matched.

[0151] S200 ′: inputting the data to be matched into the target semantic matching model for semantic matching processing to obtain a semantic matching result of the data to be matched.

[0152] The target semantic matching model is obtained by training the second data set, and the second data set is generated by the data set generation method in any one of the above implementation modes.

[0153] The second dataset generated by the aforementioned dataset generation method contains dense pixel-level image annotations. The target semantic matching model trained using the second dataset has stronger generalization ability and accuracy. Therefore, the semantic matching results of the data to be matched are also more accurate.

[0154] Further, if Figure 15 As shown, the present application provides a method for training a sense matching model, comprising the following steps:

[0155] S100": Determine a second data set.

[0156] The second training data set is generated by the data set generation method in any one of the above implementations.

[0157] S200”: Input the second data set into the initial semantic matching model for model training to obtain a target semantic matching model.

[0158] The second dataset generated by the aforementioned dataset generation method contains dense pixel-level image annotations. Therefore, the target semantic matching model trained on the second dataset has stronger generalization ability and accuracy.

[0159] The dataset generation, semantic matching and semantic matching model training methods provided in this application are a weakly supervised learning algorithm for learning high-quality semantic matching relationships from large-scale datasets using only image-level labels. Following the spirit of Multiple Instance Learning (MIL), the weakly supervised semantic matching problem is decomposed into three stages: image-level matching, region-level matching and pixel-level matching. This application proposes a novel weakly supervised semantic matching algorithm based on cascade matching optimization, integrating MIL and matching screening and refinement processes into a single deep neural network, and training the network end-to-end using only image-level supervision. During the learning process, the pixel-level matching relationships inferred from the weak supervision signal are propagated, filtered and enhanced through voting in Hough space and a calibration mechanism based on foreground information. In addition, this application also provides a matching consistency check algorithm for selecting key point pairs and image pairs with high semantic consistency, thereby generating pseudo labels for classic semantic matching algorithms. Finally, this application screened approximately 110,000 images from the ImageNet ILSVRC training set and constructed a dataset called SC-ImageNet. Experimental results based on standard datasets show that pre-training on SC-ImageNet can effectively improve the performance of state-of-the-art semantic matching algorithms.

[0160] like Figure 16 As shown, the rows from top to bottom show the pseudo labels (i.e., category labels of image pairs) generated by this application in the image-level, region-level, and pixel-level matching stages, respectively. The last row is the matching relationship annotated by humans for reference.

[0161] The dataset generation, semantic matching, and semantic matching model training methods provided in this application are designed to train high-performance neural networks for semantic matching tasks based on limited data. A weakly supervised learning method is used to learn semantic matching relationships from the ImageNet ILSVRC training set, as this dataset contains a large number of images and a variety of object categories. Given a pair of images from ImageNet, the Multiple Instance Learning (MIL) process is followed, treating the images as bags and pixel-level matching pairs as instances to train a binary classification matcher. The ImageNet ILSVRC training set is selected as the basis for subsequent work, and the MIL process is integrated into OICR (for details, see TANG P, WANG X, BAI X, et al. Multiple instance detection network with online instance classifier refinement [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 2843-2851) and MEFF (for details, see GE W, YANG S, YUY. Multi-evidence filtering and fusion for multi-label classification, object detection and semantic segmentation based on weakly supervised learning [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 1277-1286) to decompose the problem into image-level matching stage, region-level matching stage, and pixel-level matching stage.

[0162] In order to learn reliable matching relationships from the ImageNet dataset, this application provides a novel weakly supervised semantic matching algorithm based on cascade matching optimization to integrate image-level, region-level, and pixel-level matching modules into a single neural network and train it in an end-to-end manner. Figure 16As shown in , in the image-level matching module, the image category label is used to determine whether there is a suitable semantic matching relationship in the image pair, and the matches in the positive image pairs (images containing similar objects) with high confidence are considered to be reliable matching relationships. The region-level matching module accepts supervision from the previous stage and performs fine-grained region-level matching to further improve the matching accuracy. Finally, the pixel-level matching module obtains better supervision from the previous stage and trains the semantic matching head according to the previous method. In order to ensure learning efficiency, unlike the most advanced semantic matching algorithms, such as Figure 8 As shown in the figure, this application uses ViT (Vision Transformer) as the backbone network and introduces a semantic matching pipeline based on gating and attention mechanisms, including a feature aggregation module and a relevance aggregation module. This design has been experimentally verified to have superior performance. Another feature of this pipeline is that it provides a matching filtering and optimization module to improve the quality of matching relationships in different matching stages. In addition, a saliency detector SelfReformer (see YUN YK, LIN W. Selfreformer: Self-refined network with transformer for salient object detection [A]. 2022) is integrated to segment foreground objects, and regularized Hough matching (RHM, see MIN J, LEE J, PONCE J, et al. Hyperpixel flow: Semantic correspondence with multi-layer neural features [C] / / Proceedings of the IEEE / CVF International Con-ference on Computer Vision. 2019: 3395-3404) is used to further ensure matching consistency.

[0163] After learning matching relationships from image-level annotations, we generated a semantic matching dataset containing high-quality pseudo-labels, named SC-ImageNet. In ImageNet, some image categories are not instance-centric, lack significant and unique keypoints, or contain overly complex background information, making them unsuitable for training semantic matching models. These categories were manually removed, and 679 image categories were selected to be suitable for the semantic matching task. To further improve the quality of the dataset, we calculated the matching quality of each image and selected images containing more high-quality keypoints to construct our new dataset. Specifically, the matching quality was measured using a matching consistency score. The top 30% of the images in ImageNet were selected to form SC-ImageNet and used as a pre-training set for several popular semantic matching algorithms. Experimental results show that our weakly supervised semantic matching algorithm, based on cascaded matching optimization, outperforms existing weakly supervised semantic matching algorithms on the PF-PASCAL and PF-WILLOW datasets. It is worth mentioning that if we pre-train the most advanced semantic matching algorithms on SC-ImageNet and fine-tune them in a conventional fully supervised environment, this application can further improve the performance of each algorithm on various datasets.

[0164] In summary, this application introduces a new weakly supervised semantic matching paradigm that can learn pixel-level matching relationships between image pairs. This paradigm provides a new perspective for semantic matching methods to learn reliable matching relationships from large-scale, weakly annotated datasets.

[0165] Furthermore, this application proposes a novel weakly supervised semantic matching algorithm based on cascade matching optimization, which integrates multi-instance learning and matching filtering and optimization functions, and can be integrated into a neural network that can be trained end-to-end.

[0166] Furthermore, this application constructs a new semantic matching dataset based on ImageNet, named SC-ImageNet. This dataset contains 679 object categories, 113,516 images, and 794,612 image pairs for semantic matching tasks. Its scale is significantly larger than existing semantic matching datasets. In addition, models pre-trained on SC-ImageNet demonstrate strong generalization capabilities, which in turn benefits subsequent fine-tuning.

[0167] Next, combined with specific experimental data, the effectiveness and advantages of the dataset generation, semantic matching and semantic matching model training methods provided in this application in semantic matching model generation are explained.

[0168] like Figure 17As shown in the figure, the visualization results of the dataset generation method provided in this application on the SPair-71k dataset. Among them, TM represents the TransforMatcher algorithm. The other methods are CATs, DHPF and VAT. +Ours represents that the model is first pre-trained on SC-ImageNet and then fine-tuned on the SPair-71k dataset. Crosses of the same color represent key points containing similar semantic information (i.e., manually labeled). The gray line M2 represents a correct match, and the black line M1 represents an incorrect match (evaluated using the PCK@0.1 evaluation indicator).

[0169] As can be seen from the figure, the model obtained by the method provided by this application greatly improves the semantic matching results of images.

[0170] Below, we will provide further explanation based on specific data.

[0171] In order to demonstrate the effectiveness of the pseudo-label generator and SC-ImageNet provided in this application, first, the pseudo-label generator is directly compared with the weakly supervised method, and then the performance of some state-of-the-art methods pre-trained with SC-ImageNet is evaluated, such as CATs (see CHO S, HONG S, JEON S, et al. Cats: Cost aggregation transformers for visual correspondence [J]. Advances in Neural Information Processing Systems, 2021, 34: 9011-9023) and VAT (see HONG S, CHO S, NAM J, et al. Cost aggregation with 4d convolutional swin transformer for few-shot segmentation [C] / / Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXIX. Springer, 2022: 108-126). The performance of the pre-trained model is compared with that of the model trained with the original settings on PF-PASCAL, PF-WILLOW[8], and SPair-71k. At the same time, in order to verify the design of the pseudo-label generator, a series of ablation experiments are designed for the key components in the workflow.

[0172] In the process of implementing the pseudo-label generator, the SGD (Stochastic Gradient De-scent) optimizer was used for training, and the learning rate of all network layers was set to 3×10 -5 The momentum is 0.9, and the weights of the iBOT-B backbone network (for details, see ZHOU J, WEI C, WANG H, et al. ibot: Image bert pre-training with online tok-enizer [J]. International Conference on Learning Representations (ICLR), 2022) are locked. In addition, the pseudo-label generator is implemented based on PyTorch-GPU (for details, see PASZKEA, GROSS S, MASSA F, et al. Pytorch: An imperative style, high-performance deep learning library [J]. Advances in neural information processing systems, 2019, 32), using 8 NVIDIA RTX 3090 GPUs, each GPU processing 4 images simultaneously, and the resolution of each input image is adjusted to 512×512. In addition, the pseudo-label generator uses a SelfReformer trained on DUTS-TR (see WANG L, LU H, WANGY, et al. Learning to detect salient objects with image-level supervision [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 136-145) for saliency detection. The pre-training and fine-tuning of the state-of-the-art method follows its original implementation. For fair comparison, the hyperparameters, data augmentation settings, and input resolution remain the same as the original training settings.

[0173] During the experimental setup, widely used semantic matching datasets such as SPair-71k and PF-PASCAL were selected for experimentation. To further evaluate the generalization ability of the algorithm, experiments were conducted using PF-PASCAL for fine-tuning and PF-WILLOW for testing. PF-PASCAL consists of 1,351 image pairs selected from 20 object categories and roughly divided into 700 (training), 300 (validation), and 300 (testing) pairs. PF-WILLOW also includes an additional 900 image pairs for further evaluation. These images cover 10 categories. SPair-71k is a large-scale dataset consisting of 70,958 image pairs covering 18 categories with significant intra-class variation, scale variation, occlusion, and truncation. In SPair-71k, 53,340 image pairs are used for training, 5,384 pairs are used for validation, and 12,234 pairs are used for testing. However, even though existing datasets provide a large number of image pairs, the size of the images they contain is still limited. For example, SPair-71k consists of only 1800 images.

[0174] This application uses PCK@α (the percentage of correctly predicted keypoints among all keypoints in the current image pair under a threshold α) as the evaluation metric. When calculating PCK@α, a predicted keypoint is considered correct if it falls within a circle with a radius of α×d centered on the corresponding manual annotation, where d is the length of the long side of the image (for PF-PASCAL) or the long side of the object bounding box (for PF-WILLOW and SPair-71k). α is also a hyperparameter that represents accuracy.

[0175] In the stage of verifying the matching results, first, since there are no sparse keypoint annotations in ImageNet, the performance of the pseudo-label generator is evaluated on the test sets of PF-PASCAL and PF-WILLOW. As shown in Table 1, when trained only on PF-PASCAL with image-level supervision, the pseudo-label generator provided by this application surpasses the current best weakly supervised semantic matching algorithm (weakly supervised DHPF (see ROCCO I, CIMPOI M, ARANDJELOVIC R, et al. Ncnet: Neighbourhood con-sensus networks for estimating image correspondences [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020)) by 2.2% / 10.5% (PCK@0.05) and 4.1% / 8.5% (PCK@0.10) on PF-PASCAL and PF-WILLOW, respectively. Furthermore, even when trained only on a filtered subset of ImageNet without further fine-tuning, the pseudo-label generator presented in this application still outperforms the weakly supervised DHPF on PF-PASCAL and PF-WILLOW, achieving a 0.7% / 7.4% improvement in performance as measured by PCK@0.05.

[0176] Table 1 Quantitative evaluation of pseudo-label generators on the PF-PASCAL and PF-WILLOW datasets

[0177]

[0178] In Table 1, PF-P. stands for PF-PASCAL, PF-W. stands for PF-WILLOW, and FImN. stands for ImageNet after filtering out categories and multi-instance images that are not suitable for semantic matching tasks. In addition, DHPF is trained in a weakly supervised configuration.

[0179] Next, this application compares the performance of the model pre-trained on SC-ImageNet with the model trained using the original setting to demonstrate that high-quality pseudo-labels are helpful for training semantic matching models. Specifically, the state-of-the-art model is first pre-trained on SC-ImageNet. The pre-trained model is then fine-tuned on the target datasets, such as SPair-71k and PF-PASCAL, for final evaluation on the corresponding test sets. Compared with the version using the original training setting, the model pre-trained with SC-ImageNet can predict more accurate key points, especially on the test set of SPair-71k, which is the most challenging due to its obvious intra-class differences, scale differences, occlusions, and truncation. These results in Table 2 demonstrate that the effectiveness and robustness of the model have been significantly improved by adopting the pre-training setting provided by this application. In addition, according to Table 3, most of the excellent accuracy rates come from the pre-trained models, which supports the effectiveness of the SC-ImageNet provided by this application. The visualization results of the above experiments are shown in Figure 2. Figure 17 shown.

[0180] In addition, the pre-training strategy provided in this application introduces additional, automatically annotated image pairs, and therefore can be regarded as a semi-supervised method. However, the current most advanced semi-supervised semantic matching methods mainly adopt the strategy of introducing additional key points. Therefore, experiments were conducted to analyze the consistency and differences of these methods. As shown in Tables 2 and 3, the method provided in this application is competitive with SemiMatch (see KIM J, RYOO K, SEO J, et al. Semi-supervised learning of semantic correspondence with pseudo-labels [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022: 19699-19709 for details), especially on the challenging SPair-71k dataset. At the same time, compared with the original setting, SC-ImageNet pre-trained SCorrSAN (for details, see

[40] HUANG S,YANG L,HE B,et al.Learning semantic correspondence with sparse annotations[C] / / Computer Vision–ECCV 2022:17th European Conference,Tel Aviv,Israel,October23–27,2022,Proceedings,Part XIV.Springer,2022:267-284) performs better on all target datasets, indicating that the collaboration between additional image pairs and additional keypoint pairs helps improve the effectiveness, robustness and generalization ability of the model.

[0181] Table 2 Quantitative evaluation on PF-PASCAL, PF-WILLOW and SPair-71k datasets

[0182]

[0183] The best results in Table 2 are in bold, and the second-best results are underlined. Note that the input resolution of the original algorithm remains unchanged. GP. represents the pseudo-label generation process, while PT. and FT. denote the pre-training and fine-tuning processes, respectively.

[0184] Table 3 Quantitative evaluation on the SPair-71k dataset (refined to each object category in the dataset)

[0185]

[0186] The best results are in bold and the second best results are underlined in Table 3. It is worth noting that the input resolution of the original algorithm is not changed.

[0187] This application sets up an ablation experiment. First, the effect of the network components is verified. As shown in Table 4, the necessity of the network components in the pseudo-label generator (I) provided by this application is evaluated by removing each component from the complete process. Comparing Experiment (I) and Experiment (II), the decrease in PCK shows that with the help of saliency detection, the model provided by this application can more accurately focus on the foreground information in the input image pair. In addition, comparing Experiment (I) and Experiment (III), the more serious performance degradation shows that the geometric information in the Hough space is crucial to the region-level matching relationship. Further, Experiments (IV) to (VI) prove that the appropriate superpixel size is beneficial to RHM. Among them, superpixels that are too small lack geometric information, while superpixels that are too large will lead to a waste of computing resources. However, when the network weights of the feature aggregation module based on the gating and attention mechanism are shared in the three-stage pipeline, PCK will undergo the most significant decrease. Therefore, image features of different granularities should be processed independently.

[0188] Table 4 Ablation experiments on network components in the pseudo-label generator

[0189]

[0190] All models in Table 4 are trained on ImageNet. SD stands for saliency detection, L(RHM) stands for the length of superpixels in the RHM module, and SharedAttn. indicates that the network weights of the feature aggregation module based on gating and attention mechanisms are shared in the three-stage pipeline.

[0191] This application also conducted an impact test on network structure and input resolution, specifically verifying the importance of the pipeline and input resolution of each stage to the pseudo-label generator. As shown in Table 5, based on the results of experiments (II) to (IV), on PF-PASCAL, a smaller input resolution will result in a slight performance degradation. However, when tested on PF-WILLOW, such a slight degradation even disappeared in experiments (II) and (III), which reflects the powerful generalization ability of the model provided by this application. As for the network structure, experiment (V) shows that pixel-level matching plays a key role in correcting the position of key points. Experiment (VI) further reveals the importance of region-level matching to the method provided by this application, because geometric information is necessary in weakly supervised semantic matching.

[0192] Table 5 Ablation experiments on pipeline stages of pseudo-label generator and input image resolution

[0193]

[0194] All models in Table 5 are trained on ImageNet, where IIR represents the resolution of the input image.

[0195] This application also provides a data set generation device, such as Figure 18 As shown, it includes: a first input module, used to determine a first data set, the first data set includes a plurality of first image pairs, and each first image pair has corresponding category information.

[0196] A dataset generation module is used to input the first dataset into a dataset processing model, and the dataset processing model performs dataset generation processing based on the first dataset to obtain a second dataset. The second dataset includes multiple second image pairs, each second image pair has a corresponding pixel-level label, the pixel-level label is obtained based on the category information of the first image pair, and the pixel-level label is used to represent the pixel-level matching relationship of the second image pair.

[0197] The data set generation module may specifically include the aforementioned data set processing model, the data set processing module included in the data set processing model, and the aggregation unit and label generation unit included in the data set processing module.

[0198] This application also provides a semantic matching device, such as Figure 19 As shown, it includes: a second input module, used to determine the data to be matched; a semantic matching module, used to input the data to be matched into the target semantic matching model for semantic matching processing to obtain the semantic matching results of the data to be matched, wherein the target semantic matching model is obtained by training the second data set, and the second data set is generated by the aforementioned data set generation method.

[0199] This application also provides a semantic matching model training device, such as Figure 20 As shown, it includes: a third input module, used to determine the second data set, and the second training data set is generated by the aforementioned data set generation method; a model training module, used to input the second data set into the initial semantic matching model for model training to obtain a target semantic matching model.

[0200] The aforementioned data set generation, semantic matching and semantic matching model training methods or devices are applied to the client. It can be understood that the client applicable to the technical solution of the present application can be a hardware electronic device that can execute the data set generation method, or the semantic matching method, or the semantic matching model training method, such as a smart phone, tablet computer and other electronic devices, or it can be an application that can execute the data set generation method, or the semantic matching method, or the semantic matching model training method.

[0201] See Figure 21 , Figure 21 FIG2 is a block diagram of a structure of an electronic device provided by an implementation of the present application. The electronic device may include one or more processors 1002, system control logic 1008 connected to at least one of the processors 1002, system memory 1004 connected to the system control logic 1008, non-volatile memory (NVM) 1006 connected to the system control logic 1008, and a network interface 1010 connected to the system control logic 1008.

[0202] The processor 1002 may include one or more single-core or multi-core processors. The processor 1002 may include any combination of general-purpose processors and special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In the implementation of this document, the processor 1002 may be configured to execute the aforementioned data set generation method, or to cause a computing device cluster to execute the aforementioned semantic matching method, or to cause a computing device cluster to execute the aforementioned semantic matching model training method.

[0203] In some implementations, the system control logic 1008 may include any suitable interface controller to provide any suitable interface to at least one of the processors 1002 and / or any suitable device or component in communication with the system control logic 1008 .

[0204] In some implementations, the system control logic 1008 may include one or more memory controllers to provide an interface to the system memory 1004. The system memory 1004 may be used to load and store data and / or instructions. In some implementations, the system memory 1004 of the electronic device may include any suitable volatile memory, such as a suitable dynamic random access memory (DRAM).

[0205] NVM / memory 1006 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some implementations, NVM / memory 1006 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as at least one of a hard disk drive (HDD), a compact disc (CD) drive, and a digital versatile disc (DVD) drive.

[0206] NVM / storage 1006 may include a portion of storage resources installed on a device of an electronic device, or it may be accessible by the device but not necessarily a part of the device. For example, NVM / storage 1006 may be accessed over a network via network interface 1010 .

[0207] In particular, system memory 1004 and NVM / storage 1006 may respectively include a temporary copy and a permanent copy of instructions 1020. Instructions 1020 may include instructions that, when executed by at least one of processors 1002, cause an electronic device to implement the aforementioned dataset-based generation method, or cause a computing device cluster to perform the aforementioned semantic matching method, or cause a computing device cluster to perform the aforementioned semantic matching model training method. In some implementations, instructions 1020, hardware, firmware, and / or software components thereof may additionally or alternatively be located in system control logic 1008, network interface 1010, and / or processor 1002.

[0208] In one implementation, at least one of the processors 1002 may be packaged with logic for one or more controllers of the system control logic 1008 to form a System in a Package (SiP). In one implementation, at least one of the processors 1002 may be integrated with logic for one or more controllers of the system control logic 1008 on the same die to form a System on Chip (SoC).

[0209] The present application also provides a computing device 10. Figure 22 As shown, computing device 10 includes: bus 102, processor 104, and memory 106. Processor 104 and memory 106 communicate with each other via bus 102. Computing device 10 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 10.

[0210] like Figure 22 As shown, the memory 106 stores executable program codes, and the processor 104 executes the executable program codes to respectively implement the functions of the aforementioned data set generation device, thereby implementing the data set generation method. That is, the memory 106 stores instructions for executing the data set generation method.

[0211] Alternatively, the memory 106 stores executable codes, and the processor 104 executes the executable codes to respectively implement the functions of the aforementioned semantic matching device, thereby implementing the semantic matching method. That is, the memory 106 stores instructions for executing the semantic matching method.

[0212] Alternatively, the memory 106 stores executable code, and the processor 104 executes the executable code to respectively implement the functions of the aforementioned semantic matching model training device, thereby implementing the semantic matching model training method. That is, the memory 106 stores instructions for executing the semantic matching model training method.

[0213] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0214] like Figure 23 As shown, the computing device cluster includes at least one computing device 10. The memory 106 in one or more computing devices 10 in the computing device cluster may store the same instructions for executing the data set generation method.

[0215] In some possible implementations, the memory 106 of one or more computing devices 10 in the computing device cluster may also store some instructions for executing the dataset generation method. In other words, the combination of one or more computing devices 10 can jointly execute the instructions for executing the dataset generation method.

[0216] The memories 106 in different computing devices 10 in the computing device cluster may store different instructions, each for executing part of the functions of the data set generating apparatus.

[0217] Alternatively, the computing device cluster includes at least one computing device 10. The memory 106 in one or more computing devices 10 in the computing device cluster may store the same instructions for executing the semantic matching method.

[0218] In some possible implementations, the memory 106 of one or more computing devices 10 in the computing device cluster may also store some instructions for executing the semantic matching method. In other words, the combination of one or more computing devices 10 can jointly execute the instructions for executing the semantic matching method.

[0219] The memories 106 in different computing devices 10 in the computing device cluster may store different instructions, each for executing part of the functions of the semantic matching apparatus.

[0220] Alternatively, the computing device cluster includes at least one computing device 10. The memory 106 in one or more computing devices 10 in the computing device cluster may store the same instructions for executing the semantic matching model training method.

[0221] In some possible implementations, the memory 106 of one or more computing devices 10 in the computing device cluster may also store partial instructions for executing the semantic matching model training method. In other words, the combination of one or more computing devices 10 can jointly execute the instructions for executing the semantic matching model training method.

[0222] The memories 106 in different computing devices 10 in the computing device cluster may store different instructions, each used to execute part of the functions of the semantic matching model training apparatus.

[0223] The present application also provides a computer program product comprising instructions. The computer program product may be software or a program product comprising instructions that can be executed on a computing device or stored in any available medium. When the computer program product is executed on at least one computing device, the at least one computing device is caused to execute a dataset generation method, a semantic matching method, or a semantic matching model training method.

[0224] The implementation of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute a data set generation method, a semantic matching method, or a semantic matching model training method.

Claims

1. A method for generating a data set, characterized in that: Applied to a client, the method includes: Determine a first data set, the first data set including a plurality of first image pairs, each of the first image pairs having corresponding category information; The first data set is input into a data set processing model, and the data set processing model performs multi-level data set generation processing on the first data set to obtain a third data set, a fourth data set, and a second data set in sequence, wherein the data set obtained by each level of data set generation processing includes multiple image pairs, each image pair has a corresponding category label, and the category label is obtained based on the category information of the first image pair, and the density of the category labels of the image pairs included in the data set obtained by the latter level of data set generation processing is greater than the density of the category labels of the image pairs included in the data set obtained by the previous level of data set generation processing, then, The third data set includes a plurality of third image pairs, each of the third image pairs has a corresponding image-level label, and the image-level label is used to represent the image-level matching relationship of the third image pairs. The fourth data set includes a plurality of fourth image pairs, each of the fourth image pairs has a corresponding region-level label, and the region-level label is used to represent a region-level matching relationship of the fourth image pairs. The second data set includes a plurality of second image pairs, each of the second image pairs has a corresponding pixel-level label, and the pixel-level label is used to represent a pixel-level matching relationship of the second image pairs.

2. The data set generation method according to claim 1, wherein: The data set processing model includes at least two data set processing modules cascaded in sequence, wherein the data set processing modules are used to perform data set generation processing, input the first data set into the data set processing model, and the data set processing model performs multi-level data set generation processing on the first data set to sequentially obtain a third data set, a fourth data set, and a second data set, including: The data sets output by the data set processing module of the previous level are sequentially input into the data set processing module of the next level for data set generation processing, until the data set processing module of the last level outputs the corresponding data set, so as to obtain the third data set, the fourth data set and the second data set in sequence, wherein the input of the data set processing module of the first level is the first data set, and the output of the data set processing module of the last level is the second data set.

3. The data set generation method according to claim 2, wherein: The data set processing module includes an aggregation unit and a label generation unit. The data set processing module performs data set generation processing, including: The aggregation unit obtains hypercube data corresponding to the image pairs included in the data set based on the received data set, and inputs the hypercube data into the label generation unit, where the hypercube data is four-dimensional data; The label generation unit obtains the data set output by the data set processing module based on the hypercube data.

4. The data set generation method according to claim 3, wherein: The aggregation unit obtains hypercube data corresponding to the image pairs included in the data set based on the received data set, including: Performing feature aggregation processing on the image pairs included in the data set to obtain initial feature pairs; Optimizing the initial feature pair to obtain a target feature pair; The hypercube data is obtained according to the target feature pair.

5. The data set generation method according to claim 4, wherein: Obtaining the hypercube data according to the target feature pair includes: Determining the cosine similarity of the target feature pair; The hypercube data is obtained according to the cosine similarity.

6. The data set generation method according to claim 5, wherein: The label generation unit obtains the data set output by the data set processing module based on the hypercube data, including: Converting the hypercube data into Hough space to obtain converted hypercube data; calculating a domain matching consistency score for the transformed hypercube data; determining a target image pair according to the domain matching consistency score; According to the target image pair, a data set output by the data set processing module is obtained.

7. The data set generation method according to claim 6, wherein: The data set processing model includes a primary data set processing module, a secondary data set processing module, and a tertiary data set processing module. The first data set is input into the data set processing model, and the data set processing model performs multi-level data set generation processing on the first data set to sequentially obtain a third data set, a fourth data set, and a second data set, including: Inputting the first data set into the primary data set processing module for data set generation processing to obtain the third data set; Inputting the third data set into the secondary data set processing module for data set generation processing to obtain the fourth data set; The fourth data set is input into the third-level data set processing module for data set generation processing to obtain the second data set.

8. The data set generation method according to claim 7, wherein: The label generation unit included in the secondary data set processing module obtains the data set output by the secondary data set processing module based on the hypercube data, including: Based on the hypercube data, a fourth initial data set is obtained, where the fourth initial data set includes a plurality of fifth image pairs, each of the fifth image pairs having a corresponding region-level label; Selecting some region-level labels of the fifth image pair whose label similarity is greater than a label threshold, and determining a position of each of the region-level labels on the corresponding image included in the first image pair; Converting the position of the partial region-level label on the corresponding image in the fifth image pair to the position of the partial region-level label on the corresponding image included in the first image pair; The part of the region-level labels after the position conversion is processed by a non-maximum suppression algorithm to obtain the fourth data set.

9. The data set generation method according to claim 8, wherein: Inputting the fourth data set into the third-level data set processing module for data set generation processing to obtain the second data set includes: Inputting the fourth data set into the third-level data set processing module for data set generation processing to obtain a second initial data set, wherein the second initial data set includes a plurality of sixth image pairs, each of the sixth image pairs having a corresponding pixel-level label; determining the first image pair corresponding to the sixth image pair, and determining a third image based on the first image pair; determining a first feature point on an image in the first image pair; obtaining a second feature point according to an image matching path, wherein the image matching path is obtained according to the first image pair and the third image; calculating a Euclidean distance between the first feature point and the second feature point, and selecting the sixth image pair as a final second image pair when the Euclidean distance is less than a preset distance threshold; The second data set is obtained according to a plurality of second image pairs.

10. The data set generation method according to claim 9, wherein: The third image pair includes an image corresponding to a region of the foreground image from the first image pair and having an area equal to the target window value.

11. The data set generation method according to claim 10, wherein: The first image pair includes a first positive image pair and a first negative image pair, and the third image pair includes a second positive image pair and a second negative image pair, wherein: The images in the first positive image pair and the second positive image pair belong to the same category, and the images in the first negative image pair and the second negative image pair belong to different categories.

12. The method for generating a data set according to any one of claims 8 to 11, wherein: Determine a first data set, including: Determine the initial dataset; Removing samples that cannot perform the single-instance semantic matching task from the initial dataset to obtain an intermediate dataset; Performing object detection algorithm processing on samples in the intermediate data set, and selecting intermediate samples that contain only one instance with a confidence score greater than a confidence threshold; Calculating a consistency score for each of the intermediate samples, and selecting a preset proportion of sample data from the intermediate samples as target samples based on the consistency score; The first data set is determined according to the plurality of target samples.

13. A semantic matching method, characterized in that: The method comprises: Determine the data to be matched; The data to be matched is input into a target semantic matching model for semantic matching processing to obtain a semantic matching result of the data to be matched, wherein the target semantic matching model is obtained by training a second data set, and the second data set is generated by the data set generation method described in any one of claims 1-12.

14. A semantic matching model training method, characterized in that: The method comprises: Determine a second data set, where the second data set is generated by the data set generation method according to any one of claims 1 to 12; The second data set is input into the initial semantic matching model for model training to obtain a target semantic matching model.

15. A data set generating device, characterized in that: include: A first input module is configured to determine a first data set, wherein the first data set includes a plurality of first image pairs, each of the first image pairs having corresponding category information; A dataset generation module is used to input the first dataset into a dataset processing model, and the dataset processing model performs multi-level dataset generation processing on the first dataset to obtain a third dataset, a fourth dataset, and a second dataset in sequence, wherein the dataset obtained by each level of dataset generation processing includes multiple image pairs, each image pair has a corresponding category label, and the category label is obtained based on the category information of the first image pair, and the density of the category labels of the image pairs included in the dataset obtained by the latter level of dataset generation processing is greater than the density of the category labels of the image pairs included in the dataset obtained by the previous level of dataset generation processing. Then, the third dataset includes multiple third image pairs, each of the third image pairs has a corresponding image-level label, and the image-level label is used to represent the image-level matching relationship of the third image pairs. The fourth dataset includes multiple fourth image pairs, each of the fourth image pairs has a corresponding region-level label, and the region-level label is used to represent the region-level matching relationship of the fourth image pairs. The second dataset includes multiple second image pairs, each of the second image pairs has a corresponding pixel-level label, and the pixel-level label is used to represent the pixel-level matching relationship of the second image pairs.

16. A semantic matching device, characterized in that: include: The second input module is used to determine the data to be matched; A semantic matching module is used to input the data to be matched into a target semantic matching model for semantic matching processing to obtain a semantic matching result of the data to be matched, wherein the target semantic matching model is obtained by training a second data set, and the second data set is generated by the data set generation method described in any one of claims 1-12.

17. A semantic matching model training device, characterized in that: include: a third input module, configured to determine a second data set, where the second data set is generated by the data set generation method according to any one of claims 1 to 12; The model training module is used to input the second data set into the initial semantic matching model for model training to obtain a target semantic matching model.

18. A computing device cluster, characterized in that: It includes at least one computing device, each computing device includes a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the data set generation method according to any one of claims 1 to 12, or so that the computing device cluster executes the semantic matching method according to claim 13, or so that the computing device cluster executes the semantic matching model training method according to claim 14.

19. A computer program product comprising instructions, characterized in that When the instruction is executed by a computing device cluster, the computing device cluster executes the data set generation method according to any one of claims 1 to 12, or the computing device cluster executes the semantic matching method according to claim 13, or the computing device cluster executes the semantic matching model training method according to claim 14.

20. A computer-readable storage medium, characterized in that Including computer program instructions, when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the data set generation method according to any one of claims 1 to 12, or causes the computing device cluster to execute the semantic matching method according to claim 13, or causes the computing device cluster to execute the semantic matching model training method according to claim 14.

Images