Multi-scale semantic information enhancement method and device to improve contrastive learning model performance

Abstract

The present application relates to a multi-scale semantic information enhancement method and device for improving the performance of a contrastive learning model. The method comprises: cascading a YOLO target detection model and a SAM semantic segmentation model to obtain a multi-scale semantic information enhancement framework; the YOLO target detection model is used to identify targets in an input image and annotate the targets with bounding boxes, and the SAM semantic segmentation model is used to semantically segment targets within the bounding boxes; the multi-scale semantic information enhancement framework is used to extract enhanced data of the input original data; the original data and the enhanced data are mixed to obtain mixed sample data, and the mixed sample data is used to improve the performance of the contrastive learning model in the pre-training stage. The use of this method can improve the performance of the contrastive learning model.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a multi-scale semantic information enhancement method and device for improving the performance of a contrastive learning model. Background Art

[0002] In recent years, deep learning has performed well in computer vision tasks. However, supervised learning methods often rely on manually designed labels, while self-supervised methods can overcome this problem. Self-supervised learning generates descriptive and easy-to-understand representations based on unlabeled data to define prediction tasks. In the field of self-supervised learning, how to generate high-quality pseudo labels is a difficult problem. If there is a large deviation between the generated labels and the true labels, the model will be misguided during the learning process, which will affect the final performance. As a data-driven method paradigm, self-supervised learning methods have high requirements for data quality. It is an effective work to expand the original data set by introducing new data so that the model can obtain stronger model generalization ability. This work is called data expansion. The current common data expansion method is to use generated data as additional data, so that the model can learn the knowledge hidden in it through the data. This route does not intuitively utilize the knowledge contained in the model. Summary of the invention

[0003] Based on this, it is necessary to provide a multi-scale semantic information enhancement method and device to improve the performance of contrastive learning models in response to the above technical problems.

[0004] A multi-scale semantic information enhancement method for improving the performance of a contrastive learning model, the method comprising:

[0005] A YOLO target detection model and a SAM semantic segmentation model are cascaded to obtain a multi-scale semantic information enhancement framework; the YOLO target detection model is used to identify targets in an input image and annotate the targets with bounding boxes, and the SAM semantic segmentation model is used to semantically segment targets within the bounding boxes;

[0006] Extracting enhanced data of the input original data using the multi-scale semantic information enhancement framework;

[0007] After the original data and the enhanced data are mixed, mixed sample data is obtained, and the mixed sample data is used to improve the performance of the comparative learning model in the pre-training stage.

[0008] In one embodiment, it also includes: defining a mixed loss function as:

[0009] ;

[0010] in, is the loss of original data, To enhance the data loss, ;

[0011] Define the average distance between positive sample pairs in feature space for:

[0012] ;

[0013] in, , Represents the positive sample set The number of samples in and Positive sample set Positive sample pairs in ;

[0014] Define the average distance between negative sample pairs in feature space for:

[0015] ;

[0016] in, , Represents the negative sample set The number of samples in and Negative sample set Negative sample pairs in ;

[0017] Rewrite the loss function as:

[0018] ;

[0019] in, is the mixed loss function of positive samples, is the original loss function of negative samples;

[0020] The optimization goal of constructing semantic consistency between enhanced data and positive samples is:

[0021] ;

[0022] in, The semantic information of the enhanced data is represented, and the performance of the contrastive learning model is improved in the pre-training stage by combining the semantic consistency optimization objective and the loss function.

[0023] In one embodiment, it further includes: defining the loss function of the mixed sample data as:

[0024] ;

[0025] Where D is mixed sample data;

[0026] The mixed loss function is defined as:

[0027] ;

[0028] in, is the loss of original data, To enhance the data loss, ;

[0029] The optimization goal of building semantic consistency between enhanced data and original data is:

[0030] ;

[0031] Using scale matching loss The difference in scale distribution between the mixed data and the original data in the feature space is used to quantify:

[0032] ;

[0033] According to the loss function and scale matching loss, the optimization goal is to minimize the total loss for:

[0034] ;

[0035] When satisfied and When minimized, .

[0036] A multi-scale semantic information enhancement device for improving the performance of a contrastive learning model, the device comprising:

[0037] A framework building module, used to cascade a YOLO target detection model and a SAM semantic segmentation model to obtain a multi-scale semantic information enhancement framework; the YOLO target detection model is used to identify targets in an input image and annotate the targets with bounding boxes, and the SAM semantic segmentation model is used to perform semantic segmentation on targets within the bounding boxes;

[0038] A data enhancement module, used to extract enhanced data of the input original data using the multi-scale semantic information enhancement framework;

[0039] The performance improvement module is used to mix the original data and the enhanced data to obtain mixed sample data, and use the mixed sample data to improve the performance of the comparative learning model in the pre-training stage.

[0040] In one embodiment, the performance improvement module is further used to define a hybrid loss function as:

[0041] ;

[0042] in, is the loss of original data, To enhance the data loss, ;

[0043] Define the average distance between positive sample pairs in feature space for:

[0044] ;

[0045] in, , Represents the positive sample set The number of samples in and Positive sample set Positive sample pairs in ;

[0046] Define the average distance between negative sample pairs in feature space for:

[0047] ;

[0048] in, , Represents the negative sample set The number of samples in and Negative sample set Negative sample pairs in ;

[0049] Rewrite the loss function as:

[0050] ;

[0051] in, is the mixed loss function of positive samples, is the original loss function of negative samples;

[0052] The optimization goal of constructing semantic consistency between enhanced data and positive samples is:

[0053] ;

[0054] in, The semantic information of the enhanced data is represented, and the performance of the contrastive learning model is improved in the pre-training stage by combining the semantic consistency optimization objective and the loss function.

[0055] In one embodiment, the performance improvement module is further used to define the loss function of the mixed sample data as:

[0056] ;

[0057] Where D is mixed sample data;

[0058] The mixed loss function is defined as:

[0059] ;

[0060] in, is the loss of original data, To enhance the data loss, ;

[0061] The optimization goal of building semantic consistency between enhanced data and original data is:

[0062] ;

[0063] Using scale matching loss The difference in scale distribution between the mixed data and the original data in the feature space is used to quantify:

[0064] ;

[0065] According to the loss function and scale matching loss, the optimization goal is to minimize the total loss for:

[0066] ;

[0067] When satisfied and When minimized, .

[0068] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0069] A YOLO target detection model and a SAM semantic segmentation model are cascaded to obtain a multi-scale semantic information enhancement framework; the YOLO target detection model is used to identify targets in an input image and annotate the targets with bounding boxes, and the SAM semantic segmentation model is used to semantically segment targets within the bounding boxes;

[0070] Extracting enhanced data of the input original data using the multi-scale semantic information enhancement framework;

[0071] After the original data and the enhanced data are mixed, mixed sample data is obtained, and the mixed sample data is used to improve the performance of the comparative learning model in the pre-training stage.

[0072] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0073] A YOLO target detection model and a SAM semantic segmentation model are cascaded to obtain a multi-scale semantic information enhancement framework; the YOLO target detection model is used to identify targets in an input image and annotate the targets with bounding boxes, and the SAM semantic segmentation model is used to semantically segment targets within the bounding boxes;

[0074] Extracting enhanced data of the input original data using the multi-scale semantic information enhancement framework;

[0075] After the original data and the enhanced data are mixed, mixed sample data is obtained, and the mixed sample data is used to improve the performance of the comparative learning model in the pre-training stage.

[0076] The above-mentioned multi-scale semantic information enhancement method and device for improving the performance of the contrastive learning model integrates the YOLO target detection model and the SAM semantic segmentation model through a multi-scale semantic information enhancement framework. The model can obtain rich semantic information from large-scale target positions to small-scale target contours. This multi-scale semantic information fusion gives the model more basis in the feature learning process and can learn more discriminative features, thereby improving the accuracy of the model in downstream tasks. Then, based on the enhanced data of the initial data, mixed sample data is obtained, and the mixed sample data is used to improve the performance of the contrastive learning model in the pre-training stage. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 A schematic diagram of a process flow of a multi-scale semantic information enhancement method for improving the performance of a contrastive learning model in one embodiment;

[0078] Figure 2 FIG. 1 is a visualization diagram of the training process of SimCLR on the CIFAR-10 dataset in one embodiment; wherein, Figure 2 (a) represents the visual trend graph of data, Figure 2 (b) represents a visual bar graph of the data;

[0079] Figure 3 A structural block diagram of a multi-scale semantic information enhancement device for improving the performance of a contrastive learning model in one embodiment;

[0080] Figure 4 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0082] In one embodiment, Figure 1 As shown, a multi-scale semantic information enhancement method for improving the performance of a contrastive learning model is provided, comprising the following steps:

[0083] Step 102, cascading the YOLO target detection model and the SAM semantic segmentation model to obtain a multi-scale semantic information enhancement framework.

[0084] The YOLO target detection model is used to identify targets in the input image and annotate the targets with bounding boxes. The SAM semantic segmentation model is used to perform semantic segmentation on the targets within the bounding boxes.

[0085] Specifically, the YOLO target detection model is a target detection algorithm based on deep learning. Its main feature is fast speed and can detect multiple targets in an image in real time. Its prediction mechanism is to divide the input image into multiple grids. Each grid is responsible for detecting the target that falls into it. Each grid predicts multiple bounding boxes and the probability and category of these bounding boxes containing targets. By designing a suitable loss function to train the model, it can accurately predict the location, size and category of the target. The loss function usually includes positioning loss (used to penalize inaccurate prediction of the bounding box location), confidence loss (used to penalize inaccurate prediction of the confidence that the bounding box contains the target) and classification loss (used to penalize inaccurate prediction of the target category). The SAM semantic segmentation model is a powerful semantic segmentation model. Its goal is to be able to perform pixel-level segmentation on any target in any image. SAM can accept a variety of prompts as input, such as points, boxes, text, etc. These prompts are used to specify the target to be segmented. The model first extracts features from the input image, and then generates a pixel-level segmentation mask of the target through the decoder based on the input prompt information. Because it is trained on a large amount of diverse data, SAM has strong generalization capabilities and can handle various types of objects and scenes, even objects that have not appeared in the training data.

[0086] From the perspective of semantic scale, the output of the YOLO model provides preliminary object location information, and the SAM model refines the object contour information. The cascade operation of YOLO-SAM provides semantic information from two perspectives: the location and contour of the target, providing more detailed contextual information, which is expected to improve the performance of self-supervised learning.

[0087] Step 104: extract enhanced data from the input original data using a multi-scale semantic information enhancement framework.

[0088] Step 106, after mixing the original data and the enhanced data, mixed sample data is obtained, and the mixed sample data is used to improve the performance of the comparative learning model in the pre-training stage.

[0089] In the above-mentioned multi-scale semantic information enhancement method for improving the performance of the contrastive learning model, the YOLO target detection model and the SAM semantic segmentation model are integrated through the multi-scale semantic information enhancement framework, and the model can obtain rich semantic information from large-scale target positions to small-scale target contours. This multi-scale semantic information fusion gives the model more basis in the feature learning process and can learn more discriminative features, thereby improving the accuracy of the model in downstream tasks. Then, based on the enhanced data of the initial data, mixed sample data is obtained, and the mixed sample data is used to improve the performance of the contrastive learning model in the pre-training stage.

[0090] In one embodiment, the hybrid loss function is defined as:

[0091] ;

[0092] in, is the loss of original data, To enhance the data loss, ;

[0093] Define the average distance between positive sample pairs in feature space for:

[0094] ;

[0095] in, , Represents the positive sample set The number of samples in and Positive sample set Positive sample pairs in ;

[0096] Define the average distance between negative sample pairs in feature space for:

[0097] ;

[0098] in, , Represents the negative sample set The number of samples in and Negative sample set Negative sample pairs in ;

[0099] Rewrite the loss function as:

[0100] ;

[0101] in, is the mixed loss function of positive samples, is the original loss function of negative samples;

[0102] The optimization goal of constructing semantic consistency between enhanced data and positive samples is:

[0103] ;

[0104] in, The semantic information of the enhanced data is represented, and the performance of the contrastive learning model is improved in the pre-training stage by combining the semantic consistency optimization objective and the loss function.

[0105] In this embodiment, the core of contrastive learning is to learn the feature representation of data by comparing positive samples (i.e. similar samples) with negative samples (dissimilar samples). and augmented data The collection of Introduce semantic information to improve learning effect. With the introduction of enhanced data, the number of positive sample pairs will increase, and the distance between these positive sample pairs will become closer, and the expected positive sample loss will decrease because It becomes smaller with the augmented data. At the same time, the negative sample loss The contribution of will also be dispersed by introducing more negative sample pairs, which can improve the effect of contrastive learning.

[0106] In summary, combining semantically enhanced data with contrastive loss function can effectively improve the model's ability to distinguish and promote good feature learning, thereby enhancing the performance of self-supervised contrastive learning. By introducing semantic information, it is expected to achieve a lower total loss, thereby achieving the goal of optimizing model performance.

[0107] In one embodiment, the loss function of the mixed sample data is defined as:

[0108] ;

[0109] Where D is mixed sample data;

[0110] The mixed loss function is defined as:

[0111] ;

[0112] in, is the loss of original data, To enhance the data loss, ;

[0113] The optimization goal of building semantic consistency between enhanced data and original data is:

[0114] ;

[0115] Using scale matching loss The difference in scale distribution between the mixed data and the original data in the feature space is used to quantify:

[0116] ;

[0117] According to the loss function and scale matching loss, the optimization goal is to minimize the total loss for:

[0118] ;

[0119] When satisfied and When minimized, .

[0120] Specifically, mixing raw data and semantic information data and combining scale matching is expected to reduce the total loss , thereby enhancing the performance of self-supervised contrastive learning. Introducing samples with consistent semantics and matching scales will greatly promote the model's ability to distinguish features, thereby verifying the effectiveness of mixed data.

[0121] The following experiments verify the superiority of the technical solution of this application:

[0122] Experiments were conducted on three benchmark datasets of different scales: CIFAR-10, CIFAR-100, and TinyImageNet (200 categories). By default, YOLO 11 and SAM are used to enhance the semantic information of images, and the generated data is the same as the original data. The semantic images are directly mixed with the original images, and then the data is put into the pre-training of the model. The model can only access the semantic data during the pre-training phase, and only access the original data during the test phase. Model training relies on the solo-learn library. In this part, SimCLR, MoCo V2, BYOL, and Barlow Twins are included. In order to fairly compare the expansion and non-expansion training, the model is trained for 500k steps in all cases to ensure that the overall data volume of the model is consistent, and consistent data enhancement methods and other self-supervised parameter settings are used to ensure that the training is not affected by hyperparameters. Finally, three expansion methods are compared: no expansion, original data expansion, and multi-scale semantic enhancement (MSSA).

[0123] The model was trained for 500k steps. Figure 2The accuracy of different enhancement methods used during training is shown, where Figure 2 (a) represents the visual trend graph of data, Figure 2 (b) shows the data visualization bar graph. It can be found that for the CIFAR-10 dataset, not all the data using the YOLO method is optimal on the SimCLR model. On the contrary, at the beginning, the smaller-scale YOLO-SAM method is better than the larger-scale YOLO method. It is believed that this may be because in the early stages of training, data with more semantic information helps the model understand the target features faster and thus reach the optimal state of training earlier. However, as the training phase continues, a large amount of additional semantic information covers the original data, making the model too dependent on semantic data during training, and not sensitive enough to the original data without semantic enhancement, so that it does not reach the optimal state.

[0124] The effectiveness of the method is demonstrated experimentally below, with the best data in the experiment shown in bold and the suboptimal data in the experiment shown in underline. The performance of different self-supervised contrastive learning models on the CIFAR-10 dataset is significantly affected by the application of semantic enhancement methods, as shown in Table 1. The multi-scale semantic enhancement (MSSA) strategy selects the appropriate enhancement scale according to the characteristics of the dataset, and its performance exceeds the performance of the baseline and AdaInf methods in all tested models. The MSSA method achieves the highest accuracy, with an improvement of 0.83% to 1.23% compared to the AdaInf method, and in some cases 1.45% higher than the baseline. This shows that the MSSA method is particularly effective in improving model performance by leveraging the strengths of each model and dataset combination. The results highlight the importance of scale-matched semantic enhancement in achieving state-of-the-art performance for self-supervised contrastive learning models.

[0125] Table 1 Linear evaluation accuracy of semantic enhancement methods on different models

[0126]

[0127] SimCLR models are trained on multiple datasets of different scales, including CIFAR-10, CIFAR-100, and TinyImageNet, to evaluate the performance of different semantic augmentation methods. As shown in Table 2, our Multi-Scale Semantic Augmentation (MSSA) strategy significantly outperforms the Baseline and AdaInf methods on all three datasets. The Baseline results show a moderate level of accuracy, with CIFAR-10 showing the highest baseline performance due to its simpler and more homogeneous nature. The Origin method, which may involve some form of data expansion, performs slightly worse on CIFAR-100 and Tiny ImageNet compared to the Baseline, suggesting that indiscriminate data expansion does not always lead to performance gains. The AdaInf method, known for its adaptive data expansion strategy, outperforms the Baseline and Origin methods, but our MSSA method achieves the highest accuracy on all datasets, especially on CIFAR-100 and Tiny ImageNet, which have greater complexity and diversity, and the improvement is particularly significant. The MSSA method is able to adapt to the scale and complexity of the dataset, enabling it to achieve state-of-the-art results, demonstrating that choosing semantic augmentation at an appropriate scale that matches the characteristics of the dataset is crucial to maximizing the performance of self-supervised contrastive learning models.

[0128] Table 2 Linear evaluation accuracy of semantic enhancement methods on different datasets

[0129]

[0130] In general, five semantic enhancement experiments of different scales were conducted on four models for three datasets of different scales. The results are shown in Table 3. The experimental results consistently show that the semantic enhancement method can significantly improve the accuracy of the model, whether on the smaller CIFAR-10 dataset or the more complex CIFAR-100 and Tiny ImageNet-200 datasets with more categories. This is reflected in all tested model architectures (BYOL, SIMCLR, BARLOW, MOCO), proving the universal benefits of semantic enhancement strategies.

[0131] The proposed multi-scale semantic information enhancement framework combines the YOLO object detection and SAM image segmentation models to capture the semantic information of images at different scales. Experimental results show that this multi-scale semantic enhancement method achieves the best performance on all tested datasets and models, especially on more complex datasets, which emphasizes the importance of multi-scale semantic information in improving the performance of self-supervised learning models. In addition, the experimental results also reveal the interaction between semantic enhancement and different model architectures. Different model architectures may require semantic information of different scales to achieve optimal performance.

[0132] Table 3 Linear evaluation accuracy of multiple models of semantic enhancement methods on multiple scale datasets

[0133]

[0134] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0135] In one embodiment, Figure 3 As shown, a multi-scale semantic information enhancement device for improving the performance of a contrastive learning model is provided, comprising: a framework construction module 302, a data enhancement module 304 and a performance improvement module 306, wherein:

[0136] The framework construction module 302 is used to cascade the YOLO target detection model and the SAM semantic segmentation model to obtain a multi-scale semantic information enhancement framework; the YOLO target detection model is used to identify the target in the input image and annotate the target with a bounding box, and the SAM semantic segmentation model is used to perform semantic segmentation on the target in the bounding box;

[0137] A data enhancement module 304 is used to extract enhanced data of the input original data using the multi-scale semantic information enhancement framework;

[0138] The performance improvement module 306 is used to mix the original data and the enhanced data to obtain mixed sample data, and use the mixed sample data to improve the performance of the comparative learning model in the pre-training stage.

[0139] In one embodiment, the performance improvement module 306 is further configured to define a hybrid loss function as:

[0140] ;

[0141] in, is the loss of original data, To enhance the data loss, ;

[0142] Define the average distance between positive sample pairs in feature space for:

[0143] ;

[0144] in, , Represents the positive sample set The number of samples in and Positive sample set Positive sample pairs in ;

[0145] Define the average distance between negative sample pairs in feature space for:

[0146] ;

[0147] in, , Represents the negative sample set The number of samples in and Negative sample set Negative sample pairs in ;

[0148] Rewrite the loss function as:

[0149] ;

[0150] in, is the mixed loss function of positive samples, is the original loss function of negative samples;

[0151] The optimization goal of constructing semantic consistency between enhanced data and positive samples is:

[0152] ;

[0153] in, The semantic information of the enhanced data is represented, and the performance of the contrastive learning model is improved in the pre-training stage by combining the semantic consistency optimization objective and the loss function.

[0154] In one embodiment, the performance improvement module 306 is further configured to define the loss function of the mixed sample data as:

[0155] ;

[0156] Where D is mixed sample data;

[0157] The mixed loss function is defined as:

[0158] ;

[0159] in, is the loss of original data, To enhance the data loss, ;

[0160] The optimization goal of building semantic consistency between enhanced data and original data is:

[0161] ;

[0162] Using scale matching loss The difference in scale distribution between the mixed data and the original data in the feature space is used to quantify:

[0163] ;

[0164] According to the loss function and scale matching loss, the optimization goal is to minimize the total loss for:

[0165] ;

[0166] When satisfied and When minimized, .

[0167] For the specific definition of the multi-scale semantic information enhancement device for improving the performance of the contrastive learning model, please refer to the definition of the multi-scale semantic information enhancement method for improving the performance of the contrastive learning model mentioned above, which will not be repeated here. Each module in the above-mentioned multi-scale semantic information enhancement device for improving the performance of the contrastive learning model can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0168] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multi-scale semantic information enhancement method for improving the performance of a contrastive learning model is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0169] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0170] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.

[0171] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.

[0172] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0173] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0174] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A multi-scale semantic information enhancement method for improving the performance of contrastive learning models, characterized in that: The method comprises: A YOLO target detection model and a SAM semantic segmentation model are cascaded to obtain a multi-scale semantic information enhancement framework; the YOLO target detection model is used to identify targets in an input image and annotate the targets with bounding boxes, and the SAM semantic segmentation model is used to semantically segment targets within the bounding boxes; Extracting enhanced data of the input original data using the multi-scale semantic information enhancement framework; After mixing the original data and the enhanced data, mixed sample data is obtained, and the mixed sample data is used to improve the performance of the contrastive learning model in the pre-training stage; The mixed sample data is used to improve the performance of the contrastive learning model in the pre-training stage, including: The mixed loss function is defined as: ； in, is the loss of original data, To enhance the data loss, ; Define the average distance between positive sample pairs in feature space for: ； in, , Represents the positive sample set The number of samples in and Positive sample set Positive sample pairs in ; Define the average distance between negative sample pairs in feature space for: ； in, , Represents the negative sample set The number of samples in and Negative sample set Negative sample pairs in ; Rewrite the loss function as: ； in, is the mixed loss function of positive samples, is the original loss function of negative samples; The optimization goal of constructing semantic consistency between enhanced data and positive samples is: ； in, The semantic information of the enhanced data is represented, and the performance of the contrastive learning model is improved in the pre-training stage by combining the semantic consistency optimization objective and the loss function.

2. The method according to claim 1, characterized in that The mixed sample data is used to improve the performance of the contrastive learning model in the pre-training stage, including: The loss function of the mixed sample data is defined as: ； Where D is mixed sample data; The mixed loss function is defined as: ； in, is the loss of original data, To enhance the data loss, ; The optimization goal of building semantic consistency between enhanced data and original data is: ； Using scale matching loss The difference in scale distribution between the mixed data and the original data in the feature space is used to quantify: ； According to the loss function and scale matching loss, the optimization goal is to minimize the total loss for: ； When satisfied and When minimized, .

3. A multi-scale semantic information enhancement device for improving the performance of contrastive learning models, characterized in that: The device comprises: A framework building module, used to cascade a YOLO target detection model and a SAM semantic segmentation model to obtain a multi-scale semantic information enhancement framework; the YOLO target detection model is used to identify targets in an input image and annotate the targets with bounding boxes, and the SAM semantic segmentation model is used to perform semantic segmentation on targets within the bounding boxes; A data enhancement module, used to extract enhanced data of the input original data using the multi-scale semantic information enhancement framework; A performance improvement module, used for mixing the original data and the enhanced data to obtain mixed sample data, and using the mixed sample data to improve the performance of the comparative learning model in the pre-training stage; The performance improvement module is also used to define a hybrid loss function as: ； in, is the loss of original data, To enhance the data loss, ; Define the average distance between positive sample pairs in feature space for: ； in, , Represents the positive sample set The number of samples in and Positive sample set Positive sample pairs in ; Define the average distance between negative sample pairs in feature space for: ； in, , Represents the negative sample set The number of samples in and Negative sample set Negative sample pairs in ; Rewrite the loss function as: ； in, is the mixed loss function of positive samples, is the original loss function of negative samples; The optimization goal of constructing semantic consistency between enhanced data and positive samples is: ； in, The semantic information of the enhanced data is represented, and the performance of the contrastive learning model is improved in the pre-training stage by combining the semantic consistency optimization objective and the loss function.

4. The device according to claim 3, characterized in that The performance improvement module is also used to define the loss function of the mixed sample data as: ； Where D is mixed sample data; The mixed loss function is defined as: ； in, is the loss of original data, To enhance the data loss, ; The optimization goal of building semantic consistency between enhanced data and original data is: ； Using scale matching loss The difference in scale distribution between the mixed data and the original data in the feature space is used to quantify: ； According to the loss function and scale matching loss, the optimization goal is to minimize the total loss for: ； When satisfied and When minimized, .

Images