Method and device for multi-source image counterfeiting region segmentation and storage medium

Abstract

The invention provides a method and a device for multi-source image counterfeiting region segmentation and a storage medium, and belongs to the technical field of digital image processing. The method comprises the steps that a to-be-detected image and one or more corresponding point prompts arranged according to a grid are acquired and input into a trained image recognition model, and each point prompt in the one or more point prompts is used for indicating a source area of the to-be-detected image; using an encoder of the image recognition model to operate the to-be-detected image to obtain a feature representation of the to-be-detected image; for each point prompt, performing segmentation prediction on the feature representation of the to-be-detected image by using a mask decoder of the image recognition model to obtain a prediction region and a prediction result corresponding to each point prompt; and integrating prediction results by using a preset clustering algorithm to form a multi-source partition map. According to the method, different source areas in the image are accurately identified and segmented, the problem of image counterfeiting positioning is solved comprehensively, and the method has important application value in multiple fields.

Description

Technical Field

[0001] The present invention relates to the technical field of digital image processing, and in particular to a method, a device and a storage medium for multi-source image forgery region segmentation. Background Art

[0002] With the rapid development of digital technology, image editing and generation tools have become increasingly popular, greatly enriching people's visual expression. However, this also provides convenience for malicious tampering and forgery of images, and the authenticity of images faces severe challenges. In news reports, false images may mislead public cognition and affect the direction of social public opinion; in the judicial field, forged image evidence may interfere with judicial justice and undermine the legal order; in commercial advertisements, false product images may mislead consumers and damage market integrity.

[0003] Traditional image forgery localization (IFL) methods mostly use binary segmentation, marking the original area as 0 and the tampered area as 1. However, this method can only distinguish between true and false, and cannot further analyze multiple source areas in the image, which limits its application in complex situations. Summary of the invention

[0004] The present invention provides a method, a device and a storage medium for multi-source image forgery region segmentation, which are used to solve the problem that the prior art cannot accurately identify and segment different source regions in an image.

[0005] In order to achieve the above-mentioned purpose, an embodiment of the present invention provides a method for segmenting forged areas of multi-source images, the method comprising the following steps: obtaining an image to be detected and corresponding one or more point prompts arranged in a grid, and inputting them into a trained image recognition model, each of the one or more point prompts is used to indicate the source area of ​​the image to be detected; using an encoder of the image recognition model to operate on the image to be detected to obtain a feature representation of the image to be detected; for each point prompt, using a mask decoder of the image recognition model to perform segmentation prediction on the feature representation of the image to be detected to obtain a predicted area and a predicted result corresponding to each point prompt; using a preset clustering algorithm to integrate the prediction results to form a multi-source partition map.

[0006] Optionally, before acquiring the image to be detected and the corresponding one or more point prompts arranged in a grid, the method further includes: pre-training the image recognition model by using regional contrast learning.

[0007] Optionally, the pre-training of the image recognition model using regional contrast learning includes: converting the input image into an image embedding through an image encoder of the image recognition model; calculating the similarity and difference between the image embeddings using an InfoNCE loss function, and applying the result calculated by the InfoNCE loss function to all source regions of the input image to obtain a contrast loss between regions. , according to the contrast loss between the regions , adjust the parameters of the image encoder to complete the pre-training of the image recognition model.

[0008] Optionally, the contrast loss between the regions It is expressed as:

[0009] in, (·) represents the loss value, represents the number of source regions in the input image, Indicates The feature set corresponding to the source region, represents the number of elements in the first source region, Indicates that from source regions to remove the current query vector The remaining feature set after is taken as the positive sample. Indicates that except for All other features outside the source region are regarded as negative samples.

[0010] Optionally, after pre-training the image recognition model using regional contrast learning, the method further includes training the image recognition model, including: fixing the pre-trained image encoder and keeping the prompt encoder of the image recognition model in an original state; converting the binary label data set of the input image into a point mask through connected component analysis; configuring a loss function based on the point mask, the loss function including an area adaptive source segmentation loss function and a confidence loss function; and optimizing the parameters of the mask decoder based on the loss function.

[0011] Optionally, the feature representation includes image embedding and prompt embedding, and the use of the encoder of the image recognition model to operate on the image to be detected to obtain the feature representation of the image to be detected includes: encoding the image to be detected by a trained image encoder to obtain image embedding; encoding the point prompt by a trained prompt encoder to obtain prompt embedding.

[0012] Optionally, for each point prompt, the mask decoder of the image recognition model is used to perform segmentation prediction on the feature representation of the image to be detected to obtain the prediction area and prediction result corresponding to each point prompt, including: using the mask decoder to perform segmentation prediction on the image embedding and the prompt embedding to generate a corresponding segmentation mask and confidence score for each point prompt; determining the corresponding prediction area according to the segmentation mask, and calculating the average image embedding corresponding to the area according to the determined prediction area as a representative feature; and obtaining the prediction result according to the confidence score and the representative feature.

[0013] Optionally, the prediction results are integrated using a preset clustering algorithm to form a multi-source partition map, including: in the clustering process, the segmentation mask with the highest confidence score is selected from each cluster as the final representative feature of the cluster, and the multi-source partition map is formed through integration.

[0014] On the other hand, the present invention further provides a control device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any one of the above methods.

[0015] On the other hand, the present invention further provides a machine-readable storage medium, on which instructions are stored, and the instructions enable a machine to execute any one of the above methods.

[0016] The method, device and storage medium for multi-source image forgery area segmentation provided by the present invention improve the model's ability to understand and capture subtle differences and enhance the accuracy of image recognition by pre-training the image recognition model. Through the point prompt mechanism, the refined segmentation of different source areas in the image can be achieved, and the image tampering information or multi-source composition can be fully presented. Through the above method, the present invention establishes a method for effectively segmenting multi-source image forgery areas. With the efficient reasoning process, the efficiency is improved, the accuracy is enhanced, and the problem of image forgery positioning is solved in an all-round way, which has important application value in many fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the prior art description. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings: Figure 1is a flow chart of a method for multi-source image forgery region segmentation provided by an embodiment of the present invention; Figure 2 is another flow chart of a method for segmenting forged regions of multi-source images provided by an embodiment of the present invention; Figure 3 It is a structural schematic diagram of a method for multi-source image forgery region segmentation provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.

[0019] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application are in compliance with the relevant provisions of national laws and regulations. In the embodiments of this application, some existing solutions in the industry such as certain software, components, and models may be mentioned, which should be considered as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of this application, but it does not mean that the applicant has or will necessarily use the solution.

[0020] As mentioned above, with the rapid development of digital technology, image editing and generation tools are becoming increasingly popular, which provides convenience for malicious tampering and forgery of images. The authenticity of images faces severe challenges. Existing technologies are unable to parse the specific sources of each area and it is difficult to present complete tampering information. Therefore, a method is needed that can accurately identify and segment different source areas in an image to achieve the identification of forged images.

[0021] To address this problem, the present invention provides a method, device and storage medium for multi-source image forged region segmentation. The point prompt mechanism is used to achieve accurate segmentation of multi-source regions in the image. Pre-training of inter-region contrast learning is used to enable the image recognition model to effectively capture subtle feature differences between regions and improve feature resolution. The loss function is used to ensure that large and small regions can be accurately segmented, thereby improving the reliability of the output results. The dual-region paired point sampling strategy is used to enable the image recognition model to learn different regions more evenly, thereby greatly improving the accuracy of image recognition.

[0022] Combine the following Figure 1-Figure 3 The present invention is described in detail.

[0023] Figure 1 FIG. 1 is a flow chart of a method for segmenting forged regions of multi-source images provided by an embodiment of the present invention. Figure 1 As shown, the method for multi-source image forgery region segmentation provided by the embodiment of the present invention can be performed by a controller, and the method includes the following steps: Step 101: obtaining an image to be detected and corresponding one or more point prompts arranged in a grid, and inputting them into a trained image recognition model, wherein each of the one or more point prompts is used to indicate a source area of ​​the image to be detected; Step 102: using the encoder of the image recognition model to perform operations on the image to be detected to obtain a feature representation of the image to be detected; Step 103: for each point prompt, using the mask decoder of the image recognition model, perform segmentation prediction on the feature representation of the image to be detected, and obtain the prediction area and prediction result corresponding to each point prompt; Step 104: using a preset clustering algorithm, integrating the prediction results to form a multi-source partition map.

[0024] The method for multi-source image forged area segmentation provided by the present invention uses a point prompt mechanism, which not only realizes the accurate segmentation of multi-source areas in the image and breaks the limitation of traditional binary segmentation, but also greatly enhances the interactivity between the user and the system and meets the diversified application needs. The image forged area can be accurately located through the trained image recognition model. At the same time, when detecting a large number of images, it is not necessary to recalculate the image embedding for each point prompt, which saves a lot of computing resources and time.

[0025] Please refer to Figure 2 and Figure 3 For example, preferably, before step 101, the method further comprises: pre-training the image recognition model by using region-to-region contrastive learning. Region-to-region contrastive learning can enhance the image encoder's understanding of source region features.

[0026] Further preferably, the pre-training of the image recognition model by using regional contrast learning may include: converting the input image into an image embedding by using an image encoder of the image recognition model; calculating the similarity and difference between the image embeddings by using an InfoNCE loss function, and applying the result calculated by the InfoNCE loss function to all source regions of the input image to obtain a contrast loss between regions. , according to the contrast loss between the regions , adjust the parameters of the image encoder to complete the pre-training of the image recognition model.

[0027] In a preferred embodiment of the present invention, the InfoNCE loss function can measure the degree of distinction between positive samples and negative samples, and by mining the relationship between internal regions of the data, it can effectively improve the performance of the model in various tasks. In the pre-training stage, the image embedding E=E(I) generated by the image encoder is used for subsequent inter-region contrast learning. For example, the InfoNCE loss function is used to calculate the similarities and differences between these embeddings. For feature points in the same source region, their distance in the feature space should be as close as possible; for feature points from different source regions, they should be as far as possible. Subsequently, the InfoNCE loss is applied to all source regions of the entire image to obtain the inter-region contrast loss , inter-region contrast loss It aims to minimize the distance between feature points in the same source region and maximize the distance between feature points in different source regions.

[0028] Further preferably, the loss function of the encoder can be expressed by the following formula:

[0029] in, Represents the query vector, which is the feature extracted from a specific location. Represents a positive sample vector, another feature point that belongs to the same source region as the query vector. Represents a set of negative samples, which are feature points from other source regions. τ represents a temperature parameter, which controls the scaling of the similarity score. A smaller τ value will result in a higher score for the correct match and a lower score for the wrong match, thereby increasing the probability of selecting the correct match. · Represents the dot product operation of two vectors, measuring the similarity between them. (·) represents the exponential function, which is used to convert the dot product result into a positive value and strengthen the difference between high and low scores.

[0030] Further preferably, the contrast loss between the regions It is expressed as:

[0031] in, (·) represents the loss value, represents the number of source regions in the input image, Indicates The feature set corresponding to the source region, represents the number of elements in the first source region, Indicates that from source regions to remove the current query vector The remaining feature set after is taken as the positive sample. Indicates that except for All other features outside the source region are regarded as negative samples.

[0032] Preferably, after pre-training, the method further comprises: training the image recognition model to optimize the decoder.

[0033] Further preferably, the training of the image recognition model includes: fixing the pre-trained image encoder and keeping the prompt encoder of the image recognition model in an original state; converting the binary label data set of the input image into a point mask through connected component analysis; configuring a loss function according to the point mask, the loss function including an area adaptive source segmentation loss function and a confidence loss function; and optimizing the parameters of the mask decoder according to the loss function.

[0034] In a preferred embodiment of the present invention, the image encoder maintains the pre-training state unchanged, while the prompt encoder uses the state of the original SAM. For a binary label data set, it is converted into a point mask suitable for multi-source prediction by analyzing the connected components. The process of creating a point mask is to analyze the connected components from the binary label data set and convert them into a point mask suitable for multi-source prediction. This process is automatically completed by the algorithm. For a given point, if it is located in a connected area, then this area will be marked as 1, and the adjacent but not belonging to the area will be marked as 0, and the non-adjacent part will be given an ignore label of -1. Subsequently, the area adaptive source segmentation loss function will give a smaller area a higher weight based on the area of ​​different regions represented by the point mask. For example, in an image, there are both large areas of original background areas and small areas of tampered object areas. The area adaptive source segmentation loss function guides the mask decoder to accurately segment source areas of different sizes through reasonable weight distribution (determined comprehensively according to factors such as regional area ratio, image feature distribution, and task requirements), avoiding segmentation deviations caused by regional area differences, and ensuring that both large and small areas can be accurately predicted by the image recognition model. Traditional image forgery localization technology often only outputs segmentation results, making it difficult for users to intuitively understand how credible the result is.

[0035] The present invention measures the gap between the predicted confidence and the actual situation through confidence loss, and quantifies it using metrics such as Euclidean distance and cosine similarity. During the training process, the mask decoder aims to minimize the confidence score and generate a segmentation mask with high credibility. In this way, when outputting the segmentation results, each segmentation mask has a confidence score, and the user can judge the reliability of the result based on this score, avoiding misjudgment caused by unreliable segmentation results and improving the value of the entire technology in practical applications.

[0036] In addition, the present invention adopts dual-region paired point sampling. This sampling strategy ensures that the image recognition model pays equal attention to different source regions during the training process, and evenly learns the differences in their respective features, textures, colors, etc., as well as the relationships between them, so that the image recognition model has a better understanding and discrimination ability for each type of region, thereby improving the overall segmentation performance.

[0037] Preferably, the feature representation includes image embedding and prompt embedding, and the encoder of the image recognition model is used to operate the image to be detected to obtain the feature representation of the image to be detected, which includes: encoding the image to be detected by a trained image encoder to obtain an image embedding; encoding the point prompt by a trained prompt encoder to obtain a prompt embedding. Image embedding achieves efficient processing, cross-modal fusion and feature sharing by extracting and converting image features into low-dimensional vectors, and has the advantages of high efficiency, semantic preservation and transferability; prompt embedding provides additional guidance information for the model to help it focus on key points, enhance semantic understanding and customize tasks, and has the advantages of flexibility, strong interpretability and improved performance.

[0038] Preferably, in step 103, for each point prompt, the mask decoder of the image recognition model is used to perform segmentation prediction on the feature representation of the image to be detected to obtain the prediction area and prediction result corresponding to each point prompt, including: using the mask decoder to perform segmentation prediction on the image embedding and the prompt embedding to generate a corresponding segmentation mask and confidence score for each point prompt; determining the corresponding prediction area according to the segmentation mask, and calculating the average image embedding corresponding to the area according to the determined prediction area as a representative feature; and obtaining the prediction result according to the confidence score and the representative feature.

[0039] In a preferred embodiment of the present invention, the calculation of image embedding is independent of the point prompt, and only one encoder operation is required for each image to obtain a complete feature representation. This one-time calculation method reduces repeated operations and greatly improves efficiency compared to traditional methods. The mask decoder generates a segmentation mask and its confidence score for each point prompt based on image embedding and prompt embedding, combines image features with prompt information to predict possible source areas and provide reliability measures, paving the way for subsequent screening of accurate results. After determining the area based on the segmentation mask, the average image embedding is calculated as a representative feature, which provides a basis for cluster analysis. By extracting representative regional features, it is convenient for subsequent integrated analysis of the relationship between different regions.

[0040] Preferably, in step 104, the prediction results are integrated using a preset clustering algorithm to form a multi-source partition map, including: in the clustering process, the segmentation mask with the highest confidence score is selected from each cluster as the final representative feature of the cluster, and the multi-source partition map is formed through integration.

[0041] In a preferred embodiment of the present invention, a clustering algorithm (such as k-means) is used to integrate preliminary results to form a final multi-source partition map. The number of clusters can be set according to the number of sources that are known or automatically inferred, and the mask with the highest confidence level is selected from each cluster as the final representative feature to ensure that the final output is both accurate and reliable, thereby realizing the process from scattered preliminary predictions to the presentation of an overall accurate multi-source region segmentation map.

[0042] As an example, the process of the method provided in the embodiment of the present invention is as follows: (1) Input preparation. The user provides the image to be detected and a set of point prompts arranged in a grid. These point prompts are used to guide the model to identify the various source areas in the image. The user is required to provide point prompts during the inference phase, but it does not rely entirely on the user's expert level. The point prompts are designed as a set of point prompts arranged in a grid, which ensures that even if a single point prompt is not accurate or reasonable, the image can be fully covered as a whole, thereby ensuring the integrity and accuracy of the segmentation result. The grid-shaped point prompts ensure full coverage of the entire image, thereby improving the integrity of the segmentation result.

[0043] (2) Feature extraction. The feature representation of the image is obtained through the encoder. Since the calculation of image embedding is independent of the point prompt, only one encoder operation is required for each image to obtain a complete feature representation. This one-time calculation method reduces repeated operations, improves efficiency and reduces resource consumption.

[0044] (3) Segmentation prediction. For each point cue, the mask decoder generates a preliminary segmentation mask and its confidence score based on the image embedding and cue embedding. Each segmentation mask represents a possible source region, and the confidence score reflects the reliability of the prediction.

[0045] (4) Representative feature calculation. Based on the area determined by the preliminary mask, the average image embedding corresponding to the area is calculated as the representative feature. This step provides the basis for subsequent clustering analysis.

[0046] (5) Aggregation and clustering. Use a clustering algorithm (such as k-means) to integrate all preliminary results to form the final multi-source partition map. When the number of sources is known, the number of clusters is set in advance; otherwise, let the algorithm automatically infer the appropriate number. During the clustering process, the mask with the highest confidence is selected from each cluster as the final representative feature of the cluster to ensure that the final output is both accurate and reliable.

[0047] (6) Final output. After the above steps, one or more segmentation maps representing different source regions will be generated, which intuitively shows the true source of each part of the image.

[0048] The method for multi-source image forgery region segmentation provided by the present invention guides the model to identify each source region in the image by introducing a point prompt mechanism. This method allows the user or system to obtain accurate segmentation results by simply providing a few key points, greatly enhancing the interactivity and flexibility of the system. Through inter-region contrast learning pre-training, the model can better separate regions from different sources in the feature space, improving the model's ability to understand and capture subtle differences. Through the area-adaptive source segmentation loss function, smaller regions are given higher weights to ensure that all regions can receive full attention and accurate predictions. Through confidence loss, it helps to improve the reliability of the final output and enables the model to self-evaluate the accuracy of its predictions. In addition, this method effectively solves the label agnostic problem in the forgery positioning task (i.e., it is impossible to clearly distinguish between tampered and untampered regions), and achieves a stable learning effect by considering the interchangeability between tampered regions and original regions in the real world.

[0049] An embodiment of the present invention further provides a control device, which may include: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above method.

[0050] An embodiment of the present invention further provides a machine-readable storage medium, on which instructions are stored, and the instructions enable a machine to execute the above method.

[0051] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0052] In addition, the terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0053] It should be understood that in the embodiment of the present invention, "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that determining B according to A does not mean determining B only according to A, and B can also be determined according to A and / or other information.

[0054] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0055] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0056] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0057] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0058] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0059] Through the description of the above embodiments, it can be clearly understood by those skilled in the art that the present invention can be implemented in hardware, firmware, or a combination thereof. When implemented using software, the above functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any medium that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a computer. Taking this as an example but not limited to: a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer. In addition. Any connection can be appropriately a computer-readable medium. For example, if the software is transmitted from a website, server or other remote source using a coaxial cable, optical fiber cable, twisted pair, digital subscriber line (DSL) or wireless technologies such as infrared, radio and microwave, then the coaxial cable, optical fiber cable, twisted pair, DSL or wireless technologies such as infrared, wireless and microwave are included in the fixing of the medium. As used in the present invention, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and blue-ray disc, wherein disk usually reproduces data magnetically, while disc uses laser to reproduce data optically. The above combination should also be included in the scope of protection of computer-readable media.

[0060] In short, the above is only a preferred embodiment of the technical solution of the present invention, and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for multi-source image forgery region segmentation, characterized in that: The method comprises the following steps: Obtaining an image to be detected and corresponding one or more point prompts arranged in a grid, and inputting them into a trained image recognition model, wherein each of the one or more point prompts is used to indicate a source area of ​​the image to be detected; Using the encoder of the image recognition model, operating the image to be detected to obtain a feature representation of the image to be detected; For each of the point prompts, using the mask decoder of the image recognition model, segmentation prediction is performed on the feature representation of the image to be detected to obtain a prediction area and a prediction result corresponding to each of the point prompts; The prediction results are integrated using a preset clustering algorithm to form a multi-source partition map.

2. The method according to claim 1, characterized in that Before acquiring the image to be detected and the corresponding one or more point prompts arranged in a grid, the method further includes: pre-training the image recognition model by using regional contrast learning.

3. The method according to claim 2, characterized in that The pre-training of the image recognition model by using regional contrast learning includes: Converting an input image into an image embedding via an image encoder of the image recognition model; The InfoNCE loss function is used to calculate the similarity and difference between the image embeddings, and the result calculated by the InfoNCE loss function is applied to all source regions of the input image to obtain the contrast loss between regions. , According to the contrast loss between the regions , adjust the parameters of the image encoder to complete the pre-training of the image recognition model.

4. The method according to claim 3, characterized in that The contrast loss between the regions It is expressed as: in, (·) represents the loss value, represents the number of source regions in the input image, Indicates The feature set corresponding to the source region, represents the number of elements in the first source region, Indicates that from source regions to remove the current query vector The remaining feature set after is taken as the positive sample. Indicates that except for All other features outside the source region are regarded as negative samples.

5. The method according to claim 3, characterized in that: After pre-training the image recognition model by using regional contrast learning, the method further includes training the image recognition model, including: Fixing the pre-trained image encoder and keeping the prompt encoder of the image recognition model in an original state; converting the binary labeled dataset of the input image into a point mask by connected component analysis; According to the point mask, configuring a loss function, the loss function comprising an area adaptive source segmentation loss function and a confidence loss function; Optimizing parameters of the mask decoder according to the loss function.

6. The method according to claim 1, characterized in that The feature representation includes image embedding and prompt embedding, and the encoder of the image recognition model is used to operate the image to be detected to obtain the feature representation of the image to be detected, which includes: Encoding the image to be detected by using a trained image encoder to obtain image embedding; The point prompt is encoded by the trained prompt encoder to obtain the prompt embedding.

7. The method according to claim 6, characterized in that For each of the point prompts, using the mask decoder of the image recognition model, segmenting and predicting the feature representation of the image to be detected to obtain the prediction area and prediction result corresponding to each of the point prompts, including: Using the mask decoder, performing segmentation prediction on the image embedding and the cue embedding to generate a corresponding segmentation mask and a confidence score for each point cue; Determine a corresponding prediction region according to the segmentation mask, and calculate an average image embedding corresponding to the region according to the determined prediction region as a representative feature; The prediction result is obtained according to the confidence score and the representative features.

8. The method according to claim 7, characterized in that The method of utilizing a preset clustering algorithm to integrate the prediction results to form a multi-source partition map includes: In the clustering process, the segmentation mask with the highest confidence score is selected from each cluster as the final representative feature of the cluster, and is integrated to form the multi-source partition map.

9. A control device, characterized in that: The control device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the method according to any one of claims 1-8.

10. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores instructions, which enable the machine to execute the method according to any one of claims 1-8.

Images