Large model-based hidden target detection method and system

Abstract

The invention provides a hidden target detection method and system based on a large model. The method comprises the steps that a to-be-detected target RGB image is acquired; inputting the target RGB image into a pre-trained large model to obtain a hidden target; wherein the pre-training large model comprises an encoder and a decoder which are embedded into a LoRA layer. According to the method, the pre-trained large model is adopted as the baseline model, and the LoRA (Low-Rank Adaptive) layer is embedded in the encoder, so that the hidden target positioning and segmentation precision is improved, the generalization ability of large model hidden target detection is improved, and efficient and reliable hidden target image segmentation and detection are realized.

Description

Technical Field

[0001] The present invention relates to the field of image target detection, and in particular to a method and system for detecting hidden targets based on a large model. Background Art

[0002] Hidden target detection is to separate the hidden target from the background in the image through deep learning image segmentation technology to achieve the purpose of target detection. Image segmentation technology is a key technology in the field of computer vision, which is divided into semantic segmentation, instance segmentation and panoptic segmentation. Among them, semantic segmentation classifies all pixels in the image and distinguishes the foreground and background, but different instances of the same category in the foreground are indiscriminately assigned the same label. Since the hidden target and the environmental background have extremely high similarity, the process of distinguishing the foreground and background is extremely challenging. The existing image segmentation schemes for hidden target detection all use traditional deep learning frameworks to build lightweight neural networks, represented by SINet intelligent identification network and C2F-Net context-aware cross-level fusion network. By building a lightweight model to realize an end-to-end deep learning framework, the pixels in the image are classified, and the target pixel points and the surrounding environment are clearly distinguished by boundaries, so as to realize the location and segmentation of hidden targets, thereby achieving the hidden target detection task. However, due to the small size of the model, its segmentation accuracy is low.

[0003] However, the existing open source pre-trained large models are not capable of capturing high-dimensional features of hidden objects, resulting in low generalization ability of hidden target detection, unable to meet the requirements of hidden target segmentation, and unable to achieve zero-shot instance segmentation, which limits its practical application areas. Summary of the invention

[0004] In order to solve the problems of small model scale, low segmentation accuracy and poor generalization ability in the prior art, the present invention proposes a hidden target detection method based on a large model, comprising:

[0005] Get the RGB image of the target to be detected;

[0006] Input the target RGB image into the pre-trained large model to obtain a hidden target;

[0007] The pre-trained large model includes an encoder and a decoder embedded in the LoRA layer.

[0008] Preferably, the step of inputting the target RGB image into a pre-trained large model to obtain a hidden target comprises:

[0009] Performing preprocessing based on the input target RGB image;

[0010] Based on the preprocessed RGB image, high-dimensional features are captured by the encoder embedded in the LoRA layer;

[0011] Performing dimensionality reduction decoding on the high-dimensional features through a decoder to obtain a decoding result;

[0012] Based on the decoding results, predictions are made according to different task modes to generate segmentation results of hidden targets.

[0013] Preferably, the preprocessing based on the input target RGB image comprises:

[0014] For RGB images of different sizes input as NumPy array format, the sizes are aligned according to the longest side of the RGB images to obtain RGB images of fixed size;

[0015] The fixed-size RGB image is converted, the converted image is cut into 16 sub-regions, and each sub-region is vectorized.

[0016] Preferably, the converting according to the fixed-size RGB image comprises:

[0017] According to the fixed-size RGB image, a grayscale image of a fixed size is obtained by converting the RGB color space into a grayscale color space;

[0018] According to the grayscale image of the fixed size, a grayscale image in the Torch tensor format is obtained by converting the NumPy array format to the Torch tensor format.

[0019] Preferably, the vectorization processing of each sub-region includes:

[0020] Expand the tensor data of each sub-region into a one-dimensional tensor and perform position embedding operation;

[0021] The expansion dimension of each sub-region is dim;

[0022] The position embedding operation adopts a combination function of a sine function and a cosine function.

[0023] Preferably, the capturing and obtaining high-dimensional features according to the encoder embedded in the LoRA layer includes:

[0024] Forward deduction, based on the preprocessed RGB image as the input data of the first coupling structure, the input data is respectively input to the attention block and the LoRA layer in the first coupling structure, and the integrated output result is fused after tensor operation, and the output result is used as the input data of the next coupling structure until the result is output from the last coupling mechanism;

[0025] Reverse deduction: Based on the LoRA layer parameters in the last coupling structure, adjust the LoRA layer parameters in the first coupling structure in turn, and freeze the parameters of the attention blocks in each coupling structure at the same time to obtain the final high-dimensional features of the image;

[0026] The encoder embedded in the LoRA layer includes: N serially coupled structures;

[0027] Each coupled structure consists of an attention block and a LoRA layer.

[0028] Preferably, the input data is respectively input into the attention block and the LoRA layer in the first coupling structure, and the fused integrated output result after tensor operation includes:

[0029] Based on the input data input into the attention block in the first coupling structure, the data obtained through tensor operation is fused and integrated with the input data input into the LoRA layer in the first coupling structure, and the output result is output.

[0030] Preferably, the input data is input to the LoRA layer in the first coupling structure, and the data obtained through tensor operation includes:

[0031] The first linear layer of the LoRA layer uses the expanded dimension dim of each sub-region as the input channel, and the input tensor data is mapped from the dim dimension space to the 256-dimensional space through the weight of the first linear layer and output;

[0032] The second linear layer through the LoRA layer is based on dimension 256 as the input channel. The tensor data output by the first linear layer is mapped from the 256-dimensional space back to the original dim-dimensional space through the weight of the second linear layer.

[0033] Among them, the first linear layer weight or the second linear layer weight of the LoRA layer in each coupling structure is different.

[0034] Preferably, the task mode includes:

[0035] Point mode, foreground or background determination based on pixel coordinates (x, y);

[0036] Box mode, based on border (x min , x max ,y min ,y max )Separate the hidden target and the environment in the area from front to back;

[0037] Global mode, performs front / back segmentation based on hidden objects and the environment in the entire image.

[0038] Based on the same inventive concept, the present invention also provides a concealed target detection system based on a large model, comprising:

[0039] An acquisition module is used to obtain an RGB image of the target to be detected;

[0040] A detection module, inputting the RGB image into a pre-trained large model to obtain a hidden target;

[0041] The pre-trained large model includes an encoder and a decoder embedded in the LoRA layer.

[0042] Preferably, the pre-trained large model includes:

[0043] An image processing module, used for preprocessing the input target RGB image;

[0044] The encoder module is used to obtain high-dimensional features based on the preprocessed RGB image according to the encoder embedded in the LoRA layer;

[0045] A decoder module, used for performing dimension reduction decoding on the high-dimensional features through a decoder to obtain a decoding result;

[0046] The predictor module is used to make predictions based on the decoding results according to different task modes and generate segmentation results of hidden targets.

[0047] Preferably, the image processing module includes:

[0048] An alignment unit, used for aligning the RGB images of different sizes input as NumPy array format according to the longest side of the RGB images to obtain an RGB image of fixed size;

[0049] A conversion unit, used for performing conversion according to the fixed-size RGB image;

[0050] A cropping unit, used to crop the converted image into 16 sub-regions;

[0051] A processing unit is used to perform vectorized processing on each sub-region.

[0052] Preferably, the conversion unit comprises:

[0053] A color conversion subunit, configured to obtain a grayscale image of fixed size by converting the RGB color space into a grayscale color space according to the RGB image of fixed size;

[0054] The format conversion subunit is used to obtain a grayscale image in Torch tensor format by converting the grayscale image of the fixed size from NumPy array format to Torch tensor format.

[0055] Preferably, the processing unit comprises:

[0056] An expansion subunit, used to expand the tensor data of each sub-region into a one-dimensional tensor, wherein the expansion dimension of each sub-region is dim;

[0057] The embedding subunit is used to perform a position embedding operation on the unfolded one-dimensional tensor, wherein the position embedding operation adopts a combination function of a sine function and a cosine function.

[0058] Preferably, the encoder module comprises:

[0059] A forward derivation unit is used to use the preprocessed RGB image as input data of the first coupling structure, the input data is respectively input to the attention block and the LoRA layer in the first coupling structure, and the output result is integrated after tensor operation, and the output result is used as the input data of the next coupling structure until the result is output from the last coupling mechanism;

[0060] A reverse derivation unit is used for reverse derivation. Based on the LoRA layer parameters in the last coupling structure, the LoRA layer parameters in the first coupling structure are adjusted in turn, and the parameters of the attention blocks in each coupling structure are frozen at the same time to obtain the final high-dimensional features of the image.

[0061] The encoder embedded in the LoRA layer includes: N serially coupled structures;

[0062] Each coupled structure consists of an attention block and a LoRA layer.

[0063] Preferably, the forward derivation unit specifically includes:

[0064] Based on the input data input into the attention block in the first coupling structure, the data obtained through tensor operation is fused and integrated with the input data input into the LoRA layer in the first coupling structure, and the output result is output.

[0065] Preferably, the input data is input to the LoRA layer in the first coupling structure, and the data obtained through tensor operation includes:

[0066] The first linear layer of the LoRA layer uses the expanded dimension dim of each sub-region as the input channel, and the input tensor data is mapped from the dim dimension space to the 256-dimensional space through the weight of the first linear layer and output;

[0067] The second linear layer through the LoRA layer is based on dimension 256 as the input channel. The tensor data output by the first linear layer is mapped from the 256-dimensional space back to the original dim-dimensional space through the weight of the second linear layer.

[0068] Among them, the first linear layer weight or the second linear layer weight of the LoRA layer in each coupling structure is different.

[0069] Preferably, the task mode includes:

[0070] Point mode, foreground or background determination based on pixel coordinates (x, y);

[0071] Box mode, based on border (x min , x max ,y min ,y max )Separate the hidden target and the environment in the area from front to back;

[0072] Global mode, performs front / back segmentation based on hidden objects and the environment in the entire image.

[0073] Compared with the prior art, the present invention has the following beneficial effects:

[0074] The present invention provides a method and system for detecting hidden targets based on a large model, comprising: obtaining an RGB image of a target to be detected; inputting the target RGB image into a pre-trained large model to obtain a hidden target; wherein the pre-trained large model comprises an encoder and a decoder embedded with a LoRA layer. The present invention adopts a pre-trained large model as a baseline model and embeds a LoRA (Low-Rank Adaption) layer in the encoder, which not only improves the hidden target positioning and segmentation accuracy, but also improves the generalization ability of the large model hidden target detection, thereby achieving efficient and reliable hidden target image segmentation and detection.

[0075] The present invention adopts an encoder embedded in the LoRA layer and uses hidden target image data to complete large model fine-tuning, realize zero-shot instance segmentation, and provide technical support for downstream tasks (such as complex environment detection, medical image applications, etc.). BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 A flowchart of a concealed target detection method based on a large model provided by the present invention;

[0077] Figure 2 A schematic diagram of image preprocessing of a large model-based hidden target detection method provided by the present invention;

[0078] Figure 3 A schematic diagram of the encoder structure of a large model-based hidden target detection method provided by the present invention;

[0079] Figure 4A schematic diagram of prediction results of a hidden target detection method based on a large model provided by the present invention. DETAILED DESCRIPTION

[0080] Different from the traditional lightweight neural network structure, the present invention adopts a pre-trained large model as the baseline model for secondary development, and proposes a hidden target detection method and system based on the large model, which gives full play to the heavyweight encoder structure of the large model, realizes the advantages of high-dimensional feature capture of hidden objects, and improves the accuracy of target segmentation. At the same time, the LoRA layer is embedded in the large model encoder, and the model is fine-tuned using the hidden target image, which improves the generalization ability of the model's hidden target detection and zero-shot zero-sample instance segmentation, and finally realizes efficient and reliable hidden target image segmentation and detection.

[0081] The specific embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings.

[0082] Embodiment 1:

[0083] The present invention proposes a hidden target detection method based on a large model, comprising:

[0084] Get the RGB image of the target to be detected;

[0085] Input the target RGB image into the pre-trained large model to obtain a hidden target;

[0086] The pre-trained large model includes an encoder and a decoder embedded in the LoRA layer.

[0087] like Figure 1 As shown, the step of inputting the target RGB image into the pre-trained large model to obtain the hidden target includes:

[0088] Performing preprocessing based on the input target RGB image;

[0089] Based on the preprocessed RGB image, high-dimensional features are captured by the encoder embedded in the LoRA layer;

[0090] Performing dimensionality reduction decoding on the high-dimensional features through a decoder to obtain a decoding result;

[0091] Based on the decoding results, predictions are made according to different task modes to generate segmentation results of hidden targets.

[0092] The present invention takes the RGB image to be detected as input and outputs a panoramic segmentation result of the same resolution, which predicts the semantic category label and instance number for each pixel.

[0093] like Figure 2 As shown, the preprocessing based on the input target RGB image includes:

[0094] For RGB images of different sizes input as NumPy array format, the sizes are aligned according to the longest side of the RGB images to obtain RGB images of fixed size;

[0095] The fixed-size RGB image is converted, the converted image is cut into 16 sub-regions, and each sub-region is vectorized.

[0096] Specifically, RGB images of different sizes are read and input as NumPy arrays. The image is resized by the image preprocessing module, and the longest side is adjusted to 1024 pixels, and the overall image is aligned to 1024×1024.

[0097] The converting according to the fixed-size RGB image comprises:

[0098] According to the fixed-size RGB image, a grayscale image of a fixed size is obtained by converting the RGB color space into a grayscale color space;

[0099] According to the grayscale image of the fixed size, a grayscale image in the Torch tensor format is obtained by converting the NumPy array format to the Torch tensor format.

[0100] The fixed-size RGB image is converted to match the needs of subsequent model training.

[0101] The vectorization processing of each sub-region comprises:

[0102] Expand the tensor data of each sub-region into a one-dimensional tensor and perform position embedding operation;

[0103] The expansion dimension of each sub-region is dim;

[0104] The position embedding operation adopts a combination function of a sine function and a cosine function.

[0105] Specifically, vectorization is the process of converting image data into low-dimensional continuous vectors. First, the tensor data of each pacth sub-region is expanded into a one-dimensional tensor, and the expansion dimension of each sub-region is dim, and the position embedding operation is performed. Position embedding is implemented using a combination of sine and cosine functions, which enables the large model to capture the relative position relationship of each token in the context. In the field of computer vision images, a token refers to a pixel point and the position and category information contained in the point.

[0106] The position embedding is calculated by the following formula:

[0107] PE=(pos,2i)=sin(pos / 10002i / dim)

[0108] PE=(pos,2i+1)=cos(pos / 10002i / dim)

[0109] Among them, dim is the expansion dimension of each pacth, pos is the token position, and i is the index.

[0110] like Figure 3 As shown, the high-dimensional features obtained by capturing the encoder embedded in the LoRA layer include:

[0111] Forward deduction, based on the preprocessed RGB image as the input data of the first coupling structure, the input data is respectively input to the attention block and the LoRA layer in the first coupling structure, and the integrated output results are fused after tensor operation. The output results are used as the input data of the next coupling structure until the result is output from the last coupling mechanism;

[0112] Reverse deduction: Based on the LoRA layer parameters in the last coupling structure, adjust the LoRA layer parameters in the first coupling structure in turn, and freeze the parameters of the attention blocks in each coupling structure at the same time to obtain the final high-dimensional features of the image;

[0113] The encoder embedded in the LoRA layer includes: N serially coupled structures;

[0114] Each coupled structure consists of an attention block and a LoRA layer.

[0115] The input data is respectively input into the attention block and the LoRA layer in the first coupling structure, and the fused integrated output results after tensor operation include:

[0116] Based on the input data input into the attention block in the first coupling structure, the data obtained through tensor operation is fused and integrated with the input data input into the LoRA layer in the first coupling structure, and the output result is output.

[0117] Specifically, the depth coefficient of the encoder determines the number of coupling structures of the encoder. If the depth coefficient of the encoder is 12, it means that the encoder has a total of 12 coupling structures in series. The attention block and LoRA layer in each coupling structure accept the same data input, derive the data in parallel, and fuse the output results to form the output result of the coupling structure. During the forward derivation of data, the feature tensor output by the previous coupling structure is used as the input data of the next coupling structure. The dimension of the feature tensor is dim, and the feature dimension size of the output result remains unchanged.

[0118] The input data is input to the LoRA layer in the first coupling structure, and the data obtained after tensor operation includes:

[0119] The first linear layer of the LoRA layer uses the expanded dimension dim of each sub-region as the input channel, and the input tensor data is mapped from the dim dimension space to the 256-dimensional space through the weight of the first linear layer and output;

[0120] The second linear layer through the LoRA layer is based on dimension 256 as the input channel. The tensor data output by the first linear layer is mapped from the 256-dimensional space back to the original dim-dimensional space through the weight of the second linear layer.

[0121] Among them, the first linear layer weight or the second linear layer weight of the LoRA layer in each coupling structure is different.

[0122] The output data Output of the LoRA layer is calculated by the following formula:

[0123] Output=Input×W 1 ×W 2

[0124] Among them, Input is the input data, W 1 is the weight of the first linear layer, W 2 is the weight of the second linear layer.

[0125] The weight of the first linear layer in each coupling structure is different, and the weight of the second linear layer in each coupling structure is also different.

[0126] The encoder also includes a prompt encoder, which encodes prompts in different forms, such as point mode, box mode, and global mode, as embedding vectors for the large model to understand the specific requirements of different mode segmentation tasks.

[0127] The decoder includes a picture decoder, a prompt decoder, and a mask decoder, which are used to decode the data obtained by the encoder. The decoder does not perform LoRA coupling processing. The mask decoder combines the information of the image decoder and the prompt decoder to predict the result, and the prediction result is tensor data.

[0128] The task modes include:

[0129] Point mode, foreground or background determination based on pixel coordinates (x, y);

[0130] Box mode, based on border (x min , x max ,y min ,ymax )Separate the hidden target and the environment in the area from front to back;

[0131] Global mode, performs front / back segmentation based on hidden objects and the environment in the entire image.

[0132] like Figure 4 As shown in , the point mode is used to determine the category of a pixel at a specific coordinate (x, y), predict the pixel as foreground or background, and output the prediction result. The box (BBox) mode performs tasks on the characteristic area in the image, performs front / back segmentation processing on the hidden target and the environment in the area, and outputs the predicted tensor data as a mask image as the result. The global mode uses the entire image as the segmentation area, and performs front / back segmentation processing on the hidden target and the environment in the entire image, and finally outputs the predicted tensor data as a mask image as the result.

[0133] Embodiment 2:

[0134] The present invention based on the same inventive concept also provides a concealed target detection system based on a large model, comprising:

[0135] An acquisition module is used to obtain an RGB image of the target to be detected;

[0136] A detection module, inputting the RGB image into a pre-trained large model to obtain a hidden target;

[0137] The pre-trained large model includes an encoder and a decoder embedded in the LoRA layer.

[0138] The pre-trained large model includes:

[0139] An image processing module, used for preprocessing the input target RGB image;

[0140] The encoder module is used to obtain high-dimensional features based on the preprocessed RGB image according to the encoder embedded in the LoRA layer;

[0141] A decoder module, used for performing dimension reduction decoding on the high-dimensional features through a decoder to obtain a decoding result;

[0142] The predictor module is used to make predictions based on the decoding results according to different task modes and generate segmentation results of hidden targets.

[0143] The image processing module comprises:

[0144] An alignment unit, used for aligning the RGB images of different sizes input as NumPy array format according to the longest side of the RGB images to obtain an RGB image of fixed size;

[0145] A conversion unit, used for performing conversion according to the fixed-size RGB image;

[0146] A cropping unit, used to crop the converted image into 16 sub-regions;

[0147] A processing unit is used to perform vectorized processing on each sub-region.

[0148] The conversion unit comprises:

[0149] A color conversion subunit, configured to obtain a grayscale image of fixed size by converting the RGB color space into a grayscale color space according to the RGB image of fixed size;

[0150] The format conversion subunit is used to obtain a grayscale image in Torch tensor format by converting the grayscale image of the fixed size from NumPy array format to Torch tensor format.

[0151] The processing unit comprises:

[0152] An expansion subunit, used to expand the tensor data of each sub-region into a one-dimensional tensor, wherein the expansion dimension of each sub-region is dim;

[0153] The embedding subunit is used to perform a position embedding operation on the unfolded one-dimensional tensor, wherein the position embedding operation adopts a combination function of a sine function and a cosine function.

[0154] The encoder module comprises:

[0155] A forward derivation unit is used to use the preprocessed RGB image as input data of the first coupling structure, the input data is respectively input to the attention block and the LoRA layer in the first coupling structure, and the output result is integrated after tensor operation, and the output result is used as the input data of the next coupling structure until the result is output from the last coupling mechanism;

[0156] A reverse derivation unit is used for reverse derivation. Based on the LoRA layer parameters in the last coupling structure, the LoRA layer parameters in the first coupling structure are adjusted in turn, and the parameters of the attention blocks in each coupling structure are frozen at the same time to obtain the final high-dimensional features of the image.

[0157] The encoder embedded in the LoRA layer includes: N serially coupled structures;

[0158] Each coupled structure consists of an attention block and a LoRA layer.

[0159] The forward derivation unit specifically includes:

[0160] Based on the input data input into the attention block in the first coupling structure, the data obtained through tensor operation is fused and integrated with the input data input into the LoRA layer in the first coupling structure, and the output result is output.

[0161] The input data is input to the LoRA layer in the first coupling structure, and the data obtained after tensor operation includes:

[0162] The first linear layer of the LoRA layer uses the expanded dimension dim of each sub-region as the input channel, and the input tensor data is mapped from the dim dimension space to the 256-dimensional space through the weight of the first linear layer and output;

[0163] The second linear layer through the LoRA layer is based on dimension 256 as the input channel. The tensor data output by the first linear layer is mapped from the 256-dimensional space back to the original dim-dimensional space through the weight of the second linear layer.

[0164] Among them, the first linear layer weight or the second linear layer weight of the LoRA layer in each coupling structure is different.

[0165] The output data Output of the LoRA layer is calculated by the following formula:

[0166] Output=Input×W 1 ×W 2

[0167] Among them, Input is the input data, W 1 is the weight of the first linear layer, W 2 is the weight of the second linear layer.

[0168] The weight of the first linear layer in each coupling structure is different, and the weight of the second linear layer in each coupling structure is also different.

[0169] The encoder module also includes a prompt encoder, which encodes prompts in different forms, such as point mode, box mode, and global mode, as embedding vectors for the large model to understand the specific requirements of different mode segmentation tasks.

[0170] The decoder module includes an image decoder, a prompt decoder, and a mask decoder, which are used to decode the data obtained by the encoder. The decoder does not perform LoRA coupling processing. The mask decoder combines the information of the image decoder and the prompt decoder to predict the result, and the prediction result is tensor data.

[0171] The task modes include:

[0172] Point mode, foreground or background determination based on pixel coordinates (x, y);

[0173] Box mode, based on border (x min , x max ,y min ,y max )Separate the hidden target and the environment in the area from front to back;

[0174] Global mode, performs front / back segmentation based on hidden objects and the environment in the entire image.

[0175] The predictor module outputs the prediction result as a mask image according to different task modes, such as point mode, box mode, and global mode.

[0176] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0177] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0178] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0179] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0180] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are included in the scope of the claims of the present invention to be approved.

Claims

1. A method for detecting hidden targets based on a large model, characterized in that: include: Get the RGB image of the target to be detected; Input the target RGB image into the pre-trained large model to obtain a hidden target; The pre-trained large model includes an encoder and a decoder embedded in the LoRA layer.

2. The detection method according to claim 1, characterized in that: The step of inputting the target RGB image into the pre-trained large model to obtain the hidden target comprises: Performing preprocessing based on the input target RGB image; Based on the preprocessed RGB image, high-dimensional features are captured by the encoder embedded in the LoRA layer; Performing dimensionality reduction decoding on the high-dimensional features through a decoder to obtain a decoding result; Based on the decoding results, predictions are made according to different task modes to generate segmentation results of hidden targets.

3. The detection method according to claim 2, characterized in that: The preprocessing of the target RGB image based on the input includes: For RGB images of different sizes input as NumPy array format, the sizes are aligned according to the longest side of the RGB images to obtain RGB images of fixed size; The fixed-size RGB image is converted, the converted image is cut into 16 sub-regions, and each sub-region is vectorized.

4. The detection method according to claim 3, characterized in that: The converting according to the fixed-size RGB image comprises: According to the fixed-size RGB image, a grayscale image of a fixed size is obtained by converting the RGB color space into a grayscale color space; According to the grayscale image of the fixed size, a grayscale image in the Torch tensor format is obtained by converting the NumPy array format to the Torch tensor format.

5. The detection method according to claim 4, characterized in that: The vectorization processing of each sub-region comprises: Expand the tensor data of each sub-region into a one-dimensional tensor and perform position embedding operation; The expansion dimension of each sub-region is dim; The position embedding operation adopts a combination function of a sine function and a cosine function.

6. The detection method according to claim 5, characterized in that: The high-dimensional features are obtained by capturing the encoder embedded in the LoRA layer, including: Forward deduction, based on the preprocessed RGB image as the input data of the first coupling structure, the input data is respectively input to the attention block and the LoRA layer in the first coupling structure, and the integrated output result is fused after tensor operation, and the output result is used as the input data of the next coupling structure until the result is output from the last coupling mechanism; Reverse deduction: Based on the LoRA layer parameters in the last coupling structure, adjust the LoRA layer parameters in the first coupling structure in turn, and freeze the parameters of the attention blocks in each coupling structure at the same time to obtain the final high-dimensional features of the image; The encoder embedded in the LoRA layer includes: N serially coupled structures; Each coupled structure consists of an attention block and a LoRA layer.

7. The detection method according to claim 6, characterized in that: The input data is respectively input into the attention block and the LoRA layer in the first coupling structure, and the fused integrated output results after tensor operation include: Based on the input data input into the attention block in the first coupling structure, the data obtained through tensor operation is fused and integrated with the input data input into the LoRA layer in the first coupling structure, and the output result is output.

8. The detection method according to claim 7, characterized in that: The input data is input to the LoRA layer in the first coupling structure, and the data obtained after tensor operation includes: The first linear layer of the LoRA layer uses the expanded dimension dim of each sub-region as the input channel, and the input tensor data is mapped from the dim dimension space to the 256-dimensional space through the weight of the first linear layer and output; The second linear layer through the LoRA layer is based on dimension 256 as the input channel. The tensor data output by the first linear layer is mapped from the 256-dimensional space back to the original dim-dimensional space through the weight of the second linear layer. Among them, the first linear layer weight or the second linear layer weight of the LoRA layer in each coupling structure is different.

9. The detection method according to claim 2, characterized in that: The task modes include: Point mode, foreground or background determination based on pixel coordinates (x, y); Box mode, based on border (x min , x max ,y min ,y max )Separate the hidden target and the environment in the area from front to back; Global mode, performs front / back segmentation based on hidden objects and the environment in the entire image.

10. A concealed target detection system based on a large model, characterized in that: include: An acquisition module is used to obtain an RGB image of the target to be detected; A detection module, inputting the RGB image into a pre-trained large model to obtain a hidden target; The pre-trained large model includes an encoder and a decoder embedded in the LoRA layer.

Images