Target detection method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a target detection method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining a to-be-processed vehicle image, inputting the to-be-processed vehicle image into a block embedding layer of an SAM model image encoder, and generating block embedding features; acquiring data set sample information, inputting the data set sample information into an adapter of the SAM image encoder, and generating prompt features; inputting the block embedding features and the prompt features into an encoding layer of an SAM image encoder to generate an encoding vector; and inputting the coding vector into an SAM model image decoder to generate a salient target detection result corresponding to the vehicle image. The adapter model can learn data sample information and transmit the data sample information to the SAM image encoder as a prompt, adaptive training is efficiently performed in a salient target detection task through additional training of a small amount of data, the salient target detection accuracy is improved, and the segmentation efficiency of the vehicle image is further improved.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and financial technology, and in particular to a target detection method, device, computer equipment and storage medium. Background Art

[0002] Vehicle component identification plays a very important role in vehicle damage assessment. It can provide accurate location information for further damage assessment. In the traditional vehicle collision damage assessment scenario, after a traffic accident occurs, the vehicle damage assessment must first be reported by the user to the insurance company, and then the user must wait at the scene of the accident for the insurance company's damage assessor to assess the damage on site. The process is lengthy and the waiting time is long. The entire process gives the user a very poor experience and is prone to traffic congestion.

[0003] With the development of artificial intelligence vision technology, intelligent identification of vehicle parts has gradually been widely used, which can quickly locate damage, help companies save manpower, reduce customer waiting time, and improve customer satisfaction. In the existing technology, large language models pre-trained on network-scale datasets are revolutionizing the field of natural language processing through strong zero-shot and few-shot generalization. These basic models can be generalized to tasks and data distributions beyond those seen during training through prompt engineering. In SAM (Segment Anything Model), the author transforms the prompt concept from NLP into segmentation, which can be a set of foreground background points or a rough box, etc. A basic model for image segmentation is established, which can use prompts to solve a series of downstream segmentation tasks on new data distributions.

[0004] Salient object detection aims to identify the most eye-catching object from the input image. This study hopes to be able to identify the subject of the image. It is one of the important research areas of computer vision. Salient object detection is usually divided into two directions: traditional methods and deep learning methods.

[0005] Since computer vision covers a wide range of tasks, the incompleteness of SAM is obvious, which is similar to other basic models because the training data cannot contain the entire corpus and the working scene will change. In the prior art, although the initial SAM model is used for salient object detection, the segmentation edges are very fine, but it is difficult to distinguish between salient objects and non-salient objects, resulting in low accuracy of salient object detection and reduced efficiency of subsequent image segmentation. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the present invention provides a target detection method, apparatus, computer equipment and storage medium, aiming to solve the problem in the prior art that although the initial SAM model for salient target detection has very fine segmentation edges, it is difficult to distinguish between salient targets and non-salient targets, resulting in low accuracy of salient target detection and reduced efficiency of subsequent image segmentation.

[0007] The technical solution of the present invention is as follows:

[0008] A first embodiment of the present invention provides a target detection method, the method comprising:

[0009] Acquire a vehicle image to be processed, input the vehicle image to be processed into a block embedding layer of a SAM model image encoder, and generate a block embedding feature;

[0010] Acquire data set sample information, input the data set sample information into an adapter of a SAM image encoder, and generate prompt features;

[0011] Input the block embedding feature and the prompt feature into the encoding layer of the SAM image encoder to generate an encoding vector;

[0012] The encoding vector is input into the SAM model image decoder to generate a salient object detection result corresponding to the vehicle image.

[0013] Another embodiment of the present invention provides a target detection device, the device comprising:

[0014] An image embedding module is used to obtain a vehicle image to be processed, input the vehicle image to be processed into a block embedding layer of a SAM model image encoder, and generate a block embedding feature;

[0015] A prompt feature generation module, used for obtaining sample information of a data set, inputting the sample information of the data set into an adapter of a SAM image encoder, and generating prompt features;

[0016] An encoding module, used for inputting the block embedding feature and the prompt feature into the encoding layer of the SAM image encoder to generate an encoding vector;

[0017] The target detection module is used to input the encoding vector into the SAM model image decoder to generate a significant target detection result corresponding to the vehicle image.

[0018] Another embodiment of the present invention provides a computer device, the computer device comprising at least one processor; and

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the above-mentioned target detection method.

[0021] Another embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can perform the steps of the above-mentioned target detection method.

[0022] Beneficial effects: The target detection method of the embodiment of the present invention can obtain the vehicle image to be processed, input the vehicle image to be processed into the block embedding layer of the SAM model image encoder, generate the block embedding feature; obtain the data set sample information, input the data set sample information into the adapter of the SAM image encoder, generate the prompt feature; input the block embedding feature and the prompt feature into the encoding layer of the SAM image encoder to generate the encoding vector; input the encoding vector into the SAM model image decoder to generate the salient target detection result corresponding to the vehicle image. In the embodiment of the present invention, a lightweight adapter network is designed to fuse the salient target task information as a prompt to be passed into the SAM encoder to enhance the generalization of the model; the adapter model in the implementation of the present invention can learn the guidance information of the salient target detection task and pass it into the SAM image encoder as a prompt, so that the adaptive training can be efficiently performed in the salient target detection task through additional training with a small amount of data, the accuracy of salient target detection is improved, and the efficiency of vehicle image segmentation is further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0024] Figure 1 A schematic diagram of an application environment of an embodiment of a target detection method of the present invention;

[0025] Figure 2 A flow chart of a preferred embodiment of a target detection method of the present invention;

[0026] Figure 3 for Figure 2 A flowchart of a specific embodiment of step S100;

[0027] Figure 4 for Figure 2A flowchart of a specific embodiment of step S200;

[0028] Figure 5 for Figure 2 A flowchart of another specific embodiment of step S200;

[0029] Figure 6 for Figure 2 A flowchart of a specific embodiment of step S300;

[0030] Figure 7 for Figure 2 A flowchart of a specific embodiment of step S400;

[0031] Figure 8 for Figure 2 A flowchart of another specific embodiment of step S400

[0032] Fig. 9 A schematic diagram of functional modules of a preferred embodiment of a target detection device of the present invention;

[0033] Fig.10 A structural schematic diagram of a preferred embodiment of a computer device of the present invention;

[0034] Fig.11 Another structural schematic diagram of a preferred embodiment of a computer device of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0036] The embodiments of the present invention are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0037] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. Here, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0038] The method provided in this application can be applied to artificial intelligence (AI) scenarios. AI is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making. Research in the field of artificial intelligence includes robots, natural speech processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, basic AI theory, etc.

[0039] The target detection method provided by the embodiment of the present invention can be applied in Figure 1 In the application environment, the client obtains the vehicle image to be processed, inputs the vehicle image to be processed into the block embedding layer of the SAM model image encoder, and generates the block embedding feature; obtains the data set sample information, inputs the data set sample information into the adapter of the SAM image encoder, and generates the prompt feature; inputs the block embedding feature and the prompt feature into the encoding layer of the SAM image encoder to generate the coding vector; inputs the coding vector into the SAM model image decoder to generate the salient target detection result corresponding to the vehicle image. In the present invention, the adapter model can learn the guidance information of the task and pass it into the SAM image coding network as a prompt, so that the adaptive training can be efficiently performed in the salient target detection task through additional training of a small amount of data, thereby improving the accuracy of salient target detection and further improving the segmentation efficiency of the vehicle image. Among them, the client can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0040] In view of the above problems, an embodiment of the present invention provides a target detection method, please refer to Figure 2 , Figure 2 FIG. 1 is a flow chart of a preferred embodiment of a target detection method of the present invention. Figure 2 As shown, it includes:

[0041] Step S100: Obtain a vehicle image to be processed, input the vehicle image to be processed into a block embedding layer of a SAM model image encoder, and generate a block embedding feature.

[0042] The target detection method in the embodiment of the present invention is applied to a server that implements a significant target detection function in a vehicle damage assessment scenario. The server can detect significant targets in a vehicle image to be processed according to a corresponding algorithm.

[0043] Vehicle damage assessment refers to the process of evaluating and determining the extent of a vehicle's damage after it has been damaged. This process is usually performed by professional assessors from insurance companies, who determine the amount of the vehicle's loss based on the damage to the vehicle, combined with factors such as market price and repair costs. However, traditional manual damage assessment methods are highly subjective and inefficient, so intelligent damage assessment systems have emerged.

[0044] Image segmentation is an important step in the image processing process. It can divide the image into several specific regions with unique properties and propose objects of interest. In vehicle damage assessment, image segmentation technology is mainly used to identify and locate the damaged parts of the vehicle.

[0045] The positioning methods for image segmentation include:

[0046] Global recognition and positioning: Deep learning-based methods, such as convolutional neural networks (CNN) and their improved versions (such as Mask R-CNN), can perform global recognition and positioning of vehicle component structures. These methods can accurately identify various vehicle components, such as the engine compartment, body, wheels, etc., by training a large amount of vehicle image data.

[0047] Local annotation and segmentation: Based on global recognition and positioning, target detection methods (such as Yolo-v3) are used to locally annotate the damaged parts. These methods can accurately detect the damaged areas in the image and segment them. The segmented images can more clearly show the specific situation of the damaged parts, which helps the damage assessor to make an accurate assessment.

[0048] Damage degree assessment: After segmenting the damaged parts, the damage degree can be further assessed using image processing techniques (such as morphological processing, edge detection, etc.). These techniques can extract characteristic information such as the shape, size, and area of ​​the damaged parts, thereby helping the damage assessor to more accurately determine the degree of damage.

[0049] The method for obtaining the vehicle image to be processed in the present invention mainly refers to that when the user's vehicle is damaged, the damaged vehicle photo can be uploaded to the damage assessment platform. The server in the background of the damage assessment platform will identify the significant targets in the uploaded damaged vehicle photo, thereby facilitating the subsequent identification of the vehicle's damage results and improving the efficiency of damage assessment.

[0050] The image to be processed is input into the block embedding layer of the SAM model image encoder to generate the corresponding block embedding features.

[0051] The SAM model is a basic model for image segmentation. The SAM model aims to break through the boundaries of image segmentation and achieves strong zero-sample generalization capabilities by training on large-scale datasets. It can extract the corresponding elements of any object from a photo or video with one click, and quickly split and migrate them to a new picture. This feature makes the SAM model have broad application potential in image processing, augmented reality (AR), virtual reality (VR), and content creation. The SAM model mainly consists of three parts: image encoder, prompt encoder, and mask decoder.

[0052] Image Encoder: Based on a scalable and powerful pre-training method, it uses a ViT model pre-trained with MAE, capable of handling high-resolution inputs. The image encoder is run once for each image and applied before the prompt model.

[0053] Hint Encoder: Two sets of hints are considered: sparse (points, boxes, text) and dense (masks). Points and boxes are represented by positional encodings, and the learned embeddings and free-form text for each hint type are summed with the off-the-shelf text encodings from CLIP. Dense hints (i.e., masks) are embedded using convolutions and element-wise summed over the image embeddings.

[0054] Mask Decoder: Efficiently maps image embeddings, hint embeddings, and output tokens to masks. Given an image embedding, the hint encoder and mask decoder can predict a mask from a hint in about 50 milliseconds in the browser.

[0055] Before step S100, the SAM model needs to be constructed. First, the image encoder of the SAM model is constructed. The image encoder includes a block embedding layer and a plurality of image coding layers connected in sequence. An adapter is also provided in the image encoder, and the number of adapters is the same as that of the image coding layers, and the adapter is connected to the image coding layers. The adapter enhances the generalization ability of the model for the salient target detection task by learning the information of the specific task and passing it into each layer in the SAM image coding network as a prompt. The specific task in the present invention refers to salient target detection. Among them, salient target detection is intended to simulate the perception process of the human visual system to the salient targets in the image. The human visual system can quickly identify the salient targets according to some salient principles, such as color contrast, brightness contrast, direction contrast, etc. Therefore, the salient target detection algorithm is also usually based on these principles to calculate the salient score of each pixel in the image, and the pixel area with high salient score is used as a salient target.

[0056] The hint encoder of the SAM model is one of the key components in its architecture, which is mainly responsible for mapping the input hint information into the feature space so as to combine it with the image features output by the image encoder and send it to the mask decoder for segmentation prediction.

[0057] Here is how the SAM model hint encoder is constructed:

[0058] When building the hint encoder of the SAM model, the hints can be sparse or dense. Sparse hints include points, boxes, and text, while dense hints are masks. To handle these different types of hints, the hint encoder uses different representation methods: Sparse hints: Points: The position information of the points is represented by position encoding. Boxes: The four corners or boundary information of the box are also represented by position encoding. Text: Free-form text hints are added to the ready-made text encoding in CLIP (Contrastive Language–Image Pre-training) to obtain the feature representation of text hints. Dense hints (masks): The masks are embedded using convolution operations and element-wise summed through image embedding to obtain the feature representation of mask hints.

[0059] The prompt encoder is a neural network structure that converts the input prompt information into a feature vector. In the SAM model, the specific structure of the prompt encoder may vary depending on the prompt type, but in general it can be seen as a mapper that maps the prompt information to the feature space.

[0060] After obtaining the output of the image encoder and the hint encoder, the two feature vectors need to be fused. In the SAM model, this fusion is achieved by combining the image embedding and the hint embedding and feeding them into a lightweight mask decoder. The mask decoder effectively decodes the two feature vectors into the final segmentation mask.

[0061] The mask decoder of the SAM model is one of the core components in its architecture, which is mainly responsible for combining the outputs of the image encoder and the hint encoder and generating the final segmentation mask.

[0062] The mask decoder is usually a lightweight neural network structure that adopts the Transformer decoder block as its core component. In the SAM model, the mask decoder is designed to efficiently map image embeddings, hint embeddings, and output tokens to segmentation masks.

[0063] The mask decoder includes:

[0064] Transformer decoder block: This is the core part of the mask decoder, which adopts self-attention and cross-attention mechanisms to update the image embedding and hint embedding.

[0065] Dynamic Mask Prediction Head: After the Transformer decoder block, a dynamic mask prediction head is usually connected to generate the final segmentation mask. This prediction head may be a multi-layer perceptron (MLP) or other types of neural network structures.

[0066] In the mask decoder, the generation and processing of output tokens is a key step. These tokens usually include two parts: iou tokens and mask tokens:

[0067] iou token: This is a special token used to predict the IoU (Intersection over Union) score between the segmentation mask output by the model and the true mask. The IoU score is an important indicator for measuring segmentation quality. The ioutoken is generated by a separate embedding layer and occupies the first position in the output of the Transformer model.

[0068] Mask tokens: These tokens correspond to each possible segmentation mask predicted by the model. In the SAM model, multiple candidate masks are usually generated to handle possible ambiguity (for example, three may be generated by default). Each mask token is also generated through an embedding layer and follows the iou token in the output of the Transformer model.

[0069] After the SAM image model is built, it is necessary to train the SAM image model. First, obtain at least one vehicle image dataset and train the SAM model based on the vehicle image dataset. The vehicle image dataset can be a public dataset or manually annotated. For example, the SAM model is trained on a large-scale and diverse image segmentation dataset called SA-1B (Segment Anything 1-Billionmask dataset). The dataset contains 11 million images and a total of more than 1 billion mask images. Through interactive and promptable model design, the SAM model can be transferred to new image distributions and tasks through zero-sample learning. During training, the parameters of the encoding layer model can be kept frozen.

[0070] Among them, frozen parameters refer to keeping the parameters of large pre-trained models (such as large language models LLM) unchanged during the training process, that is, not updating these parameters, but training or fine-tuning other parts (such as specific task layers or modules) to adapt to new tasks or data sets. This approach aims to utilize the knowledge and representation capabilities of pre-trained models while reducing the consumption of computing resources and time.

[0071] After obtaining the vehicle image dataset, the cross-entropy loss function is used as the loss function of the SAM model. Therefore, in the process of training the improved SAM model, the cross-entropy loss function is used to train the improved SAM model. The cross-entropy loss function is a commonly used loss function in machine learning and deep learning, especially in classification problems. It measures the difference between two probability distributions, one of which is the distribution predicted by the model and the other is the true label distribution. The goal of the cross-entropy loss function is to make the distribution predicted by the model as close as possible to the true label distribution.

[0072] Among them, Figure 3 As shown, step S100, i.e., obtaining a vehicle image to be processed, inputting the vehicle image to be processed into a block embedding layer of a SAM model image encoder, and generating a block embedding feature, includes:

[0073] Step S101, obtaining the vehicle image to be processed, and inputting the vehicle image to be processed into the block embedding layer of the SAM model encoder;

[0074] Step S102, cutting the vehicle image to be processed based on the block embedding layer to generate image blocks of fixed size;

[0075] Step S103, converting the image block into an embedding vector;

[0076] Step S104: acquiring position information of the image block, and generating a position code based on the position information;

[0077] Step S105: adding the position code to the embedding vector to generate the block embedding feature.

[0078] Obtain the vehicle image to be processed and input it into the block embedding layer. The block embedding layer cuts the vehicle image to be processed into small blocks of fixed size, such as 16*16 pixel image blocks. Each block is regarded as a "token", similar to a word in NLP. Expand the generated image blocks and convert them into a series of embedding vectors through a linear layer. In order to retain the position information of the image blocks, obtain the position information of the image blocks in the vehicle image to be processed, add the position vector to the embedding vector, and generate the block embedding feature.

[0079] By introducing the block embedding layer, the amount of computation for each processing can be reduced and the processing efficiency can be improved by dividing the text data into small blocks. Block processing allows the model to process multiple small blocks in parallel, thereby further improving the processing speed. The block embedding layer can be flexibly adjusted according to different task requirements and data characteristics to adapt to different application scenarios.

[0080] Step S200: Acquire data set sample information, input the data set sample information into the adapter of the SAM image encoder, and generate prompt features.

[0081] Since it is difficult to distinguish between salient targets and non-salient targets when the initial SAM model is directly used for salient target detection, although the segmentation edges are very fine, this may be due to the lack of samples for salient detection in the training data. Therefore, an improved SAM model is proposed in an embodiment of the present invention. The improved SAM model is an adapter enhanced on the basis of the original SAM model. The SAM adapter method in the present invention can apply a large pre-trained SAM to specific downstream tasks with enhanced performance. Using internal knowledge and external control signals, a lightweight additional network is designed. The network learns the guidance information of the task from a specific downstream task and passes the information into the SAM as a visual cue. The additional training of the frozen large base model to adapt to a small amount of data is usually efficient in many downstream tasks.

[0082] Specifically, at least one vehicle image data set is obtained, sample set information of the vehicle data is extracted, and the sample set information is input into an adapter of a SAM image encoder to generate prompt features. The model of the adapter may adopt an MLP model.

[0083] Among them, Figure 4 As shown, in step S200, data set sample information is obtained, and the data set sample information is input into the adapter of the SAM image encoder to generate prompt features, including:

[0084] Step S201, obtaining a data set sample corresponding to a vehicle image, and collecting texture or frequency information of the data set sample;

[0085] Step S202: Input the texture or frequency information of the data set sample into the adapter of the SAM image encoder to generate the prompt feature.

[0086] After obtaining the data set samples corresponding to the vehicle image, the texture or frequency information of the data set samples is collected as the coding hint information, and the texture or frequency information of the data set samples is input into the adapter of the SAM image encoder to generate hint features.

[0087] The texture of the dataset samples is a key feature, especially in the fields of computer vision and image processing. Texture is a spatial arrangement of pixels in an image, which provides important information about the surface properties of the image. In computer vision, texture is one of the main features for identifying objects or regions of interest in an image.

[0088] Texture can be divided into two types: tactile texture and visual texture. Tactile texture is the surface texture we feel through touch, while visual texture is the image texture we observe through vision. In the field of image processing, visual texture is usually the focus.

[0089] Textures can be used for: Texture classification: The dataset can be used to train and evaluate deep learning models, such as convolutional neural networks (CNNs), to automatically classify texture images. Feature extraction: Researchers can use the dataset to compare and improve different feature extraction techniques, such as local binary patterns (LBP), Gabor filters, etc. Machine learning algorithms: The dataset is often used to test and optimize machine learning algorithms, including support vector machines (SVM), random forests, etc. Texture synthesis: By analyzing the textures in the dataset, new algorithms can be developed to synthesize new textures with specific properties. Augmented reality and virtual reality: High-quality texture images can be applied to AR and VR systems to enhance the realism of virtual environments.

[0090] The frequency of a data sample set is the ratio of the number of times a value appears in the data set to the total sample size. It is a statistic that describes the relative frequency of each value in the data set. The frequency can be calculated by dividing the frequency by the total sample size, usually expressed as a percentage or decimal. For example, in a data set with a sample size of 20, the value 2 appears 3 times, so the frequency of the value 2 is 3 / 20 = 0.15 or 15%.

[0091] Among them, the image encoder adapter includes a linear layer and an up-projection layer, such as Figure 5 As shown, in step S200, data set sample information is obtained, and the data set sample information is input into the adapter of the SAM image encoder to generate prompt features, including:

[0092] Step S211, obtaining data set sample information, and inputting the data set sample information into the linear layer of the image encoder adapter;

[0093] Step S212, extracting salient object detection prompt information in the data set sample information based on the linear layer to generate prompt information;

[0094] Step S213: input the prompt information into the upward projection layer of the image encoder adapter to generate the prompt feature for adjusting the shared weight of the encoding layer.

[0095] The adapters are pre-built. Each adapter consists of a linear layer that generates a salient object detection task. The upward projection layer MLP that shares weights with the input SAM Transformer encoder size up The adapter receives input information F iGet the output prompt P i , where GELU is the GELU activation function, and the specific formula is as follows:

[0096]

[0097] This method can be used not only for salient object detection tasks but also for other downstream tasks of computer vision, where the input information F i Different inputs can be designed according to different downstream tasks, such as F i It can be the texture or frequency information of the dataset sample or other manually formulated information. The salient object detection task F in this scheme i are features extracted by the middle layer of a VST saliency detection network.

[0098] Among them, the linear layer, which is usually called the fully connected layer or dense layer in the neural network, is one of the most basic layers in the neural network. The main function of the linear layer is to perform a linear transformation on the input data, that is, to perform the product operation of the input and the weight matrix and add a bias term. This transformation can map the input data to a new feature space, thereby changing the dimension of the data and facilitating further processing by subsequent layers. In the linear layer, the input data is usually represented as a matrix X with a dimension of (N, din), where N is the batch size and din is the dimension of the input feature. The weight matrix is ​​usually represented as W with a dimension of (dout, din), where dout is the dimension of the output feature. The bias vector is usually represented as b with a dimension of (dout). The output data is obtained through linear transformation, represented as Y = WX + b, where the dimension of Y is (N, dout).

[0099] An up-projection layer is a layer that increases the feature dimension or resolution. This layer is often used in tasks such as Generative Adversarial Networks (GANs), image super-resolution, and semantic segmentation, where low-resolution or low-dimensional features need to be converted to high-resolution or high-dimensional features. The implementation of the up-projection layer may include but is not limited to the following: Transposed Convolution: Transposed convolution is a special convolution operation that can upsample low-resolution feature maps to high resolution. Transposed convolution is often used in GANs and image super-resolution tasks. Bilinear Interpolation: Bilinear interpolation is a simple and effective upsampling method that estimates the value of a new pixel based on the values ​​of surrounding pixels. This method has a wide range of applications in image processing and computer vision. Nearest Neighbor Interpolation: Nearest Neighbor Interpolation is a simple upsampling method that selects the value of the known pixel closest to the new pixel as the value of the new pixel. Although this method is simple, it may not work well in some cases. Other methods: In addition to the above methods, there are other upsampling methods, such as sub-pixel convolution, max pooling inverse, etc. These methods may have different advantages and applicability in different tasks.

[0100] The GELU (Gaussian Error Linear Units) activation function is a nonlinear activation function based on the Gaussian error function. GELU introduces nonlinear transformations, allowing the neural network to learn more complex mapping relationships, thereby improving the expressiveness of the model. GELU is a smooth function with continuous derivatives, which avoids the problem of gradient truncation. This helps the stable propagation of the gradient, making it easier for the neural network to converge during training. When the input value is close to zero, the output value of GELU is also close to zero. This allows GELU to approximately keep the amplitude of the input signal unchanged, reducing information loss. When the input value approaches positive infinity, the output value of GELU approaches 1. This enables the neural network to better judge the category of the input data when performing classification tasks.

[0101] The VST network is a unified model based on Transformer, designed for RGB and RGB-D saliency detection tasks. The VST saliency detection network has powerful global information modeling capabilities and multi-task learning capabilities. The VST network includes:

[0102] Encoder: The encoder generates multi-level tokens from the cropped image block sequence. These tokens contain different features and information of the image, providing a basis for subsequent global long-range dependency modeling.

[0103] Transformer: The transformer module is responsible for transforming the patch tokens obtained by the encoder from the encoder space to the decoder space. For the RGB-D saliency detection task, a cross-modal Transformer (CMT) is also designed to fuse the RGB patch tokens and depth patch tokens extracted from the encoder to explore the interaction between multimodal information.

[0104] Decoder: The decoder is the core part of the VST network, which aims to decode the patch-token into a saliency map. To achieve this goal, the decoder adopts a token-based multi-task decoder design and introduces a new token upsampling method, multi-level token fusion, and patch-task-attention mechanism. These designs enable the decoder to perform saliency detection and edge detection tasks simultaneously and generate high-resolution saliency maps.

[0105] Step S300: input the block embedding features and the prompt features into the encoding layer of the SAM image encoder to generate an encoding vector.

[0106] In the embodiment of the present invention, the image coding vector refers to dividing the image into small blocks (or called image blocks, image regions), and selecting some representative vectors (also called vectors) from these small blocks, which can better represent the characteristics and information of the image. Vector quantization (VQ) is a commonly used method in image coding. It reduces the amount of data by mapping a continuous value data set into a discrete codebook. In this process, each image block is mapped to a discrete vector (i.e., a representative vector in the codebook), thereby achieving image compression and coding.

[0107] The process of constructing an image encoding vector usually includes the following steps:

[0108] Image segmentation: Divide the image into multiple non-overlapping blocks, each containing a certain number of pixels.

[0109] Feature extraction: Extract feature vectors from each small block. These feature vectors can reflect certain statistical characteristics or spatial distribution of pixels in the small block.

[0110] Clustering and codebook construction: Use a clustering algorithm (such as K-Means) to cluster the extracted feature vectors to obtain multiple cluster centers. These cluster centers are the representative vectors in the codebook.

[0111] Vector mapping: The feature vector of each image block is mapped to the nearest cluster center (i.e., a representative vector in the codebook), thereby achieving image compression and encoding.

[0112] Therefore, after obtaining the block embedding features and the prompt features, the block embedding features and the prompt features are concatenated and input into the encoding layer to generate the encoding vector.

[0113] The encoding layer may adopt a pre-trained model, including but not limited to a Transformer network.

[0114] Among them, the encoding layer of the SAM image encoder adopts the Transformer network. Figure 6 As shown, step S300, that is, inputting the block embedding feature and the prompt feature into the encoding layer of the SAM image encoder to generate an encoding vector, includes:

[0115] Step S301, inputting the block embedding features and the prompt features into the Transformer network;

[0116] Step S302: determine the weight of the block embedding feature through the multi-head attention mechanism, perform weighted summation on the block embedding feature based on the weight, and generate a multi-head attention feature;

[0117] Step S303: Fuse the multi-head attention feature with the prompt feature to generate a coding vector.

[0118] The block embedding features are input into the Transformer network to obtain multi-head self-attention features. The multi-head self-attention features are weighted by determining the weights of the block embedding features through the multi-head attention mechanism, and the block embedding features are weighted summed based on the weights to generate multi-head attention features. The multi-head self-attention features are normalized, downsampled, and activated, and then merged with the prompt features to obtain merged features. The multi-head attention features are fused with the prompt features. The fusion process includes normalization, triple attention mechanism, upsampling, and generating encoding vectors.

[0119] The core idea of ​​the Transformer model is the self-attention mechanism, which allows the model to focus on different parts of the input sequence at each step, without having to process the sequence step by step through structures such as RNN. This parallel operation enables the Transformer to have good performance when processing long sequences, while also reducing training time.

[0120] In the Transformer model, the input sequence is first converted to an embedding representation (with positional encoding) and then fed into the encoder for encoding. The output of the encoder is used as the input to the decoder, which gradually generates an output sequence based on the output of the encoder. In the self-attention mechanism, the model calculates a query vector, a key vector, and a value vector for each word in the sequence. The similarity between words is then evaluated by calculating the dot product of the query vector and the key vector, and the attention scores are normalized using the softmax function to obtain the attention weights. Finally, the value vectors are weighted summed using the attention weights to obtain the self-attention mechanism output for each word.

[0121] Step S400: input the encoding vector into the SAM model image decoder to generate a salient object detection result corresponding to the vehicle image.

[0122] After obtaining the encoding vector generated by the encoder, the encoding vector is input into the SAM model image decoder for decoding, and then the vehicle image to be processed is segmented, and a salient target detection result corresponding to the vehicle image is generated according to the segmented image.

[0123] Among them, Figure 7 As shown, step S400, inputting the encoding vector into the SAM model image decoder to generate a salient target detection result corresponding to the vehicle image, includes:

[0124] Step S401, inputting the encoding vector into the SAM model image decoder;

[0125] Step S402, generating a segmentation mask corresponding to the encoding vector based on the SAM model image decoder;

[0126] Step S403: Segment the vehicle image to be processed based on the segmentation mask to generate a salient object detection result corresponding to the vehicle image.

[0127] The encoded vector is fed into the image decoder, which decodes the encoded vector to generate a segmentation mask, and the image is segmented based on the segmentation mask. The mask decoder can efficiently map the image embedding, the hint embedding, and the output token to a mask. A modified Transformer decoder block is used, followed by a dynamic mask prediction head. Our modified decoder block uses hint self-attention and cross-attention in both directions (from hint to image embedding and from image embedding to hint). After running both blocks, we upsample the image embedding, and a multi-layer perceptron (MLP) maps the output token to a dynamic linear classifier, which then computes the foreground mask probability at each position of the image.

[0128] The output of the SAM model image decoder is the semantic image salient object detection result, which can be divided into True Positive, False Positive, True Negative, and False Negative.

[0129] Negative refers to the part without object labels (which can be directly understood as background), and Positive refers to the part with labels. The vehicle image salient object detection result in the embodiment of the present invention can be pixel-level image segmentation. After obtaining the salient object, segmentation is performed according to the salient object to improve segmentation efficiency.

[0130] Among them, Figure 8 As shown, step S400, that is, inputting the encoding vector into the SAM model image decoder to generate a salient target detection result corresponding to the vehicle image, includes:

[0131] Step S411, obtaining prompt content, inputting the prompt content into the prompt encoder of the SAM model, and generating a prompt vector;

[0132] Step S412: concatenate the prompt vector and the encoding vector and input them into the SAM model image decoder to generate a salient object detection result corresponding to the vehicle image.

[0133] In some other embodiments, the prompt content can also be obtained by obtaining the prompt input by the user. The prompt content prompt information can be a mask, points, box or text. If it is a mask, it is represented by convolution, and if it is the other three prompt words, it is represented in the form of position encoding. Among them, the prompt content is a technology for optimizing and designing instructions for interacting with artificial intelligence models, aiming to improve the performance and output quality of the model. Prompt engineering refers to the process of designing and optimizing inputs for natural language processing (NLP) models, such as ChatGPT, to obtain the desired response. It involves selecting appropriate elements such as vocabulary, grammar, context, and topic, and using different techniques and strategies to influence the generation behavior and results of the model. By optimizing the prompts, the model can be guided to understand the user's needs more accurately, thereby generating more relevant, creative or accurate answers.

[0134] After obtaining the prompt content, the prompt encoder is input to obtain the prompt encoding vector. The image encoding vector and the prompt encoding vector are concatenated and input to the mask decoder to obtain the image salient object detection result.

[0135] Compared with the prior art, the target detection method of the embodiment of the present invention has the following technical advantages:

[0136] Based on the pre-trained large language model SAM, the adapter model can be optimized for specific downstream tasks to improve the generalization ability and performance of the model.

[0137] The adapter model learns task guidance information and passes it into each layer in the SAM image encoding network as a hint, which can be efficiently adapted in many downstream tasks with additional training on a small amount of data.

[0138] The input information of the adapter model can be designed according to different downstream tasks, which has certain flexibility and scalability. The adapter model is not only suitable for salient object detection tasks, but also can be used in other downstream tasks of computer vision.

[0139] It should be noted that there is not necessarily a certain order between the above-mentioned steps. A person skilled in the art can understand, based on the description of the embodiments of the present invention, that in different embodiments, the above-mentioned steps may have different execution orders, that is, they may be executed in parallel, may be executed interchangeably, and so on.

[0140] Another embodiment of the present invention provides a target detection device, which corresponds one-to-one to the data verification method of the above embodiment. Fig. 9 As shown, the device 1 comprises:

[0141] The image embedding module 100 is used to obtain a vehicle image to be processed, input the vehicle image to be processed into a block embedding layer of a SAM model image encoder, and generate a block embedding feature;

[0142] A prompt feature generation module 200 is used to obtain sample information of a data set, input the sample information of the data set into an adapter of a SAM image encoder, and generate prompt features;

[0143] The encoding module 300 is used to input the block embedding feature and the prompt feature into the encoding layer of the SAM image encoder to generate an encoding vector;

[0144] The target detection module 400 is used to input the encoding vector into the SAM model image decoder to generate a significant target detection result corresponding to the vehicle image.

[0145] The specific implementation method is shown in the method embodiment, which will not be described in detail here.

[0146] In one embodiment, the image embedding module 100 is specifically used for:

[0147] Acquire the vehicle image to be processed, and input the vehicle image to be processed into the block embedding layer of the SAM model encoder;

[0148] Cutting the vehicle image to be processed based on the block embedding layer to generate image blocks of fixed size;

[0149] Converting the image block into an embedding vector;

[0150] Acquire position information of the image block, and generate a position code based on the position information;

[0151] The position code is added to the embedding vector to generate the block embedding feature.

[0152] The specific implementation method is shown in the method embodiment, which will not be described in detail here.

[0153] In one embodiment, the prompt feature generation module 200 is specifically used for:

[0154] Acquire data set sample information, and input the data set sample information into a linear layer of the image encoder adapter;

[0155] Extracting salient target detection prompt information from the data set sample information based on the linear layer to generate prompt information;

[0156] The hint information is input into the upward projection layer of the image encoder adapter to generate the hint features for adjusting the shared weights of the encoding layer.

[0157] The specific implementation method is shown in the method embodiment, which will not be described in detail here.

[0158] In one embodiment, the encoding module 300 is specifically used for:

[0159] Inputting the block embedding features and the prompt features into the Transformer network;

[0160] Determine the weight of the block embedding feature through a multi-head attention mechanism, perform weighted summation on the block embedding feature based on the weight, and generate a multi-head attention feature;

[0161] The multi-head attention feature is fused with the prompt feature to generate an encoding vector.

[0162] The specific implementation method is shown in the method embodiment, which will not be described in detail here.

[0163] In one embodiment, the target detection module 400 is specifically used for:

[0164] Inputting the encoding vector into the SAM model image decoder;

[0165] Generate a segmentation mask corresponding to the encoding vector based on the SAM model image decoder;

[0166] The vehicle image to be processed is segmented based on the segmentation mask to generate a salient object detection result corresponding to the vehicle image.

[0167] The specific implementation method is shown in the method embodiment, which will not be described in detail here.

[0168] In one embodiment, the prompt feature generation module 200 is further used to:

[0169] Obtaining a data set sample corresponding to the vehicle image, and collecting texture or frequency information of the data set sample;

[0170] The texture or frequency information of the dataset samples is input into the adapter of the SAM image encoder to generate the hint features.

[0171] The specific implementation method is shown in the method embodiment, which will not be described in detail here.

[0172] In one embodiment, the target detection module 400 is further configured to:

[0173] Obtain prompt content, and input the prompt content into a prompt encoder of a SAM model to generate a prompt vector;

[0174] The prompt vector and the encoding vector are concatenated and input into a SAM model image decoder to generate a salient object detection result corresponding to the vehicle image.

[0175] The specific implementation method is shown in the method embodiment, which will not be described in detail here.

[0176] The present invention provides a target detection device. By introducing an adapter, since the adapter model can learn the guidance information of the task and pass it into the SAM image coding network as a prompt, it is possible to efficiently perform adaptive training in the salient target detection task through additional training with a small amount of data, thereby improving the accuracy of salient target detection and further improving the segmentation efficiency of vehicle images.

[0177] Another embodiment of the present invention provides a computer device, which may be a server, and its internal structure diagram may be as shown in Fig.10 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it implements a function or step on the server side of a data verification method.

[0178] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Fig.11 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, the functions or steps on the client side of a target detection method are implemented.

[0179] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0180] Acquire a vehicle image to be processed, input the vehicle image to be processed into a block embedding layer of a SAM model image encoder, and generate a block embedding feature;

[0181] Acquire data set sample information, input the data set sample information into an adapter of a SAM image encoder, and generate prompt features;

[0182] Input the block embedding feature and the prompt feature into the encoding layer of the SAM image encoder to generate an encoding vector;

[0183] The encoding vector is input into the SAM model image decoder to generate a salient object detection result corresponding to the vehicle image.

[0184] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0185] Acquire a vehicle image to be processed, input the vehicle image to be processed into a block embedding layer of a SAM model image encoder, and generate a block embedding feature;

[0186] Acquire data set sample information, input the data set sample information into an adapter of a SAM image encoder, and generate prompt features;

[0187] Input the block embedding feature and the prompt feature into the encoding layer of the SAM image encoder to generate an encoding vector;

[0188] The encoding vector is input into the SAM model image decoder to generate a salient object detection result corresponding to the vehicle image.

[0189] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0190] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0191] The embodiments described above are merely illustrative, in which the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0192] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can exist in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0193] It should be noted that if software tools or components other than those of our company appear in the embodiments of the present application, they are only used for illustrative purposes and do not represent actual use.

[0194] Conditional language such as "can," "can," "might," or "may," among other things, unless specifically stated otherwise or otherwise understood within the context as used, is generally intended to convey that particular embodiments can include (while other embodiments do not) particular features, elements, and / or operations. Thus, such conditional language is also generally intended to imply that features, elements, and / or operations are anyway required for one or more embodiments or that one or more embodiments must include logic for determining, with or without input or prompting, whether such features, elements, and / or operations are included or will be performed in any particular embodiment.

[0195] What has been described herein in this specification and the accompanying drawings includes examples that can provide target detection methods and devices. Of course, it is not possible to describe every conceivable combination of elements and / or methods for the purpose of describing the various features of the present disclosure, but it is recognized that many other combinations and permutations of the disclosed features are possible. Therefore, it is apparent that various modifications can be made to the present disclosure without departing from the scope or spirit of the present disclosure. In addition, or in an alternative, other embodiments of the present disclosure may be apparent from consideration of this specification and the accompanying drawings and from the practice of the present disclosure as presented herein. It is intended that the examples proposed in this specification and the accompanying drawings are considered to be illustrative and not restrictive in all respects. Although specific terms are employed herein, they are used in a general and descriptive sense and are not used for the purpose of limitation.

Claims

1. A target detection method, characterized in that , the method comprises: Acquire a vehicle image to be processed, input the vehicle image to be processed into a block embedding layer of a SAM model image encoder, and generate a block embedding feature; Acquire data set sample information, input the data set sample information into an adapter of a SAM image encoder, and generate prompt features; Input the block embedding feature and the prompt feature into the encoding layer of the SAM image encoder to generate an encoding vector; The encoding vector is input into the SAM model image decoder to generate a salient object detection result corresponding to the vehicle image.

2. The target detection method according to claim 1, characterized in that: The step of obtaining a vehicle image to be processed, inputting the vehicle image to be processed into a block embedding layer of a SAM model image encoder, and generating a block embedding feature includes: Acquire the vehicle image to be processed, and input the vehicle image to be processed into the block embedding layer of the SAM model encoder; Cutting the vehicle image to be processed based on the block embedding layer to generate image blocks of fixed size; Converting the image block into an embedding vector; Acquire position information of the image block, and generate a position code based on the position information; The position code is added to the embedding vector to generate the block embedding feature.

3. The target detection method according to claim 1, characterized in that: The image encoder adapter includes a linear layer and an upward projection layer, and the acquiring of data set sample information, inputting the data set sample information into the adapter of the SAM image encoder, and generating prompt features, includes: Acquire data set sample information, and input the data set sample information into a linear layer of the image encoder adapter; Extracting salient target detection prompt information from the data set sample information based on the linear layer to generate prompt information; The hint information is input into the upward projection layer of the image encoder adapter to generate the hint features for adjusting the shared weights of the encoding layer.

4. The target detection method according to claim 1, characterized in that: The encoding layer of the SAM image encoder adopts a Transformer network, and the block embedding feature and the prompt feature are input into the encoding layer of the SAM image encoder to generate an encoding vector, including: Inputting the block embedding features and the prompt features into the Transformer network; Determine the weight of the block embedding feature through a multi-head attention mechanism, perform weighted summation on the block embedding feature based on the weight, and generate a multi-head attention feature; The multi-head attention feature is fused with the prompt feature to generate an encoding vector.

5. The target detection method according to claim 1, characterized in that: The step of inputting the encoding vector into a SAM model image decoder to generate a salient target detection result corresponding to the vehicle image comprises: Inputting the encoding vector into the SAM model image decoder; Generate a segmentation mask corresponding to the encoding vector based on the SAM model image decoder; The vehicle image to be processed is segmented based on the segmentation mask to generate a salient object detection result corresponding to the vehicle image.

6. The target detection method according to claim 1, characterized in that: The step of obtaining sample information of a data set, inputting the sample information of the data set into an adapter of a SAM image encoder, and generating a prompt feature comprises: Obtaining a data set sample corresponding to the vehicle image, and collecting texture or frequency information of the data set sample; The texture or frequency information of the dataset samples is input into the adapter of the SAM image encoder to generate the hint features.

7. The target detection method according to claim 1, characterized in that: The step of inputting the encoding vector into a SAM model image decoder to generate a salient target detection result corresponding to the vehicle image comprises: Obtain prompt content, and input the prompt content into a prompt encoder of a SAM model to generate a prompt vector; The prompt vector and the encoding vector are concatenated and input into a SAM model image decoder to generate a salient object detection result corresponding to the vehicle image.

8. A target detection device, characterized in that: The device comprises: An image embedding module is used to obtain a vehicle image to be processed, input the vehicle image to be processed into a block embedding layer of a SAM model image encoder, and generate a block embedding feature; A prompt feature generation module, used for obtaining sample information of a data set, inputting the sample information of the data set into an adapter of a SAM image encoder, and generating prompt features; An encoding module, used for inputting the block embedding feature and the prompt feature into the encoding layer of the SAM image encoder to generate an encoding vector; The target detection module is used to input the encoding vector into the SAM model image decoder to generate a significant target detection result corresponding to the vehicle image.

9. A computer device, characterized in that: The computer device comprises at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the target detection method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by one or more processors, the one or more processors can execute the steps of the target detection method described in any one of claims 1-7.

Images