Target detection method and device based on visual language large model

Abstract

The invention discloses a target detection method and device based on a visual language large model. The method comprises the following steps: constructing a target detection data set for training; analyzing the image by using area detection and target center point regression, and extracting center point coordinates of a target; and taking the central point as a prompt, and inputting a segmentation model to obtain an initial segmentation mask of the target. The initial segmentation result is verified by means of a visual language large model, the initial result is subjected to reasoning analysis by means of a text prompt function guide model, and false positive and adhesion targets are recognized; and removing target adhesion and false positive objects by using an improved watershed method. Compared with the prior art, the scheme of the invention effectively alleviates the problem that the existing automatic target detection method fails under the complex conditions of fuzzy target gray distribution, adhesion and the like, and can improve the reliability and accuracy of target detection.

Description

Technical Field

[0001] The present invention relates to the field of simulation technology, and in particular to a target detection method and device based on a large visual language model. Background Art

[0002] In image analysis, accurate target detection is particularly critical, especially in complex scenarios. This process is the core foundation of many applications (such as automated inspection, surgical planning, etc.). However, automatic target detection faces many challenges in practical applications. First, imaging artifacts caused by adjacent targets or high-density targets often cause targets to appear stuck together in the image, making it difficult to separate and identify the targets. Second, some targets may be detected from multiple perspectives, and due to factors such as the size, shape, and grayscale of the target, it is even more difficult to distinguish them from the background or other targets.

[0003] Despite extensive research in the field of automatic object detection dedicated to developing new methods, existing technologies still have significant shortcomings in handling situations where objects are clumping or difficult to distinguish. In particular, in complex situations where objects are clumping, existing methods often struggle to effectively identify clumping objects. This leads to operators spending considerable time manually adjusting the automatic detection results by comparing features (such as grayscale distribution and shape continuity), and even requiring the removal of incorrectly detected objects to avoid interference. Summary of the Invention

[0004] In view of this, the present invention proposes a target detection method and device based on a large visual language model to alleviate the shortcomings of the existing technology.

[0005] In the first aspect, the present invention provides a target detection method based on a large visual language model. First, an instruction fine-tuning dataset for target detection and a large visual language model is constructed. The dataset includes images containing targets and target instance annotation information, which are used to train target region detection and target center point regression networks. Then, the region detection and target center point regression methods are used to analyze the images containing targets and extract the center point coordinates of the targets. Based on these center points, the segmentation network model is input to obtain the initial segmentation mask of the target. Subsequently, the initial segmentation results are verified using the large visual language model, and the model is guided by the text prompt function to perform reasoning and analysis on the initial segmentation results to identify false positives and adhesion targets. Finally, the improved watershed method is used to remove target adhesions and remove false positive objects to obtain accurate target instance segmentation results, thereby improving the accuracy and reliability of target detection.

[0006] In the second aspect, the present invention provides a target detection device based on a large visual language model, including: a training data construction module, which is used to construct and manage a target detection training data set and an instruction fine-tuning data set of a large visual language model; a target area detection and center point regression module, which is used to detect the target area and regress the center point of an image containing a target, and extract the location information of each target; a target segmentation module, which is used to input the initial segmentation mask of the target into a segmentation network model based on the coordinates of the target center point as a prompt; an initial result verification module, which is used to input the initial segmentation result into the large visual language model, and screen false positives and adhesion targets through text prompts and reasoning analysis; an adhesion removal module, which is used to use an improved watershed method to remove target adhesions and remove false positive objects, and finally generate accurate target instance segmentation results.

[0007] The beneficial effects of the present invention are as follows:

[0008] The technical solution provided by the present invention may include the following beneficial effects: providing a target detection method and device based on a large visual language model, which combines target area detection, target center point regression and multimodal semantic reasoning for the first time, significantly improving the accuracy and robustness of target detection. By introducing a large visual language model for semantic verification, the interference of false positive targets in the target detection process is effectively reduced, especially in complex situations such as adhesion and grayscale blur. This method also makes full use of the advantages of the large visual language model in reasoning and analysis, and can verify the preliminary segmentation results based on the target center point prompt, identify false positive and adhesion targets, and achieve effective separation between targets through an improved watershed method, and further separate adhesion targets. This technical solution breaks through the robustness bottleneck of traditional automatic target detection methods in complex scenarios such as metal artifacts and target adhesions, and has significant clinical application value.

[0009] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the specific implementation of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific implementation or the description of the prior art. Obviously, the drawings described below are an implementation of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0011] Figure 1 This is a flowchart of a target detection method based on a large visual language model according to the first embodiment of the present invention;

[0012] Figure 2This is a structural diagram of a target detection simulation device based on a visual language large model according to the second embodiment of the present invention. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments.

[0014] First embodiment:

[0015] Figure 1 FIG1 is a flow chart of a target detection method based on a large visual language model according to the first embodiment of the present invention. Figure 1 As shown, the method includes step S1, step S2, step S3, step S4 and step S5.

[0016] Step S1: Construct an object detection dataset and a visual language model instruction fine-tuning dataset. First, construct a dataset for object detection, including images and their corresponding object instance annotations. This dataset is used to train the object region detection and object center point regression networks. Simultaneously, construct a visual language model instruction fine-tuning dataset, including object adhesion and false positive screening tasks, for training and fine-tuning the visual language model.

[0017] Step S2: Input the target image into the target region detection network to detect the target region and extract the target's 3D coordinate information through target center point regression. This process generates preliminary target location data and provides initial information for subsequent target segmentation.

[0018] Step S3: The target center point coordinates obtained in step S2 are used as a hint to input into the target segmentation network model, and the initial segmentation mask of the target is generated by the segmentation network. This mask provides a preliminary segmentation result for subsequent target verification and adhesion removal.

[0019] Step S4: Input the preliminary segmentation results into the visual language model, construct a text prompt template, and combine it with the image context information. The visual language model is used to perform semantic understanding and image context comparison on the initial segmentation results to determine whether the target to which each foreground pixel belongs is a false positive target or target adhesion.

[0020] Step S5: Use the improved watershed method to remove the adhesion between targets and remove false positive objects to obtain the final target instance segmentation result.

[0021] Second embodiment:

[0022] Figure 2FIG. 1 is a schematic diagram of a target detection device based on a large visual language model according to an embodiment of the present invention. Figure 2 As shown, the device includes a training data construction module 201, a target area detection and center point regression module 202, a segmentation module 203, a verification module 204 and an adhesion removal module 205.

[0023] The training data construction module 201 is used to construct the object detection training dataset and the instruction fine-tuning dataset for the visual language model. The data construction module organizes all object instance annotations and images into a standardized format to facilitate subsequent training and reasoning.

[0024] The target region detection and center point regression module 202 is used to detect the target region in the image containing the target and extract the target center point coordinates. Through the target center point regression network, the target position data is accurately extracted to provide initial information for the subsequent segmentation step.

[0025] The target segmentation module 203 uses the target center point coordinates as a hint and inputs the target segmentation network model to generate an initial segmentation mask for the target. The initial segmentation result provides a basis for subsequent verification and adhesion removal.

[0026] Initial result verification module 204 is used to input the preliminary segmentation results into the visual language model and determine whether there are false positive targets by combining language cues with image context information. This step can effectively screen and remove false positive targets, improving the accuracy of target detection.

[0027] Adhesion removal module 205 uses the improved watershed method to post-process the target segmentation results, remove the adhesion between targets, and eliminate false positive objects that are misdetected, and finally generate accurate target instance segmentation results to ensure the efficiency and accuracy of the detection process.

[0028] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A target detection method and device based on a large visual language model, characterized in that: The following steps are involved: Build a dataset of instruction fine-tuning for large object detection and visual language models; Use region detection and target center point regression to analyze the image containing the target and extract the target center point coordinates; The center point is used as a hint and input into the segmentation network model to obtain the initial segmentation mask of the target; Use the visual language model to verify the initial segmentation results, and use the text prompt function to guide the model to perform reasoning and analysis on the initial results to identify false positives and adhesion targets; An improved watershed method is used to remove target adhesion and remove false positive objects.

2. The method according to claim 1, wherein the target detection dataset includes images containing targets and their corresponding target instance annotation information, which is used for region detection and target center point regression network.

3. The method according to claim 1, wherein the region detection step comprises inputting an image containing targets into a region detection network and outputting the region of interest (ROI) locations of all targets in the image.

4. According to the method of claim 1, the visual language large model instruction fine-tuning dataset includes a part for determining whether the foreground pixel in the output image block belongs to the target, and detecting whether the target is adhered.

5. The method of claim 1, wherein the large visual language model receives target image feature encoding and target detection prompt text, and is fine-tuned using low-rank adaptation instructions to complete false positive target screening and adherent target recognition tasks. The large visual language model is constrained based on binary cross entropy to optimize the accuracy and robustness of target detection.

6. A target detection method and device based on a large visual language model, characterized in that: include: The training data construction module is used to build and manage the target detection training dataset and the instruction fine-tuning dataset of the large visual language model; The target area detection and center point regression module is used to detect the target area and perform center point regression on the image containing the target to obtain the position coordinates of each target; The segmentation module is used to input the target center point coordinates as a hint into the segmentation network model to generate the initial segmentation mask of the target; The verification module verifies the initial segmentation results using a large visual language model, performs reasoning analysis based on textual cues, and screens out false positives and adhesion targets. The adhesion removal module is used to remove target adhesion and false positive objects using the improved watershed method to obtain the final target instance segmentation result.

Images