Target identification and positioning method and system based on zero sample detection
By combining classification network and external object detection, using interactive information and SAM module for image segmentation and depth filtering, the problem of object detection in the prior art relying on pre-training data is solved, and zero-sample object detection and accurate spatial positioning are realized, improving the The accuracy of target recognition is achieved.
Patent Information
- Application Number
- CN202411101894.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-12
AI Technical Summary
Existing object detection methods rely heavily on pretraining data and cannot detect objects outside the pretraining data. The bounding box cannot accurately describe the shape of the target object, resulting in poor positioning accuracy and mislocalization, especially the identification of special-shaped objects and externally distributed samples. Low accuracy.
Using a zero-sample detection method, combining classification network and external object detection, the target bounding box is generated through interactive information, and the SAM module is used to segment images, obtain depth images for ground filtering, realize spatial information positioning of the target object, and avoid new Misclassification of targets.
Zero-sample object detection is achieved, and newly emerging targets can be accurately positioned, which improves the accuracy of target classification, avoids the error classification of new targets into existing categories, and enhances the ability to identify distinctive objects and distributed samples.
Smart Images

Figure CN119068175A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field related to computer vision, and in particular to a method and system for target recognition and positioning based on zero-sample detection. Background Art
[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.
[0003] As an important research direction in the field of computer vision, object detection and recognition technology has been widely used in many fields in recent years, such as autonomous driving, intelligent monitoring, face recognition, medical image analysis, and drone navigation.
[0004] The inventors found in their research that the current mainstream target detection methods usually rely on pre-trained weights to perform target detection and use the detected bounding box information for positioning. However, the detection effect of this method is heavily dependent on the data used for pre-training, and it is impossible to detect objects outside the pre-training data. At the same time, the bounding box used by the mainstream detection method is just a simple rectangle, which cannot accurately describe the shape and boundary of the target object. For objects such as Y-shaped objects with a small area of the minimum circumscribed rectangle, there are poor positioning accuracy and even wrong positioning. At the same time, with the expansion of the scope of application, the large amount of data and labor cost required for pre-training have gradually become prominent, and there is still a problem of positioning difficulty in the detection of certain special-shaped objects; and, based on the model training of labeled data, the category recognition of the target, for out-of-distribution samples, will be recognized within the category set by the model, which reduces the accuracy of target category recognition. Summary of the invention
[0005] In order to solve the above problems, the present disclosure proposes a target recognition and positioning method based on zero-sample detection, which solves the problems of pre-training dependence of target detection, poor target detection accuracy and wrong positioning, and solves the object recognition problem caused by the introduction of zero-sample segmentation algorithm by combining classification network with out-of-distribution object detection (OOD).
[0006] In order to achieve the above objectives, the present disclosure adopts the following technical solutions:
[0007] One or more embodiments provide a target recognition and positioning method based on zero-sample detection, comprising the following steps:
[0008] Acquire image data to be recognized;
[0009] Based on the interactive information, a target bounding box is initially generated to obtain the target area image;
[0010] Based on the generated target area image, a classification model including out-of-distribution object detection is used to classify the target to obtain a classification category of the target;
[0011] The obtained bounding box is used as the interactive information to input into the SAM module for image segmentation to obtain the segmentation mask of the target;
[0012] Obtain the depth image of the image to be identified, perform ground filtering, filter the depth value based on the segmentation mask, and obtain the spatial information of the target object.
[0013] One or more embodiments provide a target recognition and positioning system based on zero-sample detection, including:
[0014] An acquisition module: configured to acquire image data to be identified;
[0015] The interactive information part is configured to preliminarily generate a target bounding box based on the interactive information to obtain a target area image;
[0016] Target classification module: configured to perform target classification based on the generated target area image using a classification model including out-of-distribution object detection to obtain a classification category of the target;
[0017] Mask segmentation module: configured to input the obtained bounding box as interactive information into the SAM module for image segmentation to obtain the segmentation mask of the target;
[0018] Target recognition module: It is configured to obtain the depth image of the image to be recognized, perform ground filtering, filter the depth value based on the segmentation mask, and obtain the spatial information of the target object.
[0019] It should be noted here that the various modules in this embodiment correspond one-to-one to the various steps in Example 1, and the specific implementation process is the same, which will not be repeated here.
[0020] An electronic device includes a memory and a processor, and computer instructions stored in the memory and running on the processor. When the computer instructions are run by the processor, the steps in the above-mentioned target recognition and positioning method based on zero-sample detection are completed.
[0021] A computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the steps in the above-mentioned target recognition and positioning method based on zero-sample detection are completed.
[0022] Compared with the prior art, the present invention has the following beneficial effects:
[0023] The present disclosure realizes zero-sample target detection, and can determine the position and size of the target in the image to be identified, and then determine the spatial position of the target. The target positioning model does not require training data to train the model. Even if the sample to be detected is a sample with a new target, the spatial information of the target can be located, that is, the positioning and detection of the newly appeared target can be realized, and zero-sample target detection is realized. In addition, the new target is identified based on out-of-distribution detection and will not be classified into the existing category in the model, which improves the accuracy of target classification.
[0024] The advantages of the present disclosure and the advantages of additional aspects will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The exemplary embodiments of the present disclosure and the description thereof are used to explain the present disclosure but do not constitute a limitation of the present disclosure.
[0026] Figure 1 is a flow chart of a target identification and positioning method according to Embodiment 1 of the present disclosure;
[0027] Figure 2 is a schematic diagram of the structure of the target recognition and positioning model of Embodiment 1 of the present disclosure;
[0028] FIG3( a ) is an example effect diagram of generating a bounding box based on YOLACT according to Embodiment 1 of the present disclosure;
[0029] FIG3( b ) is a diagram showing an example of segmentation effect based on the repeated labeled boundary box labeled in FIG3( a ) according to Embodiment 1 of the present disclosure;
[0030] FIG4( a ) is an example effect diagram of the embodiment 1 of the present disclosure after removing duplicate annotations of the bounding box;
[0031] FIG4( b ) is a diagram showing an example of segmentation of the frame interaction information after removing duplicate annotations of the bounding box in FIG4( a ) according to Embodiment 1 of the present disclosure;
[0032] FIG5( a ) is a schematic diagram of the geometric relationship between the camera and the ground in Embodiment 1 of the present disclosure;
[0033] FIG5( b ) is a top view of the positional relationship between the camera and the target object in Embodiment 1 of the present disclosure;
[0034] Figure 6 is a visualization image of a simulated ground discrete point set with depth values in Embodiment 1 of the present disclosure;
[0035] FIG. 7( a ) is an unprocessed depth image in Embodiment 1 of the present disclosure;
[0036] FIG. 7( b ) is a depth image obtained after the depth image in Embodiment 1 of the present disclosure is ground filtered. DETAILED DESCRIPTION
[0037] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.
[0038] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.
[0039] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof. It should be noted that, in the absence of conflict, the various embodiments in the present disclosure and the features in the embodiments can be combined with each other. The embodiments will be described in detail below in conjunction with the accompanying drawings.
[0040] The present disclosure proposes a target recognition and positioning method based on zero-sample detection to address the problems that the current mainstream detection and recognition algorithms rely heavily on pre-training data, cannot detect objects outside the pre-training data, and have difficulty in positioning. The zero-sample segmentation algorithm solves the problem of pre-training dependence, poor accuracy, and wrong positioning of target detection. The combination of the classification network and out-of-distribution object detection (OOD) solves the object recognition problem caused by the introduction of the zero-sample segmentation algorithm. At the same time, it further gives unknown class prompts for objects outside the training data that cannot be detected by the mainstream target detection method. Finally, zero-sample object detection, spatial positioning, and target recognition tasks are achieved. This is explained below with specific embodiments.
[0041] Example 1
[0042] In the technical solutions disclosed in one or more embodiments, Figure 1 As shown in FIG7 , a target recognition and positioning method based on zero-sample detection includes the following steps:
[0043] Step 1: Obtain image data to be identified;
[0044] Step 2: Based on the interactive information, a target bounding box is initially generated to obtain a target area image;
[0045] Step 3: Based on the generated target area image, a classification model including out-of-distribution object detection is used to classify the target to obtain the classification category of the target;
[0046] Step 4: Input the obtained bounding box as interactive information into the SAM module for image segmentation to obtain the segmentation mask of the target;
[0047] Step 5: Obtain the depth image of the image to be identified, perform ground filtering, and perform depth value filtering based on the segmentation mask to obtain the spatial information of the target object;
[0048] The above scheme of this embodiment realizes zero-sample target detection, can determine the position and size of the target in the image to be identified, and then determine the spatial position of the target, and realize the target positioning model of the target object spatial information. No training data is required to train the model. Even if the sample to be detected is a sample with a new target, the spatial information of the target can be located, that is, the positioning and detection of the newly appeared target can be realized, and zero-sample target detection is realized. In addition, the new target is identified based on out-of-distribution detection and will not be classified into the existing category in the model, which improves the accuracy of target classification.
[0049] The above steps are implemented based on the constructed target recognition and positioning model, such as Figure 2 As shown, the target recognition and positioning model includes an interactive information generation part and a bounding box recognition module; the bounding box recognition module is respectively connected to the target recognition branch and the target positioning branch;
[0050] The target recognition branch includes the image slicing unit, the Visual Transformer (referred to as ViT network) and the out-of-distribution detection module (Virtual-logit Matching, referred to as ViM) set after the target slicing unit; the Visual Transformer (ViT) is used as the classification network, combined with Virtual-logit Matching (ViM) for out-of-distribution detection; the target recognition branch is used to execute step 3 for target classification and obtain the category information of the corresponding object;
[0051] The target positioning branch includes a SAM network, a depth image acquisition module, a ground filtering module, a depth value filtering module and a spatial information acquisition module which are connected in sequence. The target positioning branch is used to implement steps 4 and 5.
[0052] In some embodiments, the interaction information generation part may use an instance segmentation algorithm;
[0053] Specifically, this embodiment uses the real-time instance segmentation algorithm YOLACT (You Only Look At CoefficienTs) to extract frame interaction information, including the following steps:
[0054] Step 21, obtaining an image to be segmented, such as an RGB image;
[0055] Step 22: Extract features from the acquired image to obtain a feature map. A convolutional neural network (such as ResNet-50) can be used to extract feature maps from the input image.
[0056] Step 23, generate a set number of original masks through the feature map;
[0057] In this step, the mask of each instance is not directly generated, but a set of prototype masks (PrototypeMasks) are generated first and then assigned to each instance;
[0058] Step 24: Use the detection head to predict the bounding box of each instance from the feature map; and predict the linear combination coefficient of each instance and the prototype mask;
[0059] Step 25: Generate instance mask: Linearly combine the original mask with the predicted one to generate the final mask for each instance.
[0060] Step 26, interaction information generation: Extract interaction information based on the generated instance mask and bounding box, and use the position and size of the bounding box as the box interaction information.
[0061] Through the above process, YOLACT can quickly and accurately achieve image segmentation and generate interactive information for further processing. The bounding box generated based on YOLACT is shown in Figure 3(a).
[0062] In step 2, by Figure 2 The bounding boxes initially generated in the interactive information generation part of the algorithm may be repeatedly labeled in different parts of the same object. As shown in Figure 3(a), the bounding boxes overlap;
[0063] A further technical solution uses the area ratio and the overlap between the bounding boxes to determine the subordinate relationship of the bounding boxes, thereby preventing repeated annotation of the same object and removing the repeated annotation of the bounding boxes. The specific process is as follows:
[0064] Step 2-1, set the ratio threshold of the bounding box, and remove the bounding boxes that exceed the ratio threshold according to the ratio threshold;
[0065] The area of the image to be processed is A frame , the area of a bounding box after marking the bounding box is Abox, then the area ratio r is defined as:
[0066]
[0067] Optionally, a proportion threshold, including upper and lower limits of the area proportion, directly removes bounding boxes with an area proportion less than 5% (r<0.05) and greater than 80% (r>0.8);
[0068] Step 2-2, loop through all the remaining bounding boxes, calculate the overlap of any two bounding boxes according to the intersection area of the two bounding boxes, and remove the bounding box with the smaller area between the two bounding boxes that exceeds the overlap threshold, until the overlap of all bounding boxes meets the requirement;
[0069] The overlap ratio is calculated using the Degree Of Overlap (DOO);
[0070] For any two bounding boxes B i and B j , and its overlap is defined as follows:
[0071]
[0072] Among them, Area(B i ∩B j ) represents the intersection area of two bounding boxes, Area(B i ) and Area(B j ) represent the areas of the two bounding boxes respectively. The overlap measures the proportion of the overlapping parts of the two bounding boxes relative to the smaller box.
[0073] Given a bounding box set {B 1 ,B 2 ,...,B n}, the goal is to remove all the things that make DOO(B i ,B j )>threshold, where threshold is the set overlap threshold.
[0074] After processing, we can get a unique bounding box for each target object. By inputting these bounding boxes as interaction information into SAM for image segmentation, we can get a complete segmentation mask for the target. As shown in Figure 4(a), the bounding box result is optimized based on the overlap, and the example effect after removing the repeated annotation of the bounding box is shown in Figure 4(b). This is an example of the segmentation effect based on the optimized bounding box interaction information. After segmentation, the integrity of the target can be guaranteed, and the same target will not be segmented into different parts. Figure 3(b) is the segmentation result obtained under the box interaction information of the repeated bounding box in Figure 3(a), and some target objects are incomplete.
[0075] In step 4, image segmentation is performed to obtain a complete segmentation mask of the target, using the Segment Anything Model (SAM module);
[0076] The SAM module is a suggestible model designed for general image segmentation, which can perform image segmentation without pre-training data, i.e., zero-sample segmentation.
[0077] The SAM module is a model that relies on interactive information. The interactive information supported by SAM includes multiple modes. In this embodiment, stable, efficient and fast frame interactive information is used as the input interactive information of the model.
[0078] Step 3: Use Figure 2 The upper branch in is the target recognition branch, which uses Visual Transformer (ViT) as the classification network and combines Virtual-logit Matching (ViM) for out-of-distribution detection.
[0079] Visual Transformer (ViT) is a model that applies the Transformer architecture to image classification tasks. ViT uses the self-attention mechanism of the Transformer architecture to capture long-range dependencies in images.
[0080] Virtual-logit Matching (ViM) is an algorithm for out-of-distribution detection (OOD). Its purpose is to identify data that the model has not seen and ensure that the model does not make wrong high-confidence predictions when processing this data.
[0081] Specifically, the target classification method in step 3 includes the following steps:
[0082] Step 31, obtain the image slice sample to be detected, send it to the ViT classification network for feature extraction, and obtain the identified target category;
[0083] Step 32: Process the feature map output by the ViT classification network through the ViM algorithm to obtain the detection score of the image slice sample to be detected;
[0084] Step 33: According to the distribution threshold of the in-distribution samples obtained through training, determine whether the sample to be detected belongs to the out-of-distribution sample; if it belongs to the out-of-distribution sample, mark the target classification result as unknown category; otherwise, output the category label predicted by the ViT classification network.
[0085] There is usually a difference between the scores of samples within the distribution and samples outside the distribution after ViM algorithm. A threshold is set according to the score distribution of samples within the distribution, and the threshold is used to guide the classification of whether a new sample belongs to a sample outside the distribution.
[0086] Specifically, if the detection score of the sample to be detected is within the distribution threshold of the sample within the distribution, it belongs to the sample within the distribution; otherwise, it belongs to the sample outside the distribution;
[0087] Optionally, in step 33, the method for training and determining the distribution threshold of samples within the distribution includes the following steps:
[0088] Step 331: According to the actual classification task, obtain the data samples of the samples in the distribution to construct the training set, and perform transfer learning on the ViT classification network;
[0089] The trained ViT classification network is used to identify the target category in the image to be identified;
[0090] In the process of transfer learning, the object samples of the category we hope to obtain are called in-distribution samples, and the object samples outside of them are called out-of-distribution samples.
[0091] Step 332: extract features of the in-distribution samples involved in the transfer learning process based on the ViT classification network, and perform out-of-distribution detection using the ViM algorithm to determine the distribution threshold to be set based on the scores of the in-distribution samples;
[0092] Specifically, the in-distribution samples (ID) involved in the transfer learning process are sent to the ViT classification network for feature extraction. The extracted features are tested out of the distribution using the ViM algorithm to obtain the detection scores of the in-distribution samples, and then the distribution of the sample scores is obtained. The distribution threshold is set according to the scores of the in-distribution samples.
[0093] For example, the target object categories of the data samples of the constructed training set include: "table", "chair", "person" and "book", which are in-distribution samples; when the identified targets in the image are "vehicle", "ship", etc., they are out-of-distribution samples; when the trained ViT classification network is used for classification, the category of the out-of-distribution samples will be divided into any one of the training categories, such as the label of the "vehicle" in the image after identification may be the wrong category of "book". In this embodiment, out-of-distribution detection is added. When an out-of-distribution sample appears, it is identified as an out-of-distribution sample target through out-of-distribution detection, rather than being identified as a wrong category, that is, the label of the "vehicle" in the image after identification is "unknown", but it is not any of "table", "chair", "person" and "book".
[0094] In this embodiment, a VIT classification network and an out-of-distribution detection module (ViM) are set up at the same time to output accurate categories for samples within the distribution, and identify out-of-distribution samples as unknown categories, so that the unknown categories will not be randomly assigned to the sample categories during pre-training, thereby improving the accuracy of target object category recognition.
[0095] Step 5 is the target positioning and detection phase, using Figure 2 The lower half branch in is the target location branch.
[0096] Step 5: Perform ground filtering on the depth image to implement preprocessing of the depth image, including the following:
[0097] Step 51, calculate the depth range of the camera image according to the camera parameters, and obtain the longitudinal distance value Y of the ground point according to the set depth step;
[0098] The relative position of the camera and the camera carrier remains unchanged, and the camera is fixedly mounted on the camera carrier. The following figure is a schematic diagram of the geometric relationship between the camera and the ground:
[0099] The horizontal and vertical field of view angles are calculated through the camera internal parameter matrix K. The horizontal field of view angle (HFOV) is used. h As the symbol in the formula; Vertical field of view angle (abbreviated as VFOV), FOV v As the symbol in the formula and based on the maximum longitudinal detection range of the camera [y min ,y max ].
[0100] As well as the camera's installation height h and pitch angle θ, the minimum detection distance can be obtained for:
[0101]
[0102] According to the comparison between the pitch angle and VFOV, the maximum detection distance is determined as:
[0103]
[0104] To generate discrete ground points, in the depth range A series of longitudinal distance values Y are generated with a step length of 0.1 m.
[0105] Among them, the longitudinal distance value Y is: the longitudinal distance from the simulated ground point to the camera;
[0106] Step 52: Calculate the horizontal distance of the ground corresponding to the longitudinal distance value Y according to the obtained longitudinal distance value Y, extract the ground point, that is, obtain the ground point in the camera coordinate system, project it to the image coordinate system, and obtain the pixel point corresponding to the ground point in the image to construct the point set data;
[0107] The horizontal distance on the ground corresponding to each depth value is:
[0108]
[0109] In this horizontal range, the corresponding ground horizontal discrete point X is generated with a step size of 0.1 meters. At this point, a set of discrete point sets will be obtained, which contains the ground points (X, Y, -h) in the simulated camera coordinate system.
[0110] By projecting these ground points in the camera coordinate system to the image coordinate system, these pixels are retained if they are within the image range.
[0111] Step 53: Calculate the depth value Dep of the ground point according to the point set data to obtain a simulated ground discrete point set with depth values:
[0112]
[0113] Where k is a constant calculated from the internal parameter.
[0114] Through the above process, we can get a simulated ground discrete point set with depth values, which can be visualized as follows Figure 6 As shown, the positions of the simulated discrete ground points after conversion to the image coordinate system are shown as white dots in the figure.
[0115] Step 54: upsample the discrete point set data, perform depth interpolation filling between the sampling points, so that the resolution of the interpolated image matches the resolution of the depth image, and obtain a list of ground point depth information;
[0116] The above upsampling realizes filling sparse data into dense data;
[0117] Optionally, the depth interpolation filling may adopt a row-by-row interpolation or a column-by-column interpolation method;
[0118] Optional, row-by-row interpolation method, which can start from the bottom row of the image, traverse upward row by row, find the ground point, determine the left and right boundaries, and perform depth interpolation;
[0119] Specifically, first, start from the bottom row of the image and traverse upward row by row. For each row, find all ground points with pixel values and sort the ground points by horizontal coordinates, denoted as x list Then, by minimizing x left and the maximum value x right, determine the left and right boundaries of each row.
[0120] Because the ground sampling is sparse points, the actual ground points projected into the image are still sparse, such as Figure 6 As shown in the figure, most of the image pixel values are 0 (black). Only when the simulated discrete ground point is projected onto a pixel in the image coordinate system, the point has a pixel value. The value is the depth Dep or the distance from the simulated ground point to the camera, which is the white point on the figure. Find all the ground points with pixel values. Figure 6 The white dots in the
[0121] Between the left and right boundaries, check the horizontal coordinate x of each point in turn tem , for each x tem , determine whether it is already a sampling point; if so, directly add the point and its corresponding depth value to the ground point depth information list; otherwise, find the sampling points x on its left and right neighbors left_num and x right_num , and interpolate its depth value, the depth step length D between two sampling points value The calculation method is:
[0122]
[0123] Among them, Dep[x] is the depth value corresponding to point x. According to the step size, the point uniform difference between the sampling points is calculated. For each point x to be interpolated, the depth value is:
[0124] Dep[x]=Dep[x left_num ]+D value ·(xx left_num )
[0125] Optional, column-by-column interpolation method, which can start from the leftmost column of the image, traverse column by column to the right, find the ground point, determine the upper and lower boundaries, and perform depth interpolation;
[0126] After completing the horizontal interpolation, perform similar processing for each column, starting from the leftmost column of the image and traversing column by column to the right. For each column, find all ground points with pixel values and sort them by vertical coordinates. Then, determine the upper and lower boundaries of each column through the minimum and maximum values.
[0127] Between the upper and lower boundaries, check each ordinate in turn. If it is not a sampling point, find its upper and lower neighboring sampling points and interpolate them using a similar method. The calculation process is exactly the same as the horizontal interpolation method.
[0128] After interpolation, the depth information of all ground points constitutes a ground point depth information list;
[0129] Step 55: Match the points in the ground point depth information list with the corresponding pixels of the depth image to be processed. The successfully matched points are filtered as ground points to obtain a depth image without ground data.
[0130] FIG7(a) shows the original visualization of the depth map, and FIG7(b) shows the effect after ground filtering of FIG7(a).
[0131] Acquiring the depth map is based on stereo matching technology. This method will cause a flickering phenomenon like snowflakes on a black-and-white TV in the corresponding area when the object is too close or too far away from the camera or there is reflection. At this time, the depth value output from this area is unstable and incorrect, and objects with incomplete or even wrong depth information will appear. To address this problem, this embodiment performs depth value filtering after ground filtering to exclude objects with incomplete or even wrong depth information due to matching characteristics. Such objects are usually outside the detection range of the binocular camera and the detection task.
[0132] Furthermore, the depth value filtering method extracts the depth values in all masked areas from the ground-filtered depth map D through the segmentation mask M, filters the depth value points in the non-masked area, and locates the target object in the depth map.
[0133] Segmentation mask data output by the SAM module is a depth map Binary matrix of the same dimension, element M i,j 1 indicates that the pixel at the corresponding position is within the mask, and 0 indicates that the pixel is outside the mask. Each element D i,j Represents the depth value at pixel (i, j).
[0134] The depth values in all masked areas are extracted from the depth map D through the mask M. The remaining data after removing invalid points and points outside the range are called valid depth points. If the proportion of valid depth points to the total depth information set of the depth map D is less than the threshold of 75%, it is considered that the object corresponding to the masked area is not within the detection range, and the target is skipped.
[0135] The list of objects retained after calculation and screening is the object to be located. The distance information is obtained by sorting the effective depth points from small to large according to the depth value d to obtain an ordered list. Wherein n is the number of valid depth values d in the list L, that is, n=|L|.
[0136] According to the length n of the list L, calculate the median of the depth value of the target object as the distance D of the object from the camera obj , the calculation formula is as follows:
[0137]
[0138] Wherein, d represents the effective depth value;
[0139] Furthermore, for the target object located in the depth image, the centroid and width of the target object are further solved through the segmentation mask M;
[0140] The row coordinates and column coordinates of the non-zero pixels in the segmentation mask M can be expressed as rows and cols respectively;
[0141] Barycentric coordinates (x center ,y center ) is obtained by calculating the mean of the column coordinates and the mean of the row coordinates:
[0142]
[0143] Among them, N clos and N rows are the total number of non-zero columns and non-zero rows in the masked region, respectively;
[0144] The width W of the target object is determined by the leftmost pixel x leftmost and the rightmost pixel x rightmost The difference between them is determined;
[0145] Furthermore, the offset ratio R of the center of gravity of the target object in the width direction is center , the formula is:
[0146]
[0147] At this point, detection information such as the category, spatial orientation, and related dimensions of the target object can be obtained.
[0148] Example 2
[0149] Based on Example 1, this embodiment provides a target recognition and positioning system based on zero-sample detection, including:
[0150] An acquisition module: configured to acquire image data to be identified;
[0151] The interactive information part is configured to preliminarily generate a target bounding box based on the interactive information to obtain a target area image;
[0152] Target classification module: configured to perform target classification based on the generated target area image using a classification model including out-of-distribution object detection to obtain a classification category of the target;
[0153] Mask segmentation module: configured to input the obtained bounding box as interactive information into the SAM module for image segmentation to obtain the segmentation mask of the target;
[0154] Target recognition module: It is configured to obtain the depth image of the image to be recognized, perform ground filtering, filter the depth value based on the segmentation mask, and obtain the spatial information of the target object.
[0155] It should be noted here that the various modules in this embodiment correspond one-to-one to the various steps in Example 1, and the specific implementation process is the same, which will not be repeated here.
[0156] Example 3
[0157] This embodiment provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor. When the computer instructions are run by the processor, the steps in the target recognition and positioning method based on zero-sample detection in Embodiment 1 are completed.
[0158] Example 4
[0159] This embodiment provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps in the target recognition and positioning method based on zero-sample detection in Embodiment 1 are completed.
[0160] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
[0161] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Technical personnel in the relevant field should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.
Claims
1. A target recognition and positioning method based on zero-sample detection, characterized in that: The steps include: Acquire image data to be recognized; Based on the interactive information, a target bounding box is initially generated to obtain the target area image; Based on the generated target area image, a classification model including out-of-distribution object detection is used to classify the target to obtain a classification category of the target; The obtained bounding box is used as the interactive information to input into the SAM module for image segmentation to obtain the segmentation mask of the target; Obtain the depth image of the image to be identified, perform ground filtering, filter the depth value based on the segmentation mask, and obtain the spatial information of the target object.
2. The target recognition and positioning method based on zero-sample detection according to claim 1, characterized in that: Construct a target recognition and positioning model, including the interactive information generation part, the bounding box recognition module, the target recognition branch and the target positioning branch; The interactive information generation part is connected to the bounding box recognition module, and the bounding box recognition module is respectively connected to the target recognition branch and the target localization branch; The target recognition branch includes an image slicing unit, a ViT network set after the target slicing unit, and an out-of-distribution detection module; The target positioning branch includes a SAM network, a depth image acquisition module, a ground filtering module, a depth value filtering module and a spatial information acquisition module which are connected in sequence.
3. The target recognition and positioning method based on zero-sample detection according to claim 1, characterized in that: The method of removing duplicated marked bounding boxes is based on the area ratio and the overlap between the bounding boxes. The specific process is as follows: Set the ratio threshold of the bounding box, and remove the bounding boxes that exceed the ratio threshold according to the ratio threshold; Loop through all the remaining bounding boxes, calculate the overlap of any two bounding boxes based on the intersection area of the two bounding boxes, and remove the bounding box with the smaller area between the two bounding boxes that exceeds the overlap threshold until the overlap of all bounding boxes meets the requirement.
4. The target recognition and positioning method based on zero-sample detection according to claim 1, characterized in that: Use the SAM module to segment the image and obtain a complete segmentation mask of the target; Alternatively, the method for target classification comprises the following steps: Obtain the image slice sample to be detected, send it to the ViT classification network for feature extraction, and obtain the identified target category; The feature map output by the ViT classification network is processed by the ViM algorithm to obtain the detection score of the image slice sample to be detected; According to the distribution threshold of the in-distribution samples obtained through training, it is determined whether the sample to be detected belongs to the out-of-distribution sample; if it belongs to the out-of-distribution sample, the target classification result is marked as unknown category; otherwise, the category label predicted by the ViT classification network is output.
5. The target recognition and positioning method based on zero-sample detection according to claim 1, characterized in that: The method for training and determining the distribution threshold of samples within a distribution comprises the following steps: According to the actual classification task, we obtain the data samples of the distribution samples to construct the training set, and perform transfer learning on the ViT classification network; The in-distribution samples involved in the transfer learning process are feature extracted based on the ViT classification network, and out-of-distribution detection is performed using the ViM algorithm to determine the distribution threshold set by the score of the in-distribution samples.
6. The target recognition and positioning method based on zero-sample detection according to claim 1, characterized in that: Perform ground filtering on the depth image, including the following: The depth range of the camera image is calculated according to the camera parameters, and the longitudinal distance value of the ground point is obtained according to the set depth step length; According to the obtained longitudinal distance value, the ground lateral distance corresponding to the longitudinal distance value is calculated, and the ground point is extracted to obtain the ground point in the camera coordinate system, which is projected to the image coordinate system to obtain the pixel points corresponding to the ground point in the image to construct the point set data; Calculate the depth value of the ground point according to the point set data to obtain a simulated ground discrete point set with depth value: Upsample the discrete point set data and perform depth interpolation filling between the sampling points so that the resolution of the interpolated image matches that of the depth image, and obtain a list of ground point depth information; The points in the ground point depth information list are matched with the corresponding pixels of the depth image to be processed. The successfully matched points are filtered as ground points to obtain a depth image without ground data.
7. The target recognition and positioning method based on zero-sample detection according to claim 1, characterized in that: The depth value filtering method extracts the depth values in all masked areas from the ground-filtered depth map D through the segmentation mask M, filters the depth value points in the non-masked area, and locates the target object in the depth map; For the target object in the located depth image, the centroid and width of the target object are further solved through the segmentation mask M to obtain the spatial information of the target.
8. The target recognition and positioning system based on zero-sample detection is characterized by: include: An acquisition module: configured to acquire image data to be identified; The interactive information part is configured to preliminarily generate a target bounding box based on the interactive information to obtain a target area image; A target classification module is configured to classify the target based on the generated target area image using a classification model including out-of-distribution object detection to obtain a classification category of the target; Mask segmentation module: configured to input the obtained bounding box as interactive information into the SAM module for image segmentation to obtain the segmentation mask of the target; Target recognition module: It is configured to obtain the depth image of the image to be recognized, perform ground filtering, filter the depth value based on the segmentation mask, and obtain the spatial information of the target object.
9. An electronic device, characterized in that: The method comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps in the target recognition and positioning method based on zero-sample detection according to any one of claims 1 to 7 are completed.
10. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the steps of the target recognition and positioning method based on zero-sample detection as described in any one of claims 1 to 7.
Citation Information
Patent Citations
Target space positioning method
CN110599489A
Topic accurate segmentation method and device based on deep learning, equipment and medium
CN111652140A
Computer-implemented method, computer program product and system for analysis of cell images
CN112655003A
Mobile robot target positioning method and system
CN115496898A
Cited By
Intelligent manufacturing anomaly detection method based on intrinsic parameter learning
CN119669866A
An intelligent manufacturing anomaly detection method based on intrinsic parameter learning
CN119669866B
Target positioning method for mobile terminal
CN119784827A
A target positioning method for mobile terminals
CN119784827B
