Target recognition and positioning method and system based on zero-sample detection

Abstract

The present disclosure relates to the field of computer vision technology and proposes a target recognition and positioning method and system based on zero-shot detection, comprising the following steps: obtaining image data to be recognized; preliminarily generating a target bounding box based on interaction information to obtain a target region image; classifying the target using a classification model including out-of-distribution object detection based on the generated target region image to obtain a classification category of the target; inputting the obtained bounding box as interaction information into a SAM module to perform image segmentation to obtain a segmentation mask of the target; obtaining a depth image of the image to be recognized, performing ground filtering, and filtering depth values based on the segmentation mask to obtain spatial information of the target object. The present disclosure solves the problems of pre-training dependency, poor target detection accuracy, and incorrect positioning of target detection, and solves the object recognition problem caused by the introduction of a zero-shot segmentation algorithm by combining a classification network with out-of-distribution object detection.

Description

Technical Field

[0001] The present disclosure relates to the technical field related to computer vision, and more specifically, to a method and system for target recognition and positioning based on zero-sample detection. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] Object detection and recognition technology is an important research direction in the field of computer vision. In recent years, it has been widely used in many fields such as autonomous driving, intelligent monitoring, face recognition, medical image analysis, and drone navigation.

[0004] The inventors found in their research that the current mainstream target detection methods usually rely on pre-trained weights to perform target detection and use the detected bounding box information for positioning. However, the detection effect of this method is heavily dependent on the data used for pre-training, and it is unable to detect objects outside the pre-training data. At the same time, the bounding box used by the mainstream detection method is just a simple rectangle, which cannot accurately describe the shape and boundary of the target object. For objects with a small area of the minimum circumscribed rectangle, such as the Y-shape, there are poor positioning accuracy or even wrong positioning. At the same time, with the expansion of the scope of application, the large amount of data and labor cost required for pre-training have gradually become prominent, and there is still the problem of positioning difficulty in the detection of certain special-shaped objects; and, based on the model training of labeled data, the category recognition of the target, for out-of-distribution samples, will be identified within the category set by the model, which reduces the accuracy of target category recognition. Summary of the Invention

[0005] In order to solve the above problems, the present disclosure proposes a target recognition and positioning method based on zero-shot detection, which solves the problems of pre-training dependence, poor target detection accuracy and incorrect positioning of target detection. By combining the classification network with out-of-distribution object detection (OOD), the object recognition problem caused by the introduction of the zero-shot segmentation algorithm is solved.

[0006] In order to achieve the above objectives, the present disclosure adopts the following technical solutions:

[0007] One or more embodiments provide a target recognition and positioning method based on zero-sample detection, comprising the following steps:

[0008] Acquire image data to be recognized;

[0009] Based on the interactive information, the target bounding box is preliminarily generated to obtain the target area image;

[0010] Based on the generated target area image, a classification model including out-of-distribution object detection is used to classify the target to obtain the classification category of the target;

[0011] The obtained bounding box is input into the SAM module as interactive information for image segmentation to obtain the target segmentation mask;

[0012] Obtain the depth image of the image to be identified, perform ground filtering, and filter the depth value based on the segmentation mask to obtain the spatial information of the target object.

[0013] One or more embodiments provide a target recognition and positioning system based on zero-sample detection, including:

[0014] Acquisition module: configured to acquire image data to be identified;

[0015] Interaction information part: is configured to preliminarily generate a target bounding box based on the interaction information and obtain a target area image;

[0016] Target classification module: configured to perform target classification based on the generated target area image using a classification model including out-of-distribution object detection to obtain a classification category of the target;

[0017] Mask segmentation module: is configured to input the obtained bounding box as interactive information into the SAM module for image segmentation to obtain the target segmentation mask;

[0018] Target recognition module: It is configured to obtain the depth image of the image to be recognized, perform ground filtering, and filter the depth value based on the segmentation mask to obtain the spatial information of the target object.

[0019] It should be noted here that the various modules in this embodiment correspond one-to-one to the various steps in Example 1, and the specific implementation processes are the same, which will not be repeated here.

[0020] An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps in the above-mentioned target recognition and positioning method based on zero-sample detection are completed.

[0021] A computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the steps in the above-mentioned target recognition and positioning method based on zero-sample detection are completed.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] This disclosure achieves zero-shot target detection, capable of determining the position and size of targets in the image to be identified, and thus determining the spatial position of the target. The target localization model does not require training data to train the model. Even if the sample to be detected is a sample containing a new target, the spatial information of the target can be located, that is, it can achieve the location and detection of newly appeared targets, achieving zero-shot target detection. In addition, new targets are identified based on out-of-distribution detection and are not classified into existing categories in the model, thereby improving the accuracy of target classification.

[0024] The advantages of the present disclosure and additional advantages will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which constitute a part of the present disclosure, are used to provide a further understanding of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure but do not constitute a limitation of the present disclosure.

[0026] Figure 1 is a flow chart of the target identification and positioning method of embodiment 1 of the present disclosure;

[0027] Figure 2 This is a schematic diagram of the structure of the target recognition and positioning model of Example 1 of the present disclosure;

[0028] FIG3( a ) is an example effect diagram of generating a bounding box based on YOLACT according to Example 1 of the present disclosure;

[0029] FIG3( b ) is a diagram illustrating an example of segmentation based on the repeatedly labeled bounding box labeled in FIG3( a ) according to Example 1 of the present disclosure;

[0030] FIG4( a ) is an example effect diagram after removing duplicate annotations of the bounding box according to Example 1 of the present disclosure;

[0031] FIG4( b ) is a diagram illustrating an example of segmentation of the interactive information of the frames after removing duplicate annotations of the bounding boxes in FIG4( a ) according to Example 1 of the present disclosure;

[0032] FIG5( a ) is a schematic diagram of the geometric relationship between the camera and the ground in Example 1 of the present disclosure;

[0033] FIG5( b ) is a top view of the positional relationship between the camera and the target object in Example 1 of the present disclosure;

[0034] Figure 6 is a visualization image of a simulated ground discrete point set with depth values in Example 1 of the present disclosure;

[0035] FIG7( a ) is an unprocessed depth image in Example 1 of the present disclosure;

[0036] FIG7( b ) is a depth image obtained after the depth image in Example 1 of the present disclosure is ground filtered. DETAILED DESCRIPTION

[0037] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0038] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present disclosure belongs.

[0039] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof. It should be noted that, in the absence of conflict, the various embodiments in the present disclosure and the features in the embodiments can be combined with each other. The embodiments will be described in detail below with reference to the accompanying drawings.

[0040] This disclosure proposes a target recognition and positioning method based on zero-sample detection to address the problems that current mainstream detection and recognition algorithms rely heavily on pre-training data, cannot detect objects outside the pre-training data, and have difficulty in positioning. The zero-sample segmentation algorithm solves the problem of pre-training dependence, poor accuracy, and incorrect positioning of target detection. The combination of the classification network and out-of-distribution object detection (OOD) solves the object recognition problem caused by the introduction of the zero-sample segmentation algorithm. At the same time, it further provides unknown class prompts for objects outside the training data that cannot be detected by mainstream target detection methods. Ultimately, zero-sample object detection, spatial positioning, and target recognition tasks are achieved. This will be explained below with specific embodiments.

[0041] Example 1

[0042] In the technical solutions disclosed in one or more embodiments, Figure 1 As shown in FIG7 , a target recognition and positioning method based on zero-sample detection includes the following steps:

[0043] Step 1: Obtain image data to be identified;

[0044] Step 2: Based on the interaction information, a target bounding box is preliminarily generated to obtain the target area image;

[0045] Step 3: Based on the generated target area image, a classification model including out-of-distribution object detection is used to classify the target to obtain the classification category of the target;

[0046] Step 4: Input the obtained bounding box as interactive information into the SAM module for image segmentation to obtain the target segmentation mask;

[0047] Step 5: Obtain the depth image of the image to be identified, perform ground filtering, and filter the depth value based on the segmentation mask to obtain the spatial information of the target object;

[0048] The above-mentioned scheme of this embodiment achieves zero-shot target detection, capable of determining the position and size of the target in the image to be identified, and thus determining the target's spatial position. This target localization model, which measures the spatial information of the target object, does not require training data to train the model. Even if the sample to be detected contains a new target, the target's spatial information can be located, thus enabling the location and detection of newly appeared targets, thus achieving zero-shot target detection. Furthermore, new targets are identified based on out-of-distribution detection and are not classified into existing categories in the model, thereby improving the accuracy of target classification.

[0049] The above steps are implemented based on the constructed target recognition and positioning model, such as Figure 2 As shown in FIG, the target recognition and positioning model includes an interactive information generation part and a bounding box recognition module; the bounding box recognition module is connected to the target recognition branch and the target positioning branch respectively;

[0050] The object recognition branch includes the image slicing unit, the Visual Transformer (ViT network) set after the object slicing unit, and the out-of-distribution detection module (ViM). The Visual Transformer (ViT) is used as the classification network, combined with Virtual-logit Matching (ViM) for out-of-distribution detection. The object recognition branch is used to perform step 3 to classify the object and obtain the corresponding object category information.

[0051] The target positioning branch includes a SAM network, a depth image acquisition module, a ground filtering module, a depth value filtering module and a spatial information acquisition module connected in sequence. The target positioning branch is used to implement steps 4 and 5.

[0052] In some embodiments, the interaction information generation part may use an instance segmentation algorithm;

[0053] Specifically, this embodiment uses the real-time instance segmentation algorithm YOLACT (You Only Look At CoefficienTs) to extract box interaction information, including the following steps:

[0054] Step 21: Obtain an image to be segmented, such as an RGB image;

[0055] Step 22: Extract features from the acquired image to obtain feature maps. A convolutional neural network (e.g., ResNet-50) can be used to extract feature maps from the input image.

[0056] Step 23: Generate a set number of original masks through the feature map;

[0057] In this step, instead of directly generating a mask for each instance, a set of prototype masks (PrototypeMasks) is generated first and then assigned to each instance;

[0058] Step 24: Use the detection head to predict the bounding box of each instance from the feature map; and also predict the linear combination coefficient of each instance and the prototype mask.

[0059] Step 25: Generate instance mask: Linearly combine the original mask with the predicted one to generate the final mask for each instance.

[0060] Step 26: Interaction information generation: Based on the generated instance mask and bounding box, extract the interaction information, and use the position and size of the bounding box as the box interaction information.

[0061] Through the above process, YOLACT can quickly and accurately achieve image segmentation and generate interactive information for further processing. The bounding box generated based on YOLACT is shown in Figure 3(a).

[0062] In step 2, by Figure 2 The bounding boxes initially generated by the interactive information generation part in the algorithm may be repeatedly labeled in different parts of the same object. As shown in Figure 3(a), the bounding boxes overlap;

[0063] A further technical solution uses the area ratio and the overlap between the bounding boxes to determine the subordinate relationship of the bounding boxes, thereby preventing repeated annotation of the same object. The specific process is as follows:

[0064] Step 2-1: Set the ratio threshold of the bounding box, and remove the bounding boxes that exceed the ratio threshold according to the ratio threshold;

[0065] The area of the image to be processed is A frame , after marking the bounding box, the area of a bounding box is Abox, and the area ratio r is defined as:

[0066]

[0067] Optionally, a proportion threshold, including upper and lower limits of area proportion, will directly remove bounding boxes with area proportions less than 5% (r < 0.05) and greater than 80% (r > 0.8);

[0068] Step 2-2: Loop through all remaining bounding boxes, calculate the overlap of any two bounding boxes based on the intersection area of the two bounding boxes, and remove the bounding box with the smaller area between the two bounding boxes that exceeds the overlap threshold until the overlap of all bounding boxes meets the requirement;

[0069] The overlap ratio is calculated using the degree of overlap (DOO).

[0070] For any two bounding boxes B i and B j , and its overlap is defined as follows:

[0071]

[0072] Among them, Area(B i ∩B j ) represents the intersection area of the two bounding boxes, Area(B i ) and Area(B j ) represent the areas of the two bounding boxes respectively. The overlap measures the ratio of the overlapping parts of the two bounding boxes relative to the smaller box.

[0073] Given a set of bounding boxes {B1,B2,...,B n}, the goal is to remove all the things that make DOO(B i ,B j )>threshold, where threshold is the set overlap threshold.

[0074] After processing, unique bounding boxes are obtained for each target object. These bounding boxes are input into SAM as interaction information for image segmentation, resulting in a complete segmentation mask for the target. As shown in Figure 4(a), the bounding box results are optimized based on the degree of overlap, removing duplicate bounding box annotations. Figure 4(b) shows an example of the segmentation effect based on the optimized bounding box interaction information. This ensures target integrity after segmentation, preventing the same target from being divided into different parts. Figure 3(b) shows the segmentation result obtained using the repeated bounding box interaction information in Figure 3(a), showing that some target objects are incomplete.

[0075] In step 4, image segmentation is performed to obtain a complete segmentation mask of the target, using the Segment Anything Model (SAM module);

[0076] The SAM module is a suggestible model designed for general image segmentation, which can perform image segmentation without pre-training data, that is, zero-sample segmentation.

[0077] The SAM module is a model that relies on interactive information. The interactive information supported by SAM includes multiple methods. In this embodiment, stable, efficient and fast frame interactive information is used as the input interactive information of the model.

[0078] Step 3: Use Figure 2 The upper branch in

[15] is the target recognition branch, which uses Visual Transformer (ViT) as the classification network and combines Virtual-logit Matching (ViM) for out-of-distribution detection.

[0079] Visual Transformer (ViT) is a model that applies the Transformer architecture to image classification tasks. ViT uses the self-attention mechanism of the Transformer architecture to capture long-range dependencies in images.

[0080] Virtual-logit Matching (ViM) is an algorithm for out-of-distribution (OOD) detection. Its purpose is to identify data that the model has not seen and ensure that the model does not produce incorrect high-confidence predictions when processing such data.

[0081] Specifically, the target classification method in step 3 includes the following steps:

[0082] Step 31: Obtain the image slice sample to be detected and send it to the ViT classification network for feature extraction to obtain the identified target category;

[0083] Step 32: Process the feature map output by the ViT classification network through the ViM algorithm to obtain the detection score of the image slice sample to be detected;

[0084] Step 33: Based on the distribution threshold of the in-distribution samples obtained through training, determine whether the sample to be tested is an out-of-distribution sample; if it is an out-of-distribution sample, mark the target classification result as an unknown category; otherwise, output the category label predicted by the ViT classification network.

[0085] There is usually a difference in the scores of in-distribution samples and out-of-distribution samples after the ViM algorithm. A threshold is set according to the score distribution of in-distribution samples, and this threshold is used to guide the classification of whether a new sample belongs to the out-of-distribution sample.

[0086] Specifically, if the detection score of the sample to be detected is within the distribution threshold of the sample within the distribution, it is considered an in-distribution sample; otherwise, it is considered an out-of-distribution sample;

[0087] Optionally, in step 33, the method for training and determining the distribution threshold of samples within the distribution includes the following steps:

[0088] Step 331: According to the actual classification task, obtain data samples of the distribution samples to construct the training set, and perform transfer learning on the ViT classification network;

[0089] The trained ViT classification network is used to identify the target category in the image to be identified;

[0090] In the transfer learning process, the object samples of the category we hope to obtain are called in-distribution samples, and the object samples outside of the category are called out-of-distribution samples.

[0091] Step 332: Extract features of the in-distribution samples involved in the transfer learning process based on the ViT classification network, perform out-of-distribution detection using the ViM algorithm, and determine the distribution threshold to be set based on the score of the in-distribution samples;

[0092] Specifically, the in-distribution samples (ID) involved in the transfer learning process are fed into the ViT classification network for feature extraction. The extracted features are then used for out-of-distribution detection using the ViM algorithm to obtain the detection scores of the in-distribution samples. Furthermore, the distribution of the sample scores is obtained, and the distribution threshold is set based on the scores of the in-distribution samples.

[0093] For example, the target object categories of the data samples of the constructed training set include: "table", "chair", "person" and "book", which are in-distribution samples; when the target identified in the image is "vehicle", "ship", etc., it is an out-of-distribution sample; when the trained ViT classification network is used for classification, the category of the out-of-distribution sample will be divided into any one of the training categories, such as the label after the "vehicle" in the image may be the wrong category output of "book". In this embodiment, out-of-distribution detection is added. When an out-of-distribution sample appears, it is identified as an out-of-distribution sample target through out-of-distribution detection, rather than being identified as an incorrect category, that is, the label after the "vehicle" in the image is identified is "unknown", but it is not any of "table", "chair", "person" and "book".

[0094] In this embodiment, a VIT classification network and an out-of-distribution detection module (ViM) are set up at the same time to output accurate categories for samples within the distribution, and identify out-of-distribution samples as unknown categories, so that unknown classes will not be randomly assigned to the sample categories during pre-training, thereby improving the accuracy of target object category recognition.

[0095] Step 5 is the target positioning and detection step, using Figure 2 The lower half branch in is the target location branch;

[0096] Step 5: Perform ground filtering on the depth image to implement preprocessing of the depth image, including the following:

[0097] Step 51: Calculate the depth range of the camera image according to the camera parameters, and obtain the longitudinal distance value Y of the ground point according to the set depth step size;

[0098] The relative position of the camera and the camera carrier remains unchanged, and the camera is fixedly mounted on the camera carrier. The following figure shows the geometric relationship between the camera and the ground:

[0099] The horizontal and vertical field of view angles are calculated through the camera intrinsic parameter matrix K. The horizontal field of view angle (HFOV) is used. h As the symbol in the formula; Vertical field of view angle (abbreviated as VFOV), using FOV v As the symbol in the formula and based on the maximum longitudinal detection range of the camera [y min ,y max ].

[0100] As well as the camera's installation height h and pitch angle θ, the minimum detection distance can be obtained for:

[0101]

[0102] Based on the comparison between the pitch angle and VFOV, the maximum detection distance is determined as:

[0103]

[0104] To generate discrete ground points, in the depth range Within, a series of longitudinal distance values Y are generated with a step length of 0.1 meters;

[0105] Among them, the longitudinal distance value Y is: the longitudinal distance from the simulated ground point to the camera;

[0106] Step 52: Calculate the horizontal distance of the ground corresponding to the longitudinal distance value Y based on the obtained longitudinal distance value Y, extract the ground point, that is, obtain the ground point in the camera coordinate system, project it to the image coordinate system, and obtain the pixel points corresponding to the ground point in the image to construct point set data;

[0107] The horizontal distance on the ground corresponding to each depth value is:

[0108]

[0109] Within this horizontal range, the corresponding ground horizontal discrete points X are generated with a step size of 0.1 meters. At this point, a set of discrete points is obtained, which contains the ground points (X, Y, -h) in the simulated camera coordinate system.

[0110] By projecting these ground points in the camera coordinate system to the image coordinate system, these pixels are retained if they are within the image range.

[0111] Step 53: Calculate the depth value Dep of the ground point according to the point set data to obtain a simulated ground discrete point set with depth values:

[0112]

[0113] Where k is a constant calculated from the internal reference.

[0114] Through the above process, a simulated ground discrete point set with depth values can be obtained, which can be visualized as follows Figure 6 As shown, the positions of the simulated discrete ground points after conversion to the image coordinate system are shown as white dots in the figure.

[0115] Step 54: upsample the discrete point set data and perform depth interpolation filling between the sampling points so that the resolution of the interpolated image matches the resolution of the depth image, thereby obtaining a list of ground point depth information;

[0116] The above upsampling realizes filling sparse data into dense data;

[0117] Optionally, depth interpolation filling can adopt row-by-row interpolation or column-by-column interpolation method;

[0118] Alternatively, a row-by-row interpolation method can start from the bottom row of the image and traverse upwards row by row to find the ground points, determine the left and right boundaries, and perform depth interpolation;

[0119] Specifically, first, start from the bottom row of the image and traverse upwards row by row. For each row, find all ground points with pixel values and sort the ground points by horizontal coordinate, denoted as x list Then, by minimizing x left and the maximum value x right , determine the left and right boundaries of each row.

[0120] Because the ground sampling is sparse points, the actual ground points projected into the image are still sparse, such as Figure 6As shown, most of the image pixel values are 0 (black). Only when the simulated discrete ground point is projected onto a pixel in the image coordinate system, the point has a pixel value. The value is Dep or the distance between the simulated ground point and the camera, which is the white point on the figure. Find all the ground points with pixel values. Figure 6 The white dots in

[0121] Between the left and right boundaries, check the horizontal coordinate x of each point in turn tem , for each x tem , determine whether it is already a sampling point; if so, directly add the point and its corresponding depth value to the ground point depth information list; otherwise, find the sampling points x on its left and right sides left_num and x right_num , and interpolate its depth value, the depth step length D between two sampling points value The calculation method is:

[0122]

[0123] Among them, Dep[x] is the depth value corresponding to point x, and the point uniform difference between the sampling points is calculated according to the step size. For each point x to be interpolated, the depth value is:

[0124] Dep[x]=Dep[x left_num ]+D value ·(xx left_num )

[0125] Alternatively, a column-by-column interpolation method can be used, starting from the leftmost column of the image and traversing column by column to the right to find the ground point, determine the upper and lower boundaries, and perform depth interpolation;

[0126] After completing the horizontal interpolation, a similar process is performed for each column, starting from the leftmost column of the image and traversing column by column to the right. For each column, all ground points with pixel values are found and sorted by vertical coordinate. Then, the minimum and maximum values are used to determine the upper and lower boundaries of each column.

[0127] Between the upper and lower boundaries, each vertical coordinate is checked in turn. If it is not a sampling point, the adjacent sampling points above and below it are found and interpolated using a similar method. The calculation process is exactly the same as the horizontal interpolation method.

[0128] After interpolation, the depth information of all ground points constitutes a ground point depth information list;

[0129] Step 55: Match the points in the ground point depth information list with the corresponding pixels of the depth image to be processed. The successfully matched points are filtered as ground points to obtain a depth image without ground data.

[0130] Figure 7(a) shows the original visualization of the depth map. Figure 7(b) shows the effect after ground filtering of Figure 7(a).

[0131] Acquiring a depth map is based on stereo matching technology. This method can cause flickering in the corresponding area, similar to snow on a black-and-white TV, when an object is too close or too far from the camera, or has reflective properties. In this case, the depth value output in this area is unstable and incorrect, resulting in incomplete depth information or even erroneous objects. To address this issue, this embodiment performs depth value filtering after ground filtering to exclude objects whose depth information is incomplete or even erroneous due to matching characteristics. Such objects are usually outside the detection range of the binocular camera and the detection task.

[0132] Furthermore, the depth value filtering method extracts the depth values in all masked areas from the ground-filtered depth map D through the segmentation mask M, filters the depth value points in the non-masked area, and locates the target object in the depth map.

[0133] Segmentation mask data output by the SAM module Is a depth map A binary matrix of the same dimension, element M i,j 1 indicates that the pixel at the corresponding position is within the mask, and 0 indicates that the pixel is outside the mask. Each element D i,j Represents the depth value at pixel (i, j).

[0134] The depth values within the masked area of the depth map D are extracted using mask M. After removing invalid points and points outside the range, the remaining data is called valid depth points. If the proportion of valid depth points in the total depth information set of the depth map D is less than a threshold of 75%, the object corresponding to the masked area is considered out of the detection range and is skipped.

[0135] The list of objects retained after calculation and screening is the object to be located. The distance information is obtained by sorting the valid depth points from small to large according to the depth value d to obtain an ordered list. Here, n is the number of valid depth values d in the list L, that is, n=|L|.

[0136] According to the length n of the list L, calculate the median of the depth value of the target object as the distance D of the object from the camera obj , the calculation formula is as follows:

[0137]

[0138] Where d represents the effective depth value;

[0139] Furthermore, for the target object in the located depth image, the center of gravity and width of the target object are further solved through the segmentation mask M;

[0140] The row coordinates and column coordinates of the non-zero pixels in the segmentation mask M can be expressed as rows and cols respectively;

[0141] Barycentric coordinates (x center ,y center ) is obtained by calculating the mean of the column coordinates and the mean of the row coordinates:

[0142]

[0143] Among them, N clos and N rows are the total number of non-zero columns and non-zero rows in the masked region, respectively;

[0144] The width W of the target object is determined by the leftmost pixel x leftmost and the rightmost pixel x rightmost Determine the difference between

[0145] Furthermore, the offset ratio R of the center of gravity of the target object in the width direction is center , the formula is:

[0146]

[0147] At this point, detection information such as the target object's category, spatial orientation, and related dimensions can be obtained.

[0148] Example 2

[0149] Based on Example 1, this embodiment provides a target recognition and positioning system based on zero-sample detection, including:

[0150] Acquisition module: configured to acquire image data to be identified;

[0151] Interaction information part: is configured to preliminarily generate a target bounding box based on the interaction information and obtain a target area image;

[0152] Target classification module: configured to perform target classification based on the generated target area image using a classification model including out-of-distribution object detection to obtain a classification category of the target;

[0153] Mask segmentation module: is configured to input the obtained bounding box as interactive information into the SAM module for image segmentation to obtain the target segmentation mask;

[0154] Target recognition module: It is configured to obtain the depth image of the image to be recognized, perform ground filtering, and filter the depth value based on the segmentation mask to obtain the spatial information of the target object.

[0155] It should be noted here that the various modules in this embodiment correspond one-to-one to the various steps in Example 1, and the specific implementation processes are the same, which will not be repeated here.

[0156] Example 3

[0157] This embodiment provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of the target recognition and positioning method based on zero-sample detection in Example 1 are completed.

[0158] Example 4

[0159] This embodiment provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps of the target recognition and positioning method based on zero-sample detection in Example 1 are completed.

[0160] The foregoing description is merely a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Those skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present disclosure shall be included within the scope of protection of the present disclosure.

[0161] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Those skilled in the art should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. A target recognition and positioning method based on zero-sample detection, characterized in that: The steps include: Acquire image data to be recognized; Based on the interactive information, the target bounding box is preliminarily generated to obtain the target area image; Based on the generated target area image, a classification model including out-of-distribution object detection is used to classify the target to obtain the classification category of the target; The obtained bounding box is input into the SAM module as interactive information for image segmentation to obtain the target segmentation mask; Obtain the depth image of the image to be identified, perform ground filtering, and filter the depth value based on the segmentation mask to obtain the spatial information of the target object; Perform ground filtering on the depth image, including the following: Calculate the depth range of the camera image based on the camera parameters, and obtain the longitudinal distance value of the ground point according to the set depth step size; Based on the obtained longitudinal distance value, the ground horizontal distance corresponding to the longitudinal distance value is calculated, and the ground point is extracted to obtain the ground point in the camera coordinate system, which is projected to the image coordinate system to obtain the pixel points corresponding to the ground point in the image to construct the point set data; Calculate the depth value of the ground point according to the point set data to obtain a simulated ground discrete point set with depth values: Upsample the discrete point set data and perform depth interpolation between the sampling points so that the resolution of the interpolated image matches that of the depth image, obtaining a list of ground point depth information. Match the points in the ground point depth information list with the corresponding pixels in the depth image to be processed. The successfully matched points will be filtered as ground points to obtain a depth image without ground data. Depth value filtering method, through segmentation mask Depth map filtered from the ground Extract the depth values in all mask areas, filter the depth value points in non-mask areas, and locate the target object in the depth map; For the target object in the located depth image, the segmentation mask Continue to solve the center of gravity and width of the target object to obtain the spatial information of the target.

2. The target recognition and positioning method based on zero-sample detection according to claim 1, wherein: Build a target recognition and positioning model, including the interactive information generation part, the bounding box recognition module, the target recognition branch, and the target positioning branch; The interactive information generation part is connected to the bounding box recognition module, and the bounding box recognition module is connected to the target recognition branch and the target localization branch respectively; The target recognition branch includes an image slicing unit, a ViT classification network set after the image slicing unit, and an out-of-distribution detection module; The target positioning branch includes a SAM network, a depth image acquisition module, a ground filtering module, a depth value filtering module and a spatial information acquisition module which are connected in sequence.

3. The target recognition and positioning method based on zero-sample detection according to claim 1, wherein: The method of removing duplicated bounding boxes is based on the overlap between area ratio and bounding box judgment. The specific process is as follows: Set the ratio threshold of the bounding box, and remove the bounding boxes that exceed the ratio threshold according to the ratio threshold; Loop through all remaining bounding boxes, calculate the overlap of any two bounding boxes based on the intersection area of the two bounding boxes, and remove the bounding box with the smaller area between the two bounding boxes that exceeds the overlap threshold until the overlap of all bounding boxes meets the requirement.

4. The target recognition and positioning method based on zero-sample detection according to claim 1, wherein: The SAM module is used to perform image segmentation and obtain a complete segmentation mask of the target.

5. The target recognition and positioning method based on zero-sample detection according to claim 1, wherein: The target classification method includes the following steps: Obtain the image slice sample to be detected and send it to the ViT classification network for feature extraction to obtain the identified target category; The feature map output by the ViT classification network is processed by the ViM algorithm to obtain the detection score of the image slice sample to be detected; According to the distribution threshold of the in-distribution samples obtained through training, it is judged whether the sample to be tested belongs to the out-of-distribution sample; if it belongs to the out-of-distribution sample, the target classification result is marked as the category unknown; otherwise, the category label predicted by the ViT classification network is output.

6. The target recognition and positioning method based on zero-sample detection according to claim 1, wherein: The method for training and determining the distribution threshold of samples within a distribution includes the following steps: According to the actual classification task, we obtain the data samples of the distribution samples to construct the training set, and perform transfer learning on the ViT classification network; The in-distribution samples involved in the transfer learning process are feature extracted based on the ViT classification network, and out-of-distribution detection is performed using the ViM algorithm to determine the distribution threshold set by the score of the in-distribution samples.

7. A target recognition and positioning system based on zero-sample detection based on the target recognition and positioning method based on zero-sample detection according to claim 1, characterized in that: include: Acquisition module: configured to acquire image data to be identified; Interaction information part: is configured to preliminarily generate a target bounding box based on the interaction information and obtain a target area image; Target classification module: configured to perform target classification based on the generated target area image using a classification model including out-of-distribution object detection to obtain a classification category of the target; Mask segmentation module: is configured to input the obtained bounding box as interactive information into the SAM module for image segmentation to obtain the target segmentation mask; Target recognition module: It is configured to obtain the depth image of the image to be recognized, perform ground filtering, and perform depth value filtering based on the segmentation mask to obtain the spatial information of the target object.

8. An electronic device, characterized in that: The method comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the steps of the target recognition and positioning method based on zero-sample detection according to any one of claims 1 to 6 are completed.

9. A computer-readable storage medium, characterized in that Used to store computer instructions, which, when executed by a processor, complete the steps of the target recognition and positioning method based on zero-sample detection according to any one of claims 1 to 6.

Images