An unmanned vehicle surrounding target identification method and system based on open target detection

Abstract

The application provides an unmanned vehicle surrounding target recognition method and system based on open target detection, and relates to the technical field of computer vision and artificial intelligence. The method comprises the following steps: collecting multiple environment images around the unmanned vehicle; inputting each environment image into a YOLOv5 open target detection model to output a target information set in each environment image; according to the target information set, cutting the corresponding environment image to obtain multiple target images; extracting an image feature vector of each target image; inputting the image feature vector of each target image into a K-Means clustering model to output multiple category clusters containing each target image; inputting each category cluster into an LLaMA3 visual language model to output a category label and a natural language description corresponding to each category cluster; and determining the target category in each target image according to the category label and the natural language description corresponding to each category cluster to complete the surrounding target recognition of the unmanned vehicle.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and artificial intelligence, in particular to an unmanned vehicle surrounding target recognition method and system based on open target detection. BACKGROUND

[0002] With the rapid development of artificial intelligence, autonomous driving and computer vision technology, real-time perception and understanding of surrounding targets in complex traffic environment by unmanned vehicles has become one of the core research directions. An efficient and accurate target detection and recognition system is not only a prerequisite for realizing safe decision-making and path planning of autonomous driving, but also a key foundation for promoting intelligent driving vehicles from assisted driving to fully autonomous driving.

[0003] In the prior art, traditional open target detection algorithms mostly rely on closed-class training mechanisms and can only recognize targets in predefined class sets. When facing a large number of unknown classes in actual open environments, misjudgments and missed detections often occur, which significantly affects the reliability and environmental adaptability of the detection algorithm.

[0004] In addition, traditional unmanned vehicle surrounding target recognition methods usually only output target boxes and their class labels, lack deep understanding of target semantic information and natural language expression ability, and are difficult to meet higher-level scene understanding, human-computer interaction or auxiliary decision-making needs. SUMMARY

[0005] To solve the technical problems in the prior art that traditional open target detection algorithms mostly rely on closed-class training mechanisms, can only recognize targets in predefined class sets, and when facing a large number of unknown classes in actual open environments, misjudgments and missed detections often occur, which significantly affects the reliability and environmental adaptability of the detection algorithm, and traditional unmanned vehicle surrounding target recognition methods usually only output target boxes and their class labels, lack deep understanding of target semantic information and natural language expression ability, and are difficult to meet higher-level scene understanding, human-computer interaction or auxiliary decision-making needs, the present application provides an unmanned vehicle surrounding target recognition method and system based on open target detection.

[0006] The technical solutions provided by the embodiments of the present application are as follows:

[0007] First aspect:

[0008] The unmanned vehicle surrounding target recognition method based on open target detection provided by the embodiments of the present application comprises:

[0009] S1: Collecting multiple environment images around the unmanned vehicle;

[0010] S2: Inputting each environment image into a YOLOv5 open target detection model to output a target information set in each environment image;

[0011] S3: According to the target information set, the corresponding environment image is cropped to obtain multiple target images;

[0012] S4: Extract the image feature vector of each target image;

[0013] S5: Input the image feature vector of each target image into the K-Means clustering model, and output multiple category clusters containing each target image;

[0014] S6: Input each category cluster into the LLaMA3 visual language model to output the category label and natural language description corresponding to each category cluster;

[0015] S7: According to the category label and natural language description corresponding to each category cluster, determine the target category in each target image, and complete the surrounding target recognition of the unmanned vehicle.

[0016] Second aspect:

[0017] The embodiment of the application provides an unmanned vehicle surrounding target recognition system based on open target detection, which comprises:

[0018] A processor;

[0019] A memory, wherein the memory stores computer readable instructions, and the computer readable instructions are executed by the processor to realize the unmanned vehicle surrounding target recognition method based on open target detection as described in the first aspect.

[0020] Third aspect:

[0021] The embodiment of the application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the unmanned vehicle surrounding target recognition method based on open target detection as described in the first aspect.

[0022] The technical scheme provided by the embodiment of the application has at least the following beneficial effects:

[0023] In the embodiment of the present application, by inputting each environmental image into the YOLOv5 open target detection model, the target information set in each environmental image is output, and according to the target information set, the corresponding environmental image is cropped to obtain multiple target images, which no longer relies on the training mechanism of closed categories, can identify targets beyond the predefined class set, and when facing a large number of unknown categories in the actual open environment, will not misjudge and miss detection, significantly improving the reliability and environmental adaptability of the detection algorithm. By inputting the image feature vectors of each target image into the K-Means clustering model, multiple class clusters containing each target image are output, and each class cluster is input into the LLaMA3 visual language model to output the class label and natural language description corresponding to each class cluster. Not only the target frame and its class label are output, but also the in-depth understanding of the semantic information of the target and the natural language expression ability are output, which can meet the higher scene understanding, human-computer interaction or auxiliary decision-making needs. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0025] Figure 1 A flowchart of an unmanned vehicle surrounding target recognition method based on open target detection provided by the embodiment of the present application is shown.

[0026] Figure 2 A structural diagram of an unmanned vehicle surrounding target recognition system based on open target detection provided by the embodiment of the present application is shown. DETAILED DESCRIPTION

[0027] The technical solutions in the present application will be described below with reference to the drawings.

[0028] In the embodiments of the present application, the words such as "example", "for example" are used to represent as an example, illustration or description. Any embodiment or design scheme described as "example" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present application, the meaning expressed by "and / or" can be both, or can be one of the two.

[0029] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0030] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0031] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0032] Reference manual attached Figure 1 The diagram shows a flowchart of a method for identifying targets around an unmanned vehicle based on open target detection, provided by an embodiment of the present invention.

[0033] This invention provides a method for recognizing targets around an unmanned vehicle based on open target detection. This method can be implemented by an unmanned vehicle target recognition device based on open target detection, which can be a terminal or a server. The processing flow of the unmanned vehicle target recognition method based on open target detection may include the following steps:

[0034] S1: Collect multiple environmental images around the unmanned vehicle.

[0035] In one possible implementation, S1 specifically refers to:

[0036] Multiple environmental images of the vehicle's surroundings are captured by cameras installed in the vehicle.

[0037] In this embodiment of the invention, by acquiring images of the surrounding environment through cameras installed in the unmanned vehicle, not only can the timeliness and consistency of the images be ensured, but the system can also directly utilize native sensor data, thereby improving data acquisition efficiency, target detection accuracy, and environmental adaptability.

[0038] In one possible implementation, the process after S1 and before S2 includes:

[0039] S1A: Resize each environmental image:

[0040]

[0041]

[0042] in, This represents the t-th environmental image after resizing, and Resize() represents the resizing operation. t Let t represent the t-th environment image, W represent the width of the input image for the YOLOv5 open object detection model, H represent the height of the input image for the YOLOv5 open object detection model, J represent the environment image sequence, and T represent the total number of environment images.

[0043] S1B: Normalize each of the resized environmental images:

[0044]

[0045] in, This represents the t-th environmental image after normalization, and Normalize() represents the normalization operation.

[0046] In this embodiment of the invention, by performing size adjustment and normalization on each image after image acquisition, it is possible to effectively ensure that the image data is consistent with the input requirements of the YOLOv5 open target detection model, improve the detection accuracy and processing efficiency of the model, and enhance the system's adaptability to complex environments such as illumination changes and image noise, thereby providing a stable and reliable input foundation for the subsequent target recognition and classification process.

[0047] S2: Input each environmental image into the YOLOv5 open object detection model and output the set of target information in each environmental image.

[0048] It's worth noting that the YOLOv5 open object detection model is a high-efficiency object detection algorithm based on deep learning. While maintaining the YOLO series' strengths in real-time performance and high accuracy, it possesses the ability to accurately detect objects of known categories in images. Furthermore, it can be combined with open set recognition mechanisms to extend its ability to perceive and label objects of unknown categories. By rapidly regressing the input image to generate bounding boxes, their category labels, and confidence scores, YOLOv5 is widely used in scenarios such as autonomous driving and security monitoring, and is particularly suitable for tasks requiring rapid multi-object recognition in complex environments.

[0049] Specifically, the target information set includes:

[0050]

[0051]

[0052] Among them, P t Let p represent the set of target information in the t-th environmental image, YOLOv5() denotes the YOLOv5 open object detection model, and p t,iLet x represent the set of information about the i-th target in the t-th environmental image. t,i The x-axis coordinate of the detection box corresponding to the i-th target in the t-th environmental image is y. t,i w represents the Y-axis coordinate of the detection box corresponding to the i-th target in the t-th environmental image. t,i h represents the width of the detection box corresponding to the i-th target in the t-th environmental image. t,i c represents the height of the detection box corresponding to the i-th target in the t-th environmental image. t,i s represents the category number of the i-th target in the t-th environmental image. t,i Let represent the confidence score of the i-th target in the t-th environmental image.

[0053] In this embodiment of the invention, the YOLOv5 open target detection model is used to detect targets in environmental images. It can efficiently and accurately extract the spatial location, category identification and confidence information of various known and unknown targets, and realize the structured output of targets. This not only improves the perception accuracy and real-time performance of the unmanned vehicle to the surrounding environment, but also provides a stable and reliable input foundation for subsequent target image screening, cluster analysis and language description modules, and enhances the system's adaptability to complex dynamic scenes.

[0054] S3: Based on the target information set, crop the corresponding environmental image to obtain multiple target images.

[0055] In this embodiment of the invention, by cropping the environmental image according to the target information set, the target region image can be effectively extracted, irrelevant background can be removed, the quality and discriminability of the target image can be improved, and high-quality input can be provided for subsequent feature extraction and cluster analysis.

[0056] In one possible implementation, S3 specifically includes sub-steps S301 to S303:

[0057] S301: Based on a confidence threshold, filter the target information set to determine the target information set in each environmental image:

[0058]

[0059] in, Let θ represent the set of target information to be filtered in the t-th environmental image. s This represents the confidence threshold.

[0060] It should be noted that those skilled in the art can set the confidence threshold according to actual needs, and this invention does not limit it.

[0061] S302: By using a non-maximum suppression algorithm, overlapping targets in the filtered target information set are eliminated, and the final target information set in each environmental image is determined.

[0062]

[0063] in, Let θ represent the final target information set in the t-th environmental image, NMS() denotes the non-maximum suppression algorithm, and θ iou This represents the overlap threshold of the cross-union ratio in the nonmaximum suppression algorithm.

[0064] It's important to note that Non-Maximum Suppression (NMS) is a post-processing algorithm commonly used in object detection. It aims to retain the optimal bounding box from multiple overlapping candidate bounding boxes. Its basic principle is to prioritize the box with the highest confidence among multiple predicted bounding boxes of the same class, and then suppress other boxes with excessive overlap with this box based on a set Intersection over Union (IoU) threshold, thereby removing redundant and duplicate detection results. NMS effectively improves the accuracy and clarity of object detection, ensuring that each object corresponds to only one high-quality bounding box.

[0065] It should be noted that those skilled in the art can set the overlap threshold of the intersection-union ratio according to actual needs, and this invention does not limit it.

[0066] In one possible implementation, S302 specifically includes sub-steps S3021 and S3022:

[0067] S3021: Calculate the cross-union ratio between two adjacent detection boxes:

[0068]

[0069] Where IoU(B1,B2) represents the intersection-union ratio between the first detection box B1 and the second detection box B2. This represents the area of ​​the intersection region between the first detection box B1 and the second detection box B2. This represents the area of ​​the union region of the first detection box B1 and the second detection box B2.

[0070] S3022: If the cross-union ratio is greater than the cross-union ratio overlap threshold, it is determined that there is an overlap relationship between the two adjacent detection boxes.

[0071] S3023: For two adjacent detection boxes that have an overlapping relationship, retain the target box with the higher confidence score and delete the target box with the lower confidence score to determine the final target information set in each environmental image.

[0072] S303: Based on the final target information set, crop the corresponding environmental image to obtain multiple target images.

[0073] In this embodiment of the invention, by setting a confidence threshold to filter detection targets, low-confidence targets can be effectively eliminated, improving data quality. Non-maximum suppression algorithms are used to eliminate target overlap, ensuring that only one representative detection box is retained for each real target, thus improving recognition accuracy. By cropping the original image based on the final target information set, high-quality target image regions can be extracted, providing clear and discriminative input for subsequent feature extraction and classification / clustering steps, thereby comprehensively enhancing the system's stability, accuracy, and processing efficiency in complex environments.

[0074] S4: Extract the image feature vectors of each target image.

[0075] In this embodiment of the invention, by extracting image feature vectors from the cropped target image, the image can be converted from pixel-level representation to semantic vector representation, enabling the system to measure the similarity and differences between images mathematically, and providing standard input for subsequent clustering analysis and classification of unknown targets.

[0076] In one possible implementation, S4 specifically includes sub-steps S401 to S403:

[0077] S401: Extract local image feature vectors from each target image using the VGG19 algorithm.

[0078] It's worth noting that VGG19 is a classic convolutional neural network (CNN) architecture proposed by the Visual Geometry Group at Oxford University. It comprises a 19-layer deep network structure, extracting image features layer by layer through multiple small (3x3) convolutional kernels and max-pooling operations. VGG19 is widely used in image classification and feature extraction tasks due to its simple structure, ease of understanding, and controllable parameters. This model achieved excellent results in the ImageNet image recognition challenge, particularly excelling at extracting local texture information (such as edges, corners, and textures), and is therefore widely used for low-level feature extraction in visual perception systems.

[0079] Optionally, the local image feature vector can be extracted according to the following formula:

[0080]

[0081] Among them, f VGG Represents a local image feature vector, VGG19 represents the VGG19 algorithm, I m This represents the target image.

[0082] S402: Extract the global image feature vector of each target image using the DINOv2 algorithm.

[0083] It's worth noting that DINOv2 is an advanced visual representation learning algorithm based on self-supervised learning and the Vision Transformer (ViT) architecture, developed by MetaAI. It requires no manually labeled data and learns high-level semantic features of images through contrastive learning. DINOv2 can capture global structural information in images, modeling the shape, relationships, and semantics of objects within them, exhibiting excellent cross-frame consistency and transferability. Compared to traditional CNNs, DINOv2 excels at handling time-dependent data such as video frames, making it ideal for tasks requiring stable visual representations, such as video analysis, object tracking, and unsupervised clustering.

[0084] Optionally, the global image feature vector can be extracted according to the following formula:

[0085]

[0086] Among them, f DINO DINOv2 represents the global image feature vector, and DINOv2 represents the DINOv2 algorithm.

[0087] S403: Concatenate the local image feature vector and the global image feature vector to obtain the image feature vector, thus completing the extraction of image feature vectors for each target image.

[0088] Optionally, the local image feature vector and the global image feature vector are concatenated according to the following formula to obtain the image feature vector:

[0089]

[0090] Where f represents the image feature vector, and Concat() represents the concatenation operation.

[0091] In this embodiment of the invention, the VGG19 algorithm is used to extract local image feature vectors and the DINOv2 algorithm is used to extract global image feature vectors. The feature dimensions of the two are then concatenated to construct a joint image feature representation that simultaneously possesses local texture perception and global semantic modeling capabilities. This not only improves the integrity and discriminativeness of image representation but also enhances cross-frame consistency and stability when processing consecutive video frames. As a result, a more accurate, robust, and efficient feature foundation is provided for subsequent image clustering analysis and unknown target classification tasks.

[0092] S5: Input the image feature vectors of each target image into the K-Means clustering model, and output multiple category clusters containing each target image.

[0093] It's important to note that K-Means clustering is a commonly used unsupervised learning algorithm designed to automatically divide samples into K clusters based on the similarity between data points (usually measured by Euclidean distance). Its basic process involves: randomly initializing K cluster centers; iteratively assigning each sample to the nearest cluster; and then updating the center of each cluster until the cluster centers converge or the maximum number of iterations is reached. K-Means is suitable for clustering high-dimensional feature vectors and can discover the inherent structure of data without pre-labeling, making it widely applicable in tasks such as image classification and object recognition.

[0094] In this embodiment of the invention, by inputting the image feature vectors of each target image into the K-Means clustering model for cluster analysis, the target images can be automatically classified into multiple category clusters based on semantic similarity, thus realizing open target classification under unknown category conditions.

[0095] In one possible implementation, S5 specifically includes sub-steps S501 and S502:

[0096] S501: Based on the image feature vectors of each target image, determine the optimal number of category clusters according to the elbow rule.

[0097] It's important to note that the elbow rule is a heuristic method for determining the optimal number of clusters (K value) in a clustering algorithm. Its core idea is that as the number of clusters increases, the total squared error (SSE) of the clustering results continuously decreases. However, when the number of clusters reaches a certain critical point, the rate of decrease in SSE slows significantly, forming an "elbow" shape. The number of clusters corresponding to this "elbow" represents a choice that offers optimal clustering performance with moderate complexity. By balancing cluster compactness and model simplicity, the elbow rule helps to rationally determine the number of clusters in unsupervised learning, improving the scientific rigor and practicality of cluster analysis.

[0098] In this embodiment of the invention, by dynamically determining the optimal number of category clusters based on the elbow rule, the target distribution under different scenarios can be automatically adapted, avoiding classification bias caused by a fixed number of clusters, thereby improving the rationality of clustering and the system's generalization ability.

[0099] In one possible implementation, S501 specifically includes sub-steps S5011 to S5016:

[0100] S5011: Set the maximum number of category clusters.

[0101] S5012: Calculate the total squared error of the current category cluster number based on the image feature vectors of each target image.

[0102]

[0103] Where SEE(K) represents the total squared error of the current number of clusters K, and K represents the current number of clusters, where K = 1, 2, ... K max K max f represents the maximum number of class clusters. j Let C represent the image feature vector of the j-th target image. k Let μ represent the set of members of the k-th class cluster. k Let ||k|| represent the central feature vector of the k-th cluster, and ||k|| represent the Euclidean norm.

[0104] S5013: Calculate the first-order difference of the current number of clusters based on the total squared error:

[0105]

[0106] Where ΔD(K) represents the first difference of the current number of clusters K, D(K-1) represents the total squared error of the previous number of clusters K-1, and D(K) represents the total squared error of the current number of clusters K.

[0107] S5014: Calculate the second-order difference of the current number of clusters based on the first-order difference:

[0108]

[0109] Where, ΔD 2 (K) represents the second difference of the current number of clusters K, and ΔD(K+1) represents the first difference of the next number of clusters K+1.

[0110] S5015: Determine if the maximum number of category clusters has been reached. If yes, proceed to step S5016. Otherwise, increment the current number of category clusters by 1 and return to step S5012.

[0111] S5016: Output the number of class clusters with the largest second-order difference as the optimal number of class clusters:

[0112]

[0113] Among them, K opt This represents the optimal number of class clusters, and arg max indicates taking the maximum value.

[0114] In this embodiment of the invention, by automatically calculating the optimal number of clusters based on the elbow rule, the clustering granularity can be flexibly adjusted according to the distribution of target image features in different scenarios. This avoids clustering errors caused by a fixed number of clusters, improving the accuracy of target classification and the system's adaptive capability. Simultaneously, this step provides a precise category basis for subsequent clustering analysis and text description generation, enhancing the system's open recognition capability and semantic expression consistency, thereby significantly improving the intelligent target perception and recognition level of autonomous vehicles in complex environments.

[0115] S502: Based on the optimal number of clusters, the K-Means clustering algorithm is used to cluster each target image, and the output contains multiple clusters of each target image.

[0116] In this embodiment of the invention, the K-Means clustering algorithm is used to classify target images, which can automatically aggregate semantically similar targets, providing structured and semantically consistent basic data support for subsequent text generation, visualization management and new class recognition, and significantly enhancing the target understanding and classification capabilities of unmanned vehicles in open environments.

[0117] In one possible implementation, S502 specifically includes sub-steps S5021 to S5026:

[0118] S5021: Initialize the cluster center:

[0119]

[0120] in, Indicates the first The central feature vectors of each category cluster, where, K opt Indicates the optimal number of class clusters. This represents the set of image feature vectors for each target image to be clustered.

[0121] S5022: Set the maximum number of iterations.

[0122] S5023: Calculate the Euclidean distance between the image feature vector of each target image and the cluster center, and assign the image feature vector of each target image to the cluster with the closest Euclidean distance to perform clustering of each target image:

[0123]

[0124] in, Indicates the first The set of members of a class cluster, f j This represents the image feature vector of the j-th target image. This represents the image feature vector of the j-th target image and the image feature vector of the j-th target image. The distance between the central feature vectors of each category cluster This represents the image feature vector of the j-th target image and the image feature vector of the j-th target image. The distance between the central feature vectors of each category cluster.

[0125] S5024: Update the cluster center:

[0126]

[0127] in, Indicates the updated number The central feature vector of each category cluster.

[0128] S5025: Determine if the maximum number of iterations has been reached. If yes, proceed to step S5026. Otherwise, return to step S5023.

[0129] S5026: Output contains multiple category clusters for each target image.

[0130] In this embodiment of the invention, by employing the K-Means clustering algorithm to cluster image feature vectors, target images can be automatically divided into multiple semantically consistent clusters based on feature similarity. This process, through reasonable initialization, distance calculation, iterative optimization, and cluster center updates, not only ensures the accuracy and stability of the clustering results but also considers system operating efficiency. It enhances the system's self-organizing and open recognition capabilities when processing large-scale, unlabeled target images, thus providing a solid data foundation for subsequent semantic understanding and text generation modules.

[0131] S6: Input each category cluster into the LLaMA3 visual language model and output the category label and natural language description corresponding to each category cluster.

[0132] It's important to note that the LLaMA3 visual language model is a large-scale multimodal language model that combines image understanding and natural language processing capabilities. It can simultaneously receive image information and text prompts, generating semantic understanding, textual descriptions, or category judgments of image content. Based on the LLaMA3 architecture from Meta, this model, by integrating a visual encoder and a language decoder, possesses powerful cross-modal reasoning and semantic generation capabilities. It is widely used in tasks such as image and text understanding, object recognition, and image question answering, and is particularly suitable for scenarios involving the automatic description and labeling of unknown or complex targets.

[0133] In this embodiment of the invention, by inputting the clusters obtained from clustering into the visual language model LLaMA3, the system can automatically output the category label and natural language description corresponding to each cluster. This not only improves the system's ability to interpret and express the target category, but also enhances its ability to identify and describe unknown category targets.

[0134] In one possible implementation, S6 specifically includes sub-steps S601 and S602:

[0135] S601: Construct the input tuples for the LLaMA3 visual language model based on each category cluster:

[0136]

[0137] in, Indicates the first The input tuples of the LLaMA3 visual language model corresponding to each category cluster. Indicates the first A unique identifier for each category cluster Indicates the first A preliminary list of textual descriptions for all images in each category cluster.

[0138] S602: Input the input tuples into the LLaMA3 visual language model, and output the category labels and natural language descriptions for each category cluster:

[0139]

[0140] in, Indicates the first Category labels for each category cluster, Indicates the first The natural language description of each category cluster, LLaMA3() represents the LLaMA3 visual language model.

[0141] In this embodiment of the invention, a structured input tuple is constructed, combining category cluster identifiers with image semantic information and inputting them into the visual language model LLaMA3, outputting corresponding category labels and natural language descriptions. This approach not only achieves semantic conversion from image features to linguistic information but also enhances the system's semantic expression capabilities for both known and unknown targets, improving its adaptability in open environments. Simultaneously, the generated labels and descriptions possess good readability and interactivity, aiding human-machine understanding, result verification, and decision support, significantly improving the intelligence level and user-friendliness of the autonomous vehicle perception system.

[0142] S7: Based on the category labels and natural language descriptions corresponding to each category cluster, determine the target category in each target image to complete the recognition of surrounding targets for the autonomous vehicle.

[0143] In this embodiment of the invention, by determining the target category for each target image based on the category label and natural language description corresponding to the category cluster, the specific mapping and category assignment of image semantic information can be realized, thereby enhancing the interpretability and interactivity of the recognition results.

[0144] Furthermore, by combining the category labels and natural language descriptions corresponding to each category cluster, the target image is visualized to construct a graphic display interface that includes example images, text descriptions, and category attribution information, for decision-makers to refer to and to support user interaction.

[0145] In this embodiment of the invention, by combining and displaying the labels, natural language descriptions and example images corresponding to the category clusters, a visual interface combining text and graphics is constructed, which can improve the interpretability and intuitiveness of the recognition results, making it easier for decision-makers to quickly understand and confirm the target category. At the same time, it supports human feedback and correction, enhancing the human-computer interaction capability of the system.

[0146] Furthermore, the target category identified in each target image is considered a known category, and multiple environmental images around the autonomous vehicle are re-acquired based on this. These environmental images are then input into the YOLOv5 open-world object detection model to extract the target information set (including bounding boxes, category confidence, etc.) and determine whether the target in the image belongs to a known category: if the target belongs to a known category, the processing is skipped, and new environmental images are acquired. If the target belongs to an unknown category, the corresponding target information set is selected, and the image is cropped accordingly to obtain multiple images containing the unknown target. Next, the feature vectors of these images are extracted and input into the K-Means clustering model to cluster the unknown targets, resulting in multiple category clusters. Subsequently, each category cluster is input into the LLaMA3 visual language model, which outputs the corresponding category label and natural language description, and these newly identified unknown categories and their labels are incorporated into the known category set. This process is repeated in a loop to discover and identify new categories of targets appearing in the environment in real time.

[0147] In this embodiment of the invention, by introducing a sequential, cyclical new category discovery and update mechanism, the system can set the initially identified target category as a known category during the operation of the autonomous vehicle. Based on real-time acquired environmental images, it uses the YOLOv5 open-world target detection model to identify and judge targets. Combining image cropping, feature extraction, and K-Means clustering, unknown targets are effectively classified, and the corresponding category labels and natural language descriptions are output through the LLaMA3 visual language model. Finally, the newly identified categories are incorporated into the known category set. Through this cyclical mechanism, the target category set can be dynamically expanded and updated in real time, thereby endowing the system with continuous perception and adaptive learning capabilities in open-world scenarios.

[0148] Reference manual attached Figure 2 The diagram shows a schematic of the structure of an unmanned vehicle surrounding target recognition system based on open target detection provided by the present invention.

[0149] The present invention also provides an unmanned vehicle surrounding target recognition system 20 based on open target detection, applied to the above-mentioned unmanned vehicle surrounding target recognition method based on open target detection, comprising:

[0150] Processor 201;

[0151] The memory 202 stores computer-readable instructions, which, when executed by the processor 201, implement the unmanned vehicle surrounding target recognition method based on open target detection as described in the method embodiment.

[0152] The unmanned vehicle surrounding target recognition system 20 based on open target detection provided by the present invention can perform the above-mentioned unmanned vehicle surrounding target recognition method based on open target detection and achieve the same or similar technical effects. To avoid repetition, the present invention will not elaborate further.

[0153] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0154] In this embodiment of the invention, by inputting each environmental image into the YOLOv5 open object detection model, the model outputs a set of target information in each environmental image. Based on the target information set, the corresponding environmental image is cropped to obtain multiple target images. This eliminates the reliance on closed-category training mechanisms and enables the identification of targets beyond the predefined class set. When faced with a large number of unknown categories in real open environments, it avoids false positives and false negatives, significantly improving the reliability and environmental adaptability of the detection algorithm. By inputting the image feature vectors of each target image into the K-Means clustering model, the model outputs multiple category clusters containing each target image. Each category cluster is then input into the LLaMA3 visual language model, which outputs the category labels and natural language descriptions corresponding to each category cluster. This not only outputs the target bounding box and its category label but also provides a deep understanding of the target's semantic information and natural language expression capabilities, which can meet higher-level needs for scene understanding, human-computer interaction, or decision support.

[0155] It should be understood that the processor in the embodiments of the present invention can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0156] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0157] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0158] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0159] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0160] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0161] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0162] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0163] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0164] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0165] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0166] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0167] This invention provides a computer-readable storage medium storing a computer program thereon, characterized in that, when executed by a processor, the program implements the unmanned vehicle surrounding target recognition method based on open target detection as described in the method embodiment.

[0168] The present invention provides a computer-readable storage medium that can implement the steps and effects of the above-described method embodiment of the unmanned vehicle surrounding target recognition method based on open target detection. To avoid repetition, the present invention will not repeat the steps.

[0169] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0170] In this embodiment of the invention, by inputting each environmental image into the YOLOv5 open object detection model, the model outputs a set of target information in each environmental image. Based on the target information set, the corresponding environmental image is cropped to obtain multiple target images. This eliminates the reliance on closed-category training mechanisms and enables the identification of targets beyond the predefined class set. When faced with a large number of unknown categories in real open environments, it avoids false positives and false negatives, significantly improving the reliability and environmental adaptability of the detection algorithm. By inputting the image feature vectors of each target image into the K-Means clustering model, the model outputs multiple category clusters containing each target image. Each category cluster is then input into the LLaMA3 visual language model, which outputs the category labels and natural language descriptions corresponding to each category cluster. This not only outputs the target bounding box and its category label but also provides a deep understanding of the target's semantic information and natural language expression capabilities, which can meet higher-level needs for scene understanding, human-computer interaction, or decision support.

[0171] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0172] The following points need to be explained:

[0173] (1) The accompanying drawings of the embodiments of the present invention only involve the structures involved in the embodiments of the present invention. Other structures can refer to the general design.

[0174] (2) For clarity, the thickness of layers or regions is enlarged or reduced in the drawings used to describe embodiments of the invention, i.e., these drawings are not drawn to scale. It is understood that when an element such as a layer, film, region or substrate is referred to as being “above” or “below” another element, the element may be “directly” located “above” or “below” the other element or there may be intermediate elements.

[0175] (3) Where there is no conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other to obtain new embodiments.

[0176] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for identifying targets around an unmanned vehicle based on open target detection, characterized in that, include: S1: Collect multiple environmental images around the unmanned vehicle; S2: Input each of the environmental images into the YOLOv5 open target detection model and output the target information set in each of the environmental images; S3: Based on the target information set, crop the corresponding environmental image to obtain multiple target images; S4: Extract the image feature vector of each of the target images; S5: Input the image feature vectors of each target image into the K-Means clustering model, and output multiple category clusters containing each target image; S6: Input each of the aforementioned category clusters into the LLaMA3 visual language model, and output the category labels and natural language descriptions corresponding to each of the aforementioned category clusters; S7: Based on the category labels and natural language descriptions corresponding to each category cluster, determine the target category in each target image to complete the surrounding target recognition of the unmanned vehicle; Specifically, the target information set includes: ； ； in, P t Indicates the first t A set of target information in a Zhang environmental image. YOLOv 5() represents the YOLOv5 open object detection model. Represents the normalized th t Zhang's environmental images, p t,i Indicates the first t The first in the environmental image i A collection of information about a target. x t,i Indicates the first t The first in the environmental image i The X-axis coordinates of the detection boxes corresponding to each target. y t,i Indicates the first t The first in the environmental image i The Y-axis coordinate of the detection box corresponding to each target. w t,i Indicates the first t The first in the environmental image i The width of the detection box corresponding to each target. h t,i Indicates the first t The first in the environmental image i The height of the detection box corresponding to each target c t,i Indicates the first t The first in the environmental image i The category number of each target, s t,i Indicates the first t The first in the environmental image i Confidence score of each target; Following S7, it also includes: The target category in each of the target images is determined as a known category. The process returns to S1 to re-acquire multiple environmental images around the unmanned vehicle. Based on the target information set of the re-acquired environmental images, it is determined whether the target category in the re-acquired environmental images belongs to the known category. If yes, the process returns to S1 to re-acquire multiple environmental images around the unmanned vehicle. Otherwise, the target category in the re-acquired environmental images is determined as an unknown category. The target information set corresponding to the unknown category is then filtered out, and the process proceeds to S3.

2. The method for identifying targets around an unmanned vehicle based on open target detection according to claim 1, characterized in that, Specifically, S1 is: Multiple environmental images of the area surrounding the unmanned vehicle are captured by cameras installed in the vehicle.

3. The method for identifying targets around an unmanned vehicle based on open target detection according to claim 1, characterized in that, The following is included after S1 and before S2: S1A: Resize each of the aforementioned environmental images: ； ； in, Indicates the number after size adjustment t Zhang's environmental images, Resize ( ) indicates a size adjustment operation. I t Indicates the first t Zhang's environmental images, W This represents the width of the input image for the YOLOv5 open object detection model. H This represents the height of the input image for the YOLOv5 open object detection model. J Represents a sequence of environmental images. T This indicates the total number of environmental images; S1B: Normalize each of the resized environmental images: ； in, Represents the normalized th t Zhang's environmental images, Normalize ( ) indicates a normalization operation.

4. The method for identifying targets around an unmanned vehicle based on open target detection according to claim 1, characterized in that, S3 specifically includes: S301: Based on a confidence threshold, the target information set is filtered to determine the filtered target information set in each of the environmental images: ； in, Indicates the first t A set of target information for filtering in Zhang's environmental images. θ s Indicates the confidence threshold; S302: Eliminate overlapping targets in the selected target information set using a non-maximum suppression algorithm to determine the final target information set in each of the environmental images: ； in, Indicates the first t The final target information set in Zhang's environmental images NMS ( ) indicates a nonmaximum suppression algorithm. θ iou This represents the overlap threshold of the cross-union ratio in the non-maximum suppression algorithm; S303: Based on the final target information set, crop the corresponding environmental image to obtain multiple target images.

5. The method for identifying targets around an unmanned vehicle based on open target detection according to claim 1, characterized in that, S5 specifically includes: S501: Based on the image feature vectors of each target image, determine the optimal number of category clusters according to the elbow rule; S502: Based on the optimal number of clusters, the K-Means clustering algorithm is used to cluster each of the target images, and multiple clusters containing each of the target images are output.

6. The method for identifying targets around an unmanned vehicle based on open target detection according to claim 5, characterized in that, S501 specifically includes: S5011: Set the maximum number of category clusters; S5012: Calculate the total squared error of the current category cluster number based on the image feature vectors of each target image: ； in, SSE ( K ) indicates the number of current category clusters K The total squared error, K This indicates the number of current category clusters, where, K =1,2,… K max , K max Indicates the maximum number of class clusters. f j Indicates the first j Image feature vector of the target image, C k Indicates the first k The set of members of a category cluster μ k Indicates the first k The central feature vector of each category cluster, || || represents the Euclidean norm; S5013: Calculate the first-order difference of the current number of clusters based on the total squared error: ； Where, Δ D ( K ) indicates the number of current category clusters K The first difference, D ( K -1) indicates the number of clusters in the previous category. K -1 total squared error D ( K ) indicates the number of current category clusters K The total squared error; S5014: Based on the first-order difference, calculate the second-order difference of the current number of category clusters: ； Where, Δ D 2 ( K ) indicates the number of current category clusters K The second difference, Δ D ( K +1 indicates the number of the next category clusters. K +1 first-order difference; S5015: Determine whether the maximum number of category clusters has been reached; if yes, proceed to step S5016; otherwise, increment the current number of category clusters by 1 and return to step S5012. S5016: Output the number of class clusters with the largest second-order difference as the optimal number of class clusters: ； in, K opt This represents the optimal number of class clusters, and arg max represents taking the maximum value.

7. The method for identifying targets around an unmanned vehicle based on open target detection according to claim 5, characterized in that, Specifically, S502 includes: S5021: Initialize the cluster center: ； in, Indicates the first The central feature vectors of each category cluster, where, , K opt Indicates the optimal number of class clusters. This represents the set of image feature vectors for each target image to be clustered; S5022: Set the maximum number of iterations; S5023: Calculate the Euclidean distance between the image feature vector of each target image and the center of the category cluster, and assign the image feature vector of each target image to the category cluster with the closest Euclidean distance to perform clustering on each target image: ； in, Indicates the first The set of members of a category cluster f j Indicates the first j Image feature vector of the target image, Indicates the first j The image feature vector of the target image and the first The distance between the central feature vectors of each category cluster Indicates the first j The image feature vector of the target image and the first The distance between the central feature vectors of each category cluster; S5024: Update the cluster center of the stated category: ； in, Indicates the updated number The central feature vector of each category cluster; S5025: Determine whether the maximum number of iterations has been reached; if yes, proceed to step S5026; otherwise, return to step S5023. S5026: Output multiple category clusters for each of the target images.

8. The method for identifying targets around an unmanned vehicle based on open target detection according to claim 1, characterized in that, S6 specifically includes: S601: Construct the input tuples of the LLaMA3 visual language model based on each of the aforementioned category clusters: ； in, Indicates the first The input tuples of the LLaMA3 visual language model corresponding to each category cluster. Indicates the first A unique identifier for each category cluster Indicates the first A preliminary list of textual descriptions for all images in each category cluster; S602: Input the input tuples into the LLaMA3 visual language model, and output the category labels and natural language descriptions corresponding to each category cluster: ； in, Indicates the first Category labels for each category cluster, Indicates the first Natural language description of each category cluster, LLaMA 3() represents the LLaMA3 visual language model.

9. A target recognition system for unmanned vehicles based on open target detection, characterized in that, include: processor; A memory storing computer-readable instructions, which, when executed by the processor, implement the unmanned vehicle surrounding target recognition method based on open target detection as described in any one of claims 1 to 8.

Images