Unmanned vehicle surrounding target identification method and system based on open target detection

Abstract

The invention provides an unmanned vehicle surrounding target identification method and system based on open target detection, and relates to the technical field of computer vision and artificial intelligence. The method comprises the following steps: collecting a plurality of environment images around the unmanned vehicle; inputting each environment image into a YOLOv5 open target detection model, and outputting a target information set in each environment image; according to the target information set, cutting the corresponding environment image to obtain a plurality of target images; extracting an image feature vector of each target image; inputting the image feature vector of each target image into a K-Means clustering model, and outputting a plurality of category clusters including each target image; inputting each category cluster into an LLaMA3 visual language model, and outputting a category label and a natural language description corresponding to each category cluster; and according to the category label and the natural language description corresponding to each category cluster, determining a target category in each target image, and completing surrounding target identification of the unmanned vehicle.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and artificial intelligence technology, and in particular to a method and system for identifying targets around an unmanned vehicle based on open target detection. Background Art

[0002] With the rapid development of artificial intelligence, autonomous driving, and computer vision technologies, enabling autonomous vehicles to perceive and understand surrounding objects in real time in complex traffic environments has become a core research direction. Efficient and accurate object detection and recognition systems are not only a prerequisite for safe autonomous driving decision-making and path planning, but also a key foundation for the evolution of intelligent vehicles from assisted driving to fully autonomous driving.

[0003] In the existing technology, traditional open target detection algorithms mostly rely on closed category training mechanisms and can only identify targets in a predefined class set. When faced with a large number of unknown categories in actual open environments, misjudgments and missed detections often occur, significantly affecting the reliability and environmental adaptability of the detection algorithm.

[0004] In addition, traditional methods for identifying objects around unmanned vehicles usually only output target boxes and their category labels, lacking in-depth understanding of target semantic information and natural language expression capabilities, making it difficult to meet more advanced scene understanding, human-computer interaction, or decision-making support needs. Summary of the Invention

[0005] In order to solve the technical problems that traditional open target detection algorithms in the prior art mostly rely on closed category training mechanisms and can only identify targets in a predefined class set, and often misjudge and miss detection when faced with a large number of unknown categories in actual open environments, which significantly affects the reliability and environmental adaptability of the detection algorithm, and traditional unmanned vehicle surrounding target recognition methods usually only output target boxes and their category labels, lack in-depth understanding of target semantic information and natural language expression capabilities, and are difficult to meet more advanced scene understanding, human-computer interaction or decision-making support needs, the present invention provides a method and system for unmanned vehicle surrounding target recognition based on open target detection.

[0006] The technical solutions provided by the embodiments of the present invention are as follows:

[0007] First aspect:

[0008] An embodiment of the present invention provides a method for identifying objects around an unmanned vehicle based on open object detection, comprising:

[0009] S1: Collect multiple environmental images around the unmanned vehicle;

[0010] S2: Input each environment image into the YOLOv5 open target detection model and output the target information set in each environment image;

[0011] S3: According to the target information set, the corresponding environment image is cropped to obtain multiple target images;

[0012] S4: extracting the image feature vectors of each target image;

[0013] S5: Input the image feature vector of each target image into the K-Means clustering model, and output multiple category clusters containing each target image;

[0014] S6: Input each category cluster into the LLaMA3 visual language model and output the category label and natural language description corresponding to each category cluster;

[0015] S7: Determine the target category in each target image based on the category labels and natural language descriptions corresponding to each category cluster, and complete the recognition of the surrounding targets of the unmanned vehicle.

[0016] Second aspect:

[0017] An embodiment of the present invention provides a system for identifying objects around an unmanned vehicle based on open object detection, comprising:

[0018] processor;

[0019] A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method for identifying objects around an unmanned vehicle based on open object detection as described in the first aspect is implemented.

[0020] The third aspect:

[0021] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the method for identifying targets around an unmanned vehicle based on open target detection as described in the first aspect is implemented.

[0022] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0023] In an embodiment of the present invention, by inputting each environmental image into the YOLOv5 open target detection model, outputting the target information set in each environmental image, and cropping the corresponding environmental image according to the target information set, a plurality of target images are obtained. It no longer relies on the training mechanism of closed categories, and can identify targets beyond the predefined class set. When faced with a large number of unknown categories in an actual open environment, there will be no misjudgment and missed detection, which significantly improves the reliability and environmental adaptability of the detection algorithm. By inputting the image feature vector of each target image into the K-Means clustering model, a plurality of category clusters containing each target image are output, and each category cluster is input into the LLaMA3 visual language model, and the category label and natural language description corresponding to each category cluster are output. Not only the target box and its category label are output, but also the in-depth understanding of the target semantic information and the natural language expression ability are output, which can meet more advanced scene understanding, human-computer interaction or decision support needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 A flowchart of a method for identifying objects around an unmanned vehicle based on open object detection provided by an embodiment of the present invention;

[0026] Figure 2 A schematic diagram of the structure of an unmanned vehicle surrounding target recognition system based on open target detection provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0028] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0029] In the embodiments of the present invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same. The terms "of," "corresponding," and "corresponding" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same.

[0030] In the embodiment of the present invention, sometimes a subscript such as W1 may be written as a non-subscript form such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0031] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0032] Reference Manual Figure 1 , which shows a flow chart of a method for identifying targets around an unmanned vehicle based on open target detection provided by an embodiment of the present invention.

[0033] An embodiment of the present invention provides a method for identifying objects around an unmanned vehicle based on open object detection. This method can be implemented by a device for identifying objects around an unmanned vehicle based on open object detection. The device can be a terminal or a server. The process flow of the method for identifying objects around an unmanned vehicle based on open object detection can include the following steps:

[0034] S1: Collect multiple environmental images around the unmanned vehicle.

[0035] In a possible implementation, S1 specifically includes:

[0036] The camera installed in the unmanned vehicle collects multiple environmental images around the unmanned vehicle.

[0037] In an embodiment of the present invention, by collecting images of the surrounding environment through a camera installed in an unmanned vehicle, not only can the timeliness and scene consistency of the image be ensured, but it also facilitates the system to directly utilize native sensor data, thereby improving data acquisition efficiency, target detection accuracy and environmental adaptability.

[0038] In a possible implementation, after S1 and before S2, the following is further included:

[0039] S1A: Resize each environment image:

[0040]

[0041]

[0042] in, Represents the tth environment image after resizing, Resize() represents the resizing operation, I t Represents the t-th environment image, W represents the width of the input image of the YOLOv5 open target detection model, H represents the height of the input image of the YOLOv5 open target detection model, J represents the environment image sequence, and T represents the total number of environment images.

[0043] S1B: Normalize each resized environment image:

[0044]

[0045] in, represents the t-th environment image after normalization, and Normalize() represents the normalization operation.

[0046] In an embodiment of the present invention, by resizing and normalizing each image after image acquisition, it is possible to effectively ensure that the image data is consistent with the input requirements of the YOLOv5 open object detection model, improve the detection accuracy and processing efficiency of the model, and at the same time enhance the system's adaptability to complex environments such as lighting changes and image noise, thereby providing a stable and reliable input basis for subsequent target recognition and classification processes.

[0047] S2: Input each environment image into the YOLOv5 open target detection model and output the target information set in each environment image.

[0048] It should be noted that the YOLOv5 open object detection model is a highly efficient object detection algorithm based on deep learning. While maintaining the YOLO series' hallmarks of high real-time performance and precision, it also accurately detects objects of known categories in images. It also integrates open set recognition mechanisms to expand its ability to perceive and label objects of unknown categories. By rapidly regressing the input image to generate object bounding boxes, their class labels, and confidence scores, YOLOv5 is widely used in scenarios such as autonomous driving and security surveillance, and is particularly well-suited for tasks requiring rapid multi-target recognition in complex environments.

[0049] The target information set is specifically:

[0050]

[0051]

[0052] Among them, P t represents the target information set in the t-th environment image, YOLOv5( ) represents the YOLOv5 open target detection model, p t,irepresents the information set of the i-th target in the t-th environment image, x t,i Indicates the X-axis coordinate of the detection box corresponding to the i-th target in the t-th environment image, y t,i represents the Y-axis coordinate of the detection box corresponding to the i-th target in the t-th environment image, w t,i Indicates the width of the detection box corresponding to the i-th target in the t-th environment image, h t,i represents the height of the detection box corresponding to the i-th target in the t-th environment image, c t,i Indicates the category number of the i-th target in the t-th environment image, s t,i represents the confidence score of the i-th target in the t-th environment image.

[0053] In an embodiment of the present invention, the YOLOv5 open target detection model is used to perform target detection on environmental images, which can efficiently and accurately extract the spatial position, category identification and confidence information of various known and unknown targets, and realize structured output of targets. It not only improves the perception accuracy and real-time performance of the unmanned vehicle to the surrounding environment, but also provides a stable and reliable input basis for subsequent target image screening, cluster analysis and language description modules, and enhances the system's adaptability to complex dynamic scenes.

[0054] S3: According to the target information set, the corresponding environment image is cropped to obtain multiple target images.

[0055] In an embodiment of the present invention, by cropping the environmental image according to the target information set, the target area image can be effectively extracted, irrelevant background can be removed, the quality and discriminability of the target image can be improved, and high-quality input can be provided for subsequent feature extraction and clustering analysis.

[0056] In a possible implementation, S3 specifically includes sub-steps S301 to S303:

[0057] S301: Filter the target information set based on the confidence threshold to determine the filtered target information set in each environment image:

[0058]

[0059] in, represents the set of filtered target information in the t-th environment image, θ s Represents the confidence threshold.

[0060] It should be noted that those skilled in the art can set the confidence threshold according to actual needs, and the present invention does not limit this.

[0061] S302: Using a non-maximum suppression algorithm, eliminate overlapping targets in the filtered target information set and determine the final target information set in each environment image:

[0062]

[0063] in, represents the final target information set in the t-th environmental image, NMS( ) represents the non-maximum suppression algorithm, θ iou Represents the intersection-over-union overlap threshold of the non-maximum suppression algorithm.

[0064] It's important to note that non-maximum suppression (NMS) is a post-processing algorithm commonly used in object detection. It aims to retain the optimal object box from multiple overlapping candidate detection boxes. Its basic principle is to prioritize the box with the highest confidence among multiple predicted boxes of the same category and suppress other boxes that overlap too closely with it based on a set intersection-over-union (IoU) threshold, thereby removing redundant and duplicate detection results. NMS effectively improves the accuracy and clarity of object detection, ensuring that each object is represented by only one high-quality detection box.

[0065] It should be noted that those skilled in the art can set the size of the intersection-over-union (IoU) overlap threshold according to actual needs, and the present invention does not limit this.

[0066] In a possible implementation, S302 specifically includes sub-steps S3021 and S3022:

[0067] S3021: Calculate the intersection-over-union ratio between two adjacent detection frames:

[0068]

[0069] Among them, IoU(B1,B2) represents the intersection-over-union ratio between the first detection box B1 and the second detection box B2. represents the area of ​​the intersection of the first detection frame B1 and the second detection frame B2, It represents the area of ​​the union of the first detection frame B1 and the second detection frame B2.

[0070] S3022: When the IoU is greater than the IoU overlap threshold, determine that the two adjacent detection frames have an overlapping relationship.

[0071] S3023: For two adjacent detection frames that have an overlapping relationship, retain the target frame with a higher confidence score and delete the target frame with a lower confidence score, thereby determining the final target information set in each environment image.

[0072] S303: According to the final target information set, the corresponding environment image is cropped to obtain multiple target images.

[0073] In an embodiment of the present invention, by setting a confidence threshold to screen detection targets, low-confidence targets can be effectively eliminated, improving data quality. A non-maximum suppression algorithm eliminates target overlap, ensuring that only a set of representative detection frames is retained for each true target, thereby improving recognition accuracy. By cropping the original image based on the final target information set, high-quality target image regions can be extracted, providing clear and discriminative input for subsequent feature extraction and classification and clustering steps, thereby comprehensively enhancing the system's stability, accuracy, and processing efficiency in complex environments.

[0074] S4: Extract the image feature vector of each target image.

[0075] In an embodiment of the present invention, by extracting image feature vectors from the cropped target image, the image can be converted from pixel-level representation to semantic vector representation, enabling the system to mathematically measure the similarities and differences between images, providing standard input for subsequent clustering analysis and unknown target classification.

[0076] In a possible implementation, S4 specifically includes sub-steps S401 to S403:

[0077] S401: Extracting local image feature vectors of each target image using the VGG19 algorithm.

[0078] It's important to note that VGG19 is a classic convolutional neural network (CNN) architecture proposed by the Visual Geometry Group at the University of Oxford. It comprises a 19-layer deep network structure that extracts image features layer by layer through multiple small (3x3) convolution kernels and max pooling operations. Due to its simple structure, ease of understanding, and controllable parameters, VGG19 is widely used in image classification and feature extraction tasks. This model achieved excellent results in the ImageNet image recognition challenge, particularly excelling at extracting local texture information (such as edges, corners, and textures) in images. Consequently, it is widely used for low-level feature extraction in visual perception systems.

[0079] Optionally, extract the local image feature vector according to the following formula:

[0080]

[0081] Among them, f VGG Represents the local image feature vector, VGG19 represents the VGG19 algorithm, I m Represents the target image.

[0082] S402: Extracting the global image feature vector of each target image using the DINOv2 algorithm.

[0083] It should be noted that DINOv2 is an advanced visual representation learning algorithm based on self-supervised learning and the Vision Transformer (ViT) architecture, launched by MetaAI. It does not require manually labeled data and learns high-level semantic features of images through comparative learning. DINOv2 can capture global structural information in images, modeling the shape, relationships, and semantics of objects in the image, and has excellent cross-frame consistency and transfer capabilities. Compared to traditional CNNs, DINOv2 is better at processing time-related data such as video frames, making it well-suited for tasks requiring stable visual representations, such as video analysis, object tracking, and unsupervised clustering.

[0084] Optionally, extract the global image feature vector according to the following formula:

[0085]

[0086] Among them, f DINO represents the global image feature vector, and DINOv2 represents the DINOv2 algorithm.

[0087] S403: Concatenate the local image feature vector and the global image feature vector to obtain an image feature vector, thereby completing the extraction of the image feature vectors of each target image.

[0088] Optionally, the local image feature vector and the global image feature vector are concatenated according to the following formula to obtain an image feature vector:

[0089]

[0090] Where f represents the image feature vector, and Concat() represents the concatenation operation.

[0091] In an embodiment of the present invention, by adopting the VGG19 algorithm to extract local image feature vectors and the DINOv2 algorithm to extract global image feature vectors, and splicing the feature dimensions of the two, an image joint feature representation with both local texture perception ability and global semantic modeling ability is constructed, which not only improves the integrity and discriminability of image expression, but also enhances the cross-frame consistency and stability when processing continuous video frames, thereby providing a more accurate, robust and efficient feature basis for subsequent image clustering analysis and unknown target classification tasks.

[0092] S5: Input the image feature vector of each target image into the K-Means clustering model, and output multiple category clusters containing each target image.

[0093] It's important to note that the K-Means clustering model is a commonly used unsupervised learning algorithm designed to automatically classify samples into K clusters based on the similarity between data points (typically measured by Euclidean distance). Its basic process involves randomly initializing K cluster centers, iteratively assigning each sample to the closest cluster, and then updating each cluster center until the cluster centers converge or the maximum number of iterations is reached. K-Means is suitable for clustering high-dimensional feature vectors and can discover the inherent structure of data without pre-labeling. It has wide applications in tasks such as image classification and object recognition.

[0094] In an embodiment of the present invention, by inputting the image feature vector of each target image into the K-Means clustering model for cluster analysis, the target image can be automatically classified into multiple category clusters based on semantic similarity, thereby realizing open target classification under unknown category conditions.

[0095] In a possible implementation, S5 specifically includes sub-steps S501 and S502:

[0096] S501: Determine the optimal number of category clusters based on the elbow rule according to the image feature vectors of each target image.

[0097] It should be noted that the elbow rule is a heuristic method for determining the optimal number of clusters (K) in a clustering algorithm. Its core idea is that as the number of clusters increases, the total squared error (SSE) of the clustering results will continue to decrease. However, after the number of clusters reaches a certain critical point, the rate of decline in SSE slows significantly, forming an "elbow" shape. The number of clusters corresponding to this "elbow" is the one that achieves optimal clustering results while maintaining a moderate level of complexity. By balancing cluster compactness with model simplicity, the elbow rule helps to appropriately determine the number of clusters in unsupervised learning, thereby improving the scientific nature and practicality of cluster analysis.

[0098] In an embodiment of the present invention, by dynamically determining the optimal number of category clusters based on the elbow rule, it can automatically adapt to the target distribution in different scenarios, avoid classification bias caused by a fixed number of clusters, and thus improve clustering rationality and system generalization ability.

[0099] In a possible implementation, S501 specifically includes sub-steps S5011 to S5016:

[0100] S5011: Set the maximum number of category clusters.

[0101] S5012: Calculate the total square error of the current number of category clusters based on the image feature vectors of each target image:

[0102]

[0103] Among them, SEE(K) represents the total square error of the current category cluster number K, K represents the current category cluster number, where K=1,2,…K max , K max Indicates the maximum number of category clusters, f j represents the image feature vector of the j-th target image, C k represents the member set of the k-th category cluster, μ k represents the central feature vector of the k-th category cluster, and || || represents the Euclidean norm.

[0104] S5013: Calculate the first-order difference of the number of current category clusters based on the total square error:

[0105]

[0106] Among them, ΔD(K) represents the first-order difference of the current number of category clusters K, D(K-1) represents the total square error of the previous number of category clusters K-1, and D(K) represents the total square error of the current number of category clusters K.

[0107] S5014: Based on the first-order difference, calculate the second-order difference of the number of current category clusters:

[0108]

[0109] Where ΔD 2 (K) represents the second-order difference of the current category cluster number K, and ΔD(K+1) represents the first-order difference of the next category cluster number K+1.

[0110] S5015: Determine whether the maximum number of category clusters has been reached. If so, proceed to step S5016. Otherwise, add 1 to the current number of category clusters and return to step S5012.

[0111] S5016: Output the number of category clusters with the largest second-order difference as the optimal number of category clusters:

[0112]

[0113] Among them, K opt Indicates the optimal number of category clusters, and arg max indicates the maximum value.

[0114] In this embodiment of the present invention, by automatically calculating the optimal number of category clusters based on the elbow rule, the clustering granularity can be flexibly adjusted according to the target image feature distribution in different scenarios, avoiding clustering errors caused by setting a fixed number of clusters, improving target classification accuracy and the system's adaptability. This step also provides a precise categorization foundation for subsequent cluster analysis and text description generation, enhancing the system's open recognition capabilities and semantic expression consistency, significantly improving the autonomous vehicle's intelligent target perception and recognition capabilities in complex environments.

[0115] S502: Clustering each target image using the K-Means clustering algorithm according to the optimal number of category clusters, and outputting multiple category clusters containing each target image.

[0116] In an embodiment of the present invention, target images are classified through the K-Means clustering algorithm, and semantically similar targets can be automatically aggregated, providing structured and semantically consistent basic data support for subsequent text generation, visualization management and new category recognition, significantly enhancing the target understanding and classification capabilities of unmanned vehicles in open environments.

[0117] In a possible implementation, S502 specifically includes sub-steps S5021 to S5026:

[0118] S5021: Initialize category cluster centers:

[0119]

[0120] in, Indicates the The central feature vector of the category cluster, where , K opt represents the optimal number of category clusters, Represents the set of image feature vectors of each target image to be clustered.

[0121] S5022: Set the maximum number of iterations.

[0122] S5023: Calculate the Euclidean distance between the image feature vector of each target image and the center of the category cluster, and assign the image feature vector of each target image to the category cluster with the closest Euclidean distance, so as to cluster each target image:

[0123]

[0124] in, Indicates the The member set of the category cluster, f j represents the image feature vector of the j-th target image, The image feature vector representing the jth target image and the The distance between the central feature vectors of the class clusters, The image feature vector representing the jth target image and the The distance between the central feature vectors of the class clusters.

[0125] S5024: Update category cluster center:

[0126]

[0127] in, Indicates the updated The central feature vector of each class cluster.

[0128] S5025: Determine whether the maximum number of iterations has been reached. If so, proceed to step S5026. Otherwise, return to step S5023.

[0129] S5026: Output multiple category clusters containing each target image.

[0130] In this embodiment of the present invention, image feature vectors are clustered using the K-Means clustering algorithm, automatically dividing target images into multiple semantically consistent clusters based on feature similarities. This process, through rational initialization, distance calculation, iterative optimization, and cluster center updates, not only ensures the accuracy and stability of clustering results, but also balances system operational efficiency. This enhances the system's self-organizing and open recognition capabilities when processing large-scale, unlabeled target images, thereby providing a solid data foundation for subsequent semantic understanding and text generation modules.

[0131] S6: Input each category cluster into the LLaMA3 visual language model and output the category label and natural language description corresponding to each category cluster.

[0132] It should be noted that the LLaMA3 visual language model is a large-scale, multimodal language model that combines image understanding and natural language processing capabilities. It can simultaneously receive image information and textual prompts to generate semantic understanding, textual descriptions, or category judgments of the image content. Based on the LLaMA3 architecture launched by Meta, this model, which integrates a visual encoder and a language decoder, possesses powerful cross-modal reasoning and semantic generation capabilities. It is widely used in tasks such as image and text understanding, object recognition, and image question and answering. It is particularly suitable for scenarios where unknown or complex objects need to be automatically described and labeled.

[0133] In an embodiment of the present invention, by inputting the category clusters obtained by clustering into the visual language model LLaMA3, the category label and natural language description corresponding to each cluster can be automatically output, which not only improves the system's ability to interpret and express semantics of target categories, but also enhances the ability to recognize and describe unknown category targets.

[0134] In a possible implementation, S6 specifically includes sub-steps S601 and S602:

[0135] S601: Based on each category cluster, construct the input tuple of the LLaMA3 visual language model:

[0136]

[0137] in, Indicates the The input tuple of the LLaMA3 visual language model corresponding to the category cluster, Indicates the A unique identifier for each category cluster, Indicates the A list of preliminary textual descriptions of all images in the category cluster.

[0138] S602: Input the input tuple to the LLaMA3 visual language model, and output the category label and natural language description corresponding to each category cluster:

[0139]

[0140] in, Indicates the The category labels of the category clusters, Indicates the The natural language description of the category clusters, LLaMA3 ( ) represents the LLaMA3 visual language model.

[0141] In this embodiment of the present invention, a structured input tuple is constructed, combining category cluster identifiers with image semantic information and inputting them into the visual language model LLaMA3, which then outputs corresponding category labels and natural language descriptions. This approach not only achieves the semantic conversion from image features to language information, but also improves the system's ability to express semantics for both known and unknown targets, enhancing recognition adaptability in open environments. Furthermore, the generated labels and descriptions are highly readable and interactive, facilitating human-machine understanding, result verification, and decision-making support, significantly enhancing the intelligence and user-friendliness of unmanned vehicle perception systems.

[0142] S7: Determine the target category in each target image based on the category labels and natural language descriptions corresponding to each category cluster, and complete the recognition of the surrounding targets of the unmanned vehicle.

[0143] In an embodiment of the present invention, by determining the target category for each target image based on the category label corresponding to the category cluster and the natural language description, the specific mapping of image semantic information and category assignment can be achieved, thereby enhancing the interpretability and interactive friendliness of the recognition results.

[0144] Furthermore, the target image is visualized by combining the category labels and natural language descriptions corresponding to each category cluster, and a graphic display interface containing example images, text descriptions and category attribution information is constructed for decision makers' reference and to support user interaction.

[0145] In an embodiment of the present invention, by combining the labels, natural language descriptions and example images corresponding to the category clusters to display them, a visual interface combining pictures and text is constructed, which can improve the interpretability and intuitiveness of the recognition results, facilitate decision makers to quickly understand and confirm the target category, and at the same time support manual feedback and correction, thereby enhancing the system's human-computer interaction capabilities.

[0146] Furthermore, the object category identified in each target image is considered a known category, and multiple images of the environment surrounding the unmanned vehicle are recollected based on this. These images are then fed into the YOLOv5 open-world object detection model, which extracts the target information set (including bounding boxes, category confidence, etc.) and determines whether the object in the image belongs to a known category. If the object belongs to a known category, processing is skipped and new environment images are collected. If the object belongs to an unknown category, the corresponding target information set is filtered and the image is cropped accordingly, resulting in multiple images containing the unknown object. Feature vectors of these images are then extracted and fed into a K-Means clustering model to cluster the unknown objects and generate multiple category clusters. Each category cluster is then fed into the LLaMA3 visual language model, which outputs the corresponding category label and natural language description. These newly identified unknown categories and their labels are then incorporated into the set of known categories. This cycle continues, enabling real-time discovery and recognition of new categories of objects in the environment.

[0147] In an embodiment of the present invention, by introducing a cyclical new category discovery and update mechanism, the system can set the initially identified target category as a known category during the operation of the unmanned vehicle. Based on the real-time environmental imagery, the system uses the YOLOv5 open-world target detection model to identify and judge targets. Combined with image cropping, feature extraction, and K-Means clustering, it effectively classifies unknown targets and outputs corresponding category labels and natural language descriptions through the LLaMA3 visual language model. Ultimately, the identified new categories are incorporated into the set of known categories. This cyclical mechanism enables the dynamic expansion and real-time updating of the target category set, thereby endowing the system with continuous perception and adaptive learning capabilities in open-world scenarios.

[0148] Reference Manual Figure 2 , which shows a structural schematic diagram of an unmanned vehicle surrounding target recognition system based on open target detection provided by the present invention.

[0149] The present invention further provides an unmanned vehicle surrounding target recognition system 20 based on open target detection, which is applied to the above-mentioned unmanned vehicle surrounding target recognition method based on open target detection, comprising:

[0150] Processor 201;

[0151] The memory 202 stores computer-readable instructions. When the computer-readable instructions are executed by the processor 201 , the method for identifying objects around an unmanned vehicle based on open object detection as described in the method embodiment is implemented.

[0152] The unmanned vehicle surrounding target recognition system 20 based on open target detection provided by the present invention can execute the above-mentioned unmanned vehicle surrounding target recognition method based on open target detection and achieve the same or similar technical effects. To avoid repetition, the present invention will not go into details.

[0153] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0154] In an embodiment of the present invention, by inputting each environmental image into the YOLOv5 open target detection model, outputting the target information set in each environmental image, and cropping the corresponding environmental image according to the target information set, a plurality of target images are obtained. It no longer relies on the training mechanism of closed categories, and can identify targets beyond the predefined class set. When faced with a large number of unknown categories in an actual open environment, there will be no misjudgment and missed detection, which significantly improves the reliability and environmental adaptability of the detection algorithm. By inputting the image feature vector of each target image into the K-Means clustering model, a plurality of category clusters containing each target image are output, and each category cluster is input into the LLaMA3 visual language model, and the category label and natural language description corresponding to each category cluster are output. Not only the target box and its category label are output, but also the in-depth understanding of the target semantic information and the natural language expression ability are output, which can meet more advanced scene understanding, human-computer interaction or decision support needs.

[0155] It should be understood that the processor in the embodiments of the present invention may be a central processing unit (CPU), but may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0156] It should also be understood that the memory in the embodiments of the present invention may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0157] The above embodiments can be implemented in whole or in part via software, hardware (e.g., circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are fully or partially performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0158] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0159] In this disclosure, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0160] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0161] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0162] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0163] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of the device or unit, which can be electrical, mechanical or other forms.

[0164] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0165] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0166] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical disks.

[0167] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the method for identifying targets around an unmanned vehicle based on open target detection as described in the method embodiment is implemented.

[0168] The computer-readable storage medium provided by the present invention can implement the steps and effects of the method for identifying targets around an unmanned vehicle based on open target detection in the above-mentioned method embodiment. To avoid repetition, the present invention will not go into details.

[0169] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0170] In an embodiment of the present invention, by inputting each environmental image into the YOLOv5 open target detection model, outputting the target information set in each environmental image, and cropping the corresponding environmental image according to the target information set, a plurality of target images are obtained. It no longer relies on the training mechanism of closed categories, and can identify targets beyond the predefined class set. When faced with a large number of unknown categories in an actual open environment, there will be no misjudgment and missed detection, which significantly improves the reliability and environmental adaptability of the detection algorithm. By inputting the image feature vector of each target image into the K-Means clustering model, a plurality of category clusters containing each target image are output, and each category cluster is input into the LLaMA3 visual language model, and the category label and natural language description corresponding to each category cluster are output. Not only the target box and its category label are output, but also the in-depth understanding of the target semantic information and the natural language expression ability are output, which can meet more advanced scene understanding, human-computer interaction or decision support needs.

[0171] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

[0172] There are a few points to note:

[0173] (1) The drawings of the embodiments of the present invention only relate to the structures related to the embodiments of the present invention. Other structures may refer to conventional designs.

[0174] (2) For the sake of clarity, the thickness of layers or regions in the drawings used to describe the embodiments of the present invention are exaggerated or reduced, that is, these drawings are not drawn to scale. It is understood that when an element such as a layer, film, region, or substrate is referred to as being "on" or "under" another element, the element may be "directly on" or "under" the other element or intervening elements may be present.

[0175] (3) In the absence of conflict, the embodiments of the present invention and the features therein may be combined with each other to form new embodiments.

[0176] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for identifying targets around an unmanned vehicle based on open target detection, characterized in that: include: S1: Collect multiple environmental images around the unmanned vehicle; S2: Input each of the environment images into the YOLOv5 open target detection model, and output a set of target information in each of the environment images; S3: cropping the corresponding environment image according to the target information set to obtain multiple target images; S4: extracting image feature vectors of each target image; S5: inputting the image feature vectors of each of the target images into a K-Means clustering model, and outputting a plurality of category clusters including each of the target images; S6: Input each of the category clusters into the LLaMA3 visual language model, and output the category label and natural language description corresponding to each of the category clusters; S7: Determine the target category in each of the target images based on the category labels and natural language descriptions corresponding to each of the category clusters, and complete the recognition of the surrounding targets of the unmanned vehicle.

2. The method for identifying targets around an unmanned vehicle based on open target detection according to claim 1, characterized in that: The S1 is specifically: A plurality of environmental images around the unmanned vehicle are collected by a camera installed in the unmanned vehicle.

3. The method for identifying targets around an unmanned vehicle based on open target detection according to claim 1, characterized in that: After S1 and before S2, the following is also included: S1A: Resize each of the environment images: ； ； in, Represents the tth environment image after resizing, Resize() represents the resizing operation, I t represents the t-th environment image, W represents the width of the input image of the YOLOv5 open target detection model, H represents the height of the input image of the YOLOv5 open target detection model, J represents the environment image sequence, and T represents the total number of environment images; S1B: Normalize each resized environment image: ； in, represents the t-th environment image after normalization, and Normalize() represents the normalization operation.

4. The method for identifying targets around an unmanned vehicle based on open target detection according to claim 3, characterized in that: The target information set is specifically: ； ； Among them, P t represents the target information set in the t-th environment image, YOLOv5( ) represents the YOLOv5 open target detection model, p t,i represents the information set of the i-th target in the t-th environment image, x t,i Indicates the X-axis coordinate of the detection box corresponding to the i-th target in the t-th environment image, y t,i represents the Y-axis coordinate of the detection box corresponding to the i-th target in the t-th environment image, w t,i Indicates the width of the detection box corresponding to the i-th target in the t-th environment image, h t,i represents the height of the detection box corresponding to the i-th target in the t-th environment image, c t,i Indicates the category number of the i-th target in the t-th environment image, s t,i represents the confidence score of the i-th target in the t-th environment image.

5. The method for identifying targets around an unmanned vehicle based on open target detection according to claim 4, characterized in that: The S3 specifically includes: S301: Filtering the target information set based on a confidence threshold to determine a filtered target information set in each of the environment images: ； in, represents the set of filtered target information in the t-th environment image, θ s represents the confidence threshold; S302: Eliminate overlapping targets in the screened target information set by using a non-maximum suppression algorithm, and determine a final target information set in each of the environment images: ； in, represents the final target information set in the t-th environmental image, NMS( ) represents the non-maximum suppression algorithm, θ iou Represents the intersection-over-union overlap threshold of the non-maximum suppression algorithm; S303: According to the final target information set, the corresponding environment image is cropped to obtain a plurality of target images.

6. The method for identifying targets around an unmanned vehicle based on open target detection according to claim 1, characterized in that: The S5 specifically includes: S501: determining the optimal number of category clusters based on the elbow rule according to the image feature vectors of each target image; S502: Clustering each of the target images using a K-Means clustering algorithm according to the optimal number of category clusters, and outputting a plurality of category clusters including each of the target images.

7. The method for identifying targets around an unmanned vehicle based on open target detection according to claim 6, characterized in that: The S501 specifically includes: S5011: Set the maximum number of category clusters; S5012: Calculate the total square error of the current number of category clusters based on the image feature vectors of each target image: ； Among them, SEE(K) represents the total square error of the current category cluster number K, K represents the current category cluster number, where K=1,2,…K max , K max Indicates the maximum number of category clusters, f j represents the image feature vector of the j-th target image, C k represents the member set of the k-th category cluster, μ k represents the central feature vector of the k-th category cluster, || || represents the Euclidean norm; S5013: Calculate the first-order difference of the number of current category clusters based on the total square error: ； Among them, ΔD(K) represents the first-order difference of the current category cluster number K, D(K-1) represents the total square error of the previous category cluster number K-1, and D(K) represents the total square error of the current category cluster number K; S5014: Calculate the second-order difference of the number of current category clusters based on the first-order difference: ； Where ΔD 2 (K) represents the second-order difference of the current category cluster number K, and ΔD(K+1) represents the first-order difference of the next category cluster number K+1; S5015: Determine whether the maximum number of category clusters has been reached; if so, proceed to step S5016; otherwise, add 1 to the current number of category clusters and return to step S5012; S5016: Output the number of category clusters with the largest second-order difference as the optimal number of category clusters: ； Among them, K opt Indicates the optimal number of category clusters, and arg max indicates the maximum value.

8. The method for identifying targets around an unmanned vehicle based on open target detection according to claim 6, characterized in that: The S502 specifically includes: S5021: Initialize category cluster centers: ； in, Indicates the The central feature vector of the category cluster, where , K opt represents the optimal number of category clusters, A set of image feature vectors representing each target image to be clustered; S5022: Set the maximum number of iterations; S5023: Calculate the Euclidean distance between the image feature vector of each target image and the center of the category cluster, and assign the image feature vector of each target image to the category cluster with the closest Euclidean distance, so as to cluster each target image: ； in, Indicates the The member set of the category cluster, f j represents the image feature vector of the j-th target image, The image feature vector representing the jth target image and the The distance between the central feature vectors of the class clusters, The image feature vector representing the jth target image and the The distance between the central feature vectors of the class clusters; S5024: Update the category cluster center: ； in, Indicates the updated The central feature vector of each category cluster; S5025: Determine whether the maximum number of iterations has been reached; if so, proceed to step S5026; otherwise, return to step S5023; S5026: Output multiple category clusters containing each of the target images.

9. The method for identifying targets around an unmanned vehicle based on open target detection according to claim 1, characterized in that: The S6 specifically includes: S601: Constructing the input tuple of the LLaMA3 visual language model according to each of the category clusters: ； in, Indicates the The input tuple of the LLaMA3 visual language model corresponding to the category cluster, Indicates the A unique identifier for each category cluster, Indicates the A list of preliminary text descriptions of all images in the category cluster; S602: Input the input tuple to the LLaMA3 visual language model, and output the category label and natural language description corresponding to each category cluster: ； in, Indicates the The category labels of the category clusters, Indicates the The natural language description of the category clusters, LLaMA3 ( ) represents the LLaMA3 visual language model.

10. An unmanned vehicle surrounding target recognition system based on open target detection, characterized in that: include: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method for identifying objects around an unmanned vehicle based on open target detection according to any one of claims 1 to 9 is implemented.

Images