Target identification method, device, non-volatile storage medium and computer equipment

Abstract

The present invention discloses a target recognition method, device, non-volatile storage medium and computer equipment. The method comprises: collecting image data in a traffic scene; inputting the image data and target text into a preset target multimodal neural network model to obtain target image data, wherein the target image data is image data annotated based on the target text, and the target text represents the text describing the scene in the image data; inputting the target image data into a target recognition model to obtain a target recognition result, wherein the target recognition model is used to recognize traffic targets in the image, and the traffic targets include one of the following: traffic signs, pedestrians and vehicles. The present invention solves the technical problem that the current target detection in traffic scenes is that the acquired image data is not clear enough due to the complexity of the traffic scenes, and the target recognition results are not accurate enough.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a target recognition method, device, non-volatile storage medium and computer equipment. Background Art

[0002] Current models often perform poorly for complex traffic scenarios (e.g., dynamic environments, multi-target scenarios). In a crowded traffic environment, vehicles, pedestrians, traffic signals, and other targets may occlude or overlap each other, making the target recognition task more difficult. Environmental complexity: The diversity of traffic scenarios makes target detection more challenging, especially in complex dynamic and multi-target scenarios, where occlusion, blur, and changes between targets make it difficult for models to detect accurately.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present invention provide a target recognition method, apparatus, non-volatile storage medium and computer device to at least solve the technical problem that the current target detection in traffic scenes is not clear enough due to the complexity of traffic scenes, and the target recognition results are not accurate enough.

[0005] According to one aspect of an embodiment of the present invention, there is provided a target recognition method, comprising: collecting image data in a traffic scene; inputting the image data and target text into a preset target multimodal neural network model to obtain target image data, wherein the target image data is image data annotated based on the target text, and the target text represents text describing the scene in the image data; inputting the target image data into a target recognition model to obtain a target recognition result, wherein the target recognition model is used to recognize traffic targets in the image, and the traffic targets include one of the following: traffic signs, pedestrians, and vehicles.

[0006] Optionally, before inputting the image data and the target text into a preset target multimodal neural network model to obtain the target image data, the method includes: obtaining an original multimodal neural network model and a data set, wherein the data set includes traffic targets and description texts corresponding to the traffic targets; and training the original multimodal neural network model using the data set to obtain a target multimodal neural network model.

[0007] Optionally, the image data and the target text are input into a preset target multimodal neural network model to obtain target image data, including: inputting the image data and the target text into an attention mechanism module in the target multimodal neural network to obtain target relevance of the image data and the target text; based on the target relevance, annotating the image data to obtain the target image data.

[0008] Optionally, before inputting the target image data into the target recognition model to obtain the target recognition result corresponding to the image data, it includes: acquiring a plurality of sample images of labeled traffic targets; using a plurality of sample images of labeled traffic targets to train a preset original target recognition model to obtain a target recognition model.

[0009] Optionally, obtaining a plurality of sample images of labeled traffic targets includes: obtaining a plurality of sample images and description texts corresponding to each of the plurality of sample images; inputting the plurality of sample images and the description texts corresponding to each of the plurality of sample images into a target multimodal neural network to obtain a plurality of sample images of labeled traffic targets.

[0010] Optionally, when there is a traffic target in the image data that is identified as a new target, a description text of the new target is received based on the target account input, wherein the new target represents a traffic target that has not obtained a target recognition result; the image data and the description text are input into the target multimodal neural network model to obtain labeled image data; and the target recognition model is trained using the labeled image data to obtain a trained target recognition model.

[0011] According to another aspect of an embodiment of the present invention, a target recognition device is also provided, including: an acquisition module for acquiring image data in a traffic scene; an input module for inputting the image data and target text into a preset target multimodal neural network model to obtain target image data, wherein the target image data is image data annotated based on the target text, and the target text represents text describing the scene in the image data; a recognition module for inputting the target image data into a target recognition model to obtain a target recognition result, wherein the target recognition model is used to identify traffic targets in the image, and the traffic targets include one of the following: traffic signs, pedestrians, and vehicles.

[0012] According to another aspect of an embodiment of the present invention, a non-volatile storage medium is provided, the non-volatile storage medium including a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute any one of the above-mentioned target recognition methods.

[0013] According to yet another aspect of an embodiment of the present invention, a computer device is provided. The computer device includes a processor, and the processor is used to run a program. When the program is run, any one of the above-mentioned target recognition methods is executed.

[0014] According to yet another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, any one of the above target recognition methods is implemented.

[0015] In an embodiment of the present invention, a target recognition method is adopted to collect image data in a traffic scene; the image data and the target text are input into a preset target multimodal neural network model to obtain target image data, wherein the target image data is image data annotated based on the target text, and the target text represents the text describing the scene in the image data; the target image data is input into a target recognition model to obtain a target recognition result, wherein the target recognition model is used to recognize traffic targets in the image, and the traffic targets include one of the following: traffic signs, pedestrians and vehicles, thereby achieving the purpose of combining image data and text information for target recognition, thereby achieving the technical effect of improving the accuracy of target recognition, and further solving the technical problem that the current target detection in traffic scenes is not clear enough due to the complexity of the traffic scenes, and further leading to the inaccurate result of target recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0017] Figure 1 A hardware structure block diagram of a computer terminal for implementing a target recognition method is shown;

[0018] Figure 2 is a flow chart of a target recognition method provided according to an embodiment of the present invention;

[0019] Figure 3 is a schematic diagram of the overall structure of a target recognition method provided according to an optional embodiment of the present invention;

[0020] Figure 4 is a structural block diagram of a target recognition device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] According to an embodiment of the present invention, a method embodiment of a target identification method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0024] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a target recognition method. Figure 1 As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0025] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0026] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the target recognition method in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the target recognition method of the above-mentioned application program. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0027] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0028] Figure 2 is a flow chart of a target recognition method according to an embodiment of the present invention. Figure 2 As shown, the method comprises the following steps:

[0029] Step S202: collecting image data in a traffic scene.

[0030] In this step, fixed surveillance cameras can be used to collect image data in traffic scenes. Fixed cameras can be installed at key locations such as major traffic routes, intersections, and accident-prone areas to continuously collect data. Mobile devices such as vehicle-mounted cameras, drones, and mobile phones can also be used to collect data while driving or flying to obtain more dynamic and changing traffic scenes.

[0031] Step S204, inputting the image data and the target text into a preset target multimodal neural network model to obtain target image data, wherein the target image data is image data annotated based on the target text, and the target text represents text describing the scene in the image data.

[0032] In this step, the image data and the target text are input into the target multimodal neural network model. The convolutional neural network module in the multimodal neural network model will extract the image features from the image data. The first layer of the convolutional neural network module (CNN) is usually a convolutional layer, which applies a series of convolution kernels (or filters) to slide on the image to detect local features in the image, such as edges, textures, etc. Each convolution kernel is responsible for detecting different features. The convolutional layer is usually followed by an activation function, such as ReLU (Rectified Linear Unit), which is used to introduce nonlinearity and enhance the learning ability of the model. The activation function transforms the output of the convolutional layer, removes negative values, retains positive values, and further highlights the important features in the image. The pooling layer is usually located after the convolutional layer to reduce the size of the feature map, reduce the computational complexity, and maintain the key features of the image. Common pooling operations include max pooling and average pooling, which retain the maximum value or average value of the local area respectively. The CNN model usually contains multiple layers of convolution and pooling operations, and each layer further extracts the deep features of the image. As the number of layers increases, the model can learn more abstract and complex feature expressions, such as the shape and structure of objects. At the end of CNN, there are usually one or more fully connected layers to convert the feature map output by the convolution layer into a fixed-length feature vector. The fully connected layer can further integrate the extracted features for classification or recognition tasks. After the above processing, the CNN model will eventually generate a feature vector, which contains the deep feature representation of the image. Specifically, in traffic scenes, CNN can learn high-level features of the scene, such as road type (urban roads, highways), weather conditions, traffic flow density, etc. These features are crucial for understanding traffic conditions. CNN can also estimate the position, size and direction of targets in the image, which is very important for positioning in target detection tasks. For complex traffic scenes, the model can learn the relative position and spatial relationship between different targets, such as the distance between vehicles and pedestrians, the actual position relationship between traffic signs and roads, etc. These features are extracted and processed by CNN's convolutional layers, pooling layers, fully connected layers, and possible attention mechanisms, and finally form a vector, which contains the comprehensive feature representation of the image in the traffic scene. These features can be further fed into models for classification, detection, or other high-level vision tasks to achieve recognition and understanding of traffic objects. In specific tasks, such as traffic object detection, the model may focus on certain types of features, such as the shape and color of specific traffic signs, to improve the accuracy and reliability of detection.

[0033] In the language analysis module of the multimodal neural network model, text features in the target text can be extracted. The input text is first converted into a vector representation through a word embedding layer and then passed to a series of Transformer encoder layers. Each encoder layer contains a multi-head self-attention mechanism and a feed-forward neural network, which work together on the input word embedding to generate a text feature representation that takes into account contextual dependencies. After being processed by the Transformer encoder layer, the model generates feature vectors for each word or text sequence. These feature vectors contain the semantic information of the words in the text as well as their position and contextual relationship in the text. The final text features can be a vector representation of the entire text sequence or a vector representation of a specific word or phrase, depending on the task requirements and model structure.

[0034] After obtaining the image features and text features, these features can be fused through the attention mechanism network in the cross-modal neural network to generate a joint representation of the image and text, that is, the target image data. Here, "target image data" actually refers to the image data representation that accurately matches the text annotation information after model fusion and understanding. The image features and text features are input into a specific fusion layer in the multimodal neural network. The fusion layer can use different techniques to combine the two features, such as using the attention mechanism to calculate the correlation between the image features and the text features, assign weights to each feature, and thus highlight the task-related parts. The attention mechanism can help the model locate the target area that matches the text description in a complex image. It is also possible to establish a shared feature space through a joint encoder, in which both the image features and the text features are converted into representations in this space. The joint encoder strengthens the correlation between the features by learning the correspondence between the two modalities, which helps the model to reason without labeled data. It is also possible to combine the attention of the image and text through a multimodal attention network, while considering the local and global information of the two modalities to generate a richer feature representation. The fused features are deeply interacted and processed through a multi-layer neural network. These network layers may include fully connected layers, more complex attention mechanisms, sequence models such as LSTM (Long Short-Term Memory) or GRU (Gated Recurrent Unit) to capture the temporal or contextual relationships between features.

[0035] The model uses the fused and interacted features to perform cross-modal reasoning, that is, reasoning based on the relationship between text descriptions and image content. For example, in the task of traffic target detection, the model can understand the text description of "traffic light" and the actual position and status of the traffic light in the image, and can assist in recognition through text information even in the case of poor image quality or target occlusion. After processing by the multimodal neural network, the model generates an output representation that combines image and text information. This representation can be a more accurate target detection box, target category probability distribution, a detailed description of the target, etc., depending on the needs of the downstream task.

[0036] Specifically, a multimodal neural network model including a convolutional neural network and a language model may be used, and the image data and the target text may be directly input into the model to obtain the target image data.

[0037] Through the above process, the multimodal neural network model can effectively combine image features and text features to generate richer and more accurate feature representations, thereby achieving more accurate recognition and classification in target detection tasks, especially when dealing with complex traffic scenarios and zero-sample or small-sample learning tasks. It can significantly improve the performance and generalization ability of the model.

[0038] Step S206, inputting the target image data into a target recognition model to obtain a target recognition result, wherein the target recognition model is used to recognize traffic targets in the image, and the traffic targets include one of the following: traffic signs, pedestrians, and vehicles.

[0039] In this step, the target image data, i.e. the annotated image data, is input into the target recognition model to obtain the recognition result of the traffic target in the image, wherein the target recognition model, such as a deep learning convolutional neural network (CNN) or a Transformer-based model, can learn the mapping relationship between image features and traffic targets based on the training data. The annotated image data is input into the target recognition model, and the feature extraction layer in the model, such as the convolution layer, extracts the features describing the traffic target from the image. These features may include the shape, texture, color, etc. of the object. The model then uses a detection algorithm (such as Faster R-CNN, YOLO, SSD, etc.) to locate the traffic target in the image, usually in the form of a bounding box. For each detected target, the model will classify it into predefined categories such as traffic signs, pedestrians, vehicles, etc. The model will output the bounding box coordinates, category labels, and confidence scores for each target.

[0040] The target recognition model can be applied to scenarios such as real-time monitoring and the perception system of autonomous vehicles to provide accurate traffic target recognition results. In addition, as new traffic signs, pedestrian behavior patterns, or vehicle types emerge, the model needs to be regularly updated and retrained to keep it up to date and accurate. Through the above steps, the target recognition model can effectively identify key targets such as traffic signs, pedestrians, and vehicles in the image, providing real-time environmental perception capabilities for intelligent transportation systems and autonomous driving technologies, which is crucial to improving road safety, traffic efficiency, and the autonomous driving capabilities of autonomous vehicles.

[0041] Figure 3 is a schematic diagram of the overall structure of a target recognition method provided according to an optional embodiment of the present invention, such as Figure 3 As shown in the figure, traffic images can be extracted through convolutional neural networks to generate representations of visual information. Text descriptions can be extracted through language models such as BERT to generate representations of semantic information. The cross-modal attention mechanism combines the features of images and texts to strengthen the correlation between them. Finally, the visual and language information are combined to output the detection results for target recognition and classification.

[0042] Through the above steps, the purpose of combining image data and text information for target recognition is achieved, thereby realizing the technical effect of improving the accuracy of target recognition, and further solving the technical problem that the image data obtained is not clear enough and the target recognition result is not accurate enough in the current target detection in traffic scenes because the traffic scenes are relatively complex.

[0043] As an optional embodiment, before inputting image data and target text into a preset target multimodal neural network model to obtain target image data, it includes: obtaining an original multimodal neural network model and a data set, wherein the data set includes traffic targets and description texts corresponding to the traffic targets; and using the data set to train the original multimodal neural network model to obtain a target multimodal neural network model.

[0044] Optionally, the multimodal neural network model can be pre-trained first. For example, it can be pre-trained in multiple traffic scenarios by leveraging existing large datasets (such as COCO, Visual Genome, etc.). The original multimodal neural network model is usually based on existing architectures, such as CLIP, ALIGN, or ViLT, which have been pre-trained on large-scale image-text pairing datasets and have preliminary cross-modal information fusion capabilities. The model may contain multiple components, such as CNN for image feature extraction, Transformer for text feature extraction, and cross-modal fusion layers for learning the association between images and text. For the object detection task in traffic scenes, the dataset should contain a large number of image samples of traffic objects and text descriptions of these objects. The text description can be detailed, such as "a pedestrian is crossing a zebra crossing", or a simple category label, such as "pedestrian". The dataset needs to cover a variety of traffic objects to ensure that the model can generalize to multiple scenarios. Before training the model, the images and text in the dataset can be pre-processed. Image preprocessing may include resizing, normalization, etc.; text preprocessing may involve word segmentation and word embedding. In addition, it is necessary to ensure that each image has a corresponding text description, and these descriptions are accurate, comprehensive, and can reflect the image content, so as to facilitate the learning of the correspondence between images and texts during model training. The original multimodal neural network model is trained using the prepared dataset. The goal of model training is to enable the model to extract and learn relevant features from images and texts, and to effectively fuse these features to achieve the detection and recognition of traffic targets. During the training process, the model will try to minimize the prediction difference between image features and text features until the model performs satisfactorily on the dataset. In addition, the training may also include hyperparameter tuning such as regularization, optimizer selection, and learning rate adjustment. After sufficient training and optimization, the original multimodal neural network model will be transformed into a target multimodal neural network model, which has high performance in the task of traffic target detection. The model can process image features and text features, and through cross-modal fusion, output traffic target detection results that match the text description, including the target's category, location, and possible other information. Finally, when the model training is completed and optimized to the target performance, the image data in the traffic scene can be input into the CNN to obtain image features; the text describing the scene can be input into the language model to obtain text features. Then, these two features are input into the fusion layer of the target multimodal neural network model, and through cross-modal reasoning, the target image data is obtained, that is, a consistent representation of the text description information and image content, which is used for subsequent target detection, classification and understanding tasks.

[0045] Through the above steps, a multimodal neural network model optimized for traffic scene target detection can be constructed. It can not only extract features from images and texts, but also effectively fuse and utilize these features to achieve accurate target detection and recognition, especially without the need for a large amount of labeled data. Through zero-sample or small-sample learning, the model's generalization ability and adaptability to new traffic targets can be improved.

[0046] As an optional embodiment, the image data and the target text are input into a preset target multimodal neural network model to obtain the target image data, including: inputting the image data and the target text into an attention mechanism module in the target multimodal neural network to obtain the target correlation of the image data and the target text; based on the target correlation, the image data is labeled to obtain the target image data.

[0047] Optionally, the image data and the target text are input into a preset target multimodal neural network to obtain the target image data, i.e., the annotated image data. The image features in the image data and the text features in the target text can be extracted first, and then the image features and the text features are input into the attention mechanism module. The attention mechanism module is a core component in the multimodal neural network, which is used for alignment and fusion between cross-modal features. It allows the model to focus on the most relevant information parts when processing images and texts, thereby improving the model's ability to understand and locate the target. In traffic target recognition, the attention mechanism can learn the target relevance between the image features and the text features based on the image features and the text features. Specifically, the module calculates the similarity between each image region and the text description to determine which image regions are most relevant to the traffic targets in the text description. For example, when the text description involves a vehicle, the attention mechanism automatically assigns more weight to the vehicle region in the image and ignores the background information. Based on the target relevance, the multimodal neural network can automatically locate the traffic targets that match the text description in the image and annotate them. Annotation is usually implemented in the form of bounding boxes, each of which corresponds to a traffic target, and is accompanied by a category label (such as "vehicle", "pedestrian", "traffic sign") and a confidence score, indicating the degree of certainty of the model's recognition of the target. The annotation results can be displayed on the image through visualization software or tools for subsequent analysis and use. Specifically, the image features and text features can be mapped to a shared feature space to ensure that they can be compared and fused in the same dimension. This process may involve operations such as linear transformation and normalization. Calculate the similarity between image features and text features, such as through dot products, cosine similarity, or more complex contrastive learning strategies. The similarity matrix reflects the degree of association between features at different locations in the image and the text description. According to the similarity calculation results, an attention weight is assigned to each location in the image (or feature of each object). The part with a larger weight indicates that it is highly related to the text description and is the key area for the model to understand and locate the target. The image features are weighted using the attention weights and then fused with the text features. This step ensures that the model can focus on the image parts that are most relevant to the text description in subsequent processing, improving the accuracy and robustness of target recognition. After obtaining the target correlation between image features and text features, this information can be used to automatically annotate image data to generate target image data.

[0048] Based on the correlation between image features and text features calculated by the attention mechanism, the model can locate the target location that matches the text description. For example, if the text description is "a red traffic light", the model will identify the location of the red traffic light in the image.

[0049] Through the above steps, the multimodal neural network model can use the attention mechanism to accurately calculate the correlation between image features and text features, so as to automatically annotate image data and generate target image data without manual annotation. This method has broad application prospects in the fields of intelligent transportation systems, autonomous driving, monitoring and analysis, and can significantly improve the performance of target detection and recognition, especially when dealing with the challenges of zero-sample learning or small-sample learning.

[0050] In practical applications, attention mechanisms can have many variants, such as bidirectional attention, self-attention, interactive attention, etc. Each variant has its own specific advantages and disadvantages and applicable scenarios. Choosing the right attention mechanism and parameter configuration is crucial to improving model performance.

[0051] Through the above steps, the preset multimodal neural network can automatically identify and annotate traffic targets based on the input image data and target text, without the need for a large amount of manual annotation data. This technology is of great significance in accelerating the development of intelligent transportation systems and improving road safety and traffic management efficiency.

[0052] As an optional embodiment, before inputting the target image data into the target recognition model to obtain the target recognition result corresponding to the image data, it includes: obtaining a plurality of sample images of labeled traffic targets; using a plurality of sample images of labeled traffic targets to train a preset original target recognition model to obtain a target recognition model.

[0053] Optionally, before the labeled image data is input into the target recognition model to obtain the target recognition result corresponding to the image data, it is necessary to go through the process of obtaining labeled sample images and training the model. A large number of images can be collected in real or simulated traffic environments, including various traffic targets, such as vehicles, pedestrians, traffic signs, etc. Data annotation: The collected images are finely annotated, and the location (usually represented by a bounding box), category and possible other attributes (such as vehicle type, pedestrian behavior, etc.) of each traffic target in each image need to be clearly annotated. Using professional annotation tools such as LabelImg, VGG Image Annotator (VIA), etc. can improve the efficiency and accuracy of annotation. Dataset construction: Build a dataset containing image data and corresponding annotation information, which will be used to train the target recognition model. Make sure that the dataset covers different weather conditions, lighting changes, viewing angles and traffic scenes to improve the generalization ability of the model. Then choose a pre-trained target recognition model as the basis, such as YOLO, Faster R-CNN, SSD, etc. These models have been pre-trained on large-scale general datasets and have certain recognition capabilities. Input the labeled traffic target sample image dataset into the model for fine-tuning training. The purpose of fine-tuning training is to allow the model to learn the characteristics of specific targets in traffic scenes and improve its accuracy in target recognition in the traffic field. Then define appropriate loss functions, such as cross entropy loss, IoU loss, bounding box regression loss, etc., to guide the model to learn feature representations related to traffic targets. Set relevant training parameters, including learning rate, batch size, number of training rounds, etc., to optimize the model training process. Run the model training process and iteratively update the model parameters until the model's performance on the validation set meets expectations. Evaluate the performance of the model on the test set, check the model's accuracy, recall rate, mAP and other indicators to ensure that the model has good recognition capabilities for traffic targets. After training is completed, save the fine-tuned target recognition model for subsequent use in target detection and recognition tasks.

[0054] Few-shot learning and zero-shot learning can also be used for training. Especially when facing new category target detection, if there is not enough labeled data to directly train the model, these two learning methods can significantly improve the adaptability and generalization ability of the model. The core of few-shot learning is to use a small number of samples (usually a few to a few hundred examples) to learn the recognition model of new categories. It relies on the pre-training of the model on a large number of known categories to capture common features and patterns to assist in the learning of new categories. The goal of zero-shot learning is to recognize new categories without direct training samples. It converts the text description of the new category into a form that the model can understand by leveraging the knowledge of known categories, and then uses this information to infer the features of the new category image. The key to zero-shot learning is to build cross-modal associations between images and text.

[0055] The application of small-shot learning and zero-shot learning in intelligent transportation systems can significantly reduce the reliance on labeled data required for detecting new categories of targets, allowing the model to quickly adapt and effectively identify changing traffic environments and emerging traffic signs. This is of great value for scenarios such as traffic monitoring, road safety analysis, and visual understanding of autonomous vehicles. In particular, these two methods provide powerful target detection solutions when resources are limited and it is difficult to obtain a large amount of labeled data.

[0056] Through the above steps, the preset original target recognition model can be trained and optimized to adapt to the target recognition task in the traffic field. The fine-tuned model will be better at identifying and locating specific targets in traffic scenes, such as vehicles, pedestrians, traffic signs, etc., so that when the labeled image data is input, the target recognition results can be output more accurately. This process is a key step to achieve automated traffic target detection, improve road safety, traffic efficiency and autonomous driving technology.

[0057] As an optional embodiment, obtaining a plurality of sample images of labeled traffic targets includes: obtaining a plurality of sample images and description texts corresponding to each of the plurality of sample images; inputting the plurality of sample images and the description texts corresponding to each of the plurality of sample images into a target multimodal neural network to obtain a plurality of sample images of labeled traffic targets.

[0058] Optionally, multiple sample images of annotated traffic targets are obtained. This process combines image acquisition, text generation of visual language models and joint annotation capabilities of multimodal neural networks. A large amount of image data can be collected in traffic scenes to ensure that the sample images cover various types of traffic targets, such as vehicles, pedestrians, traffic signs, road facilities, etc. At the same time, the samples need to cover different weather conditions, light intensity, time (such as day and night), viewing angles and backgrounds to improve the generalization ability of the model. Then, the description samples corresponding to the sample images can be obtained. Specifically, the collected sample images can be input into a pre-trained visual language model (i.e., a multimodal neural network model, such as CLIP, ALIGN, etc.), and the model's text generation capabilities are used to generate one or more descriptive texts for each sample image. These text descriptions should contain key information in the image, such as "a red car is parked on the side of the road", "pedestrians are crossing the zebra crossing", etc. The generated text description will be used as auxiliary information for subsequent training of the multimodal neural network. Combining the image data and text description, the category and location information (i.e., bounding box) of the traffic target are output. These images “labeled” by the model, that is, images containing target category and location information, can be regarded as labeled traffic target sample images.

[0059] The labeled data generated by the model can be further used to train other types of models. It is particularly useful for resource-limited environments or scenarios that need to quickly adapt to new categories of targets, because it can use multimodal learning techniques with fewer labeled samples to improve the model's recognition and generalization capabilities.

[0060] As an optional embodiment, when there is a traffic target in the image data that is identified as a new target, a description text of the new target is received based on the target account input, wherein the new target represents a traffic target that has not obtained a target recognition result; the image data and the description text are input into the target multimodal neural network model to obtain labeled image data; and the labeled image data is used to train the target recognition model to obtain a trained target recognition model.

[0061] Optionally, in the case where there are traffic targets identified as new targets in the image data, by receiving the user's description text of the new target, combining the multimodal neural network to annotate the target, and then using the annotated image data to further train the target recognition model, this process can significantly improve the model's ability to recognize new targets. When the model encounters an unrecognized traffic target in the image data, the system can be designed to request a description of the new target from the user or expert. The description text should detail the target's appearance, shape, color, size, location and other features, as well as its potential role in the traffic scene. For example, for a new type of traffic sign, the description may include "a blue circle with a white arrow pointing to the right in the middle, used to indicate the direction of a right turn for vehicles." The image data containing the new target and the corresponding description text are input into the pre-trained multimodal neural network. The network uses its cross-modal learning ability to extract semantic information from the text and match it with the visual features in the image. Through the attention mechanism or joint encoding, the network can learn the features of the new target and automatically annotate it in the image. Annotation includes determining the bounding box, category label and possible attribute information of the new target. After being processed by the multimodal neural network, the labeled image data is obtained, that is, the new target in the image is accurately framed and attached with detailed category labels and descriptions. These labeled image data constitute new training samples, which can be used to optimize the target recognition model. The new training samples are input into the model for training or fine-tuning. Through iterative optimization, the model will learn the feature representation of the new target and improve the detection and recognition capabilities of the new target.

[0062] Through the above steps, the system can not only cope with new traffic targets in the image data, but also automatically complete the labeling of new targets through the description text entered by the user, and further optimize the target recognition model so that it can accurately identify these new targets.

[0063] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0064] Through the description of the above implementation methods, those skilled in the art can clearly understand that the target recognition method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0065] According to an embodiment of the present invention, a target recognition device for implementing the above target recognition method is also provided. Figure 4 is a structural block diagram of a target recognition device provided according to an embodiment of the present invention. Figure 4 As shown, the target recognition device includes: a collection module 402, an input module 404 and a recognition module 406. The target recognition device is described below.

[0066] The acquisition module 402 is used to acquire image data in traffic scenes.

[0067] The input module 404 is connected to the acquisition module 402, and is used to input the image data and the target text into a preset target multimodal neural network model to obtain the target image data, wherein the target image data is image data annotated based on the target text, and the target text represents the text describing the scene in the image data.

[0068] The recognition module 406 is connected to the input module 404 and is used to input the target image data into the target recognition model to obtain a target recognition result, wherein the target recognition model is used to recognize traffic targets in the image, and the traffic targets include one of the following: traffic signs, pedestrians and vehicles.

[0069] It should be noted that the acquisition module 402, input module 404 and identification module 406 correspond to steps S202 to S206 in the embodiment, and the examples and application scenarios implemented by the multiple modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in the embodiment.

[0070] An embodiment of the present invention may provide a computer device. Optionally, in this embodiment, the computer device may be located in at least one network device among multiple network devices of a computer network. The computer device includes a memory and a processor.

[0071] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the target recognition method and device in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned target recognition method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0072] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: collect image data in a traffic scene; input the image data and the target text into a preset target multimodal neural network model to obtain target image data, wherein the target image data is image data annotated based on the target text, and the target text represents the text describing the scene in the image data; input the target image data into a target recognition model to obtain a target recognition result, wherein the target recognition model is used to identify traffic targets in the image, and the traffic targets include one of the following: traffic signs, pedestrians and vehicles.

[0073] Optionally, the processor may also execute program code of the following steps: before inputting the image data and the target text into a preset target multimodal neural network model to obtain the target image data, it includes: obtaining the original multimodal neural network model and a data set, wherein the data set includes traffic targets and description texts corresponding to the traffic targets; and using the data set to train the original multimodal neural network model to obtain the target multimodal neural network model.

[0074] Optionally, the processor may also execute the program code of the following steps: inputting the image data and the target text into a preset target multimodal neural network model to obtain the target image data, including: inputting the image data and the target text into an attention mechanism module in the target multimodal neural network to obtain the target relevance of the image data and the target text; based on the target relevance, annotating the image data to obtain the target image data.

[0075] Optionally, the processor may also execute program code of the following steps: before inputting the target image data into the target recognition model to obtain the target recognition result corresponding to the image data, including: obtaining sample images of multiple labeled traffic targets; and training a preset original target recognition model using sample images of multiple labeled traffic targets to obtain a target recognition model.

[0076] Optionally, the processor may also execute the program code of the following steps: obtaining a plurality of sample images of labeled traffic targets, including: obtaining a plurality of sample images and description texts corresponding to each of the plurality of sample images; inputting the plurality of sample images and the description texts corresponding to each of the plurality of sample images into a target multimodal neural network to obtain a plurality of sample images of labeled traffic targets.

[0077] Optionally, the processor may also execute the program code of the following steps: in the case where there is a traffic target identified as a new target in the image data, receiving a description text of the new target input based on a target account, wherein the new target represents a traffic target for which no target recognition result has been obtained; inputting the image data and the description text into a target multimodal neural network model to obtain labeled image data; and training the target recognition model using the labeled image data to obtain a trained target recognition model.

[0078] By adopting the embodiment of the present invention, a scheme of a target recognition method is provided. By collecting image data in a traffic scene; inputting the image data and the target text into a preset target multimodal neural network model to obtain target image data, wherein the target image data is image data based on target text annotation, and the target text represents the text describing the scene in the image data; inputting the target image data into a target recognition model to obtain a target recognition result, wherein the target recognition model is used to recognize traffic targets in the image, and the traffic targets include one of the following: traffic signs, pedestrians and vehicles, so as to achieve the purpose of combining image data and text information for target recognition, thereby achieving the technical effect of improving the accuracy of target recognition, and further solving the technical problem that the acquired image data is not clear enough and the target recognition result is not accurate enough due to the relatively complex traffic scene in the current target detection in the traffic scene.

[0079] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a non-volatile storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0080] The embodiment of the present invention further provides a non-volatile storage medium. Optionally, in this embodiment, the non-volatile storage medium can be used to store the program code executed by the target identification method provided by the above embodiment.

[0081] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0082] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: collecting image data in a traffic scene; inputting the image data and target text into a preset target multimodal neural network model to obtain target image data, wherein the target image data is image data annotated based on the target text, and the target text represents text describing the scene in the image data; inputting the target image data into a target recognition model to obtain a target recognition result, wherein the target recognition model is used to identify traffic targets in the image, and the traffic targets include one of the following: traffic signs, pedestrians, and vehicles.

[0083] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: before inputting the image data and the target text into a preset target multimodal neural network model to obtain the target image data, including: obtaining an original multimodal neural network model and a data set, wherein the data set includes a traffic target and a description text corresponding to the traffic target; and using the data set to train the original multimodal neural network model to obtain a target multimodal neural network model.

[0084] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: inputting image data and target text into a preset target multimodal neural network model to obtain target image data, including: inputting image data and target text into an attention mechanism module in the target multimodal neural network to obtain target correlation between the image data and the target text; based on the target correlation, annotating the image data to obtain target image data.

[0085] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: before inputting the target image data into the target recognition model to obtain the target recognition result corresponding to the image data, including: obtaining sample images of multiple labeled traffic targets; using the sample images of multiple labeled traffic targets to train a preset original target recognition model to obtain a target recognition model.

[0086] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: obtaining sample images of multiple labeled traffic targets, including: obtaining multiple sample images and description texts corresponding to each of the multiple sample images; inputting the multiple sample images and the description texts corresponding to each of the multiple sample images into a target multimodal neural network to obtain sample images of multiple labeled traffic targets.

[0087] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: in a case where a traffic target is identified as a new target in the image data, receiving a description text of the new target based on a target account input, wherein the new target represents a traffic target for which no target recognition result is obtained; inputting the image data and the description text into a target multimodal neural network model to obtain labeled image data; and training the target recognition model using the labeled image data to obtain a trained target recognition model.

[0088] An embodiment of the present invention also provides a computer program product, including a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it can achieve: collecting image data in a traffic scene; inputting the image data and target text into a preset target multimodal neural network model to obtain target image data, wherein the target image data is image data annotated based on the target text, and the target text represents the text describing the scene in the image data; inputting the target image data into a target recognition model to obtain a target recognition result, wherein the target recognition model is used to identify traffic targets in the image, and the traffic targets include one of the following: traffic signs, pedestrians, and vehicles.

[0089] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0090] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0091] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0092] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0093] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0094] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0095] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A target recognition method, characterized in that: include: Collect image data in traffic scenes; Inputting the image data and the target text into a preset target multimodal neural network model to obtain target image data, wherein the target image data is the image data annotated based on the target text, and the target text represents text describing a scene in the image data; The target image data is input into a target recognition model to obtain a target recognition result, wherein the target recognition model is used to recognize traffic targets in the image, and the traffic targets include one of the following: traffic signs, pedestrians and vehicles.

2. The method according to claim 1, characterized in that: Before inputting the image data and the target text into a preset target multimodal neural network model to obtain the target image data, the method includes: Acquire an original multimodal neural network model and a data set, wherein the data set includes the traffic target and a description text corresponding to the traffic target; The original multimodal neural network model is trained using the data set to obtain the target multimodal neural network model.

3. The method according to claim 1, characterized in that The step of inputting the image data and the target text into a preset target multimodal neural network model to obtain the target image data comprises: Inputting the image data and the target text into an attention mechanism module in the target multimodal neural network to obtain a target relevance between the image data and the target text; Based on the target relevance, the image data is labeled to obtain the target image data.

4. The method according to claim 1, characterized in that Before inputting the target image data into the target recognition model to obtain the target recognition result corresponding to the image data, the method includes: Obtain sample images of multiple labeled traffic targets; The target recognition model is obtained by training a preset original target recognition model using the sample images of the plurality of labeled traffic targets.

5. The method according to claim 4, characterized in that The step of obtaining a plurality of sample images of labeled traffic targets includes: Acquire a plurality of sample images and description texts corresponding to the plurality of sample images; The multiple sample images and the description texts corresponding to the multiple sample images are input into the target multimodal neural network to obtain the sample images of the multiple labeled traffic targets.

6. The method according to any one of claims 1 to 5, characterized in that: Also includes: In the case where there is a traffic target identified as a new target in the image data, receiving a description text of the new target input based on a target account, wherein the new target represents a traffic target for which no target recognition result is obtained; Inputting the image data and the description text into the target multimodal neural network model to obtain labeled image data; The target recognition model is trained using the labeled image data to obtain a trained target recognition model.

7. A target recognition device, characterized in that: include: A collection module, used to collect image data in traffic scenes; An input module, used for inputting the image data and target text into a preset target multimodal neural network model to obtain target image data, wherein the target image data is the image data annotated based on the target text, and the target text represents text describing the scene in the image data; The recognition module is used to input the target image data into a target recognition model to obtain a target recognition result, wherein the target recognition model is used to recognize traffic targets in the image, and the traffic targets include one of the following: traffic signs, pedestrians and vehicles.

8. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the target recognition method according to any one of claims 1 to 6.

9. A computer device, characterized in that: include: Memory and processor, The memory stores a computer program; The processor is used to execute the computer program stored in the memory, and when the computer program is run, the processor executes the target recognition method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the target recognition method described in any one of claims 1 to 6 is implemented.

Images