Label processing method and device, server and storage medium

Abstract

The invention relates to an annotation processing method and device, a server and a storage medium. The annotation processing method comprises the following steps: performing information extraction on an image through at least two preset recognition models to obtain first description information of the image; performing first processing on the image to obtain a first processed image; and inputting the first description information of the image and the first processing image into a preset visual language model to obtain first annotation information of the image. According to the scheme provided by the invention, the accuracy and comprehensiveness of marking description can be improved, and the accurate recognition capability of the vehicle on the surrounding environment is improved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a labeling processing method, device, server and storage medium. Background Art

[0002] Intelligent autonomous driving technology involves accurate understanding and processing of images. In order to train the machine learning models in the autonomous driving system, a large amount of image data with detailed annotations is required.

[0003] In related technologies, text annotation can be performed on images based on visual language models. However, the method of text annotation on images based on visual language models lacks the ability to describe image details. In particular, the reasoning ability for the specific location and relationship of key objects in the image is poor. At the same time, the current visual language model lacks the ability to recognize text in natural scenes such as road sign text and license plate text in self-driving scenes.

[0004] Therefore, related technologies use large visual language models to annotate images, but the accuracy and comprehensiveness of the annotation descriptions are poor, which affects the vehicle's ability to accurately recognize the surrounding environment. Summary of the invention

[0005] In order to solve or partially solve the problems existing in the related art, the present application provides a labeling processing method, device, server and storage medium, which can improve the accuracy and comprehensiveness of the labeling description and improve the vehicle's ability to accurately identify the surrounding environment.

[0006] The first aspect of the present application provides a marking processing method, comprising:

[0007] Extracting information from the image using at least two preset recognition models to obtain first description information of the image;

[0008] By performing a first process on the image, a first processed image is obtained;

[0009] The first description information of the image and the first processed image are input into a preset visual language model to obtain first annotation information of the image.

[0010] In one embodiment, the method further comprises:

[0011] The first description information of the image and the first annotation information of the image are input into a preset language model to obtain second annotation information of the image.

[0012] In one embodiment, extracting information from an image using at least two preset recognition models to obtain first description information of the image includes:

[0013] The image is segmented through the semantic segmentation model to obtain the information of objects in the image that do not have instance concepts;

[0014] Perform instance detection on the image through the instance detection model to obtain the information of the object with instance concept in the image;

[0015] The information of the object without instance concept in the image and the information of the object with instance concept in the image are combined and processed to obtain the first description information of the image.

[0016] In one embodiment, extracting information from an image using at least two preset recognition models to obtain first description information of the image includes:

[0017] The image is segmented through the semantic segmentation model to obtain the information of objects in the image that do not have instance concepts;

[0018] Perform instance detection on the image through the instance detection model to obtain the information of the object with instance concept in the image;

[0019] The road sign detection model is used to detect traffic signs on the image to obtain road traffic sign information on the road in the image;

[0020] The information of the object without instance concept in the image, the information of the object with instance concept in the image and the road traffic sign information of the road in the image are combined and processed to obtain the first description information of the image.

[0021] In one embodiment, performing instance detection on an image by using an instance detection model to obtain information of an object having an instance concept in the image includes:

[0022] Detecting the traffic light device in the image using the instance detection model, and obtaining the traffic light information in the traffic light device using the traffic light detection model; and / or,

[0023] The traffic scene objects containing text in the image are detected through the instance detection model, and the text information in the traffic scene objects is recognized using the natural scene text recognition model.

[0024] In one embodiment, a first processed image is obtained by performing a first process on the image;

[0025] A 2D bounding box of an object having an instance concept is superimposed onto the image to obtain a first processed image.

[0026] In one implementation, inputting the first description information of the image and the first annotation information of the image into a preset language model to obtain the second annotation information of the image includes:

[0027] The first description information of the image, the first annotation information of the image and the acquired description requirement information are input into a preset language model to obtain second annotation information of the image.

[0028] A second aspect of the present application provides a labeling processing device, including:

[0029] A first information determination module, configured to extract information from an image using at least two preset recognition models to obtain first description information of the image;

[0030] A first image processing module, configured to obtain a first processed image by performing a first processing on the image;

[0031] The first annotation module is used to input the first description information of the image and the first processed image into a preset visual language model to obtain first annotation information of the image.

[0032] In one embodiment, the device further comprises:

[0033] The second annotation module is used to input the first description information of the image and the first annotation information of the image into a preset language model to obtain second annotation information of the image.

[0034] In one embodiment, the first information determination module includes:

[0035] The semantic segmentation submodule is used to segment the image through the semantic segmentation model to obtain the information of objects in the image that do not have instance concepts;

[0036] The instance detection submodule is used to perform instance detection on the image through the instance detection model to obtain information about objects with instance concepts in the image;

[0037] The combination processing submodule is used to combine the information of the object without instance concept in the image and the information of the object with instance concept in the image to obtain the first description information of the image.

[0038] In one embodiment, the first information determination module includes:

[0039] The semantic segmentation submodule is used to segment the image through the semantic segmentation model to obtain the information of objects in the image that do not have instance concepts;

[0040] The instance detection submodule is used to perform instance detection on the image through the instance detection model to obtain information about objects with instance concepts in the image;

[0041] A road sign detection submodule is used to detect traffic signs on an image using a road sign detection model to obtain road traffic sign information on the road in the image;

[0042] The combination processing submodule is used to combine the information of objects without instance concepts in the image, the information of objects with instance concepts in the image, and the road traffic sign information of the road in the image to obtain the first description information of the image.

[0043] In one embodiment, the instance detection submodule detects the traffic light device in the image by using the instance detection model, and obtains the traffic light information in the traffic light device by using the traffic light detection model; and / or,

[0044] The traffic scene objects containing text in the image are detected through the instance detection model, and the text information in the traffic scene objects is recognized using the natural scene text recognition model.

[0045] A third aspect of the present application provides a server, including:

[0046] Processor; and

[0047] The memory stores executable codes thereon, and when the executable codes are executed by the processor, the processor is caused to execute the method as described above.

[0048] A fourth aspect of the present application provides a computer-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as described above.

[0049] The technical solution provided by this application may have the following beneficial effects:

[0050] This application uses at least two preset recognition models to extract information from an image to obtain first description information of the image; then performs a first process on the image to obtain a first processed image; and then inputs the first description information of the image and the first processed image into a preset visual language model to obtain first annotation information of the image. Because this application integrates a variety of traditional deep visual neural networks into the annotation processing process, uses at least two preset recognition models to extract fine-grained information from the image, and then analyzes and processes the extracted information through a visual language model, combines the image and text information, and generates relevant image annotation information by analyzing the image content and context through the visual language model, thereby improving the accuracy and comprehensiveness of the annotation description and improving the vehicle's ability to accurately recognize the surrounding environment.

[0051] Furthermore, the present application can also input the first description information of the image and the first annotation information of the image into a preset language model to obtain the second annotation information of the image. In other words, the present application further utilizes the high-level understanding and generalization capabilities of the large language model to simulate the human language cognition and generation process, and ultimately achieves pixel-level description and annotation of the autonomous driving scene image; thereby further improving the accuracy and comprehensiveness of the annotation description, and further improving the vehicle's ability to accurately identify the surrounding environment.

[0052] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The above and other objects, features and advantages of the present application will become more apparent by describing in more detail the exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0054] Figure 1 is a first flow chart of the annotation processing method shown in the present application;

[0055] Figure 2 is a second flow chart of the annotation processing method shown in the present application;

[0056] Figure 3 is a third flow chart of the annotation processing method shown in the present application;

[0057] Figure 4 This application Figure 3 The corresponding application framework diagram;

[0058] Figure 5 It is a schematic diagram of the application process of the annotation processing method shown in this application;

[0059] Figure 6 is another application framework schematic diagram shown in this application;

[0060] Figure 7 is a first structural schematic diagram of the annotation processing device shown in the present application;

[0061] Figure 8 is a second structural schematic diagram of the annotation processing device shown in the present application;

[0062] Fig. 9 It is a schematic diagram of the structure of the server shown in this application. DETAILED DESCRIPTION

[0063] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0064] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0065] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the third information, and similarly, the third information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0066] Related technologies use large visual language models to annotate images, but the accuracy and comprehensiveness of the annotation descriptions are poor, which affects the vehicle's ability to accurately recognize the surrounding environment.

[0067] In response to the above problems, the present application provides a labeling processing method that can improve the accuracy and comprehensiveness of the labeling description and improve the vehicle's ability to accurately identify the surrounding environment.

[0068] The technical solution of the present application is described in detail below with reference to the accompanying drawings.

[0069] Figure 1 It is a first flow chart of the marking processing method shown in the present application.

[0070] See also Figure 1 , the method comprising:

[0071] S101, extracting information from an image using at least two preset recognition models to obtain first description information of the image.

[0072] This step may include:

[0073] The image is segmented through the semantic segmentation model to obtain the information of objects in the image that do not have instance concepts;

[0074] Perform instance detection on the image through the instance detection model to obtain the information of the object with instance concept in the image;

[0075] The information of the object without instance concept in the image and the information of the object with instance concept in the image are combined and processed to obtain the first description information of the image.

[0076] S102, obtaining a first processed image by performing a first processing on the image.

[0077] This step may include: superimposing a 2D bounding box of an object having an instance concept into the image to obtain a first processed image.

[0078] S103: Input the first description information of the image and the first processed image into a preset visual language model to obtain first annotation information of the image.

[0079] The visual language model usually includes an image recognition part and a natural language processing part. It uses deep learning technology to combine image and text information to build a model that can understand and generate the relationship between images and text. In this step, after the large visual language model obtains the input information, it analyzes and processes the model and outputs the first annotation information of the image.

[0080] It can be found from this embodiment that the present application integrates multiple traditional deep visual neural networks into the labeling process, uses at least two preset recognition models to extract fine-grained information in the image, and then analyzes and processes the extracted information through a visual language model, combining the image and text information. The visual language model generates relevant image labeling information by analyzing the image content and context, thereby improving the accuracy and comprehensiveness of the labeling description and improving the vehicle's ability to accurately recognize the surrounding environment.

[0081] Figure 2 It is a second flow chart of the marking processing method shown in the present application.

[0082] See also Figure 2 , the method comprising:

[0083] S201, extracting information from an image using at least two preset recognition models to obtain first description information of the image.

[0084] This step may include:

[0085] The image is segmented through the semantic segmentation model to obtain the information of objects in the image that do not have instance concepts;

[0086] Perform instance detection on the image through the instance detection model to obtain the information of the object with instance concept in the image;

[0087] The road sign detection model is used to detect traffic signs on the image to obtain road traffic sign information on the road in the image;

[0088] The information of the object without instance concept in the image, the information of the object with instance concept in the image and the road traffic sign information of the road in the image are combined and processed to obtain the first description information of the image.

[0089] The example detection model is used to perform instance detection on the image to obtain information about objects with instance concepts in the image, which may include:

[0090] Detecting the traffic light device in the image using the instance detection model, and obtaining the traffic light information in the traffic light device using the traffic light detection model; and / or,

[0091] The traffic scene objects containing text in the image are detected through the instance detection model, and the text information in the traffic scene objects is recognized using the natural scene text recognition model.

[0092] S202, obtaining a first processed image by performing a first processing on the image.

[0093] This step may include: superimposing a 2D bounding box of an object having an instance concept into the image to obtain a first processed image.

[0094] S203: Input the first description information of the image and the first processed image into a preset visual language model to obtain first annotation information of the image.

[0095] The large visual language model usually includes an image recognition part and a natural language processing part. The large visual language model uses deep learning technology to combine image and text information to build a model that can understand and generate the relationship between images and text. In this step, after the large visual language model obtains the input information, it analyzes and processes the model and outputs the first annotation information of the image.

[0096] S204: Input the first description information of the image and the first annotation information of the image into a preset language model to obtain second annotation information of the image.

[0097] This step may also include: inputting the first description information of the image, the first annotation information of the image and the acquired description requirement information into a preset language model to obtain second annotation information of the image.

[0098] The large language model can simulate the human language cognition and generation process to a certain extent, can generate natural language text or understand the meaning of language text, learn and simulate the complex rules of human language, and achieve text generation capabilities close to human level. In this step, after the large visual language model obtains the input information, it analyzes and processes it and outputs the second annotation information of the image.

[0099] From this embodiment, it can be found that after obtaining the first annotation information of the image, the present application can also input the first description information of the image and the first annotation information of the image into the preset language model to obtain the second annotation information of the image. In other words, the present application further utilizes the high-level understanding and generalization capabilities of the large language model to simulate the human language cognition and generation process, and finally realizes the pixel-level description annotation of the autonomous driving scene image; thereby further improving the accuracy and comprehensiveness of the annotation description, and further improving the vehicle's ability to accurately identify the surrounding environment.

[0100] Figure 3 This is a third flow chart of the annotation processing method shown in this application. Figure 4 and Figure 5 ,in Figure 4 This application Figure 3 The corresponding application framework diagram; Figure 5 It is a schematic diagram of the application process of the marking processing method shown in this application.

[0101] This application integrates a variety of traditional deep visual neural networks into the annotation processing process, and uses different types of related models, such as semantic segmentation models, instance detection models (object detection models), road sign detection models, and natural scene text recognition models, to extract fine-grained information from images, and input the extracted information into a large visual language model through visual cues and text cues, and then further utilizes the high-level understanding and generalization capabilities of the large language model to ultimately achieve pixel-level description and annotation of autonomous driving scene images.

[0102] See also Figure 3 , the method comprising:

[0103] S301, performing segmentation processing on the image through a semantic segmentation model to obtain information of objects in the image that do not have instance concepts.

[0104] After acquiring the image collected by the vehicle sensor, the present application can use the semantic segmentation model to perform semantic segmentation on the image. The vehicle sensor may include a camera, a laser radar, etc. but is not limited thereto.

[0105] Image semantic segmentation is an important technology in the field of computer vision. Its goal is to segment an image into regions with different semantic information and label each region with a corresponding semantic label. Semantic segmentation can give a specific category to each pixel in an image. For example, it can analyze objects in a picture or a video stream and mark their categories pixel by pixel. Semantic segmentation is widely used in autonomous driving, and other fields.

[0106] The semantic segmentation model can be pre-trained by a relevant deep neural network training method, which is not limited in this application. The deep neural network training method may include: a method of training a single network using fully annotated data, a method of training multiple networks according to the data annotation method, a method of continuously adjusting the training categories by incremental learning, etc.

[0107] In this step, the semantic segmentation model can be used to select objects in the image that do not have instance concepts, such as the sky, grass, and trees, and the 2D bounding boxes of these objects that do not have instance concepts can be calculated through the segmentation mask. These objects that do not have instance concepts are also uncountable objects.

[0108] Segmentation mask is a technique in computer vision that is used to accurately separate objects from the background in an image. It achieves fine-grained segmentation of image regions by classifying and labeling each pixel. Each pixel is assigned a label to indicate whether it belongs to the foreground or background, or to a different object category. Such label information forms a two-dimensional matrix, the segmentation mask. The mask can accurately describe the position and boundaries of different objects in the image.

[0109] S302, performing instance detection on the image through an instance detection model to obtain information of objects with instance concepts in the image; and obtaining a 2D bounding box of the objects with instance concepts.

[0110] In this step, the instance detection model can be used to perform instance detection on the image, filter out objects that do not conform to the driving scene, obtain objects with instance concepts in the image (instance objects), and obtain 2D bounding boxes of objects with instance concepts in the image. These objects with instance concepts are all countable objects.

[0111] Instance detection is a computer vision task that divides objects in an image into different categories. The goal is to identify the location and category of objects in a given image. The main tasks of instance detection include identifying objects in an image and classifying them into different categories. Instance detection algorithms can include feature-based methods and deep learning-based methods.

[0112] The instance detection model used in this step may detect the bounding boxes of various countable objects (instance objects). For example, for cars on the road, all vehicles can be detected and their 2D bounding boxes can be obtained.

[0113] Among them, the objects with instance concepts (instance objects) detected in this step are mainly common objects in traffic scenes. These objects with instance concepts (instance objects) include, for example, traffic lights, road signs, traffic signs, license plate locations and other objects.

[0114] S303, detecting the traffic light device in the image through the instance detection model, and obtaining the traffic light information in the traffic light device through the traffic light detection model.

[0115] This step uses the traffic light position detected by the instance detection model to extract the image content at the corresponding position, and uses the traffic light detection model to accurately identify the color of the traffic light.

[0116] The semantic information of traffic lights has a very important impact on the driving of vehicles. Therefore, traffic light detection is a very important functional module in the autonomous driving system, and the detection results are very important for vehicle motion planning. By using the traffic light detection model to detect the traffic light image, the traffic light category and the color of the traffic light corresponding to the traffic light image can be obtained. Among them, the traffic light detection model can be composed of a deformable convolutional neural network and an attention mechanism-based detection framework but is not limited to this. The deformable convolutional neural network can be used to extract features from traffic light images; the attention mechanism-based detection framework can be used to perceive the extracted features of traffic lights and obtain the classification information and color of traffic lights.

[0117] S304, detecting traffic scene objects containing text in the image by using an instance detection model, and obtaining text information in the traffic scene objects by using a natural scene text recognition model.

[0118] This step uses the positions of road signs, traffic signs, and license plates detected by the instance detection model to extract the image parts containing text information at the corresponding positions, and uses the natural scene text recognition model to perform text recognition and extraction on these image blocks to accurately obtain the specific text information in the image.

[0119] Text detection is a hot topic in the field of computer vision. It aims to detect the position of text in natural scene images for the next step of recognition, thereby converting the image into real text information that can be processed by the computer. Text recognition algorithms can include CRNN (Convolutional Recurrent Neural Network) algorithm or natural scene text recognition method based on Transformer model. CRNN is a neural network model that combines CNN (Convolutional Neural Networks) and RNN (Recurrent Neural Network). CRNN is suitable for tasks such as image processing. For example, the CRNN algorithm uses a CNN network to extract features from the image, then slices the features into feature sequences, inputs them into a bidirectional LSTM (Long Short-Term Memory) network for recognition, and finally uses CTC (Connectionist Temporal Classification) algorithm to align the recognition results to obtain the final recognition result. CTC is an algorithm commonly used in speech recognition, text recognition and other fields to solve the problem of different lengths of input and output sequences and inability to align them.

[0120] S305, performing traffic sign detection on the image using a road sign detection model to obtain road traffic sign information of the road in the image.

[0121] This step uses a road sign detection model to identify the location and information of traffic signs on the road surface, such as left-turn arrows, right-turn arrows, left-turn U-turn signs, etc., to obtain road traffic sign information of the road in the image.

[0122] Road sign detection is a fundamental and core issue that autonomous driving needs to solve. Road signs include lines, arrows, and text on the road surface that convey traffic information such as guidance, restrictions, and warnings to traffic participants, and play a very important role in controlling and guiding traffic.

[0123] Among them, the road sign detection model can use traditional classification algorithms or convolutional neural networks to perform image segmentation of road signs, thereby determining the sign type based on the segmented image. The road sign detection model can also further use the fusion of graph networks and attention mechanisms to achieve road sign detection.

[0124] S306 , combining information of objects without instance concepts in the image, information of objects with instance concepts in the image, and road traffic sign information of the road in the image to obtain first description information of the image.

[0125] This step can integrate the information obtained through the relevant visual network in the above step, combine the information of objects without instance concepts in the image, the information of objects with instance concepts in the image, and the road traffic sign information of the road in the image to obtain the first description information of the image. The first description information can be a text prompt.

[0126] The text prompt may include the following information descriptions but is not limited to these:

[0127] 1) The category name of each object;

[0128] 2) Traffic light location and color information;

[0129] 3) The location of objects containing text and the specific text content;

[0130] 4) Location and sign information of road traffic signs;

[0131] 5) Location information of other objects.

[0132] The object position information in this step may be expressed in the coordinates of a normalized 2D bounding box, such as [upper left corner x coordinate, upper left corner y coordinate, lower right corner x coordinate, lower right corner y coordinate].

[0133] S307 , superimposing the 2D bounding box of the object with the instance concept into the image to obtain a first processed image.

[0134] In this step, the 2D bounding box of the object detected in S302 is superimposed on the original image to obtain a first processed image after superimposition processing.

[0135] S308: Input the first description information of the image and the first processed image into a preset visual language model to obtain first annotation information of the image.

[0136] This step inputs the first description information (text prompt words) and the first processed image into the large visual language model, and requires it to describe the weather, main objects, traffic signs, the location status and intentions of other traffic participants, the intention of the vehicle, and possible measures to be taken in the image as detailed as possible. The large visual language model can accept text + pictures as input and output text.

[0137] Visual Language Model (VLM) is a technology that combines image and natural language processing. Its main purpose is to understand and interpret the relationship between images and text, and generate accurate and vivid natural language descriptions based on images. The visual language model generates relevant text descriptions by analyzing image content and context, and has a visual understanding ability that is closer to that of humans.

[0138] Visual language models usually include image recognition and natural language processing. Visual language models use deep learning technology to combine image and text information to build a model that can understand and generate the relationship between images and text.

[0139] In this step, after the large visual language model obtains the input information, it is analyzed and processed by the model and outputs the first annotation information of the image.

[0140] S309: Input the first description information of the image, the first annotation information of the image, and the acquired description requirement information into a preset language model to obtain second annotation information of the image.

[0141] After obtaining the first annotation information of the image, the present application can further input the first description information (text prompt word) of the image, the first annotation information of the image and the obtained description requirement information into a preset language model, requiring the language model to summarize a description of the image as detailed as possible and remove grammatical errors, repeated sentences, some meaningless information, etc. After the language model obtains this information, it analyzes and processes the model and outputs the second annotation information of the image.

[0142] Large Language Model (LLM) refers to a deep learning model trained with a large amount of text data, which enables the model to generate natural language text or understand the meaning of language text. Large language models can provide in-depth knowledge and language production on various topics by training on huge data sets. The core idea is to learn the patterns and structures of natural language through large-scale unsupervised training, and to simulate the human language cognition and generation process to a certain extent, so that it can generate natural language text or understand the meaning of language text, learn and simulate the complex laws of human language, and achieve text generation capabilities close to human level.

[0143] Large language models can generally only process text, so both the input and output are text content. In this step, after the large visual language model obtains the input information, it analyzes and processes the model and outputs the second annotation information of the image.

[0144] It should be noted that the bounding box obtained in this application is expressed in pixel coordinates, so the description of the final annotation information can be accurate to what objects are in certain pixel areas, so it is ultimately possible to achieve pixel-level description and annotation of autonomous driving scene images.

[0145] It should also be noted that after executing step S308, the present application can also directly use the first annotation information of the image obtained by the visual language model as the final annotation information. The application framework can be found in Figure 6 , Figure 6 It is another application framework schematic diagram shown in this application. Figure 6 Only a large visual language model is used in the training set, and no large language model is used.

[0146] From this example, it can be seen that this application combines relevant visual models with a large model to realize text annotation processing of images of driving scenes. This application extracts various information of the image through semantic segmentation models, instance detection models (object detection models), road sign detection models and natural scene text recognition models, and inputs the information into a large visual language model to output preliminary annotation information, and then further inputs the preliminary annotation information, the first description information of the image (text prompt words) and the description requirement information into the large language model, and uses the high-level understanding and generalization capabilities of the large language model to finally realize the pixel-level description annotation of the autonomous driving scene images, thereby improving the accuracy and comprehensiveness of the annotation description compared to related technologies, and further improving the vehicle's ability to accurately recognize the surrounding environment.

[0147] Corresponding to the aforementioned application function implementation method embodiment, the present application also provides a labeling processing device, a server and corresponding embodiments.

[0148] Figure 7 It is a first structural schematic diagram of the annotation processing device shown in this application.

[0149] See also Figure 7 The present application provides a labeling processing device 70, including: a first information determination module 71, a first image processing module 72, and a first labeling module 73.

[0150] The first information determination module 71 is used to extract information from the image through at least two preset recognition models to obtain first description information of the image. The first information determination module 71 can segment the image through a semantic segmentation model to obtain information about objects in the image that do not have instance concepts; perform instance detection on the image through an instance detection model to obtain information about objects in the image that have instance concepts; and combine information about objects in the image that do not have instance concepts with information about objects in the image that have instance concepts to obtain first description information of the image.

[0151] The first image processing module 72 is used to obtain a first processed image by performing a first processing on the image. The first image processing module 72 superimposes a 2D bounding box of an object with an instance concept into the image to obtain the first processed image.

[0152] The first annotation module 73 is used to input the first description information of the image and the first processed image into a preset visual language model to obtain the first annotation information of the image. After the large visual language model obtains the input information, it analyzes and processes the model and outputs the first annotation information of the image.

[0153] It can be found from this embodiment that the device provided by the present application integrates multiple traditional deep visual neural networks into the labeling processing process, uses at least two preset recognition models to extract fine-grained information in the image, and then analyzes and processes the extracted information through a visual language model, combines the image and text information, and generates relevant image labeling information by analyzing the image content and context through the visual language model, thereby improving the accuracy and comprehensiveness of the labeling description and improving the vehicle's ability to accurately recognize the surrounding environment.

[0154] Figure 8 It is a second structural schematic diagram of the annotation processing device shown in this application.

[0155] See also Figure 8 The present application provides a labeling processing device 70, comprising: a first information determination module 71, a first image processing module 72, a first labeling module 73, and a second labeling module 74.

[0156] The first information determination module 71, the first image processing module 72, and the first marking module 73 can be found in Figure 7 Description in .

[0157] The second annotation module 74 is used to input the first description information of the image and the first annotation information of the image into a preset language model to obtain second annotation information of the image.

[0158] The second annotation module 74 may also input the first description information of the image, the first annotation information of the image, and the acquired description requirement information into a preset language model to obtain second annotation information of the image.

[0159] Among them, the first information determination module 71 includes: a semantic segmentation submodule 711, an instance detection submodule 712, and a combination processing submodule 713.

[0160] The semantic segmentation submodule 711 is used to perform segmentation processing on the image through the semantic segmentation model to obtain information of objects in the image that do not have instance concepts;

[0161] An instance detection submodule 712 is used to perform instance detection on an image through an instance detection model to obtain information about objects with instance concepts in the image;

[0162] The combination processing submodule 713 is used to combine the information of the object without instance concept in the image with the information of the object with instance concept in the image to obtain the first description information of the image.

[0163] The first information determination module may further include: a road sign detection submodule 714 .

[0164] A road sign detection submodule 714 is used to detect traffic signs on an image using a road sign detection model to obtain road sign information on the road in the image;

[0165] The combination processing submodule 713 combines the information of the object without instance concept in the image, the information of the object with instance concept in the image and the road traffic sign information of the road in the image to obtain the first description information of the image.

[0166] The instance detection submodule 712 can detect the traffic light device in the image by using the instance detection model, and obtain the traffic light information in the traffic light device by using the traffic light detection model; and / or,

[0167] The traffic scene objects containing text in the image are detected through the instance detection model, and the text information in the traffic scene objects is recognized using the natural scene text recognition model.

[0168] In summary, the device provided by the present application extracts various information of the image through a semantic segmentation model, an instance detection model (object detection model), a road sign detection model, and a natural scene text recognition model, and inputs the information into a large visual language model to output preliminary annotation information, and then further inputs the preliminary annotation information, the first description information of the image (text prompt words) and the description requirement information into the large language model, and uses the high-level understanding and generalization capabilities of the large language model to ultimately achieve pixel-level description annotation of the autonomous driving scene image, thereby improving the accuracy and comprehensiveness of the annotation description compared to related technologies, and further improving the vehicle's ability to accurately recognize the surrounding environment.

[0169] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0170] Fig. 9 It is a schematic diagram of the structure of the server shown in this application.

[0171] See also Fig. 9, the server 1000 includes a memory 1010 and a processor 1020 .

[0172] The processor 1020 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0173] The memory 1010 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 1020 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory 1010 may include any combination of computer-readable storage media, including various types of semiconductor storage chips (such as DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 1010 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (such as a DVD-ROM, a double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a mini SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0174] The memory 1010 stores executable codes, and when the executable codes are processed by the processor 1020 , the processor 1020 can execute part or all of the methods described above.

[0175] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present application.

[0176] Alternatively, the present application can also be implemented as a computer-readable storage medium (or non-transitory machine-readable storage medium or machine-readable storage medium) on which executable code (or computer program or computer instruction code) is stored. When the executable code (or computer program or computer instruction code) is executed by a processor of an electronic device (or server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.

[0177] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A marking processing method, characterized in that: include: Extracting information from the image using at least two preset recognition models to obtain first description information of the image; By performing a first process on the image, a first processed image is obtained; The first description information of the image and the first processed image are input into a preset visual language model to obtain first annotation information of the image.

2. The method according to claim 1, characterized in that: The method further comprises: The first description information of the image and the first annotation information of the image are input into a preset language model to obtain second annotation information of the image.

3. The method according to claim 1, characterized in that: The step of extracting information from the image by using at least two preset recognition models to obtain first description information of the image includes: The image is segmented through the semantic segmentation model to obtain the information of objects in the image that do not have instance concepts; Perform instance detection on the image through the instance detection model to obtain the information of the object with instance concept in the image; The information of the object without instance concept in the image and the information of the object with instance concept in the image are combined and processed to obtain the first description information of the image.

4. The method according to claim 1, characterized in that: The step of extracting information from the image by using at least two preset recognition models to obtain first description information of the image includes: The image is segmented through the semantic segmentation model to obtain the information of objects in the image that do not have instance concepts; Perform instance detection on the image through the instance detection model to obtain the information of the object with instance concept in the image; The road sign detection model is used to detect traffic signs on the image to obtain road traffic sign information on the road in the image; The information of the object without instance concept in the image, the information of the object with instance concept in the image and the road traffic sign information of the road in the image are combined and processed to obtain the first description information of the image.

5. The method according to claim 3, characterized in that: The example detection model is used to perform instance detection on the image to obtain information about objects with instance concepts in the image, including: Detecting the traffic light device in the image using the instance detection model, and obtaining the traffic light information in the traffic light device using the traffic light detection model; and / or, The traffic scene objects containing text in the image are detected through the instance detection model, and the text information in the traffic scene objects is recognized using the natural scene text recognition model.

6. The method according to any one of claims 3 to 5, characterized in that: By performing a first process on the image, a first processed image is obtained; A 2D bounding box of an object having an instance concept is superimposed onto the image to obtain a first processed image.

7. The method according to any one of claims 2 to 5, characterized in that: The step of inputting the first description information of the image and the first annotation information of the image into a preset language model to obtain second annotation information of the image includes: The first description information of the image, the first annotation information of the image and the acquired description requirement information are input into a preset language model to obtain second annotation information of the image.

8. A marking processing device, characterized in that: include: A first information determination module, configured to extract information from an image using at least two preset recognition models to obtain first description information of the image; A first image processing module, configured to obtain a first processed image by performing a first processing on the image; The first annotation module is used to input the first description information of the image and the first processed image into a preset visual language model to obtain first annotation information of the image.

9. The device according to claim 8, characterized in that The device also includes: The second annotation module is used to input the first description information of the image and the first annotation information of the image into a preset language model to obtain second annotation information of the image.

10. A server, characterized in that: include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 7.

11. A computer-readable storage medium having executable codes stored thereon, and when the executable codes are executed by a processor of an electronic device, the processor is caused to execute the method according to any one of claims 1 to 7.

Images