Target recognition method and device based on visual language model, and medium

Abstract

The application relates to the technical field of automatic driving, and particularly provides a target identification method and device based on a visual language model and a medium, aiming to solve the problem of how to accurately identify a target. The method provided by the application comprises the following steps: acquiring an image of a driving environment of an intelligent device and a question text corresponding to the image, the question text being used for describing a question about a target to be identified in the driving environment; inputting the image and the question text into a visual language model for processing to obtain an answer text; the visual language model is acquired in the following manner: acquiring a candidate image containing the target to be identified from a driving environment video; checking the candidate image by using a to-be-trained model, if the checking is passed, taking the candidate image as an image sample, the to-be-trained model being a visual language model after pre-training; and performing incremental training on the to-be-trained model by using the image sample and its labeled information. Through the above method, target information of an environmental target can be accurately identified from the image of the driving environment of the intelligent device.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, specifically to a target recognition method, device, and medium based on a visual language model. Background Technology

[0002] When controlling a vehicle for autonomous driving, it is necessary to accurately perceive various signs in the vehicle's environment (such as speed limit signs, directional signs, etc.) and control the vehicle according to the information indicated by these signs. For example, if a speed limit sign indicates that the maximum speed of the vehicle is 60 km / h, then when controlling the vehicle, it is necessary to ensure that the speed is less than or equal to 60 km / h.

[0003] With the continuous development of Vision Language Model (VLM) technology, VLM is increasingly being used in various fields to understand image content and convert it into text information. Currently, commonly used VLM models are trained on open-source image-text datasets. In these datasets, the image content is relatively simple, and the targets are clearly defined. Therefore, current VLM models are only suitable for recognizing images with clear and distinct subjects. However, when a vehicle is in motion, the images captured by its camera represent the vehicle's driving environment. All environmental targets within the camera's field of view (such as roads, static / dynamic obstacles, buildings, trees, signs, etc.) are captured, resulting in a complex and varied image. Signs may only occupy a small portion of the image, and high-speed driving can cause signs to appear blurred. For these reasons, current VLM models cannot accurately identify sign information from images of a vehicle's driving environment.

[0004] Accordingly, a new technical solution is needed in this field to solve the above problems. Summary of the Invention

[0005] In order to overcome the above-mentioned deficiencies, this application is made to solve or at least partially solve the following technical problem: how to accurately identify target information of environmental targets (such as signs) from images of the driving environment of intelligent devices (such as vehicles).

[0006] In a first aspect, a target recognition method based on a visual language model is provided, the method comprising:

[0007] Acquire images of the driving environment of the intelligent device and corresponding question texts for the images, wherein the question texts are used to describe a question about a target to be identified in the driving environment;

[0008] The image and the question text are input into a visual language model for processing to obtain the answer text corresponding to the question text, which describes the answer to the question.

[0009] The visual language model is trained in the following way:

[0010] The video of the driving environment of the intelligent device is acquired, and images containing the target to be identified are obtained from each frame of the video as candidate images.

[0011] The candidate image is validated using the model to be trained. If the validation passes, the candidate image is used as an image sample. The model to be trained is a pre-trained visual language model. The validation is used to determine whether the candidate image contains the target to be identified.

[0012] Based on the target information of the target to be identified in the image sample, the annotation information of the image sample is obtained, and the annotation information includes a question and its answer regarding the target to be identified;

[0013] Using the image samples and their annotation information, the model to be trained is incrementally trained to obtain the final visual language model.

[0014] In one technical solution of the above target recognition method, obtaining an image containing the target to be recognized from each frame of the video as a candidate image includes: performing target detection on each frame of the video; if the target to be recognized is detected, then using the image as the candidate image.

[0015] In one technical solution of the above target recognition method, the target information of the target to be recognized includes the category of the target to be recognized and the detection box, wherein the detection box is obtained when the image is subjected to target detection.

[0016] In one technical solution of the above target recognition method, the step of verifying the candidate image using a model to be trained includes:

[0017] Obtain the segmentation region of the target to be identified in the candidate image, wherein the segmentation region is obtained by target segmentation of the candidate image;

[0018] Based on the segmented region, the target image of the target to be identified is cropped from the candidate image;

[0019] Using the model to be trained, target recognition is performed on the candidate image and the target image, respectively;

[0020] Based on the target recognition result, the verification result of the candidate image is determined; if the target to be identified is recognized in both the candidate image and the target image, then the candidate image passes the verification.

[0021] In one technical solution of the above target recognition method, there are multiple models to be trained, and the step of determining the verification result of the candidate image based on the target recognition result includes: if all models to be trained recognize the target to be recognized in both the candidate image and the target image, then the candidate image passes the verification.

[0022] In one technical solution of the above target recognition method, there are multiple models to be trained, and the method further includes obtaining the verification result through the following means:

[0023] The target image is semantically recognized using the multiple models to be trained, thereby obtaining the semantic information in the target image;

[0024] The verification result is determined based on the target recognition result and the semantic information;

[0025] If all trainable models identify the target in both the candidate image and the target image, and the semantic information obtained by all trainable models is the same, then the candidate image passes the verification.

[0026] In one technical solution of the above target recognition method, the target information of the target to be recognized includes the category of the target to be recognized, the detection box, and the semantic information, wherein the detection box is obtained when performing target detection on the image.

[0027] In one technical solution of the above target recognition method, obtaining the annotation information of the image sample based on the target information of the target to be recognized in the image sample includes:

[0028] Obtain the corpus format of the labeled information, wherein the corpus format is used to describe the expression of the question and the answer of the target to be identified;

[0029] Based on the corpus format and the target information, construct the first annotation information;

[0030] The expression of questions and / or answers in the first annotation information is adjusted using a large language model to form multiple different second annotation information.

[0031] In one technical solution of the above-mentioned target recognition method, there are multiple models to be trained, and the incremental training of the models to be trained to obtain the final visual language model includes:

[0032] Using the image samples and their annotation information, incremental training is performed on the multiple models to be trained to obtain multiple candidate models that have completed incremental training.

[0033] Obtain the performance metrics of the candidate model, which are used to represent the performance of the candidate model;

[0034] Based on the performance metrics, the candidate model with the best performance is selected as the final visual language model.

[0035] In one technical solution of the above target recognition method, the target to be recognized is a sign in the driving environment; wherein, the sign includes road signs, parking space signs and / or location signs.

[0036] In a second aspect, a smart device is provided, the smart device including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, which, when executed by the at least one processor, implements the method described in any of the technical solutions provided in the first aspect.

[0037] In a third aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored therein, the program codes being adapted to be loaded and executed by a processor to perform the method described in any of the technical solutions provided in the first aspect above.

[0038] Solution 1. A target recognition method based on a visual language model, characterized in that the method includes:

[0039] Acquire images of the driving environment of the intelligent device and corresponding question texts for the images, wherein the question texts are used to describe a question about a target to be identified in the driving environment;

[0040] The image and the question text are input into a visual language model for processing to obtain the answer text corresponding to the question text, which describes the answer to the question.

[0041] The visual language model is trained in the following way:

[0042] The video of the driving environment of the intelligent device is acquired, and images containing the target to be identified are obtained from each frame of the video as candidate images.

[0043] The candidate image is validated using the model to be trained. If the validation passes, the candidate image is used as an image sample. The model to be trained is a pre-trained visual language model. The validation is used to determine whether the candidate image contains the target to be identified.

[0044] Based on the target information of the target to be identified in the image sample, the annotation information of the image sample is obtained, and the annotation information includes a question and its answer regarding the target to be identified;

[0045] Using the image samples and their annotation information, the model to be trained is incrementally trained to obtain the final visual language model.

[0046] Solution 2. The method according to Solution 1, characterized in that, obtaining images containing the target to be identified from each frame of the video as candidate images includes:

[0047] For each frame of the image, target detection is performed on the image; if the target to be identified is detected, the image is used as the candidate image.

[0048] Scheme 3. The method according to Scheme 2, wherein the target information of the target to be identified includes the category of the target to be identified and a detection box, and the detection box is obtained when the image is subjected to target detection.

[0049] Solution 4. The method according to Solution 2, characterized in that, the step of verifying the candidate image using the model to be trained includes:

[0050] Obtain the segmentation region of the target to be identified in the candidate image, wherein the segmentation region is obtained by target segmentation of the candidate image;

[0051] Based on the segmented region, the target image of the target to be identified is cropped from the candidate image;

[0052] Using the model to be trained, target recognition is performed on the candidate image and the target image, respectively;

[0053] Based on the target recognition result, the verification result of the candidate image is determined; if the target to be identified is recognized in both the candidate image and the target image, then the candidate image passes the verification.

[0054] Solution 5. The method according to Solution 4, characterized in that the models to be trained are multiple, and the step of determining the verification result of the candidate image based on the result of the target recognition includes:

[0055] If all models to be trained identify the target in both the candidate image and the target image, then the candidate image passes the verification.

[0056] Solution 6. The method according to Solution 4, characterized in that there are multiple models to be trained, and the method further includes obtaining the verification result through the following means:

[0057] The target image is semantically recognized using the multiple models to be trained, thereby obtaining the semantic information in the target image;

[0058] The verification result is determined based on the target recognition result and the semantic information;

[0059] If all trainable models identify the target in both the candidate image and the target image, and the semantic information obtained by all trainable models is the same, then the candidate image passes the verification.

[0060] Scheme 7. The method according to Scheme 6, wherein the target information of the target to be identified includes the category of the target to be identified, the detection box and the semantic information, and the detection box is obtained when the image is detected.

[0061] Solution 8. The method according to Solution 1, characterized in that, obtaining the annotation information of the image sample based on the target information of the target to be identified in the image sample includes:

[0062] Obtain the corpus format of the labeled information, wherein the corpus format is used to describe the expression of the question and the answer of the target to be identified;

[0063] Based on the corpus format and the target information, construct the first annotation information;

[0064] The expression of questions and / or answers in the first annotation information is adjusted using a large language model to form multiple different second annotation information.

[0065] Solution 9. The method according to Solution 1, characterized in that the models to be trained are multiple, and the incremental training of the models to be trained to obtain the final visual language model includes:

[0066] Using the image samples and their annotation information, incremental training is performed on the multiple models to be trained to obtain multiple candidate models that have completed incremental training.

[0067] Obtain the performance metrics of the candidate model, which are used to represent the performance of the candidate model;

[0068] Based on the performance metrics, the candidate model with the best performance is selected as the final visual language model.

[0069] Scheme 10. The method according to any one of Schemes 1 to 9, characterized in that the target to be identified is a sign in the driving environment;

[0070] The signs include road signs, parking space signs, and / or location signs.

[0071] Option 11. A smart device, characterized in that it comprises:

[0072] At least one processor;

[0073] And, a memory communicatively connected to the at least one processor;

[0074] The memory stores a computer program, which, when executed by the at least one processor, implements the target recognition method based on a visual language model as described in any one of schemes 1 to 10.

[0075] Scheme 12. A computer-readable storage medium storing a plurality of program codes, characterized in that the program codes are adapted to be loaded and run by a processor to perform the target recognition method based on a visual language model as described in any one of Schemes 1 to 10.

[0076] The above-described technical solutions of this application have at least one or more of the following beneficial effects:

[0077] In one technical solution of the target recognition method based on the visual language model provided in this application, images of the driving environment of a smart device and corresponding question texts can be acquired. The question texts describe questions about the target to be identified in the driving environment. The images and question texts are input into a visual language model for processing to obtain answer texts corresponding to the question texts. The answer texts describe the answers to the questions. The visual language model is trained in the following way: acquiring a video of the driving environment of the smart device, and extracting images containing the target to be identified from each frame of the video as candidate images; using the model to be trained to verify the candidate images, and if the verification passes, using the candidate images as image samples. The model to be trained is a pre-trained visual language model, and the verification is used to determine whether the candidate images contain the target to be identified; obtaining the annotation information of the image samples based on the target information of the target to be identified in the image samples, including the question and its answer regarding the target to be identified; and using the image samples and their annotation information, incrementally training the model to be trained to obtain the final visual language model.

[0078] In the above implementation scheme, the image samples are images of the driving environment of the intelligent device. Incremental training of the pre-trained visual language model using these image samples and their annotation information improves the model's ability to recognize targets in the driving environment. This allows the model to accurately identify environmental targets (such as signs) from images of the driving environment. Furthermore, the final visual language model can still be used as a training model and subjected to further incremental training using the above implementation scheme. In other words, the training model can be a visual language model that has only completed pre-training, or a visual language model that has completed both pre-training and at least one incremental training iteration. As the number of incremental training iterations increases, the visual language model's ability to recognize targets from images of the driving environment of the intelligent device will become increasingly stronger. Attached Figure Description

[0079] The disclosure of this application will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application. Wherein:

[0080] Figure 1 This is a schematic flowchart of the main steps of a target recognition method based on a visual language model according to an embodiment of this application;

[0081] Figure 2 This is an image of an underground parking lot according to an embodiment of this application;

[0082] Figure 3 This is a schematic flowchart of the main steps of a visual language model acquisition method according to an embodiment of this application;

[0083] Figure 4 This is a schematic flowchart illustrating the main steps of obtaining annotation information for image samples according to an embodiment of this application;

[0084] Figure 5 This is a schematic diagram of the main steps in incrementally training the model to be trained to obtain the final visual language model according to an embodiment of this application.

[0085] Figure 6 This is a schematic diagram of the overall process of obtaining a visual language model according to an embodiment of this application;

[0086] Figure 7 yes Figure 6 A flowchart illustrating the main steps of target detection and annotation in China;

[0087] Figure 8 This is a schematic diagram of the main structure of a smart device according to an embodiment of this application.

[0088] Figure label:

[0089] 11: Memory; 12: Processor. Detailed Implementation

[0090] Some embodiments of this application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of this application and are not intended to limit the scope of protection of this application.

[0091] In the description of this application, "processor" can include hardware, software, or a combination of both. A processor can be a central processing unit, microprocessor, graphics processor, digital signal processor, or any other suitable processor. A processor has data and / or signal processing capabilities. A processor can be implemented in software, in hardware, or a combination of both. Computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B.

[0092] The relevant user personal information that may be involved in the various embodiments of this application is processed in strict accordance with the requirements of laws and regulations, following the principles of legality, legitimacy, and necessity, based on the reasonable purpose of the business scenario, and includes personal information that users actively provide or that is generated as a result of using the product / service, as well as personal information obtained with user authorization.

[0093] The personal information processed in this application will vary depending on the specific product / service scenario and will be based on the specific scenario in which the user uses the product / service. This may involve the user's account information, device information, driving information, vehicle information, or other related information. This application will treat the user's personal information and its processing with the utmost diligence.

[0094] This application attaches great importance to the security of users' personal information and has taken reasonable and feasible security protection measures that comply with industry standards to protect users' information and prevent unauthorized access, disclosure, use, modification, damage or loss of personal information.

[0095] The following describes an embodiment of the target recognition method based on a visual language model provided in this application.

[0096] See appendix Figure 1 , Figure 1 This is a schematic flowchart illustrating the main steps of a target recognition method according to an embodiment of this application. Figure 1 As shown, the target identification method in this application embodiment mainly includes the following steps S101 to S102.

[0097] Step S101: Obtain images of the driving environment of the smart device and corresponding question texts. The question texts are used to describe the questions regarding the target to be identified in the driving environment.

[0098] Smart devices can include driving equipment, smart cars, robots, and other similar devices. Each smart device is equipped with at least one camera, which can capture images of the environment in which the device operates.

[0099] The target to be identified can be signs in the driving environment, which can include road signs, parking space signs, and / or location signs. For example, road signs can be speed limit signs indicating the maximum speed of smart devices, directional signs indicating road directions, etc.; parking space signs can be signs indicating parking space numbers; location signs can be signs indicating the location of places (such as parking lots, elevators, etc.).

[0100] Question text is textual information that describes the problem in natural language. Taking a speed limit sign as an example, the question text could be "Does the image contain a speed limit sign, and what is the maximum speed indicated by the speed limit sign?"

[0101] Step S102: Input the image and question text into the Vision Language Mode (VLM) for processing to obtain the answer text corresponding to the question text. The answer text is used to describe the answer to the question.

[0102] The answer text is the textual information describing the answer in natural language. Taking the targets to be identified as elevator signs, parking payment signs, parking area numbers, and parking space numbers as examples, the question text input to the visual language model and the answer text obtained by the visual language model are explained. Specifically, the answer text will be... Figure 2 The image of the underground parking lot shown and the question text in Table 1 are input into the visual language model for processing, and the answer text shown in Table 1 can be obtained. Table 1

[0103] The visual language model is a pre-trained model that can be directly called when needed for object recognition. The training method for the visual language model is explained below.

[0104] See appendix Figure 3 In the embodiments of this application, a visual language model can be trained through the following steps S201 to S204.

[0105] Step S201: Obtain a video of the driving environment of the smart device, and extract images containing the target to be identified from each frame of the video as candidate images.

[0106] Cameras on smart devices can capture images of the driving environment. Multiple consecutive frames are superimposed to form a video; that is, the video in this application is captured by a camera on a smart device. Each frame is an image from a specific time frame within the video. The time interval between any two adjacent time frames in the video is fixed. Each time frame in the video is a still image, and each frame can be understood as a still image presented in the video at that specific time frame.

[0107] Step S202: The candidate image is validated using the model to be trained. If the validation passes, the candidate image is used as an image sample. The model to be trained is a pre-trained visual language model. If the validation fails, the candidate image does not need to be used as an image sample.

[0108] The verification is used to determine whether the candidate image contains the target to be identified; if the candidate image is found to contain the target to be identified, the candidate image passes the verification; otherwise, the candidate image fails the verification.

[0109] A pre-trained visual language model can be understood as a general visual language model. This embodiment does not specify the pre-training method for the visual language model, as long as a pre-trained visual language model can be obtained.

[0110] When validating candidate images using a model to be trained, the candidate image and its question text can be input into the model to obtain the answer text corresponding to the question text. The question text describes whether the image contains the target to be identified, and the answer text describes the answer to that question. Taking an elevator sign as an example, the question text could be "Does the image contain an elevator sign?", and the answer text could be "Yes" or "No". If the answer text is "Yes", the candidate image passes the validation; if the answer text is "No", the candidate image fails the validation.

[0111] Step S203: Based on the target information of the target to be identified in the image sample, obtain the annotation information of the image sample. The annotation information includes the question and answer for the target to be identified.

[0112] The target information may include the category of the target to be identified. Taking a sign as an example, the category may include speed limit signs, directional signs, parking space signs, parking lot signs, elevator signs, etc. This embodiment does not specifically limit the category of the target to be identified.

[0113] The questions and answers in the annotation information are all textual information described in natural language. Taking the target to be identified as an elevator sign as an example, the question could be "Does the image contain an elevator sign? A = Yes; B = No; directly reply with the option letter", and the answer could be "B".

[0114] Step S204: Using image samples and their annotation information, incrementally train the model to be trained to obtain the final visual language model.

[0115] Since the model to be trained is a pre-trained visual language model, training it using image samples and their annotation information can be understood as incremental training.

[0116] In this embodiment, conventional training methods can be used to supervise the training of the model under test using image samples and their annotation information. This embodiment does not impose specific limitations on this method. For example, the question in the image sample and its annotation information can be input into the model under test for processing to obtain the answer to the question. Then, the model parameters of the model under test can be optimized based on the difference between the answer in the annotation information and the answer output by the model under test.

[0117] It should be noted that the final visual language model described above can still be used as a training model, and incremental training can be performed again using the methods described in steps S201 to S204 to update or optimize the final visual language model. In other words, the training model in this embodiment can be a visual language model that has only completed pre-training, or a visual language model that has completed both pre-training and at least one incremental training iteration. As the number of incremental training iterations increases, the visual language model's ability to identify targets from images of the intelligent device's driving environment will become stronger, meaning the visual language model's ability to understand targets in images will become increasingly powerful.

[0118] In the method described in steps S101 to S102 above, the image samples are images of the driving environment of the intelligent device. The pre-trained visual language model is incrementally trained using the image samples and their annotation information, which can improve the visual language model's ability to recognize targets in the driving environment of the intelligent device. In this way, when using the visual language model for target recognition, the target information of environmental targets (such as signs) can be accurately identified from the images of the driving environment of the intelligent device.

[0119] The embodiments of the target recognition method provided in this application will be described below, specifically steps S201 to S204 in the above embodiments.

[0120] I. Explanation of step S201.

[0121] In some embodiments of step S201 above, when obtaining candidate images from each frame of the video, target detection can be performed on the images; if a target to be identified is detected, the image is used as a candidate image.

[0122] Specifically, conventional object detection models can be used to detect objects in the image. This embodiment does not specifically limit the type of object detection model. Object detection models include, but are not limited to: DETR (DEtectionTRansformer) model, FCOS (Fully Convolutional One-Stage Object Detection) model, Grounding DiNO (Distillation with No Labels) model, Grounding SAM (SegmentAnything Model) model, and BEIT (Bidirectional Encoder representation from ImageTransformers) model.

[0123] In some implementations, when performing target detection on an image, if a target is detected, its bounding box (bbox) can also be obtained. Based on this, the target information of the target to be identified can include the category of the target and its bounding box. When obtaining the annotation information of the image sample in step S203, the annotation information can be obtained simultaneously using the category of the target and its bounding box. Taking a parking area sign as an example, the question in the annotation information could be, "Please view the image and deduce the area number of the current parking area. Reply with the area number and provide the bbox on which the judgment is based. If you cannot determine it, directly reply 'Cannot determine'." The answer in the annotation information could be "P1". <box> 0.248,0.626,0.281,0.672 <box>", where P1 is the area number, and bbox is the rectangle." <box> 0.248,0.626,0.281,0.672 <box>" represents the coordinates of the four vertices in the bbox.

[0124] II. Explanation of step S202.

[0125] In some embodiments of step S202 above, the candidate image can be verified through the following steps S2021 to S2024.

[0126] Step S2021: Obtain the segmentation region of the target to be identified in the candidate image. The segmentation region is obtained by segmenting the target in the candidate image.

[0127] Specifically, conventional target segmentation models can be used to segment the image. This embodiment does not specifically limit the type of target segmentation model. Target segmentation models include, but are not limited to: DeepLab model, SegNet model, SAM model, and Grounding SAM model.

[0128] Step S2022: Based on the segmented region, crop the target image of the target to be identified from the candidate image. Specifically, crop the image region covered by the segmented region from the candidate image, and use this image region as the target image of the target to be identified.

[0129] Step S2023: Using the model to be trained, target recognition is performed on the candidate image and the target image respectively.

[0130] Specifically, candidate images and their question texts can be input into the model to be trained to obtain the answer text corresponding to the question text. Similarly, target images and their question texts can be input into the model to be trained to obtain the answer text corresponding to the question text. The question text describes whether the image contains the target to be identified, and the answer text describes the answer to that question. For example, if the target to be identified is an elevator sign, the question text could be "Does the image contain an elevator sign?", and the answer text could be "Yes" or "No".

[0131] Step S2024: Based on the target recognition result, determine the verification result of the candidate image; if the target to be identified is recognized in both the candidate image and the target image, the candidate image passes the verification; otherwise, the candidate image fails the verification.

[0132] Based on the method described in steps S2021 to S2024 above, the target recognition results of both candidate images and target images can be used to verify the candidate images, which improves the reliability of the candidate image verification and ensures the reliability of obtaining image samples based on the candidate images. In this way, when using image samples to incrementally train the model to be trained, it helps to improve the ability of the model to be trained to recognize the target to be identified from the image.

[0133] In some implementations of step S2024 above, multiple models to be trained can be used to verify candidate images, and these multiple models have different model architectures. For example, the multiple models to be trained can be models such as LLaVA (Large Language and Vision Assistant), InternVL (Vision Language), and QwenVL (Vision Language).

[0134] Specifically, for each training model, target recognition is performed on both the candidate image and the target image to obtain the recognition result of the training model. Based on the recognition results of all training models, the verification result of the candidate image is determined; if all training models recognize the target in both the candidate image and the target image, the candidate image passes the verification; otherwise, the candidate image fails the verification.

[0135] In some embodiments of step S2024 above, multiple training models can also be used to verify candidate images. However, unlike the above embodiments, in this embodiment, candidate images can be verified through the following steps 11 to 13.

[0136] Step 11: For each model to be trained, use the model to perform target recognition on the candidate image and the target image respectively.

[0137] Step 12: Use multiple models to perform semantic recognition on the target image to obtain the semantic information in the target image.

[0138] Specifically, for each model to be trained, the target image and its question text are input into the model to obtain the answer text corresponding to the question text. The question text describes the problem of recognizing semantic information in the target image, and the answer text describes the answer to that problem. Taking a speed limit sign as an example, the question text could be "What is the maximum driving speed indicated by the speed limit sign in the picture?", and the answer text could be "120 km / h". After obtaining the answer text, the semantic information of the target image is obtained based on the information in the answer text.

[0139] In some implementations, a model to be trained can be used to simultaneously perform target recognition and semantic recognition on the target image. Taking a speed limit sign as an example, the question text could be "Does the image contain a speed limit sign, and what is the maximum speed indicated by the speed limit sign?" The answer text obtained by the model to be trained could be "The image contains a speed limit sign, and the maximum speed indicated by the speed limit sign is 120 km / h".

[0140] Step 13: Determine the verification result based on the target recognition result and semantic information.

[0141] If all trainable models recognize the target in the candidate image, and all trainable models recognize the target in the target image, and all trainable models obtain the same semantic information, then the candidate image passes the verification; otherwise, the candidate image fails the verification.

[0142] Based on the methods described in steps 11 to 13 above, the target recognition results and semantic recognition results of multiple training models can be used to verify candidate images, further improving the reliability of candidate image verification. In some embodiments, if a candidate image passes verification, the category, detection box, and the aforementioned semantic information of the target to be identified can be obtained as the target information of the target to be identified. When obtaining the annotation information of the image sample in step S203, the category, detection box, and semantic information can be used simultaneously to obtain the annotation information. The detection box is obtained when performing target detection on the image.

[0143] III. Explanation of step S203.

[0144] In some embodiments of step S203 above, it can be achieved by... Figure 4 The following steps S2031 to S2033 are shown to obtain the annotation information of the image samples.

[0145] Step S2031: Obtain the corpus format of the annotation information. The corpus format is used to describe the expression of the question and the answer of the target to be identified.

[0146] For example, if the category of the target to be identified is a speed limit sign, the question could be expressed as "Does the image contain a speed limit sign?", and the answer could be expressed as "contains" or "does not contain". Another example is the question "Does the bounding box region in the image contain a speed limit sign?", with the answer being "contains" or "does not contain", where the bounding box is the coordinate information of the detection box. Yet another example is the question "Does the bounding box region in the image contain a speed limit sign, and what is the maximum speed indicated by the speed limit sign?", with the answer being "Contains the speed limit sign, and the maximum speed is 120 km / h".

[0147] Step S2032: Construct the first annotation information based on the corpus format and target information.

[0148] Specifically, questions are constructed based on the target information according to the way questions are expressed in the corpus format; answers are constructed based on the target information according to the way answers are expressed in the corpus format; finally, questions and answers are combined to form the first annotation information.

[0149] Step S2033: Use a Large Language Model (LLM) to adjust the way questions and / or answers are expressed in the first annotation information to form multiple different second annotation information.

[0150] By adjusting the representation, multiple different second annotations can be obtained based on each first annotation. For an image sample, the image sample and its first annotation can be considered as an image-text data pair (i.e., training data), while the image sample and each second annotation can be considered as separate image-text data pairs. This effectively expands one training data set into multiple training data sets, increasing the amount of training data.

[0151] When adjusting the way a question is expressed using a large language model, the question and its adjustment instructions from the first labeled information can be input into the large language model for processing to obtain the question with adjusted expression. The adjustment instructions are text information described in natural language, and they instruct the large language model to adjust the way the question is expressed. Similarly, when adjusting the way the answer is expressed using a large language model, the answer and its adjustment instructions from the first labeled information can also be input into the large language model for processing to obtain the answer with adjusted expression. The adjustment instructions are also text information described in natural language, and they instruct the large language model to adjust the way the answer is expressed. For example, if the question and answer in the first labeled information are in Chinese, they can be adjusted to English by the large language model. Another example: if the question in the first labeled information is "Does the image contain a speed limit sign?", and the answer is "Yes", after adjustment by the large language model, the question in the first labeled information becomes "Does the image contain a speed limit sign? A = Yes; B = No; Reply directly with the option letter", and the answer could be "A".

[0152] It should be noted that the large language model is a conventional model in the field of artificial intelligence technology, and this embodiment does not impose specific limitations on the model structure of the large language model.

[0153] Based on the method described in steps S2031 to S2033 above, the first annotation information can be expanded using a large language model, thereby increasing the amount of training data. This helps to improve the training effect of the model when incrementally training the model to be trained, enabling the model to accurately identify the target to be identified from the image.

[0154] IV. Explanation of step S204.

[0155] As can be seen from the aforementioned step S2024, multiple models to be trained can be used to verify candidate images. In some embodiments of step S204, when multiple models to be trained are available, the following can be used... Figure 5 The following steps S2041 to S2043 are used to incrementally train the model to be trained to obtain the final visual language model.

[0156] Step S2041: Using image samples and their annotation information, perform incremental training on multiple models to be trained to obtain multiple candidate models that have completed incremental training.

[0157] Specifically, the model to be trained after completing incremental training is used as a candidate model.

[0158] Step S2042: Obtain the performance metrics of the candidate model. These metrics represent the performance of the candidate model. Since the candidate model is a visual language model, performance metrics specific to visual language models can be obtained. This implementation does not specifically limit the type of performance metric. For example, the performance metric could be accuracy; the higher the accuracy, the better the performance of the candidate model.

[0159] Step S2043: Based on performance metrics, select the candidate model with the best performance as the final visual language model. For example, if the performance metric is accuracy, then the candidate model with the highest accuracy can be selected as the final visual language model.

[0160] Based on the methods described in steps S2041 to S2043 above, the optimal visual language model can be selected by utilizing the performance metrics of the models when there are multiple models to be trained.

[0161] The following is in conjunction with the appendix Figure 6 and attached Figure 7 This application describes the method for obtaining the visual language model. First, please refer to the appendix... Figure 6 , Figure 6 An example illustrates the overall process of obtaining a visual language model. For example... Figure 6 As shown, a visual language model can be obtained through the following steps S301 to S305, and then the visual language model can be used to perform target recognition on images of the driving environment of the smart device.

[0162] Step S301: Obtain video of the driving environment of the smart device.

[0163] Step S302: Target detection and annotation. Specifically, the method described in step S201 of the aforementioned embodiments is used to perform target detection on each frame of the video; if a target to be identified is detected, the image is used as a candidate image. Furthermore, the methods described in steps S202 to S203 of the aforementioned embodiments are used to obtain image samples and their first annotation information.

[0164] Step S303: Corpus expansion. Specifically, the expression of questions and / or answers in the first annotation information is adjusted using the methods described in steps S2031 to S2033 of the aforementioned embodiments to form multiple second annotation information for the image samples.

[0165] Step S304: Incrementally train the model to be trained using image samples and their second annotation information. Step S305: Obtain the final visual language model based on the model after incremental training. This final visual language model can still be used as the model to be trained again, and incremental training can be performed again using the methods described in steps S301 to S304 above to update or optimize the final visual language model. Note that the video used in step S301 during this second incremental training may be different from the video used in the previous incremental training.

[0166] In some embodiments of step S302 above, the target to be identified is a sign, and can be identified by... Figure 7 The following steps S3021 to S3029 are shown to perform target detection and annotation.

[0167] Step S3021: Perform sign detection on the image. If a sign is detected, the detection box of the sign in the image can be obtained, and the sign can be segmented to obtain the segmented region of the sign in the image.

[0168] Step S3022: Determine whether the detection was successful; if the sign is detected, the detection is successful and proceed to step S3024; if the sign is not detected, the detection fails and proceed to step S3023.

[0169] Step S3023: Manually check whether there is a sign in the image; if there is a sign, the labeler marks the detection box and segmentation area of ​​the sign in the image, and then proceeds to step S3024; if there is no sign, discard the current image and proceed to step S3021 to continue the sign detection for the next image.

[0170] Step S3024: Cropping the sign image from the image. Specifically, the image is detected and cropped based on the segmented region of the sign in the image to obtain the sign image.

[0171] Step S3025: Verify the image using multiple training models. Specifically, the method described in step S2024 of the aforementioned method embodiment is used to perform target recognition and semantic recognition on the image respectively, and the recognition results are obtained.

[0172] Step S3026: Determine whether the recognition results of all training models are consistent. Specifically, if all training models recognize the sign in the image, and all training models recognize the sign in the sign image, and all training models obtain the same semantic information, then the recognition results of all training models are consistent, and the image is confirmed to pass the verification, proceeding to step S3028; otherwise, the recognition results are inconsistent, proceeding to step S3027.

[0173] Step S3027: Manually verify whether the recognition results of all models to be trained are consistent; if consistent, proceed to step S3028; if inconsistent, discard the current image and proceed to step S3021 to continue sign detection for the next image.

[0174] Step S3028: Obtain the target information of the sign in the image. The target information includes the sign's category (e.g., speed limit sign), detection box, and semantic information.

[0175] Step S3029: Obtain image annotation information based on target information. Specifically, the annotation information is obtained using the method described in step S203 of the aforementioned method embodiment.

[0176] Based on the method described in steps S3021 to S3029 above, image samples and their annotation information for sign recognition can be quickly and accurately obtained from the video of the driving environment of the smart device. This helps to improve the training effect of the visual language model, enabling the visual language model to accurately identify sign information (e.g., whether the image contains a sign, the type of the sign, the detection box, semantic information, etc.) from the images of the driving environment of the smart device.

[0177] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of this application, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders. These adjusted solutions are equivalent to the technical solutions described in this application and therefore will also fall within the protection scope of this application.

[0178] Those skilled in the art will understand that all or part of the processes in the method of the above-described embodiment can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0179] Another aspect of this application provides a computer-readable storage medium.

[0180] In one embodiment of a computer-readable storage medium according to this application, the computer-readable storage medium can be configured to store a program that performs the target recognition method of the above-described method embodiments. This program can be loaded and run by a processor to implement the target recognition method. For ease of explanation, only the parts related to the embodiments of this application are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of this application. The computer-readable storage medium can be a storage device comprising various electronic devices. Optionally, in the embodiments of this application, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0181] Another aspect of this application provides a smart device.

[0182] In one embodiment of a smart device according to this application, the smart device may include at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, which, when executed by the at least one processor, implements the method described in any of the above embodiments. The smart device described in this application may include driving equipment, smart vehicles, robots, and other devices. See appendix. Figure 8 , Figure 8 The image exemplarily illustrates a communication connection between memory 11 and processor 12 via a bus.

[0183] In some embodiments of this application, the smart device may further include at least one sensor for sensing information. The sensor is communicatively connected to any type of processor mentioned in this application. Optionally, the smart device may further include an autonomous driving system for guiding the smart device to drive autonomously or assisting in driving. The processor communicates with the sensor and / or the autonomous driving system to perform the methods described in any of the above embodiments.

[0184] The technical solution of this application has been described above with reference to one embodiment shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. Without departing from the principles of this application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of this application.< / box> < / box> < / box> < / box>

Claims

1. A target recognition method based on a visual language model, characterized in that, The method includes: Acquire images of the driving environment of the intelligent device and corresponding question texts for the images, wherein the question texts are used to describe a question about a target to be identified in the driving environment; The image and the question text are input into a visual language model for processing to obtain the answer text corresponding to the question text, which describes the answer to the question. in, The visual language model was trained in the following way: The video of the driving environment of the intelligent device is acquired, and images containing the target to be identified are obtained from each frame of the video as candidate images. The candidate image is validated using the model to be trained. If the validation passes, the candidate image is used as an image sample. The model to be trained is a pre-trained visual language model. The validation is used to determine whether the candidate image contains the target to be identified. Based on the target information of the target to be identified in the image sample, the annotation information of the image sample is obtained, and the annotation information includes a question and its answer regarding the target to be identified; Using the image samples and their annotation information, the model to be trained is incrementally trained to obtain the final visual language model.

2. The method according to claim 1, characterized in that, The step of obtaining images containing the target to be identified as candidate images from each frame of the video includes: For each frame of the image, target detection is performed on the image; if the target to be identified is detected, the image is used as the candidate image.

3. The method according to claim 2, characterized in that, The target information of the target to be identified includes the category of the target to be identified and the detection box, wherein the detection box is obtained when the image is subjected to target detection.

4. The method according to claim 2, characterized in that, The step of verifying the candidate images using the model to be trained includes: Obtain the segmentation region of the target to be identified in the candidate image, wherein the segmentation region is obtained by target segmentation of the candidate image; Based on the segmented region, the target image of the target to be identified is cropped from the candidate image; Using the model to be trained, target recognition is performed on the candidate image and the target image, respectively; Based on the target recognition result, the verification result of the candidate image is determined; if the target to be identified is recognized in both the candidate image and the target image, then the candidate image passes the verification.

5. The method according to claim 4, characterized in that, The number of models to be trained is multiple, and the step of determining the verification result of the candidate image based on the target recognition result includes: If all models to be trained identify the target in both the candidate image and the target image, then the candidate image passes the verification.

6. The method according to claim 4, characterized in that, The method includes obtaining the verification result by means of multiple models to be trained: The target image is semantically recognized using the multiple models to be trained, thereby obtaining the semantic information in the target image; The verification result is determined based on the target recognition result and the semantic information; If all trainable models identify the target in both the candidate image and the target image, and the semantic information obtained by all trainable models is the same, then the candidate image passes the verification.

7. The method according to claim 6, characterized in that, The target information of the target to be identified includes the category of the target to be identified, the detection box, and the semantic information. The detection box is obtained when the image is detected.

8. The method according to claim 1, characterized in that, The step of obtaining the annotation information of the image sample based on the target information of the target to be identified in the image sample includes: Obtain the corpus format of the labeled information, wherein the corpus format is used to describe the expression of the question and the answer of the target to be identified; Based on the corpus format and the target information, construct the first annotation information; The expression of questions and / or answers in the first annotation information is adjusted using a large language model to form multiple different second annotation information.

9. The method according to claim 1, characterized in that, The number of models to be trained is multiple, and the incremental training of the models to be trained to obtain the final visual language model includes: Using the image samples and their annotation information, incremental training is performed on the multiple models to be trained to obtain multiple candidate models that have completed incremental training. Obtain the performance metrics of the candidate model, which are used to represent the performance of the candidate model; Based on the performance metrics, the candidate model with the best performance is selected as the final visual language model.

10. The method according to any one of claims 1 to 9, characterized in that, The target to be identified is a sign in the driving environment; The signs include road signs, parking space signs, and / or location signs.

Images