An image data processing method, device, storage medium and equipment

Abstract

This application discloses an image data processing method, apparatus, storage medium, and device. The method includes: firstly, acquiring visual cue text of a target object to be detected in a target image; wherein the target image is an environmental image of the vehicle's surroundings; then, detecting the target object from the target image based on the visual cue text using an OVD model to obtain a first detection result, and performing re-frame filtering processing on it to obtain a second detection result of the target object; next, filtering out the data with detection errors in the second detection result of the target object to obtain the target image data processing result. Since this application introduces OVD and adopts a multi-filtering strategy (such as an image data processing strategy that combines the CLIP model and the VLM model) to filter out the data with detection errors of the target object in the target image (such as data with shape and text detection errors), it improves the accuracy of the target detection result, the user's riding experience, and the vehicle's driving safety.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and in particular to an image data processing method, apparatus, storage medium and device. Background Technology

[0002] With the improvement of people's living standards and the rapid development of the social economy, the usage rate of automobiles is gradually increasing, and people's requirements for vehicle functions are also getting higher and higher. For example, for autonomous vehicles, they need to be able to perceive the environment, make decisions, and control the vehicle to drive safely without the active operation of a human driver. Among these, object detection is one of the key technologies for realizing autonomous driving, which directly affects the driving performance and safety of autonomous vehicles.

[0003] However, traditional object detection methods rely heavily on costly manually labeled data (such as Convolutional Neural Networks, CNNs) to implement object detection models. This not only increases the cost of model development but also limits the rapid applicability of models in new fields. This is because if these existing open-source object detection models are directly deployed in practical applications such as autonomous driving, they often encounter performance bottlenecks due to the lack of domain-specific optimization and adjustments. This makes it difficult to meet the detection accuracy requirements of practical applications, reduces the accuracy of detection results, and affects the user's riding experience and vehicle driving safety. Summary of the Invention

[0004] In view of the above problems, this application provides an image data processing method, apparatus, storage medium, and device. By introducing an Open Vocabulary Detection Model (OVD) and employing multiple filtering strategies (such as an image data processing strategy that combines Contrastive Language-Image Pretraining (CLIP) and a Vision Language Model (VLM) to fully leverage their collaborative filtering capabilities), more accurate model training data is selected to train a target detection model with higher detection accuracy. This improves the target detection accuracy of the model in practical applications such as autonomous driving, thereby enhancing the user's riding experience and vehicle driving safety.

[0005] This application provides an image data processing method, including:

[0006] Acquire a target image; and acquire visual cue text for the target object to be detected; the target image is an image of the environment surrounding the vehicle; the target object contains text information;

[0007] The target object is detected from the target image using an open vocabulary detection model based on the visual cue text, and a first detection result is obtained.

[0008] The first detection result is subjected to reframe filtering to obtain the second detection result of the target object;

[0009] The data with detection errors in the second detection result of the target object are filtered to obtain the target image data processing result.

[0010] In one possible implementation, the open vocabulary detection model is GLEE, a general object foundation model for large-scale images and videos.

[0011] In one possible implementation, the step of detecting the target object from the target image using an open vocabulary detection model based on the visual cue text to obtain a first detection result includes:

[0012] The visual cue text of the target object is encoded using a text encoder based on the GLEE model to obtain text features;

[0013] The target image is image encoded using an image encoder based on the GLEE model to obtain the full image features;

[0014] The entire image features are detected using a target detector based on the GLEE model to obtain detection boxes;

[0015] The detection box is input into the feature transformation module of the GLEE model to obtain the image features corresponding to the detection box;

[0016] Calculate the similarity matrix between the image features and the text features, and determine the first detection result of the target object based on the calculation result.

[0017] In one possible implementation, the step of performing re-frame filtering on the first detection result to obtain a second detection result of the target object includes:

[0018] The first detection result is processed by the intersection ratio (IOG) of the actual value and the re-frame filtering method to obtain the second detection result of the target object.

[0019] In one possible implementation, filtering out erroneous data in the second detection result of the target object to obtain the target image data processing result includes:

[0020] The CLIP model, which is pre-trained using contrastive language images, is used to filter out data with shape detection errors in the second detection result of the target object, thereby obtaining the third detection result of the target object.

[0021] The Visual Language Model (VLM) is used to filter out text detection errors in the third detection results of the target object, thus obtaining the target image data processing results.

[0022] In one possible implementation, the step of using a contrastive language image pre-trained CLIP model to filter out shape detection errors in the second detection result of the target object to obtain a third detection result of the target object includes:

[0023] The detection box region containing the target object is cropped from the image containing the second detection result to obtain the first sub-image; and the first sub-image is image encoded by the image encoder of the CLIP model to obtain the image features of the first sub-image.

[0024] The positive sample prompt text corresponding to the target object and the negative sample prompt text corresponding to N other objects are text encoded using the CLIP model text encoder to obtain N+1 categories of text features; where N is a positive integer greater than 0.

[0025] Calculate the feature similarity between the first sub-image image features and the text features of the N+1 categories respectively, and determine the data with shape detection errors in the second detection result based on the calculation results;

[0026] The data with shape detection errors in the second detection result are filtered out to obtain the third detection result of the target object.

[0027] In one possible implementation, the Visual Language Model (VLM) includes a dynamic high-resolution adaptive selection module, an image encoder, a pixel shuffling layer, a feature mapping layer, a text encoder, and a large language model. The step of using the VLM to filter out text detection errors in the third detection result of the target object, obtaining a fourth detection result of the target object, which serves as the target image data processing result, includes:

[0028] Based on the content of the target object, determine the prompt instruction question text; and input the prompt instruction question text into the text encoder of the visual language model (VLM) to obtain the text features corresponding to the prompt instruction question text;

[0029] The detection box region containing the target object is cropped from the image containing the third detection result to obtain the second sub-image; and the second sub-image is image encoded by the dynamic high-resolution adaptive selection module, image encoder, pixel shuffling layer and feature mapping layer of the visual language model VLM to obtain the image features of the second sub-image.

[0030] Input the text features corresponding to the prompt instruction question text and the image features of the second sub-graph into the large language model, and determine the data of text detection errors in the third detection result based on the model output results;

[0031] The data with text detection errors in the third detection result are filtered out to obtain the fourth detection result of the target object, which is used as the target image data processing result.

[0032] In one possible implementation, the target object is a road construction sign.

[0033] In one possible implementation, after filtering out text detection errors in the third detection result of the target object using the Visual Language Model (VLM) to obtain the fourth detection result of the target object, which is then used as the target image data processing result, the method further includes:

[0034] The target image data processing result is used as the training label. Combined with the target image and the visual cue text of the target object contained in the target image, the initial target detection model is iteratively trained to obtain the target detection model, which is used to perform the target detection task.

[0035] This application also provides an image data processing apparatus, including:

[0036] An acquisition unit is used to acquire a target image and to acquire visual cue text for the target object to be detected; the target image is an environmental image of the vehicle's surroundings.

[0037] The detection unit is used to detect the target object from the target image based on the visual cue text using an open vocabulary detection model, and obtain a first detection result;

[0038] The first filtering unit is used to perform reframe filtering on the first detection result to obtain the second detection result of the target object;

[0039] The second filtering unit is used to filter out the data with detection errors in the second detection result of the target object to obtain the target image data processing result.

[0040] In one possible implementation, the open vocabulary detection model is GLEE, a general object foundation model for large-scale images and videos.

[0041] In one possible implementation, the detection unit includes:

[0042] The first encoding subunit is used to encode the visual cue text of the target object using a text encoder of the GLEE model to obtain text features;

[0043] The second encoding subunit is used to encode the target image using an image encoder based on the GLEE model to obtain the full image features;

[0044] The detection subunit is used to detect the features of the entire image using the target detector of the GLEE model to obtain the detection box;

[0045] The first input subunit is used to input the detection box into the feature transformation module of the GLEE model to obtain the image features corresponding to the detection box;

[0046] The calculation subunit is used to calculate the similarity matrix between the image features and the text features, and determine the first detection result of the target object based on the calculation result.

[0047] In one possible implementation, the first filtering unit is specifically used for:

[0048] The first detection result is processed by the intersection ratio (IOG) of the actual value and the re-frame filtering method to obtain the second detection result of the target object.

[0049] In one possible implementation, the second filtering unit includes:

[0050] The first filtering subunit is used to filter out the shape detection errors in the second detection result of the target object using a contrastive language image pre-trained CLIP model, so as to obtain the third detection result of the target object.

[0051] The second filtering subunit is used to filter out text detection errors in the third detection result of the target object using the Visual Language Model (VLM) to obtain the target image data processing result.

[0052] In one possible implementation, the first filtering subunit includes:

[0053] The third encoding subunit is used to crop out the detection box region where the target object is located from the image where the second detection result is located to obtain the first sub-image; and to perform image encoding on the first sub-image through the image encoder of the CLIP model to obtain the image features of the first sub-image;

[0054] The fourth encoding subunit is used to encode the positive sample prompt text corresponding to the target object and the negative sample prompt text corresponding to N other objects using the text encoder of the CLIP model, so as to obtain text features of N+1 categories; where N is a positive integer greater than 0.

[0055] A subunit is determined to calculate the feature similarity between the first sub-image features and the text features of the N+1 categories, and based on the calculation results, to determine the data with shape detection errors in the second detection result;

[0056] The first processing subunit is used to filter out the data with shape detection errors in the second detection result to obtain the third detection result of the target object.

[0057] In one possible implementation, the Visual Language Model (VLM) includes a dynamic high-resolution adaptive selection module, an image encoder, a pixel shuffling layer, a feature mapping layer, a text encoder, and a large language model; the second filtering subunit includes:

[0058] The second input subunit is used to determine the prompt instruction question text based on the content of the target object; and input the prompt instruction question text into the text encoder of the visual language model (VLM) to obtain the text features corresponding to the prompt question text.

[0059] The fifth encoding subunit is used to crop out the detection box region where the target object is located from the image where the third detection result is located to obtain the second sub-image; and to encode the second sub-image through the dynamic high-resolution adaptive selection module, image encoder, pixel shuffling layer and feature mapping layer of the visual language model VLM to obtain the image features of the second sub-image.

[0060] The third input subunit is used to input the text features corresponding to the prompt instruction question text and the image features of the second sub-graph into the large language model, and determine the data of text detection errors in the third detection result based on the model output results;

[0061] The second processing subunit is used to filter out the text detection errors in the third detection result to obtain the fourth detection result of the target object, which is used as the target image data processing result.

[0062] In one possible implementation, the target object is a road construction sign.

[0063] In one possible implementation, the device further includes:

[0064] The training unit is used to use the processing results of the target image data as training labels, and combined with the target image and the visual cue text of the target object contained in the target image, to iteratively train the initial target detection model to obtain the target detection model, which is used to perform the target detection task.

[0065] This application also provides an image data processing device, including: a processor, a memory, and a system bus;

[0066] The processor and the memory are connected via the system bus;

[0067] The memory is used to store one or more programs, the one or more programs including instructions, which, when executed by the processor, cause the processor to perform any of the above-described implementations of the image data processing method.

[0068] This application also provides a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to perform any of the above-described image data processing methods.

[0069] This application also provides a computer program product, which, when run on a terminal device, causes the terminal device to execute any of the above-described image data processing methods.

[0070] This application provides an image data processing method, apparatus, storage medium, and device. First, a target image is acquired; then, visual cue text for a target object to be detected is acquired; wherein the target image is an environmental image surrounding a vehicle; the target object may contain text information. Then, an Open Vocabulary Detection (OVD) model is used to detect the target object from the target image based on the visual cue text, obtaining a first detection result. The first detection result is then subjected to re-frame filtering to obtain a second detection result for the target object. Next, erroneous data in the second detection result of the target object is filtered to obtain the target image data processing result.

[0071] As can be seen, this application introduces the Open Vocabulary Detection (OVD) model and employs multiple filtering strategies (such as combining the CLIP and VLM models in image data processing to fully leverage their collaborative filtering capabilities) to filter out erroneous data (e.g., data with shape and text detection errors) in the target image. This results in more accurate target image data processing, providing strong data support for the subsequent management and analysis of vehicle-related data, such as target detection. Consequently, it improves the accuracy of detection results, enhances the user's riding experience, and improves vehicle driving safety. Attached Figure Description

[0072] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0073] Figure 1 A flowchart illustrating an image data processing method provided in an embodiment of this application;

[0074] Figure 2 This is an example diagram illustrating the overall process of image data processing provided in the embodiments of this application;

[0075] Figure 3 An example diagram illustrating the process of detecting target objects from a target image based on visual cue text using an open vocabulary detection model, as provided in this embodiment of the application, to obtain a first detection result;

[0076] Figure 4 A comparative example diagram showing how a second detection result of a target object is obtained by performing re-frame filtering on the first detection result, as provided in an embodiment of this application.

[0077] Figure 5 An example diagram illustrating the process of using the CLIP model to filter out shape detection errors in the second detection result of the target object to obtain the third detection result of the target object, as provided in this embodiment of the application.

[0078] Figure 6 An example diagram illustrating the process of filtering text detection errors in the third detection result of a target object using a VLM model, as provided in this embodiment of the application.

[0079] Figure 7 This is a schematic diagram of the composition of an image data processing apparatus provided in an embodiment of this application. Detailed Implementation

[0080] As we all know, in the field of vehicles, autonomous driving technology has gradually entered people's lives; due to the complexity of road traffic, accurate detection of different obstacles is the key to the application of autonomous driving technology.

[0081] However, traditional object detection methods rely heavily on costly manually labeled data (such as Convolutional Neural Networks (CNN) models). This not only increases the cost of model development but also limits the rapid applicability of models in new fields. This is because if these existing open-source object detection models are directly deployed in practical applications such as autonomous driving, they often encounter performance bottlenecks due to the lack of domain-specific optimization and adjustments, making it difficult to meet the detection accuracy requirements of practical applications. This is especially true for target objects such as construction signs, which require not only visual inspection but also semantic understanding and recognition, resulting in low accuracy.

[0082] Therefore, improving the detection accuracy of different obstacles containing text information in practical applications such as autonomous driving, and filtering out those erroneous detection objects that may look similar but do not semantically conform to the definition of the target obstacle (such as a construction sign) is an urgent problem to be solved.

[0083] To address the aforementioned shortcomings, this application provides an image data processing method. First, a target image is acquired; then, visual cue text for the target object to be detected is acquired; wherein the target image is an environmental image surrounding a vehicle; the target object may contain text information. Next, an Open Vocabulary Detection (OVD) model is used to detect the target object from the target image based on the visual cue text, obtaining a first detection result for the target object. The first detection result is then subjected to bounding box filtering to obtain a second detection result for the target object. Finally, erroneous data in the second detection result are filtered to obtain the target image data processing result.

[0084] As can be seen, this application introduces the Open Vocabulary Detection (OVD) model and employs multiple filtering strategies (such as image data processing strategies that combine the CLIP model and the VLM model, giving full play to their collaborative filtering capabilities, filtering out data in the target image that has errors in the detection of the target object (such as data with errors in shape detection and text detection), thereby selecting more accurate target image data processing results. This provides strong data support for the management and analysis of vehicle-related data such as subsequent target detection, thereby improving the accuracy of the detection results and enhancing the user's riding experience and vehicle driving safety.

[0085] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0086] First Embodiment

[0087] See Figure 1 This is a flowchart illustrating an image data processing method provided in this embodiment. The method includes the following steps:

[0088] S101: Acquire the target image; and acquire the visual cue text of the target object to be detected; wherein, the target image is an image of the environment around the vehicle.

[0089] In this embodiment, any image processed using this method is defined as a target image, and the object to be detected (or identified) (which may contain text information) contained in the target image is defined as a target object. The target image typically refers to an image of the environment surrounding the vehicle, and the target object refers to any obstacle in the driving environment around the vehicle that needs to be detected promptly (such as road construction signs, traffic barriers, etc.). It should be noted that this application does not limit the method of acquiring the target image. For example, a road surface image captured by an external vision module of the vehicle can be used as the target image; alternatively, an image of the surrounding environment captured by an aerial photography device during the vehicle's movement can be used as the target image; or, a remote sensing image from a satellite can be acquired, which includes images captured during the vehicle's movement. Furthermore, this embodiment does not limit the type of target image. For example, the target image can be a color image composed of the three primary colors red (R), green (G), and blue (B), or it can be a grayscale image, etc.

[0090] In this application, the specific content of the target object contained in the target image is not limited and can be set according to the actual situation. For example, the target object can be set as a road construction sign containing text information, or it can be set as a traffic guardrail, shop sign, information board, speed limit sign, etc.

[0091] Furthermore, the visual cue text of the target object can be determined based on the text information contained in the target object to be detected in the target image, in order to perform the subsequent step S102. It should be noted that this application does not limit the specific text content or determination method of the visual cue text corresponding to the target object (such as a construction sign) to be detected in the acquired target image. For example, assuming the target image is as follows... Figure 2In the image shown in the lower left corner, the target object to be detected is a construction sign. The visual cue text content corresponding to the construction sign can be the visual cue word "construction sign", etc.

[0092] Based on this, and using the target image and its contained target objects (such as construction signs) and their visual cue text (e.g., "construction sign"), subsequent steps S102-S104 can be executed. These steps sequentially employ the Open Lexical Detection Model (GLEE) and multiple filtering methods (e.g., bounding box filtering, CLIP model filtering, and Visual Language Model (VLM)) to filter out data with shape detection errors and text detection errors in the target image. This process determines a more accurate result for the target image data processing. Figure 2 As shown.

[0093] S102: Detect the target object from the target image based on the visual cue text using an open vocabulary detection model to obtain the first detection result.

[0094] In this embodiment, after obtaining the target image and the visual cue text of the target object to be detected in the target image through step S101, in order to obtain a more accurate data processing result of the target image, the target image and the visual cue text of the target object can be further input into an existing or future open vocabulary detection model, so that the open vocabulary detection model can detect the target object from the target image according to the visual cue text, thereby identifying the detection result of the target object, such as the detection box corresponding to the target object in the target image, and defining it as the first detection result of the target object, so as to execute the subsequent step S103.

[0095] It should be noted that this application does not limit the specific structure of the open vocabulary detection model. One optional implementation is that this application uses the General Object Foundation Model for Images and Videos at Scale (GLEE) as the open vocabulary detection model, such as... Figure 3 As shown, the GLEE model can include, but is not limited to, text encoders, image encoders, object detectors, and feature transformation modules.

[0096] Based on this, an optional implementation method is that the specific implementation process of this step S102 may include the following steps S1021-S1025:

[0097] S1021: Text features are obtained by text encoding the visual cue text of the target object using a text encoder based on the GLEE model.

[0098] In this implementation, after obtaining the target image and the visual cue text (e.g., "construction sign") of the target object to be detected (e.g., a construction sign) in step S101, as follows: Figure 3 As shown, the visual cue text of the target object (such as a construction sign) (e.g., "construction sign") can be further input into the text encoder of the GLEE model (the specific structure is not limited, and any existing or future text encoder can be used). The GLEE model's text encoder then performs text encoding on the visual cue text of the target object, obtaining the text features corresponding to the visual cue text of the target object (such as "construction sign"), as shown below. Figure 3 As shown, this is used to execute the subsequent step S1025.

[0099] S1022: Image encoding of the target image is performed using the GLEE model image encoder to obtain the full image features.

[0100] In this implementation, after obtaining the target image through step S101, as follows: Figure 3 As shown, the target image can be further input into the GLEE model's image encoder to perform image encoding processing on the target image, thereby obtaining the full-image features corresponding to the target image, such as... Figure 3 As shown, this is used to execute the subsequent step S1023.

[0101] S1023: The object detector of the GLEE model is used to detect the features of the whole image and obtain the detection box.

[0102] In this implementation, after obtaining the full-image features corresponding to the target image through step S1022, as follows: Figure 3 As shown, the entire image feature can be further input into the object detector of the GLEE model to detect the entire image feature and output the detection box for each object, so as to perform the subsequent step S1024.

[0103] S1024: Input the detection box into the feature transformation module of the GLEE model to obtain the image features corresponding to the detection box.

[0104] In this implementation, after obtaining the detection boxes for each object through step S1023, as follows: Figure 3 As shown, the detection box can be further input into the feature transformation module of the GLEE model to output the image features corresponding to the detection box in the target image, which can then be used to execute the subsequent step S1025.

[0105] S1025: Calculate the similarity matrix between the image features and the text features, and determine the first detection result of the target object based on the calculation result.

[0106] In this implementation, after obtaining text features in step S1021 and image features in step S1024, as follows: Figure 3 As shown, further, existing or future feature similarity calculation methods can be used to calculate the similarity matrix between image features and text features, and the maximum similarity value can be selected from it. Then, the text category corresponding to this maximum similarity value can be used as the category corresponding to the target object detection box, such as... Figure 3 The detection result of the target object "construction sign" is... Figure 3 The bounding box enclosing the construction sign indicates the location of the target object, the "construction sign".

[0107] S103: Perform re-frame filtering on the first detection result to obtain the second detection result of the target object.

[0108] It should be noted that, because the target object detected in this application may be affected by its shape, overlapping frames may occur, such as... Figure 3 As shown, Figure 3 In the first test results of the "construction sign", because the "construction sign" has a "support", the test of each construction sign is unstable and often overlaps, that is, "construction sign without support" and "construction sign with support" will be detected.

[0109] Therefore, in this embodiment, after obtaining the first detection result of the target object through step S102, in order to obtain a more accurate data processing result for the target image, a re-frame filtering method such as the intersection over ground truth (IOG) can be further used to perform re-frame filtering on the first detection result of the target object to obtain the second detection result of the target object, which is then used to execute the subsequent step S104. For example, as Figure 4 As shown, for Figure 3 The first test result of the "construction sign" was obtained (i.e. Figure 4 (The left image in the image) can be processed using IOG re-frame filtering logic to retain the construction sign with the bracket, resulting in a second detection result for the filtered target object, such as... Figure 4 The right-hand side of the image shows the calculation formula for IOG re-frame filtering.

[0110]

[0111] Where B1 represents the first box; B2 represents the second box; area(B1,B2) calculates the total area of ​​the overlapping part of the two boxes; area(B2) calculates the area of ​​the second box.

[0112] S104: Filter the data with detection errors in the second detection result of the target object to obtain the target image data processing result.

[0113] In this embodiment, after performing re-frame filtering on the first detection result of the target object in step S103 to obtain the second detection result of the target object, in order to obtain more accurate data processing results of the target image (such as to effectively distinguish between "construction sign" and "non-construction sign" targets), existing or future data filtering methods can be used to filter out the data with detection errors in the second detection result of the target object, thereby obtaining more accurate target image data processing results. This provides strong data support for the management and analysis of vehicle-related data such as subsequent target detection, thereby improving the accuracy of the detection results and enhancing the user's riding experience and vehicle driving safety.

[0114] Specifically, one possible implementation is that, in order to obtain more accurate data processing results for the target image, after obtaining the second detection result of the target object, the Contrastive Language Image Pre-training (CLIP) model can first be used to filter out the data with shape detection errors in the second detection result of the target object to obtain the third detection result of the target object. Then, the Visual Language Model (VLM) can be used to filter out the data with text detection errors in the third detection result of the target object to obtain more accurate data processing results for the target image.

[0115] In this implementation, after performing re-frame filtering on the first detection result of the target object in step S103 to obtain the second detection result of the target object, in order to obtain a more accurate data processing result of the target image (such as to effectively distinguish between "construction sign" and "non-construction sign" targets), the CLIP model can be used to filter the data with shape detection errors in the second detection result of the target object to obtain the third detection result of the target object.

[0116] It should be noted that this application employs the CLIP model to filter out shape detection errors in the second detection results of the target object because the CLIP model is based on the principle of contrastive learning. By optimizing the model to align related images and text within the same feature space, it achieves semantic matching between image content and natural language descriptions. Compared to traditional object detection or image classification models, a significant advantage of the CLIP model is its 'zero-shot learning' capability. In this mode, the CLIP model can utilize knowledge learned during the pre-training phase to identify and classify new categories without any task-specific fine-tuning. This capability makes CLIP a flexible and powerful tool, particularly suitable for visual tasks involving numerous categories, high annotation costs, or constantly changing scenes.

[0117] Specifically, to improve the filtering effect of the CLIP model, taking "construction sign" as the target object, in order to effectively distinguish between "construction sign" and "non-construction sign" targets, it is necessary to pre-set appropriate category labels for negative sample (i.e., non-construction sign) prompt text. The N (the specific value is not limited, but must be a positive integer greater than 0) negative sample prompt texts (labels) for other objects set in this application can be, but are not limited to, objects that are prone to false detection, such as: "shop sign," "traffic barrier," "information board," "notice," "other," "arrow point sign," "crash barrel," "street name sign," "speed restriction sign," "school sign," and "zebra crossing sign." This detailed and diverse design of negative sample prompt text descriptions can more effectively use the CLIP model to filter out non-construction signs, thereby reducing the false detection rate.

[0118] Specifically, one optional implementation is that after obtaining the second detection result in step S103, during the filtering process using the CLIP model (which may include, but is not limited to, two modules: a text encoder and an image encoder), firstly, the detection box region containing the target object (such as a construction sign) can be cropped from the image containing the second detection result, as shown in Figure 5. That is, the region corresponding to the construction sign detection box (i.e., the box enclosing the construction sign) is cropped from the image containing the second detection result and used as the first sub-image (the number is not limited). Figure 5 (The first sub-image in the example is only shown here), and this first sub-image is input into the CLIP model's image encoder to perform image encoding on the first sub-image, obtaining the image features of the first sub-image, such as... Figure 5 As shown in I1. Then, the positive sample prompt text corresponding to the target object (such as a construction sign) (e.g., "construction sign") and the negative sample prompt text corresponding to N other objects (e.g., "traffic barrier, shop sign, information board, notice, other, arrow point sign, crash barrel, street name sign, speed restriction sign, school sign, zebra crossing sign, etc., representing traffic barriers, shop signs, information boards, notices, other, arrow signs, crash barrels, street signs, speed limit signs, school signs, zebra crossing signs, etc.) are input into the CLIP model's text encoder. The CLIP model's text encoder encodes the positive sample prompt text corresponding to the target object and the negative sample prompt text corresponding to the N other objects, obtaining the encoded text features of N+1 categories, such as... Figure 5 The values ​​T1, T2, ..., T(N+1) are shown. Next, the feature similarity between the image features of the first sub-image and the text features of these N+1 categories is calculated, and the text category corresponding to the highest similarity is taken as the detection category of the target object. Thus, when the detection result of the target object does not match the category label corresponding to the target object's detection result, it can be determined that the sub-image corresponds to data with shape detection errors, and it needs to be filtered out from the second detection result to obtain the third detection result of the target object.

[0119] For example, such as Figure 5 As shown, assuming the maximum feature similarity between image feature I1 and the text features T1, T2…T(N+1) of these N+1 categories is "I1·T2", then the corresponding "traffic barrier" is the category of the current detection box. In other words, the category determined by the CLIP model for the first sub-image is "traffic barrier," not "construction sign." Therefore, the current detection box is a false detection, i.e., data with incorrect shape detection, and needs to be filtered out from the second detection result.

[0120] Understandably, as a classification model, the CLIP model's filtering capabilities can be further enhanced by expanding the categories of negative samples to include more categories that are easily misidentified as construction signs. Using a negative sample filtering strategy, the CLIP model can effectively filter out negative sample data.

[0121] Furthermore, based on this, a Visual Language Model (VLM) can be used to filter out text detection errors in the third detection result of the target object, resulting in a fourth detection result of the target object, which serves as the final target image data processing result.

[0122] It should be noted that, unlike small targets such as traffic cones, large targets such as construction signs detected in this application are not only affected by their shape, resulting in overlapping frames, but also contain text information. Therefore, it may not be possible to distinguish construction signs from similar-looking directional signs based solely on the appearance of the CLIP model.

[0123] Therefore, in this embodiment, after obtaining the third detection result of the target object, in order to obtain a more accurate data processing result of the target image, the text understanding ability of the existing or future Visual Language Model (VLM) can be used to filter the text detection error data in the third detection result of the target object, thereby identifying a more accurate fourth detection result of the target object, such as the detection box corresponding to the target object in the target image, as the final target image data processing result.

[0124] It should be noted that this application does not limit the specific structure of the Visual Language Model (VLM). One optional implementation is that this application uses the InternVL2 model as the Visual Language Model (VLM), such as... Figure 6 As shown, the InternVL2 model may include, but is not limited to, a dynamic high-resolution adaptive selection module, an image encoder, a pixel shuffling layer, a feature mapping layer, a text encoder, and a Large Language Model (LLM).

[0125] Building upon this, an alternative implementation involves filtering out shape detection errors in the second detection result of the target object using the CLIP model to obtain the third detection result. Further, the prompt text can be determined based on the content of the target object. This prompt text is then input into the text encoder of a Visual Language Model (VLM) (such as the InternVL2 model) to obtain the corresponding text features. For example, still using a construction sign as the target object, the prompt text can be set as a simple yes or no question-and-answer format, such as "Is it a construction sign? Please answer 'yes' or 'no'". This text can then be input into the InternVL2 model text encoder for encoding to obtain the corresponding text features.

[0126] Then, the detection box region containing the target object can be cropped from the image containing the third detection result to obtain the second sub-image. This second sub-image is then input into the dynamic high-resolution adaptive selection module, image encoder, pixel shuffling layer, and feature mapping layer of a Visual Language Model (VLM) (such as the InternVL2 model). The second sub-image is then encoded using these components to obtain its image features, such as... Figure 6 As shown.

[0127] Next, the text features corresponding to the prompt question text and the image features of the second sub-image can be input into the Large Language Model (LLM). Based on the model output, the data with text detection errors in the third detection result can be identified and filtered out to obtain the fourth detection result of the target object, which serves as the target image data processing result. For example, based on the above example, for the prompt question text "Is it a construction sign? Please answer 'yes' or 'no'", the model output result can be either yes or no, and the detection box corresponding to 'no' (i.e., the data with text detection errors) can be filtered out.

[0128] In addition, an alternative implementation is that after determining a more accurate target image data processing result through the above steps S101-S104, the target image data processing result can be used as a training label, combined with the target image and the visual cue text of the target object contained in the target image, to iteratively train the initial target detection model (the specific model structure is not limited, such as the GLEE model, etc.) to obtain the target detection model, which is then used to perform the target detection task.

[0129] Specifically, in this implementation, the target image and the visual cue text of the target object it contains can be used as the input to the model, and the box of the target object in the corresponding target image data processing result can be used as the output. Multiple rounds of model training are performed, and the detection results obtained in each round of training are compared with the labels. The model parameters are updated according to the difference between the two until the preset conditions are met, such as the loss constraint function is very small and basically unchanged, or the preset maximum number of iterations is reached, at which point the update of the model parameters is stopped, the training of the target detection model is completed, and a trained target detection model is generated to perform the corresponding target detection task.

[0130] In summary, the image data processing method provided in this embodiment first acquires a target image; then acquires visual cue text for the target object to be detected; wherein the target image is an environmental image of the vehicle's surroundings; the target object may contain text information; then, the target object is detected from the target image based on the visual cue text using an Open Vocabulary Detection (OVD) model, obtaining a first detection result for the target object; and then, the first detection result for the target object is subjected to re-bounding filtering to obtain a second detection result for the target object; finally, the data with detection errors in the second detection result for the target object is filtered to obtain the target image data processing result.

[0131] As can be seen, this application introduces the Open Vocabulary Detection (OVD) model and employs a multi-filtering strategy (such as an image data processing strategy that combines the CLIP model and the VLM model to fully leverage their collaborative filtering capabilities) to filter out data with shape detection errors and text detection errors in the target image. This results in more accurate target image data processing results, providing strong data support for the subsequent management and analysis of vehicle-related data such as target detection. Consequently, the accuracy of the detection results is improved, enhancing the user's riding experience and vehicle driving safety.

[0132] Second Embodiment

[0133] This embodiment will introduce an image data processing apparatus; please refer to the above method embodiment for related content.

[0134] See Figure 7This is a schematic diagram of the composition of an image data processing apparatus provided in this embodiment. The apparatus 700 includes:

[0135] The acquisition unit 701 is used to acquire a target image and acquire visual cue text of the target object to be detected; the target image is an environmental image of the vehicle's surroundings; the target object includes text information.

[0136] Detection unit 702 is used to detect the target object from the target image based on the visual cue text using an open vocabulary detection model, and obtain a first detection result;

[0137] The first filtering unit 703 is used to perform reframe filtering on the first detection result to obtain the second detection result of the target object;

[0138] The second filtering unit 704 is used to filter the data with detection errors in the second detection result of the target object to obtain the target image data processing result.

[0139] In one implementation of this embodiment, the open vocabulary detection model is GLEE, a general object foundation model for large-scale images and videos.

[0140] In one implementation of this embodiment, the detection unit 702 includes:

[0141] The first encoding subunit is used to encode the visual cue text of the target object using a text encoder of the GLEE model to obtain text features;

[0142] The second encoding subunit is used to encode the target image using an image encoder based on the GLEE model to obtain the full image features;

[0143] The detection subunit is used to detect the features of the entire image using the target detector of the GLEE model to obtain the detection box;

[0144] The first input subunit is used to input the detection box into the feature transformation module of the GLEE model to obtain the image features corresponding to the detection box;

[0145] The calculation subunit is used to calculate the similarity matrix between the image features and the text features, and determine the first detection result of the target object based on the calculation result.

[0146] In one implementation of this embodiment, the first filtering unit 703 is specifically used for:

[0147] The first detection result is processed by the intersection ratio (IOG) of the actual value and the re-frame filtering method to obtain the second detection result of the target object.

[0148] In one implementation of this embodiment, the second filtering unit 704 includes:

[0149] The first filtering subunit is used to filter out the shape detection errors in the second detection result of the target object using a contrastive language image pre-trained CLIP model, so as to obtain the third detection result of the target object.

[0150] The second filtering subunit is used to filter out text detection errors in the third detection result of the target object using the Visual Language Model (VLM) to obtain the target image data processing result.

[0151] In one implementation of this embodiment, the first filtering subunit includes:

[0152] The third encoding subunit is used to crop out the detection box region where the target object is located from the image where the second detection result is located to obtain the first sub-image; and to perform image encoding on the first sub-image through the image encoder of the CLIP model to obtain the image features of the first sub-image;

[0153] The fourth encoding subunit is used to encode the positive sample prompt text corresponding to the target object and the negative sample prompt text corresponding to N other objects using the text encoder of the CLIP model, so as to obtain text features of N+1 categories; where N is a positive integer greater than 0.

[0154] A subunit is determined to calculate the feature similarity between the first sub-image features and the text features of the N+1 categories, and based on the calculation results, to determine the data with shape detection errors in the second detection result;

[0155] The first processing subunit is used to filter out the data with shape detection errors in the second detection result to obtain the third detection result of the target object.

[0156] In one implementation of this embodiment, the Visual Language Model (VLM) includes a dynamic high-resolution adaptive selection module, an image encoder, a pixel shuffling layer, a feature mapping layer, a text encoder, and a large language model; the second filtering subunit includes:

[0157] The second input subunit is used to determine the prompt text based on the content of the target object; and input the prompt text into the text encoder of the visual language model (VLM) to obtain the text features corresponding to the prompt text.

[0158] The fifth encoding subunit is used to crop out the detection box region where the target object is located from the image where the third detection result is located to obtain the second sub-image; and to encode the second sub-image through the dynamic high-resolution adaptive selection module, image encoder, pixel shuffling layer and feature mapping layer of the visual language model VLM to obtain the image features of the second sub-image.

[0159] The third input subunit is used to input the text features corresponding to the prompt instruction question text and the image features of the second sub-graph into the large language model, and determine the data of text detection errors in the third detection result based on the model output results;

[0160] The second processing subunit is used to filter out the text detection errors in the third detection result to obtain the fourth detection result of the target object, which is used as the target image data processing result.

[0161] In one implementation of this embodiment, the target object is a road construction sign.

[0162] In one implementation of this embodiment, the apparatus further includes:

[0163] The training unit is used to use the processing results of the target image data as training labels, and combined with the target image and the visual cue text of the target object contained in the target image, to iteratively train the initial target detection model to obtain the target detection model, which is used to perform the target detection task.

[0164] Furthermore, embodiments of this application also provide an image data processing device, including: a processor, a memory, and a system bus;

[0165] The processor and the memory are connected via the system bus;

[0166] The memory is used to store one or more programs, the one or more programs including instructions, which, when executed by the processor, cause the processor to perform any of the above-described implementations of the image data processing method.

[0167] Furthermore, embodiments of this application also provide a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to perform any of the above-described implementations of the image data processing method.

[0168] Furthermore, this application embodiment also provides a computer program product, which, when run on a terminal device, causes the terminal device to execute any of the above-described image data processing methods.

[0169] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0170] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0171] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0172] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image data processing method, characterized in that, include: Acquire the target image; And obtain the visual cue text of the target object to be detected; The target image is an image of the environment surrounding the vehicle; The target object is detected from the target image using an open vocabulary detection model based on the visual cue text, and a first detection result is obtained. The first detection result is subjected to reframe filtering to obtain the second detection result of the target object; The data with detection errors in the second detection result of the target object are filtered to obtain the target image data processing result.

2. The method according to claim 1, characterized in that, The open vocabulary detection model is GLEE, a general object foundation model for large-scale images and videos.

3. The method according to claim 2, characterized in that, The step of detecting the target object from the target image based on the visual cue text using an open vocabulary detection model to obtain a first detection result includes: The visual cue text of the target object is encoded using a text encoder based on the GLEE model to obtain text features; The target image is image encoded using an image encoder based on the GLEE model to obtain the full image features; The entire image features are detected using a target detector based on the GLEE model to obtain detection boxes; The detection box is input into the feature transformation module of the GLEE model to obtain the image features corresponding to the detection box; Calculate the similarity matrix between the image features and the text features, and determine the first detection result of the target object based on the calculation result.

4. The method according to any one of claims 1-3, characterized in that, The step of performing re-frame filtering on the first detection result to obtain the second detection result of the target object includes: The first detection result is processed by the intersection ratio (IOG) of the actual value and the re-frame filtering method to obtain the second detection result of the target object.

5. The method according to claim 1, characterized in that, The step of filtering out erroneous data in the second detection result of the target object to obtain the target image data processing result includes: The CLIP model, which is pre-trained using contrastive language images, is used to filter out data with shape detection errors in the second detection result of the target object, thereby obtaining the third detection result of the target object. The Visual Language Model (VLM) is used to filter out text detection errors in the third detection results of the target object, resulting in the target image data processing result.

6. The method according to claim 5, characterized in that, The third detection result of the target object is obtained by filtering out the shape detection errors in the second detection result of the target object using the contrastive language image pre-trained CLIP model, including: The detection box region containing the target object is cropped from the image containing the second detection result to obtain the first sub-image; and the first sub-image is image encoded by the image encoder of the CLIP model to obtain the image features of the first sub-image. The positive sample prompt text corresponding to the target object and the negative sample prompt text corresponding to N other objects are text encoded using the CLIP model text encoder to obtain N+1 categories of text features; where N is a positive integer greater than 0. Calculate the feature similarity between the first sub-image image features and the text features of the N+1 categories respectively, and determine the data with shape detection errors in the second detection result based on the calculation results; The data with shape detection errors in the second detection result are filtered out to obtain the third detection result of the target object.

7. The method according to claim 5, characterized in that, The Visual Language Model (VLM) includes a dynamic high-resolution adaptive selection module, an image encoder, a pixel shuffling layer, a feature mapping layer, a text encoder, and a large language model. The fourth detection result of the target object, obtained by filtering out text detection errors in the third detection result using the VLM, serves as the target image data processing result and includes: Based on the content of the target object, determine the prompt instruction question text; and input the prompt instruction question text into the text encoder of the visual language model (VLM) to obtain the text features corresponding to the prompt instruction question text; The detection box region containing the target object is cropped from the image containing the third detection result to obtain the second sub-image; and the second sub-image is image encoded by the dynamic high-resolution adaptive selection module, image encoder, pixel shuffling layer and feature mapping layer of the visual language model VLM to obtain the image features of the second sub-image. Input the text features corresponding to the prompt instruction question text and the image features of the second sub-graph into the large language model, and determine the data of text detection errors in the third detection result based on the model output results; The data with text detection errors in the third detection result are filtered out to obtain the fourth detection result of the target object, which is used as the target image data processing result.

8. The method according to claim 1, characterized in that, The target object is a road construction sign.

9. The method according to any one of claims 5-7, characterized in that, After filtering out text detection errors in the third detection result of the target object using the Visual Language Model (VLM) to obtain the fourth detection result of the target object, which is then used as the target image data processing result, the method further includes: The target image data processing result is used as the training label. Combined with the target image and the visual cue text of the target object contained in the target image, the initial target detection model is iteratively trained to obtain the target detection model, which is used to perform the target detection task.

10. An image data processing apparatus, characterized in that, include: The acquisition unit is used to acquire the target image; And obtain the visual cue text of the target object to be detected; The target image is an image of the environment surrounding the vehicle; The detection unit is used to detect the target object from the target image based on the visual cue text using an open vocabulary detection model, and obtain a first detection result; The first filtering unit is used to perform reframe filtering on the first detection result to obtain the second detection result of the target object; The second filtering unit is used to filter out the data with detection errors in the second detection result of the target object to obtain the target image data processing result.

11. An image data processing device, characterized in that, include: Processor, memory, system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, the one or more programs including instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a terminal device, cause the terminal device to perform the method described in any one of claims 1-9.

Images