Data mining method and device

Abstract

The embodiment of the invention provides a data mining method and device, and relates to the technical field of data mining. The method comprises the steps that based on a query image, at least one first image is obtained from an image set, and the first image comprises at least one second instance matched with a first instance of a to-be-mined target in the query image; obtaining an annotation box of each second instance in each first image; performing target detection on the first image by using a visual perception model, and generating a detection result comprising at least one detection frame and a target category corresponding to each detection frame; and for any target annotation box in the annotation boxes, under the condition that the corresponding target category indicates that each target detection box of the to-be-mined target is not matched with the target annotation box, determining a second instance corresponding to the target annotation box as a sample instance of the visual perception model. Therefore, the sample instance which is similar to the first instance of the to-be-mined target and is poorer in current visual perception model recognition can be efficiently mined from the mass stock data.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data mining technology, and in particular to a data mining method and apparatus. Background Art

[0002] With the development of artificial intelligence (AI) technology, more and more AI solutions are being used in the field of autonomous driving. Visual perception models, the "eyes" of autonomous driving, are the models that have made the most in-depth exploration of AI technology. However, the performance of visual perception models is highly dependent on the quality of their training datasets. Mining as diverse data as possible on motor vehicles and non-motor vehicles from public roads and public datasets is a key means of enriching the coverage of training datasets and improving data quality. Summary of the Invention

[0003] The embodiments of the present application provide a data mining method and device that can efficiently mine sample instances that are similar to the first instance of the target to be mined and are poorly recognized by the current visual perception model from massive existing data, for iterative upgrading of the visual perception model.

[0004] In a first aspect, an embodiment of the present application provides a data mining method, comprising: Based on a query image, obtaining at least one first image from an image collection; wherein the query image includes a first instance of a target to be mined, and each first image includes at least one second instance matching the first instance; Obtaining a labeling box of each second instance in each first image; Performing object detection on the first image using a visual perception model to generate a detection result, where the detection result includes at least one detection box and an object category corresponding to each detection box; For any target annotation box in each annotation box, when each target detection box does not match the target annotation box, the second instance corresponding to the target annotation box is determined as a sample instance of the visual perception model, and the target detection box is included in the detection result and the corresponding target category indicates the target to be mined.

[0005] In one embodiment, the data mining method further includes: For any target detection box, when each annotation box does not match the target detection box, the third instance corresponding to the target detection box is determined as a sample instance of the visual perception model.

[0006] In one embodiment, obtaining at least one first image from an image collection based on a query image includes: generating second feature information based on the query image; Determining, based on similarities between each piece of first feature information and the second feature information in a target database, target feature information that matches the second feature information in each piece of first feature information, wherein the target database stores a plurality of pieces of image information, each piece of image information including an image identifier of an original image in the image set and first feature information of an image region in the original image that contains the second instance; An original image indicated by an image identifier corresponding to the target feature information is obtained from the image set as the first image.

[0007] In one embodiment, generating second feature information based on the query image includes: Second feature information is generated based on the query image and text information describing the query image.

[0008] In one embodiment, the step of determining the matching status of the target annotation box and each target detection box includes: For a first image region corresponding to the target annotation frame in the first image, and a second image region corresponding to each target detection frame in the first image, determining an intersection-over-union ratio of the first image region and the second image region; When the maximum value of each intersection-over-union ratio is equal to a preset value, it is determined that each target detection frame does not match the target annotation frame; When the maximum value is greater than the preset value, it is determined that the target detection frame corresponding to the maximum value matches the target annotation frame.

[0009] In one embodiment, determining the intersection-over-union ratio of the first image region and the second image region includes: An intersection-over-union ratio is determined based on an area of ​​an overlapping portion of the first image region and the second image region and an area jointly occupied by the first image region and the second image region.

[0010] In one embodiment, determining the intersection-over-union ratio of the first image region and the second image region includes: The intersection-over-union ratio is determined based on the first number of pixels in the overlapping part of the first image area and the second image area, the second number of pixels in the other part of the first image area except the overlapping part, and the third number of pixels in the other part of the second image area except the overlapping part.

[0011] In one embodiment, determining the intersection-over-union ratio includes: When the overlapping portion is an irregular polygon, an intersection-over-union ratio is determined based on the first number of pixels, the second number of pixels, and the third number of pixels.

[0012] In one embodiment, after determining that the target detection frame corresponding to the maximum value matches the target annotation frame, the method further includes: When the above-mentioned maximum value is less than the first threshold, or the difference between the first heading angle and the second heading angle is greater than the second threshold, the second instance corresponding to the target annotation box is determined as the sample instance of the visual perception model; wherein the first heading angle is the heading angle of the second instance corresponding to the target annotation box, and the second heading angle is the heading angle of the second instance corresponding to the target detection box corresponding to the above-mentioned maximum value.

[0013] In a second aspect, an embodiment of the present application provides a data mining device, comprising: A first acquisition unit is configured to acquire at least one first image from the image collection based on a query image; wherein the query image includes a first instance of the target to be mined, and each first image includes at least one second instance matching the first instance; A second acquiring unit is configured to acquire a labeling box of each second instance in each first image; a detection unit configured to perform object detection on the first image using a visual perception model and generate a detection result, the detection result including at least one detection box and an object category corresponding to each detection box; The determination unit is configured to determine, for any target annotation box in each annotation box, when each target detection box does not match the target annotation box, a second instance corresponding to the target annotation box as a sample instance of the visual perception model, the target detection box is included in the detection result and the corresponding target category indicates the target to be mined.

[0014] In the solution provided in an embodiment of the present application, an image collection may include a vast number of images, which can be considered a vast amount of existing data. This solution can first obtain at least one first image from the image collection based on a query image; the query image includes a first instance of the target to be mined, and each first image includes at least one second instance that matches the first instance. Subsequently, an annotation box for each second instance in each first image can be obtained, and target detection can be performed on the first image using a visual perception model to generate a detection result. The detection result includes at least one detection box and the target category corresponding to each detection box. Next, for any target annotated box in the annotated boxes corresponding to the first image, if the target detection box (the target category corresponding to the detection box in the detection result indicates the detection box of the target to be mined) does not match the target annotated box, it can be determined that the visual perception model missed the second instance corresponding to the target annotated box, and this second instance is poorly recognized by the visual perception model. This second instance is then determined as a sample instance for the visual perception model. This allows efficient mining of sample instances from the vast amount of existing data that are similar to the first instance of the target to be mined and poorly recognized by the current visual perception model, for iterative upgrades of the visual perception model. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The following detailed description of the specific embodiments of the present application in conjunction with the accompanying drawings will make the technical solutions and other beneficial effects of the present application apparent.

[0016] Figure 1 is a flow chart of the data mining method in the embodiment of the present application; Figure 2 It is a schematic diagram of the data mining process in the embodiment of the present application; Figure 3 is another schematic diagram of the data mining process in an embodiment of the present application; Figure 4 is a schematic diagram of a marking frame and a detection frame in an embodiment of the present application; Figure 5 is a schematic diagram of the overlapping first image area and second image area in an embodiment of the present application; Figure 6 is another schematic diagram of the overlapping first image area and second image area in an embodiment of the present application; Figure 7 is another schematic diagram of the overlapping first image area and second image area in an embodiment of the present application; Figure 8 is another flow chart of the data mining method in an embodiment of the present application; Figure 9 It is a structural diagram of the data mining device in an embodiment of the present application.

[0017] Reference numerals: 901 - first acquisition unit, 902 - second acquisition unit, 903 - detection unit, 904 - determination unit. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0019] In the description of this application, it should be noted that, unless otherwise specified or limited, the term "and / or" herein is merely a description of an association relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, and B exists alone. Furthermore, the character " / " herein, unless otherwise specified, generally indicates that the associated objects are in an "or" relationship.

[0020] As mentioned earlier, with the development of artificial intelligence (AI) technology, an increasing number of AI solutions are being used in the field of autonomous driving. Visual perception models, the "eyes" of autonomous driving, are the models that have most deeply explored AI technologies. However, the performance of visual perception models is highly dependent on the quality of their training datasets. Mining as diverse a range of data, including both motor vehicles and non-motor vehicles, from public roads and public datasets is a key means of enriching the coverage of training datasets and improving data quality.

[0021] Due to technological and economic development, the number of unusual tricycles on the road continues to increase, including food trucks, modified mobility scooters, and overloaded delivery vehicles. Missed and false detections of these tricycles, such as failing to detect a moving mobility scooter, can pose significant safety risks and impact the comfort and safety of autonomous driving. However, this type of data is often extremely rare in the vast amount of collected and publicly available datasets. To accelerate the iteration of visual perception models and improve the accuracy of such data recognition, it is necessary to efficiently and quickly mine similar scenarios from existing data for iterative upgrades of the visual perception models.

[0022] Currently, most solutions use driver labeling, manual labeling, similarity search and other solutions, or combine two or three solutions together.

[0023] Driver labeling involves drivers using tools to mark obstacles on the vehicle when they encounter a scene similar to the one they need. Engineers then use the time the driver marked the obstacle to find the corresponding image. This approach requires a high level of driver attention, distracting drivers, and only allows drivers to label a small number of obstacle types, making it incapable of processing existing data.

[0024] Manual labeling means that every time a new unusual-shaped tricycle needs to be found, the user manually searches the image within a certain circle to find the corresponding unusual-shaped tricycle scene. This solution is very labor-intensive and extremely inefficient.

[0025] Similarity search involves using deep learning technology to encode each image in the massive amount of image data in an existing database. The user then provides an image of the desired unusual-shaped tricycle, which is then encoded using the same encoding scheme. The image is then searched for images in the existing database that match the encoding. Manual screening is then used to find similar images of unusual-shaped tricycles. However, this approach relies heavily on manual experience, and each update to the long-tail tricycle discovery requires manual re-labeling of the corresponding tricycle type from the massive amount of similar data, which is time-consuming and costly.

[0026] In the process of streamlined data production, if all produced data is included in the training dataset, it will dilute the effective data and increase training costs. Therefore, it is necessary to mine data that is truly valuable to the model, such as scenarios where the current version of the model has weak detection capabilities and long-tail data.

[0027] Long-tail data refers to data in the dataset for training the model in which a certain type of data sample accounts for a small proportion and has low sample richness. Such long-tail data not only cannot express the characteristics of this type of target and cannot be generalized, but is also easily affected by other categories of data with a large proportion and high richness during the training process, ultimately resulting in poor detection performance of the model for the target.

[0028] Three-wheeled vehicles in cities typically include: cargo tricycles, characterized by their large cargo volume that obscures the main body of the vehicle and contains cargo of varying shapes; food trucks, often obscured by multiple pedestrians; and street vendor carts, which are even larger and often obscured by pedestrians. Because tricycles represent a smaller proportion of objects on urban roads than cars, trucks, two-wheeled vehicles, and pedestrians, their detection capabilities in practical autonomous driving models are weaker than those of other objects.

[0029] To identify tricycles, which are poorly detected by visual perception models, within the massive amount of existing data, it is necessary to analyze the possible outcomes of long-tail data object detection by the visual perception model, including false detections, missed detections, and position shifts. A false detection occurs when other objects are identified as tricycles, or vice versa. A missed detection occurs when a tricycle is not detected. Position shifts occur when the detected box position deviates significantly from the actual position. These issues are primarily caused by the long-tail data for tricycles.

[0030] Therefore, it is necessary to design an automated sample instance mining solution (such as instances of long-tail targets such as tricycles) that can automatically find obstacles that are poorly recognized by existing perception models from massive stock data.

[0031] Based on the description in the foregoing, the embodiments of the present application provide a data mining method and device that can efficiently mine sample instances that are similar to the first instance of the target to be mined and are poorly recognized by the current visual perception model from massive existing data, for iterative upgrading of the visual perception model.

[0032] In some embodiments, the targets to be mined in this application are long-tail targets, which refer to target categories that have a small proportion in the training data and low sample richness, resulting in poor detection performance of the visual perception model.

[0033] In some embodiments, the present application is applied to the field of autonomous driving, and the targets to be mined in the present application include but are not limited to tricycle targets.

[0034] In some embodiments, the data mining method provided in this application can be executed by any platform, system, server, device, device cluster or apparatus with computing and processing capabilities.

[0035] Figure 1 This is a flow chart of the data mining method in the embodiment of the present application. The mining method includes the following steps: S101: Based on a query image, obtain at least one first image from an image collection; wherein the query image includes a first instance of a target to be mined, and each first image includes at least one second instance matching the first instance; S103: Obtaining a labeling box of each second instance in each first image; S105: Performing object detection on the first image using a visual perception model to generate a detection result, where the detection result includes at least one detection box and an object category corresponding to each detection box; S107: For any target annotation box in each annotation box, when each target detection box does not match the target annotation box, the second instance corresponding to the target annotation box is determined as a sample instance of the visual perception model, and the target detection box is included in the detection result and the corresponding target category indicates the target to be mined.

[0036] exist Figure 1 In the solution provided by the corresponding embodiment, the image collection may include a vast number of images, which can be considered a vast amount of existing data. This solution first obtains at least one first image from the image collection based on a query image; the query image includes a first instance of the target to be mined, and each first image includes at least one second instance matching the first instance. Subsequently, the labeled bounding boxes of each second instance in each first image are obtained, and target detection is performed on the first image using a visual perception model, generating detection results comprising at least one detection box and the target category corresponding to each detection box. Next, for any target labeled bounding box in the corresponding labeled bounding boxes of the first image, if the target detection box (the target category corresponding to the detection box in the detection result indicates the target to be mined) does not match the target labeled bounding box, it is determined that the visual perception model missed the second instance corresponding to the target labeled bounding box, indicating that the visual perception model poorly recognized the second instance. Consequently, the second instance is determined as a sample instance for the visual perception model. This allows efficient mining of sample instances from the vast amount of existing data that are similar to the first instance of the target to be mined and poorly recognized by the current visual perception model, for iterative upgrades of the visual perception model.

[0037] Next, steps S101 to S107 will be described.

[0038] In step S101, based on the query image, Figure 2As shown in , at least one first image is obtained from an image collection; wherein the query image includes a first instance of the target to be mined, and each first image includes at least one second instance matching the first instance. Figure 2 It is a schematic diagram of the data mining process in the embodiment of the present application.

[0039] Specifically, in one embodiment, the third feature information of each original image in the image set can be obtained, and the second feature information can be generated based on the query image. Then, based on the similarity between the second feature information and each third feature information, the third feature information that matches the second feature information is determined in each third feature information, and the original image corresponding to the third feature information is obtained from the image set as the first image. The third feature information of each original image can be obtained by extracting features from the original image using an existing image feature extraction algorithm. In addition, in order to improve image retrieval efficiency, the third feature information of each original image can be pre-generated and stored. In addition, in order to improve matching accuracy, the second feature information and the third feature information are generated based on the same image feature extraction algorithm.

[0040] It should be noted that in this application, the similarity between feature information can be calculated using an existing similarity calculation algorithm, such as cosine similarity or Euclidean distance.

[0041] When determining the third feature information that matches the second feature information, as one implementation method, the third feature information among each set of third feature information whose similarity with the second feature information is greater than a similarity threshold can be determined as the third feature information that matches the second feature information. As another implementation method, the first first number of third feature information among each set of third feature information that have the highest similarity with the second feature information can be determined as the third feature information that matches the second feature information. The first number can be set based on actual needs and is not specifically limited here.

[0042] In another embodiment, in order to achieve rapid retrieval of images, the following can be done: Figure 3 As shown, the target database is pre-built based on the image collection. Figure 3This is another schematic diagram of the data mining process in an embodiment of the present application. To improve the retrieval hit rate and reduce the interference of irrelevant information from the original image in the image collection on semantic encoding, a currently effective object detection model (such as the OWLv2 model) can be used to perform object detection on the original image, extracting image regions that may contain instances of objects of interest. In the field of autonomous driving, objects of interest can include any obstacle targets, such as, but not limited to, tricycles. OWLv2 is an open vocabulary object detection model based on a self-training mechanism. It is of great significance in the field of computer vision and can detect objects in images based on textual cues, providing an efficient solution for zero-shot object detection. Furthermore, to ensure object recall, the acceptable detection threshold for detection boxes can be set to a relatively low level, and detection boxes with overlapping areas can be merged. Subsequently, semantic encoding can be performed on the image regions extracted from the original image, such as using the BLIP2 model. The first feature information (such as a feature vector) obtained after encoding is merged with other relevant metadata information and injected into the target database as image information. This metadata information includes the image identifier of the original image. Furthermore, this metadata information may also include any of the following: sensor information of the original image, timestamp, image path, etc. The BLIP2 model is a highly effective lightweight model suitable for application to large-scale datasets. It supports multimodal input and alignment of input text and input images in the latent space, making it easier to supplement query images with text descriptions during retrieval, improving retrieval results.

[0043] As described above, the target database stores image information. Any piece of image information includes the image identifier of an original image in the image collection and first feature information of an image region within the original image that contains a target instance. The target instance is an instance of a target of interest, which may be the same target as or different from the target to be mined. Furthermore, the first feature information within multiple pieces of image information in the target database is generated based on an image region containing a second instance of the target to be mined.

[0044] When performing image retrieval, semantic retrieval can be performed. Specifically, the following steps can be performed: generating second feature information based on the query image; determining target feature information that matches the second feature information in each first feature information based on the similarity between the first feature information and the second feature information in the target database; and obtaining the original image indicated by the image identifier corresponding to the target feature information from the image collection as the first image. The first feature information and the second feature information are generated based on the same image feature extraction algorithm, such as the same BLIP2 model.

[0045] Furthermore, to improve retrieval performance, secondary feature information can be generated based on the query image and the textual information describing the query image. As previously described, the BLIP2 model supports multimodal input and supports alignment of input text and input image in the latent space. If the primary feature information is generated using the BLIP2 model, the BLIP2 model can be used to semantically encode the query image and the textual information describing the query image to generate the secondary feature information.

[0046] When determining the target feature information that matches the second feature information, as one implementation method, the first feature information among each set of first feature information whose similarity with the second feature information is greater than a similarity threshold can be determined as the target feature information that matches the second feature information. As another implementation method, the first second number of first feature information among each set of first feature information whose similarity with the second feature information is highest can be determined as the target feature information that matches the second feature information. The second number can be set based on actual needs and is not specifically limited here.

[0047] After each first image is retrieved from the image collection, further sample instance mining may be performed based on the first image to obtain high-value data.

[0048] In step S103, the labeled boxes of each second instance in each first image are obtained. In practice, the labeling results of each original image in the image collection can be pre-stored, and the labeling results can be generated based on the detection results of the original image by the target detection model as described above. For example, the detection result includes at least one detection box and the target category corresponding to each detection box, the labeling result includes at least one labeling box and the target category corresponding to each labeling box, and any labeling box in the labeling result is the same as a detection box in the detection result, or is generated by merging detection boxes with overlapping areas in the detection result. It should be noted that when the labeling box is generated by merging detection boxes with overlapping areas, in one example, when the detection boxes with overlapping areas correspond to the same target category, the labeling box also corresponds to the same target category; in another example, the detection result also includes the confidence corresponding to the target category. When the target categories corresponding to the detection boxes with overlapping areas are not exactly the same, the labeling box corresponds to the target category with the highest confidence among the target categories.

[0049] In the case of pre-storing the annotation results of each original image in the image set, when obtaining the annotation box of each second instance in each first image, it can be as follows Figure 2 As shown in , the annotation result of the first image is obtained, and then the annotation box of the target to be mined corresponding to the target category is obtained from the annotation result.

[0050] In step S105, Figure 2 As shown in , a visual perception model is used to perform target detection on the first image to generate a detection result, which includes at least one detection box and a target category corresponding to each detection box.

[0051] After generating the detection results of the first image, the detection box of the target to be mined corresponding to the target category in the detection results can be used as the target detection box, and then the matching status between the annotation boxes of each second instance in the first image and each target detection box is determined, so as to mine high-value sample instances based on the matching status.

[0052] Figure 4 Schematic diagram of the marking frame and the detection frame in the embodiment of the present application. Figure 4 In the figure, the solid box is the annotation box, and the hollow box is the detection box. Figure 4 Different colors represent different categories of targets (the solid red target in the middle represents the vehicle, green represents vehicles, blue represents pedestrians, and pink represents tricycles). The difference between the annotation box and the detection box can be seen intuitively, such as Figure 4 The target with ID 10 is a high-value target.

[0053] For any target annotation frame in the annotation frames of each second instance of the first image, the matching situation between the target annotation frame and each target detection frame can be as follows: Figure 2 As shown in , the DIFF (Difference) algorithm is used to determine the difference, which allows for the selection of sample instances based on the DIFF algorithm. The core idea of ​​the DIFF algorithm is to analyze the differences between the labeling results and the detection results, using the Intersection over Union (IOU) comparison dimension.

[0054] Specifically, the steps of determining the matching between the target annotation frame and each target detection frame include: For a first image region corresponding to the target annotation frame in the first image, and a second image region corresponding to each target detection frame in the first image, determining an intersection-over-union ratio of the first image region and the second image region; When the maximum value of each intersection-over-union ratio is equal to a preset value, it is determined that each target detection frame does not match the target annotation frame; When the maximum value is greater than the preset value, it is determined that the target detection frame corresponding to the maximum value matches the target annotation frame.

[0055] It should be noted that by determining the matching status of the target annotation box and each target detection box based on the intersection-over-union ratio of the first image area and the second image area, the spatial correlation of the box positions can be quantified, ensuring that the matching logic is objective and quantifiable, accurately screening sample instances with weak model detection performance, improving mining efficiency, adapting to complex scenes, and being compatible with the matching needs of irregular boxes.

[0056] The IoU ratio between the first image region and the second image region may be determined in different implementations. Specifically, in one embodiment, the IoU ratio between the first image region and the second image region may be determined based on the area of ​​the overlapping portion of the first image region and the second image region, and the area jointly occupied by the first image region and the second image region.

[0057] Figure 5 and Figure 6 Schematic diagram of the overlapping first image area and second image area in an embodiment of the present application. Figure 5 、 Figure 6 As shown, A represents the first image area, B represents the second image area, Taking the overlapping part of A and B as an example, the intersection of A and B can be determined by the following formula (1), and the polygon area can be calculated by the following formula (2), where Represents the polygon i The coordinates of the vertices: (1) (2) in, Indicates area, the unit can be square meters. represents the area of ​​the first image region A, represents the area of ​​the second image region B, express It should be understood that represents the area occupied by the first image region A and the second image region B. Formula (2) is the shoelace formula, also known as the Gaussian area formula, which is a simple and effective method for calculating the area of ​​a polygon. In Formula (2), n represents the number of vertices of the polygon, in units of vertices (or counts). i is an integer greater than or equal to 1 and less than or equal to n. If any item in is a polygon, the coordinates of each vertex in the item can be substituted into formula (2) to calculate the area of ​​the item.

[0058] Furthermore, to ensure computational efficiency, when the overlapping portion of the first image region and the second image region is a rectangular region, the intersection-over-union ratio of the first image region and the second image region is determined based on the area of ​​the overlapping portion of the first image region and the second image region and the area jointly occupied by the first image region and the second image region. Specifically, Figure 5 As shown, For a rectangular area, the above formula (1) can be used to determine the intersection-union ratio iou of A and B.

[0059] In another embodiment, the intersection-over-union ratio of the first image area and the second image area can be determined based on the first number of pixels in the overlapping part of the first image area and the second image area, the second number of pixels in the other part of the first image area except the overlapping part, and the third number of pixels in the other part of the second image area except the overlapping part.

[0060] A represents the first image area, and B represents the second image area. Taking the overlapping part of A and B as an example, the intersection-over-union ratio (iou) of A and B can be determined using the following formula (3): (3) in, Indicates the number of pixels, in units of pieces (or counts). Indicates the first image area A except the overlapping part The number of second pixels in other parts, The second image area B except the overlapping part The number of third pixels in the other parts, Indicates overlapping parts The first pixel number.

[0061] Furthermore, in order to ensure computational efficiency and avoid tedious computational processes, when the first image area and the second image area are not arranged horizontally or vertically, or when the overlapping portion is an irregular polygon, the intersection-over-union ratio of the first image area and the second image area is determined based on the first pixel number, the second pixel number, and the third pixel number. Figure 6 As shown, It is an irregular polygon, and it is very cumbersome to calculate the iou using geometric position relationships. Figure 7 Schematically shows Figure 6 The pixels of A and B. Figure 7 Schematic diagram of the overlapping first image area and second image area in an embodiment of the present application. Figure 7 In the figure, the small triangles represent the first image area A except the overlapping part. The diamond represents the pixel points of the second image area B except the overlapping part. The five-pointed star represents the overlapping part. By using the above formula (2), we can quickly calculate Figure 6 、 Figure 7 The intersection-union ratio of A and B shown is iou.

[0062] After determining each IoU, if the maximum value among the IoU is equal to a preset value, it is determined that the target detection frame does not match the target annotation frame. The preset value is 0. If the maximum value is greater than the preset value, it is determined that the target detection frame corresponding to the maximum value matches the target annotation frame.

[0063] Continue to read Figure 1 In step S107, for any target annotation frame among the annotation frames corresponding to the first image, if the target detection frames corresponding to the first image do not match the target annotation frame, it can be determined that the visual perception model missed the second instance corresponding to the target annotation frame. This second instance is an instance poorly recognized by the visual perception model, and thus the second instance is determined as a sample instance of the visual perception model. It should be understood that this second instance serves as a positive sample instance of the visual perception model.

[0064] In one embodiment, for any target detection frame, when each annotated frame does not match the target detection frame, the third instance corresponding to the target detection frame is determined as a sample instance of the visual perception model. It should be noted that when each annotated frame does not match the target detection frame, it can be determined that the visual perception model has misdetected the third instance corresponding to the target detection frame, and the third instance was misdetected as an instance of the target to be mined. Therefore, the third instance can be determined as a negative sample instance of the visual perception model to improve the accuracy of the visual perception model in identifying such negative sample instances.

[0065] In one embodiment, after determining that the target detection frame corresponding to the above-mentioned maximum value matches the target annotation frame, the second instance corresponding to the target annotation frame can also be determined as a sample instance of the visual perception model when the above-mentioned maximum value is less than the first threshold value, or the difference between the first heading angle and the second heading angle is greater than the second threshold value; wherein the first heading angle is the heading angle of the second instance corresponding to the target annotation frame, and the second heading angle is the heading angle of the second instance corresponding to the target detection frame corresponding to the above-mentioned maximum value. The first threshold value can be, for example, 0.5, and the second threshold value can be, for example, 45°. It should be understood that the first threshold value and the second threshold value can be set according to actual needs and are not specifically limited here.

[0066] It should be noted that when the above-mentioned maximum value is less than the first threshold, or the difference between the first heading angle and the second heading angle is greater than the second threshold, it means that the target detection box corresponding to the above-mentioned maximum value does not match the target annotation box well, and there is a position offset. By determining the second instance corresponding to the target detection box corresponding to the above-mentioned maximum value as the sample instance of the visual perception model, it can be used for iterative upgrading of the visual perception model to improve the accuracy of the visual perception model in identifying such sample instances.

[0067] In one embodiment, mining result information may also be generated and stored. The mining result information includes the image identifier of the first image where the sample instance is located as described above, and the instance identifier of the sample instance.

[0068] Figure 8 This is another flow chart of the data mining method in the embodiment of the present application. Figure 8 As shown, the mining method includes the following steps: S801: Based on a query image, obtain at least one first image from an image collection; wherein the query image includes a first instance of a target to be mined, and each first image includes at least one second instance matching the first instance; S803: Obtaining a labeling box of each second instance in each first image; S805: Performing object detection on the first image using a visual perception model to generate a detection result, where the detection result includes at least one detection box and an object category corresponding to each detection box; S807: For a first image region corresponding to any target annotation frame in each annotation frame in the first image, and a second image region corresponding to each target detection frame in the first image, determining an intersection-over-union ratio between the first image region and the second image region; the target detection frame is included in the detection result and the corresponding target category indicates a target to be mined; S809: When the maximum value of each intersection-over-union ratio is equal to a preset value, determining that each target detection frame does not match the target annotation frame; S811: When each target detection frame does not match the target annotation frame, determine the second instance corresponding to the target annotation frame as a sample instance of the visual perception model; S813: When the maximum value is greater than the preset value, determining that the target detection frame corresponding to the maximum value matches the target annotation frame; S815: After determining that the target detection frame corresponding to the maximum value matches the target annotation frame, if the maximum value is less than a first threshold, or if the difference between the first heading angle and the second heading angle is greater than a second threshold, determining the second instance corresponding to the target annotation frame as a sample instance of the visual perception model; wherein the first heading angle is the heading angle of the second instance corresponding to the target annotation frame, and the second heading angle is the heading angle of the second instance corresponding to the target detection frame corresponding to the maximum value; S817: For any target detection frame, when each annotation frame does not match the target detection frame, determine the third instance corresponding to the target detection frame as a sample instance of the visual perception model.

[0069] exist Figure 8In the solution provided by the corresponding embodiment, for any target annotation frame in each annotation frame corresponding to the first image, when each target detection frame (the detection frame of the target to be mined indicated by the corresponding target category in the detection result) does not match the target annotation frame, it can be known that the visual perception model has missed the second instance corresponding to the target annotation frame, and thus the second instance is determined as a sample instance of the visual perception model. In addition, for any target detection frame, when each annotation frame does not match the target detection frame, it can be known that the visual perception model has a misdetection problem with the third instance corresponding to the target detection frame, and the third instance is misdetected as an instance of the target to be mined, so that the third instance can be determined as a sample instance of the visual perception model. In addition, after determining that the target detection frame corresponding to the above-mentioned maximum value matches the target annotation frame, if the above-mentioned maximum value is less than the first threshold, or the difference between the first heading angle and the second heading angle is greater than the second threshold, it means that the target detection frame corresponding to the above-mentioned maximum value does not match the target annotation frame well, and there is a position offset, so the third instance corresponding to the target detection frame corresponding to the above-mentioned maximum value can be determined as a sample instance of the visual perception model. In this way, sample instances that are similar to the first instance of the target to be mined and poorly recognized by the current visual perception model can be efficiently mined from massive stock data for iterative upgrading of the visual perception model.

[0070] Figure 9 Schematic diagram of the structure of the data mining device in the embodiment of the present application. The mining device includes: A first acquisition unit 901 is configured to acquire at least one first image from an image collection based on a query image; wherein the query image includes a first instance of a target to be mined, and each first image includes at least one second instance matching the first instance; A second acquiring unit 902 is configured to acquire a labeling box of each second instance in each first image; a detection unit 903 configured to perform object detection on the first image using a visual perception model and generate a detection result, where the detection result includes at least one detection box and an object category corresponding to each detection box; The determination unit 904 is configured to determine, for any target annotation box in each annotation box, when each target detection box does not match the target annotation box, the second instance corresponding to the target annotation box as a sample instance of the visual perception model, and the target detection box is included in the detection result and the corresponding target category indicates the target to be mined.

[0071] In one embodiment, the determining unit 904 is further configured to: For any target detection box, when each annotation box does not match the target detection box, the third instance corresponding to the target detection box is determined as a sample instance of the visual perception model.

[0072] In one embodiment, the first acquiring unit 901 is configured to acquire at least one first image from the image set based on the query image, including: generating second feature information based on the query image; Determining, based on similarities between each piece of first feature information and the second feature information in a target database, target feature information that matches the second feature information in each piece of first feature information, wherein the target database stores a plurality of pieces of image information, each piece of image information including an image identifier of an original image in the image set and first feature information of an image region in the original image that contains the second instance; An original image indicated by an image identifier corresponding to the target feature information is obtained from the image set as the first image.

[0073] In one embodiment, the first acquiring unit 901 is configured to generate second feature information based on the query image, including: The first acquiring unit 901 is configured to generate second feature information based on a query image and text information used to describe the query image.

[0074] In one embodiment, the determining unit 904 is further configured to: For a first image region corresponding to the target annotation frame in the first image, and a second image region corresponding to each target detection frame in the first image, determining an intersection-over-union ratio of the first image region and the second image region; When the maximum value of each intersection-over-union ratio is equal to a preset value, it is determined that each target detection frame does not match the target annotation frame; When the maximum value is greater than the preset value, it is determined that the target detection frame corresponding to the maximum value matches the target annotation frame.

[0075] In one embodiment, the determining unit 904 is configured to determine an intersection-over-union ratio between the first image region and the second image region, including: The determining unit 904 is configured to determine an IoU based on an area of ​​an overlapping portion of the first image region and the second image region and an area jointly occupied by the first image region and the second image region.

[0076] In one embodiment, the determining unit 904 is configured to determine an intersection-over-union ratio between the first image region and the second image region, including: The determination unit 904 is configured to determine the intersection-over-union ratio based on the first number of pixels in the overlapping part of the first image area and the second image area, the second number of pixels in the other parts of the first image area except the overlapping part, and the third number of pixels in the other parts of the second image area except the overlapping part.

[0077] In one embodiment, the determining unit 904 is configured to determine the intersection-over-union ratio, including: The determining unit 904 is configured to determine an intersection-over-union ratio based on the first number of pixels, the second number of pixels, and the third number of pixels when the overlapping portion is an irregular polygon.

[0078] In one embodiment, the determining unit 904 is further configured to: After determining that the target detection frame corresponding to the above-mentioned maximum value matches the target annotation frame, when the above-mentioned maximum value is less than the first threshold, or the difference between the first heading angle and the second heading angle is greater than the second threshold, the second instance corresponding to the target annotation frame is determined as the sample instance of the visual perception model; wherein the first heading angle is the heading angle of the second instance corresponding to the target annotation frame, and the second heading angle is the heading angle of the second instance corresponding to the target detection frame corresponding to the above-mentioned maximum value.

[0079] It should be noted that other aspects and implementation details of the data mining device provided in the embodiment of the present application are the same as or similar to the data mining method described above and will not be repeated here.

[0080] As described above, the solution provided by the embodiments of this application utilizes the OWLv2 and BLIP2 models to construct a target database. By calculating cosine similarity between the second feature information of the query image and the first feature information in the target database, data containing tricycles can be quickly located within the massive amount of existing data. By constructing a model-based DIFF automated mining process, reading the ground truth and utilizing the model's inference results, valuable tricycle data can be quickly and conveniently mined. The quality and standard of the mined data can also be adjusted by adjusting the DIFF threshold.

[0081] The embodiment of the present application further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following is achieved: Figure 1 or Figure 8 Describe the data mining method.

[0082] The embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the following is achieved: Figure 1 or Figure 8 Describe the data mining method.

[0083] The present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the following Figure 1 or Figure 8 Describe the data mining method.

[0084] The above description is only a partial implementation of the embodiments of the present application and does not constitute any form of limitation to the application. The protection scope of the embodiments of the present application is not limited thereto. Any simple modifications, equivalent changes and modifications that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed in the embodiments of the present application should be covered within the protection scope of the embodiments of the present application.

Claims

1. A data mining method, characterized in that: include: Based on a query image, obtaining at least one first image from an image collection; wherein the query image includes a first instance of a target to be mined, and each of the first images includes at least one second instance matching the first instance; Obtaining a labeling box of each second instance in each first image; Performing object detection on the first image using a visual perception model to generate a detection result, the detection result including at least one detection box and an object category corresponding to each detection box; For any target annotation box in each of the annotation boxes, when each target detection box does not match the target annotation box, the second instance corresponding to the target annotation box is determined as a sample instance of the visual perception model, and the target detection box is included in the detection result and the corresponding target category indicates the target to be mined.

2. The data mining method according to claim 1, characterized in that: Also includes: For any one of the target detection frames, when the annotation frames do not match the target detection frame, the third instance corresponding to the target detection frame is determined as the sample instance.

3. The data mining method according to claim 1, wherein: The acquiring at least one first image from the image set based on the query image includes: generating second feature information based on the query image; determining, based on similarities between each piece of first feature information and the second feature information in a target database, target feature information that matches the second feature information in each piece of first feature information, wherein the target database stores a plurality of pieces of image information, each piece of image information including an image identifier of an original image in the image set and the first feature information of an image region in the original image that contains the second instance; The original image indicated by the image identifier corresponding to the target feature information is obtained from the image set as the first image.

4. The data mining method according to claim 3, characterized in that: Generating second feature information based on the query image includes: The second feature information is generated based on the query image and text information used to describe the query image.

5. The data mining method according to any one of claims 1 to 4, characterized in that: The step of determining the matching status of the target annotation frame and each target detection frame includes: For a first image region corresponding to the target annotation frame in the first image, and a second image region corresponding to each target detection frame in the first image, determining an intersection-over-union ratio between the first image region and the second image region; When the maximum value of each intersection-over-union ratio is equal to a preset value, determining that each target detection frame does not match the target annotation frame; When the maximum value is greater than the preset value, it is determined that the target detection frame corresponding to the maximum value matches the target annotation frame.

6. The data mining method according to claim 5, characterized in that: Determining an intersection-over-union ratio of the first image area and the second image area includes: The intersection-over-union ratio is determined based on an area of ​​an overlapping portion of the first image region and the second image region, and an area jointly occupied by the first image region and the second image region.

7. The data mining method according to claim 5, characterized in that: Determining an intersection-over-union ratio of the first image area and the second image area includes: The intersection-over-union ratio is determined based on the first number of pixels in the overlapping part of the first image area and the second image area, the second number of pixels in the other parts of the first image area except the overlapping part, and the third number of pixels in the other parts of the second image area except the overlapping part.

8. The data mining method according to claim 7, characterized in that: Determining the intersection-over-union ratio includes: In a case where the overlapping portion is an irregular polygon, the intersection-over-union ratio is determined based on the first number of pixels, the second number of pixels, and the third number of pixels.

9. The data mining method according to claim 5, characterized in that: After determining that the target detection frame corresponding to the maximum value matches the target annotation frame, the method further includes: When the maximum value is less than a first threshold, or the difference between the first heading angle and the second heading angle is greater than a second threshold, the second instance corresponding to the target marking box is determined as the sample instance; wherein the first heading angle is the heading angle of the second instance corresponding to the target marking box, and the second heading angle is the heading angle of the second instance corresponding to the target detection box corresponding to the maximum value.

10. A data mining device, characterized in that: include: A first acquisition unit is configured to acquire at least one first image from an image collection based on a query image; wherein the query image includes a first instance of a target to be mined, and each of the first images includes at least one second instance matching the first instance; a second acquiring unit, configured to acquire a labeling box of each second instance in each first image; a detection unit configured to perform object detection on the first image using a visual perception model and generate a detection result, the detection result including at least one detection box and an object category corresponding to each detection box; A determination unit is configured to determine, for any one target annotation box in each of the annotation boxes, the second instance corresponding to the target annotation box as a sample instance of the visual perception model when each target detection box does not match the target annotation box, the target detection box is included in the detection result and the corresponding target category indicates the target to be mined.

Images