Data architecture optimization method for target detection algorithm

Abstract

The invention discloses a target detection algorithm-oriented data architecture optimization method in the field of visual detection, and the method comprises a video data management module, a data automatic labeling module, a special data sample reservation module and an image data de-duplication module. Business data of different sources are subjected to renaming management in a unified mode, videos are intercepted by adopting different video interception frequencies and stored as picture data, naming of the picture data is based on the names of the source videos, and related frame numbers are added to serve as supplements. According to the method, the redundancy of the service data set is reduced by utilizing an image similarity calculation method, the dispersion degree of the service data set is improved, the time cost of subsequent manual annotation and the resource cost used by transfer learning are reduced, and meanwhile, the detection performance of an AI visual detection technology in an on-site scene is ensured.

Description

Technical Field

[0001] The present invention relates to the field of visual detection, and in particular to a data architecture optimization method for a target detection algorithm. Background Art

[0002] With the development of artificial intelligence technology, target detection technology based on artificial intelligence has become one of the primary technologies in different fields. At present, the application process of intelligent detection algorithms at home and abroad first collects image data from a certain camera as a business data set according to business needs; then uses a large number of manual methods to annotate the business data set; finally, uses the method of transfer learning to solve the generalization of business scenarios of AI visual detection technology. The above three steps realize the landing function of AI visual detection technology in real-world scenarios, but the number of business data sets determines the time cost of subsequent manual annotation, the resource cost used for transfer learning, and the detection performance of AI visual detection technology in real-world scenarios.

[0003] Patent CN117765534A describes the "Method and Device for Automatic Labeling of Traffic Images Based on Difficult Sample Mining", which uses a difficult sample labeling model to perform frame detection on video stream data, and manually judges images where no instance is detected. When the image contains a difficult sample category, it is placed in a difficult image sample set to update the difficult sample labeling model. This patent introduces too many artificial factors in the process of determining difficult samples. Patent CN CN117115557A describes the "Method, System, Electronic Device and Storage Medium for Automatic Labeling of Images", which inputs the feature vectors of the image to be detected and the target image, and sends them to the decoder to calculate the query embedding vector, and finally realizes the labeling of targets that exist and do not exist in the training set. However, the same instance in different locations in real business scenarios varies greatly, and the detection difficulty is greater than in other scenarios.

[0004] Patent CN117349027A describes a system and method for building a multimodal large model with reduced computing power requirements. The image data redundancy cleaning module mentioned in the patent is closest to the technical solution in this article. In the process of image redundancy cleaning, this method only uses pixel grayscale values ​​to calculate similarity, but pixel grayscale values ​​are difficult to represent the fine-grained feature differences between two images, so it is difficult to truly discard images whose similarity is near the threshold.

[0005] Existing research pays less attention to the number of business data sets and data architecture issues, that is, how to better organize and manage data to improve the effect and performance of target detection models. Comprehensive analysis of existing technologies still shows the following defects:

[0006] 1. The existing method of creating business data sets for landing scenarios is to simply and crudely collect data in different time periods and weather conditions to ensure the generalization of visual detection algorithms;

[0007] 2. Common image automatic annotation methods use a fine-tuned closed set detection model to automatically annotate business data sets, which often requires a large amount of manual review of the annotation results and is unable to annotate categories that have not been fine-tuned;

[0008] 3. Existing methods for deduplicating images based on image similarity calculation often compare the features of the entire image when calculating image similarity. Therefore, small target samples in the image cannot cause differences in the feature dimension to affect the similarity score calculation, resulting in the failure to retain images containing small targets.

[0009] Therefore, those skilled in the art provide a data architecture optimization method for target detection algorithms to solve the problems raised in the above background technology. Summary of the invention

[0010] The purpose of the present invention is to provide a data architecture optimization method for target detection algorithms to solve the problems raised in the above-mentioned background technology.

[0011] To achieve the above object, the present invention provides the following technical solutions:

[0012] A data architecture optimization method for target detection algorithms, including a video data management module, a data automatic labeling module, a special data sample retention module and an image data deduplication module;

[0013] The video data management module uniformly renames and manages business data from different sources by defining relevant naming rules, and uses different video capture frequencies to capture videos and save them as image data. The naming of the image data is based on the name of the source video and is supplemented by relevant frame numbers.

[0014] The data automatic labeling module can preset different target detection categories according to prior experience and the needs of business scenarios. First, according to the characteristics of the closed-set detection big model, the closed-set detection big model is used to automatically label the categories labeled in the public data set. Secondly, in view of the lack of detection ability of the closed-set detection big model for certain special categories (such as domain-specific targets or categories missing from the public data set), the text-image open-set detection big model is used to supplement the labeling to ensure both the number of detection instances and the accuracy, thereby reducing the time cost of subsequent manual labeling.

[0015] The special data sample retention module can mark certain categories as special categories based on prior knowledge (some categories are difficult to detect and some categories have a small number of samples in actual business scenarios), and mark categories with lower detection difficulty as ordinary categories. An empty list is created for each special category, and an empty list is created for ordinary categories. The detection results are traversed, and when the detection results contain different special categories, the image information is added to the corresponding list; when the detection results do not contain special categories, the image information is added to the ordinary category list. After the traversal is completed, 2 to 3 non-repetitive video frame fragments containing different special categories are taken as supplements, and the image set in the ordinary category list is sent to the deduplication step.

[0016] In order to reduce the redundancy of the business data set, the image data deduplication module sends the pictures that do not contain special categories to the image data deduplication module. The image data deduplication module first uses the closed set detection large model to extract the one-dimensional feature vector x∈(f) of all images in this part, set X, where f is the output category dimension information of the large model, and randomly extracts a one-dimensional feature vector as the reference vector subset A. Then, the matching elements B in the remaining feature vector set are respectively calculated with the reference vector subset A to obtain the similarity score S. If S is greater than or equal to the preset threshold T, the matching data B is discarded, otherwise the matching element B is added to the reference vector subset A, and the subset A is updated. The comparison is repeated until the set X is empty. After this step is completed, the elements in the reference vector subset A are not similar to each other, which can greatly reduce the redundancy of the business data set.

[0017] As a further solution of the present invention: Step S1: rename and manage the collected traffic monitoring video data using a video data management module; classify the data according to the scene, camera ID, project name and source time involved in the video. For example, the initial traffic monitoring video name of d11_20240501080000.mp4 will be renamed to 1_1_d11_XK20240501080000.mp4 using the video data management module, indicating that the video is numbered 1, comes from camera No. d11 on the main line of the road, the project comes from information technology, and the collection time is 8:00 on May 1, 2024.

[0018] Step S2: Use the built-in picture making tool in the video data management module to set the video processing rule to take 1 frame per second to intercept the video frame of the video to be processed, and obtain the preliminary screening business data set. The naming rule of each picture in the preliminary screening business data set will add a frame number field to the name of the parent video as a distinction;

[0019] Step S3: Use the data automatic annotation module to automatically annotate the initial screening business data set, use the closed set detection model (detr model) and the text-image open set detection model (grounding-dino model), and summarize the detection result list;

[0020] Step S4: Use the special data sample retention module to retain pictures containing special categories. First, the result list obtained in step S3 is allocated according to the detection results, and then 2 to 3 pictures of the list containing special detection results are randomly retained;

[0021] Step S5: Use the image data deduplication module to reduce the redundancy of the remaining images. The closed set detection large model (dinov2 model) extracts the corresponding feature vectors and uses the cosine distance algorithm to calculate the similarity values ​​between different images. When the similarity value is greater than the preset threshold, it is discarded;

[0022] Step S6: Summarize the remaining image files and label files and send them to the data annotation platform for review and modification to ensure that there are no obvious errors in the data set.

[0023] Furthermore, the step S3 specifically includes the following steps:

[0024] Step S31: According to the business scenario needs, 11 detection categories are preset, including non-motor vehicles, pedestrians, reflective clothing, triangular warning signs, small trucks, open flames, smoke, traffic cones, cars, large trucks and buses. According to the particularity of the traffic scene, reflective clothing, triangular warning signs and traffic cones are used as domain-specific targets, open flames and smoke are used as missing categories in the public dataset, and the remaining six categories are used as common categories;

[0025] Step S32: prepare a closed-set detection large model (detr model) and a text-image open-set detection large model (grounding-dino model). According to the business scenario analysis, since the closed-set detection large model is usually trained on general data sets such as coco, it lacks data on traffic facilities such as reflective clothing, triangular warning signs, traffic cones, open flames and smoke. Therefore, corresponding prompt words are set (such as setting fire as the prompt word for open flames) and the grounding-dino model is used to detect and label them.

[0026] Step S33: When combining the detection results of different models, the text-image open set detection model has direct decision-making power over the detection results of reflective clothing, triangular warning signs, traffic cones, open flames and smoke. For categories contained in public data sets such as non-motor vehicles, pedestrians, and cars, the closed set detection model has direct decision-making power. Finally, the detection results of the two models are unioned as the final output.

[0027] Furthermore, the step S4 specifically includes the following steps:

[0028] Step S41: six types of targets, namely, reflective clothing, non-motor vehicles, open flames, smoke, triangle warning signs and pedestrians, are specified as special categories, and images containing these six categories need to be additionally retained.

[0029] Step S42: Create an empty list for each type of target, and traverse the list output by step S3. When the image contains these categories, add the image information to the corresponding list to obtain the category lists L1, L2, ..., L6.

[0030] Step S43: Randomly retain 2 to 3 pictures for each category list and discard the remaining pictures.

[0031] Furthermore, the step S5 specifically includes the following steps:

[0032] Step S51: Input image i with dimensions (1, 3, 700, 700), where 1 is the batch size, 3 is the number of channels, and 700 is the length and width of the image. After the encoder-decoder structure of the dinov2 model, the feature vector x with a feature dimension of (1, 768) is output. Extract all remaining images to generate a one-dimensional vector set X.

[0033] Step S52: randomly extract a vector x as the first element of the reference vector subset A, and extract the remaining vectors x in order as matching vectors B. Using the cosine distance formula:

[0034]

[0035] Where x is the reference vector subset A, and y is the matching vector B. The similarity score S is calculated, and when S ≥ T, the matching vector B is discarded, but when S < T, B is added to the reference vector subset A.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] 1. The present invention uses the method of image similarity calculation to reduce the redundancy of business data sets, improve the discreteness of business data sets, reduce the time cost of subsequent manual annotation and the resource cost used for transfer learning, and at the same time ensure the detection performance of AI visual inspection technology in real-world scenarios.

[0038] 2. The present invention combines the closed set detection model and the text-image open set detection model to automatically annotate images, and designs different weight selection coefficients for the detection results of the two different large models according to different categories, which not only ensures the accuracy of automatic annotation but also expands the types of annotation categories, thereby reducing the time cost of subsequent manual modification of annotations;

[0039] 3. The present invention designs a special sample retention rule to compensate for the image data that is erroneously discarded due to image similarity calculation and to increase the number of instances of small objects in the business data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flow chart of the present invention;

[0041] Figure 2 It is a structural schematic diagram of the data automatic labeling module in the present invention;

[0042] Figure 3 The work of the module is reserved for the special data samples in the present invention;

[0043] Figure 4 This is a workflow diagram of the image data deduplication module in the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] See also Figures 1 to 4 ,In an embodiment of the present invention, a data architecture optimization method for target detection algorithm includes a video data management module, a data automatic annotation module, a special data sample retention module and an image data deduplication module;

[0046] The video data management module uniformly renames and manages business data from different sources by defining relevant naming rules, and uses different video capture frequencies to capture videos and save them as image data. The naming of the image data is based on the name of the source video and is supplemented by relevant frame numbers.

[0047] The data automatic labeling module can preset different target detection categories according to prior experience and the needs of business scenarios. First, according to the characteristics of the closed-set detection big model, the closed-set detection big model is used to automatically label the categories labeled in the public data set. Secondly, in view of the lack of detection ability of the closed-set detection big model for certain special categories (such as domain-specific targets or categories missing from the public data set), the text-image open-set detection big model is used to supplement the labeling to ensure both the number of detection instances and the accuracy, thereby reducing the time cost of subsequent manual labeling.

[0048] The special data sample retention module can mark certain categories as special categories based on prior knowledge (some categories are difficult to detect and some categories have a small number of samples in actual business scenarios), and mark categories with lower detection difficulty as ordinary categories. An empty list is created for each special category, and an empty list is created for ordinary categories. The detection results are traversed, and when the detection results contain different special categories, the image information is added to the corresponding list; when the detection results do not contain special categories, the image information is added to the ordinary category list. After the traversal is completed, 2 to 3 non-repetitive video frame fragments containing different special categories are taken as supplements, and the image set in the ordinary category list is sent to the deduplication step.

[0049] In order to reduce the redundancy of the business data set, the image data deduplication module sends the pictures that do not contain special categories to the image data deduplication module. The image data deduplication module first uses the closed set detection large model to extract the one-dimensional feature vector x∈(f) of all images in this part, set X, where f is the output category dimension information of the large model, and randomly extracts a one-dimensional feature vector as the reference vector subset A. Then, the matching elements B in the remaining feature vector set are respectively calculated with the reference vector subset A to obtain the similarity score S. If S is greater than or equal to the preset threshold T, the matching data B is discarded, otherwise the matching element B is added to the reference vector subset A, and the subset A is updated. The comparison is repeated until the set X is empty. After this step is completed, the elements in the reference vector subset A are not similar to each other, which can greatly reduce the redundancy of the business data set.

[0050] In this embodiment, a data architecture optimization method is implemented using a traffic monitoring scenario as an example, and specifically includes the following steps:

[0051] Step S1: Rename and manage the collected traffic monitoring video data using the video data management module; classify the data according to the scene, camera ID, project name and source time involved in the video. For example, the initial traffic monitoring video name of d11_20240501080000.mp4 will be renamed to 1_1_d11_XK20240501080000.mp4 using the video data management module, indicating that the video number is 1, comes from camera No. d11 on the main line of the road, the project comes from information technology, and the collection time is 8:00 on May 1, 2024.

[0052] Step S2: Use the built-in picture making tool in the video data management module to set the video processing rule to take 1 frame per second to intercept the video frame of the video to be processed, and obtain the preliminary screening business data set. The naming rule of each picture in the preliminary screening business data set will add a frame number field to the name of the parent video as a distinction;

[0053] Step S3: Use the data automatic annotation module to automatically annotate the initial screening business data set, use the closed set detection model (detr model) and the text-image open set detection model (grounding-dino model), and summarize the detection result list;

[0054] Step S4: Use the special data sample retention module to retain pictures containing special categories. First, the result list obtained in step S3 is allocated according to the detection results, and then 2 to 3 pictures of the list containing special detection results are randomly retained;

[0055] Step S5: Use the image data deduplication module to reduce the redundancy of the remaining images. The closed set detection large model (dinov2 model) extracts the corresponding feature vectors and uses the cosine distance algorithm to calculate the similarity values ​​between different images. When the similarity value is greater than the preset threshold, it is discarded;

[0056] Step S6: Summarize the remaining image files and label files and send them to the data annotation platform for review and modification to ensure that there are no obvious errors in the data set.

[0057] Furthermore, the step S3 specifically includes the following steps:

[0058] Step S31: According to the business scenario needs, 11 detection categories are preset, including non-motor vehicles, pedestrians, reflective clothing, triangular warning signs, small trucks, open flames, smoke, traffic cones, cars, large trucks and buses. According to the particularity of the traffic scene, reflective clothing, triangular warning signs and traffic cones are used as domain-specific targets, open flames and smoke are used as missing categories in the public dataset, and the remaining six categories are used as common categories;

[0059] Step S32: prepare a closed-set detection large model (detr model) and a text-image open-set detection large model (grounding-dino model). According to the business scenario analysis, since the closed-set detection large model is usually trained on general data sets such as coco, it lacks data on traffic facilities such as reflective clothing, triangular warning signs, traffic cones, open flames and smoke. Therefore, corresponding prompt words are set (such as setting fire as the prompt word for open flames) and the grounding-dino model is used to detect and label them.

[0060] Step S33: When combining the detection results of different models, the text-image open set detection model has direct decision-making power over the detection results of reflective clothing, triangular warning signs, traffic cones, open flames and smoke. For categories contained in public data sets such as non-motor vehicles, pedestrians, and cars, the closed set detection model has direct decision-making power. Finally, the detection results of the two models are unioned as the final output.

[0061] Furthermore, the step S4 specifically includes the following steps:

[0062] Step S41: six types of targets, namely, reflective clothing, non-motor vehicles, open flames, smoke, triangle warning signs and pedestrians, are specified as special categories, and images containing these six categories need to be additionally retained.

[0063] Step S42: Create an empty list for each type of target, and traverse the list output by step S3. When the image contains these categories, add the image information to the corresponding list to obtain the category lists L1, L2, ..., L6.

[0064] Step S43: Randomly retain 2 to 3 pictures for each category list and discard the remaining pictures.

[0065] Furthermore, the step S5 specifically includes the following steps:

[0066] Step S51: Input image i with dimensions (1, 3, 700, 700), where 1 is the batch size, 3 is the number of channels, and 700 is the length and width of the image. After the encoder-decoder structure of the dinov2 model, the feature vector x with a feature dimension of (1, 768) is output. Extract all remaining images to generate a one-dimensional vector set X.

[0067] Step S52: randomly extract a vector x as the first element of the reference vector subset A, and extract the remaining vectors x in order as matching vectors B. Using the cosine distance formula:

[0068]

[0069] Where x is the reference vector subset A, and y is the matching vector B. The similarity score S is calculated, and when S ≥ T, the matching vector B is discarded, but when S < T, B is added to the reference vector subset A.

[0070] By adopting the above technical solutions and using different automatic image annotation models for different categories, the annotation speed of business data sets can be accelerated and the corresponding labor costs can be reduced. The image data deduplication module can be used to effectively reduce the redundancy in the business data set, and reduce the training loss caused by the visual detection model without damaging the detection accuracy when the later model is implemented, speed up the training time of the visual detection model, and shorten the implementation cycle of artificial intelligence technology. By using the special data sample retention module, a part of the image data containing special samples can be retained as much as possible while ensuring the reduction of redundancy in the business data set, thereby increasing the number of instances of special categories in the business data set and ensuring the detection performance of the visual detection model when it is implemented in the field. The implementation of the entire process better organizes and manages video and image data to improve the effect and performance of the target detection model based on artificial intelligence.

[0071] In this embodiment, the present invention uses an image similarity calculation method to reduce redundancy of business data sets, improve the discreteness of business data sets, reduce the time cost of subsequent manual annotation and the resource cost used for transfer learning, while ensuring the detection performance of AI visual inspection technology in real-world scenarios; combines a closed set detection large model and a text-image open set detection large model to automatically annotate images, and designs different weight selection coefficients for the detection results of two different large models for different categories, which not only ensures the accuracy of automatic annotation but also expands the types of annotation categories, thereby reducing the time cost of subsequent manual modification of annotations; and designs special sample retention rules to make up for image data that is erroneously discarded due to image similarity calculation, thereby increasing the number of instances of small targets in the business data set.

[0072] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A data architecture optimization method for target detection algorithm, comprising a video data management module, a data automatic labeling module, a special data sample retention module and an image data deduplication module, characterized in that: The video data management module uniformly renames and manages the business data from different sources by defining relevant naming rules, and uses different video capture frequencies to capture the video and save it as picture data. The naming of the picture data is based on the name of the source video and is supplemented by the relevant frame number; The data automatic annotation module can preset different target detection categories according to prior experience and business scenario needs. First, according to the characteristics of the closed set detection large model, the closed set detection large model is used to automatically annotate the categories annotated in the public data set. Secondly, in view of the lack of detection ability of the closed set detection large model for some special categories, the text-image open set detection large model is used to supplement the annotation. The special data sample retention module can mark certain categories as special categories based on prior knowledge, mark categories with lower detection difficulty as common categories, create an empty list for each special category, and create an empty list for the common category; Traverse the detection results, and when the detection results contain different special categories, add the image information to the corresponding list; When the detection result does not contain any special category, the image information is added to the general category list. After the traversal is completed, 2 to 3 non-repetitive video frame fragments containing different special categories are taken as supplements, and the image set in the general category list is sent to the deduplication step; In order to reduce the redundancy of the business data set, the image data deduplication module sends the pictures that do not contain special categories to the image data deduplication module. The image data deduplication module first uses the closed set detection large model to extract the one-dimensional feature vector x∈(f) of all images in this part, set X, where f is the output category dimension information of the large model, and randomly extracts a one-dimensional feature vector as the reference vector subset A. Then, the matching elements B in the remaining feature vector set are respectively calculated with the reference vector subset A to obtain a similarity score S. If S is greater than or equal to a preset threshold T, the matching data B is discarded, otherwise the matching element B is added to the reference vector subset A, and the subset A is updated. The comparison is cyclic until the set X is empty. After this step is completed, the elements in the reference vector subset A are not similar to each other.

2. The data architecture optimization method for target detection algorithm according to claim 1, characterized in that: The optimization method of the data architecture includes the following steps: Step S1: renaming and managing the collected video data using the video data management module; classifying the video data according to the scene, camera ID, project name and source time involved; Step S2: Use the built-in picture making tool of the video data management module to set the video processing rules; Step S3: Automatically annotate the preliminary screening business data set using the data automatic annotation module, use the closed set detection large model and the text-image open set detection large model, and summarize the detection result list; Step S4: Use the special data sample retention module to retain pictures containing special categories. First, the result list obtained in step S3 is allocated according to the detection results, and then 2 to 3 pictures of the list containing special detection results are randomly retained; Step S5: using the image data deduplication module to reduce the redundancy of the remaining images, extracting the corresponding feature vectors through the closed set detection model, and using the cosine distance algorithm to calculate the similarity values ​​between different images. When the similarity value is greater than a preset threshold, the images are discarded; Step S6: Summarize the remaining image files and label files and send them to the data annotation platform for review and modification to ensure that there are no obvious errors in the data set and complete the optimization.

3. The data architecture optimization method for target detection algorithm according to claim 2, characterized in that: The step S3 specifically comprises the following steps: Step S31: According to the needs of the business scenario, 11 detection categories are preset, including non-motor vehicles, pedestrians, reflective clothing, triangular warning signs, small trucks, open flames, smoke, traffic cones, cars, large trucks and buses. According to the particularity of the traffic scene, reflective clothing, triangular warning signs and traffic cones are used as domain-specific targets, open flames and smoke are used as missing categories in the public dataset, and the remaining six categories are used as common categories; Step S32: prepare a closed-set detection model and a text-image open-set detection model. According to the business scenario analysis, since the closed-set detection model is usually trained on the Coco general dataset, there is a lack of data on reflective clothing, triangular warning signs, traffic cones, open flames, and smoke traffic facilities. Therefore, corresponding prompt words are set and the grounding-dino model is used to detect and annotate them. Step S33: When combining the detection results of different models, the text-image open set detection model has direct decision-making power over the detection results of reflective clothing, triangular warning signs, traffic cones, open flames and smoke. The closed set detection model has direct decision-making power over the categories contained in the public data sets of non-motor vehicles, pedestrians and cars. Finally, the detection results of the two models are unioned as the final output.

4. The data architecture optimization method for target detection algorithm according to claim 1, characterized in that: The step S4 specifically comprises the following steps: Step S41: specifying six types of targets, namely, reflective clothing, non-motor vehicles, open flames, smoke, warning triangles and pedestrians, as special categories, and requiring additional retention of images containing these six categories; Step S42: Create an empty list for each category of target, traverse the list output by step S3, but when the picture contains these categories, add the picture information to the corresponding list to obtain category lists L1, L2, ..., L6; Step S43: Randomly retain 2 to 3 pictures for each category list and discard the remaining pictures.

5. The data architecture optimization method for target detection algorithm according to claim 1, characterized in that: The step S5 specifically comprises the following steps: Step S51: Input image i with dimension (1, 3, 700, 700), where 1 is the batch size, 3 is the number of channels, and 700 is the length and width of the image. After the encoder-decoder structure of the dinov2 model, the feature vector x with feature dimension (1, 768) is output. All remaining images are extracted to generate a one-dimensional vector set X. Step S52: randomly select a vector x as the first element of the reference vector subset A, extract the remaining vectors x in order as matching vectors B, and use the cosine distance formula: Where x is the reference vector subset A, y is the matching vector B, and the similarity score S is calculated. When S≥T, the matching vector B is discarded, but when S<T, B is added to the reference vector subset A.

Images