Image data preprocessing method, device, equipment, medium and product

Abstract

The present application discloses a method, device, equipment, medium and product for preprocessing image data, which relates to the field of image processing. First, a corresponding image data set is obtained according to the task objective of the target detection model; various types of image data in the image data set are classified and screened according to the target type to be detected to obtain a target type data set; data balancing is performed on the image data of each target type in the target type data set to obtain a balanced data set; feature dimension optimization is performed on each image data in the balanced data set to obtain an optimized feature vector and store it in a vector database for use in target detection model training. The present application stores image data directly in the form of feature vectors in a vector database. When used for model training, it can not only avoid repeated operation processes, but also reduce the time complexity and space complexity of model training, and reduce the hardware resources required in the model training and parameter adjustment process under the same conditions.

Description

Technical Field

[0001] The present application relates to the field of image processing technology in artificial intelligence, and in particular to a method, device, equipment, medium and product for preprocessing image data. Background Art

[0002] When detecting and recognizing objects in images, the image quality and data size directly impact the subsequent model architecture design, loss function selection, optimization algorithm implementation, and recognition accuracy. Therefore, proper image preprocessing is necessary before detection or recognition. Image preprocessing primarily aims to remove noise, enhance the readability of useful information, and maximize data quality, thus providing optimal conditions for subsequent object detection. Existing data preprocessing techniques typically inspect the data for anomalies, perform data denoising based on business requirements, and remove useless or illegal information. The model then trains the entire image and text data (text data serves as labels for the image data). When new business requirements arise, developers must perform further data preprocessing based on the new business logic. The model then uses the image and text data as input for model training, performing internal vectorization to convert the image and text data into feature vectors. This process increases the time and cost of data preprocessing. Furthermore, a large amount of image and text data contains data irrelevant to the business requirements, further increasing the model training cost and space complexity. When adjusting the model algorithm or wanting to adjust the algorithm parameters, this preprocessing process will be repeated. Summary of the Invention

[0003] The purpose of this application is to provide a method, device, equipment, medium and product for preprocessing image data, which directly stores image data in the form of feature vectors in a vector database. When used for model training, it can not only avoid repeated operation processes, but also reduce the time complexity and space complexity of model training, and reduce the hardware resources required during model training and parameter adjustment under the same conditions.

[0004] To achieve the above objectives, this application provides the following solutions.

[0005] In a first aspect, the present application provides a method for preprocessing image data, comprising:

[0006] Obtain corresponding image datasets according to the task objectives of the target detection model; the task objectives of the target detection model include human object detection, animal object detection, vehicle object detection, lesion area detection, obstacle detection, product defect detection, abnormal object detection and pest area detection; the corresponding image datasets include human image datasets, animal image datasets, vehicle image datasets, lesion image datasets, obstacle image datasets, product image datasets, object image datasets and crop image datasets;

[0007] Classify and filter various types of image data in the image dataset according to the target type to be detected to obtain a target type dataset;

[0008] Perform data balancing on the image data of each target category in the target category dataset to obtain a balanced dataset;

[0009] Perform feature dimension optimization on each image data in the balanced dataset to obtain the optimized feature vector;

[0010] The optimized feature vector is stored in the vector database for use in target detection model training.

[0011] Optionally, the classifying and screening various types of image data in the image dataset according to the types of targets to be detected to obtain the target type dataset specifically includes:

[0012] The CLIP model is used to calculate the similarity between each image data in the image dataset and each target category, and the image data with similarity greater than the similarity threshold is retained to form the target category dataset.

[0013] Optionally, performing data balancing on the image data of each target category in the target category dataset to obtain a balanced dataset specifically includes:

[0014] Determine whether the image data ratio between any two target categories in the target category dataset is balanced;

[0015] If the ratio is balanced, the image data of each target category is compressed to the same category number Nmax; where Nmax is the number of images contained in the target category with the largest proportion after data compression;

[0016] If the proportion is unbalanced, the image data of the target category with a smaller proportion will be enhanced, and the image data of the target category with a larger proportion will be compressed, and the image data of each target category will be enhanced or compressed to the same classification number Nmax.

[0017] Optionally, performing feature dimension optimization processing on each image data in the balanced data set to obtain an optimized feature vector specifically includes:

[0018] Each image data in the balanced dataset is divided into blocks, each of which is divided into multiple blocks of M rows × N columns; M and N are both integers greater than 1;

[0019] Perform block-level classification on each image block, mark the image block containing the target object as 1, and mark the image block not containing the target object as 0, to obtain the labeled image block;

[0020] Based on the label of each image block, the multiple image blocks divided into each image data are screened and reorganized to obtain the optimized feature vector.

[0021] Optionally, the step of filtering and reorganizing the multiple image blocks into which each image data is segmented based on the label of each image block to obtain an optimized feature vector specifically includes:

[0022] Traverse each image data into multiple image blocks of M rows × N columns, and keep the image blocks with label 1 and discard the image blocks with label 0 in each row;

[0023] All image blocks with label 1 are reorganized into a 1×X dimension feature matrix. After each row of image blocks is successfully reorganized, a sequence start position marker [b] is added. After the last row of image blocks is reorganized, a sequence end position marker [e] is added to obtain the optimized feature vector. X is the number of image blocks with label 1 in each image data.

[0024] Optionally, storing the optimized feature vector in a vector database for use in target detection model training specifically includes:

[0025] The optimized feature vector is used as a feature data set and stored in the vector database;

[0026] When training the target detection model, the feature dataset is divided into a training set, a validation set, and a test set for training, validating, and testing the target detection model.

[0027] In a second aspect, the present application provides an image data preprocessing device, comprising:

[0028] An image dataset acquisition module is used to acquire corresponding image datasets according to the task objectives of the target detection model; the task objectives of the target detection model include human object detection, animal object detection, vehicle object detection, lesion area detection, obstacle detection, product defect detection, abnormal object detection and pest area detection; the corresponding image datasets include human image datasets, animal image datasets, vehicle image datasets, lesion image datasets, obstacle image datasets, product image datasets, object image datasets and crop image datasets;

[0029] The image classification and screening module is used to classify and screen various types of image data in the image dataset according to the target type to be detected, and obtain the target type dataset;

[0030] A data balancing processing module is used to perform data balancing on the image data of each target category in the target category data set to obtain a balanced data set;

[0031] The feature dimension optimization module is used to optimize the feature dimension of each image in the balanced data set to obtain the optimized feature vector;

[0032] The feature vector storage module is used to store the optimized feature vectors into the vector database for use in target detection model training.

[0033] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image data preprocessing method.

[0034] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the image data preprocessing method when executed by a processor.

[0035] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which implements the image data preprocessing method when executed by a processor.

[0036] According to the specific embodiments provided in this application, this application discloses the following technical effects.

[0037] The present application provides a method, device, equipment, medium and product for preprocessing image data. On the one hand, it automatically screens the target category data set and balances the category proportions according to the task objectives of the target detection model and the target categories to be detected. On the other hand, it optimizes the feature dimension of the image data and realizes feature dimensionality reduction. The image data preprocessed by the method of the present application is directly stored in the vector database in the form of feature vectors. In this way, when used for target detection model training, it not only avoids repeated operation processes, but also reduces the time complexity and space complexity of the model, reduces the training time of the model under the same conditions, and reduces the hardware resources required in the training and parameter adjustment process, thereby effectively reducing the cost of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0039] Figure 1 This is a flowchart of a method for preprocessing image data in this application;

[0040] Figure 2 This is a schematic diagram of the original picture of a tiger in the embodiment of this application;

[0041] Figure 3 This is a schematic diagram of block processing of an original image in an embodiment of the present application;

[0042] Figure 4 Schematic diagram of performing block-level classification on image blocks in an embodiment of the present application;

[0043] Figure 5 This is a schematic diagram of screening and reorganizing image blocks in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0045] This application proposes a method, device, equipment, medium and product for preprocessing image data, which aims to store image data directly in the form of feature vectors in a vector database through an improved data preprocessing strategy. When used for model training, it can not only avoid repetitive operation processes, but also reduce the time complexity and space complexity of model training, and reduce the hardware resources required during model training and parameter adjustment under the same conditions.

[0046] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0047] In an exemplary embodiment, Figure 1 As shown, a method for preprocessing image data is provided, including steps 1 to 5 below.

[0048] Step 1: Obtain the corresponding image dataset according to the task objectives of the target detection model.

[0049] Object detection is a core task in computer vision, aiming to identify and locate objects in images (or pictures) and generate bounding boxes and category labels for each detected object. Object detection technology is widely used in various fields. For example, in security surveillance, it is used to analyze surveillance video in real time and detect abnormal behavior or targets, such as people or unusual objects. In intelligent transportation, for example, in autonomous driving and traffic management, it detects and tracks vehicles, pedestrians, and other targets, enabling traffic congestion prediction and intelligent driving. In smart retail, it is used to detect product on shelves and analyze customer behavior, improving retail efficiency and customer experience. In medical image analysis, for example, it assists doctors in lesion detection and cell identification, improving diagnostic accuracy and efficiency.

[0050] Therefore, the task objectives of the object detection model described in this application can be human object detection, animal object detection, vehicle object detection, lesion area detection, obstacle detection, product defect detection, abnormal object detection, and pest area detection, etc. The corresponding image datasets should be human image datasets, animal image datasets, vehicle image datasets, lesion image datasets, obstacle image datasets, product image datasets, object image datasets, or crop image datasets.

[0051] Step 2: Classify and filter the various types of image data in the image dataset according to the target type to be detected to obtain the target type dataset.

[0052] The types of targets to be detected correspond to the task objectives of the target detection model. For the convenience of description, the target detection model can be referred to as a model, and the term "type" is also referred to as a category. For example, for lesion area detection, the types of targets to be detected can be lesion areas such as tumors and nodules, and normal tissue areas. For example, for obstacle detection, the types of targets to be detected can be pedestrians, other vehicles, and various objects on the road in front of the vehicle. For example, for product defect detection, the types of targets to be detected can be product surface defect types such as cracks, scratches, bubbles, and spots. For example, for abnormal object detection, the types of targets to be detected can be suspicious objects such as abandoned packages and dangerous goods. For pest and disease area detection, the types of targets to be detected can be types of pests and diseases and crop areas affected by pests and diseases, etc.

[0053] The initially acquired image dataset contains various types of image data, which can also be referred to as images. The purpose of step 2 is to extract images that meet the model task objectives through an automated method to form the target category dataset.

[0054] The CLIP (Contrastive Language-Image Pretraining) model is a multimodal machine learning model proposed by OpenAI in 2021. The core idea of ​​the CLIP model is to map images and texts to the same semantic space through large-scale image-text pair training to achieve cross-modal understanding. The training data of the CLIP model is 400 million pairs of (image, text) data publicly available on the Internet. The CLIP model has the following capabilities: Zero-Shot classification, which directly classifies images based on text descriptions without fine-tuning for specific tasks. Features of the CLIP model include: 1) No training data required: directly supports zero-shot classification; 2) Dynamic adjustment of categories: modify candidate category texts at any time without retraining; 3) Cross-domain generalization: applicable to open domain scenarios, it has been exposed to massive open domain data during training and can understand a wide range of topics, such as objects, scenes, and abstract concepts.

[0055] The CLIP model takes image and text inputs. Image input refers to a single image, such as an RGB image. Preprocessing involves resizing the image to a fixed resolution (e.g., 224×224) and normalizing the pixel values. Image features are extracted using a Vision Transformer (ViT) or ResNet encoder. Text input refers to categories or sentences described in natural language, such as "a picture of a dog." Preprocessing involves converting the text into a sequence of tokens using a tokenizer. Text features are extracted using a Transformer encoder. The output of the CLIP model is a similarity score between the input image and text. The CLIP model calculates the cosine similarity (real number, no fixed upper or lower bounds) between image and text features and, after Softmax normalization, obtains the degree of match between each text description and the image, i.e., the similarity.

[0056] Therefore, when calculating the similarity between each image data in the image dataset and each target category, the present application only needs to input any candidate category text (such as target categories such as cracks, scratches, bubbles, spots, etc.) and any image data in the image dataset, and the CLIP model can calculate the similarity between each image data and each target category text. Furthermore, image data with a similarity greater than a similarity threshold is retained to form the target category dataset. The similarity threshold ranges from 0 to 1, and the closer it is to 1, the greater the possibility of the target category. The threshold can be set according to the actual situation. For example, the similarity threshold is set to 0.75.

[0057] Let's use a specific example to illustrate this. For example, in one exemplary embodiment, the target of the object detection model is to detect animals. The corresponding image dataset includes images of various animals, such as lions, tigers, elephants, monkeys, cheetahs, zebras, penguins, and bears. If the target objects of the object detection model the user wants to build are lions, tigers, and cheetahs, then the target categories are determined to be lions, tigers, and cheetahs. Therefore, the purpose of step 2 is to classify each image in the image dataset into one of the following categories: lion, tiger, or cheetah. The specific classification and screening process includes: 1) Text encoding: Convert each target category into a text description (e.g., "picture of lion," "picture of tiger," and "picture of cheetah") and encode it into a corresponding text feature vector; 2) Image encoding: Encode the image to be classified into an image feature vector; 3) Similarity calculation: Compute the cosine similarity between the image feature vector and the three text feature vectors; 4) Probability normalization: Calculate the similarity probability distribution using Softmax, and select the category with the highest probability as the image label. Calculate the similarity between each image and the three given labels (lion, tiger, and cheetah). Keep images with a similarity > 0.75 and label the rest as "Other." Finally, the target species dataset contains images of lions, tigers, and cheetahs, each with a label for at least one of these three animals.

[0058] Step 3: Perform data balancing on the image data of each target category in the target category dataset to obtain a balanced dataset.

[0059] The target category data set obtained in step 2 contains pictures of multiple target categories. In model training, the more balanced the proportion of each category in the training data, the better the training effect of the model. If the proportion of each category is very different, the trained model may be biased: for pictures with a larger overall proportion, the training effect of the model will be much better than that of pictures with a smaller proportion. Therefore, in this step, it will be judged whether the data ratios of the three pictures of lions, tigers and cheetahs in the above specific example are balanced. For example, in the pictures containing the above three target categories, the ratio between any two of the data is 1:1, that is, the ratio of the number of pictures of the three target categories is 1:1:1, which means that the data is very balanced. Step 3 specifically includes the following steps 3.1 to 3.3.

[0060] Step 3.1: Determine whether the ratio of image data between any two target categories in the image data of each target category in the target category dataset is balanced.

[0061] The ideal state for balanced data ratios is to have an equal number of images for each target category. Therefore, for the target category dataset after classification and screening, we first determine whether the image data ratio between any two target categories is balanced. In other words, we determine whether the image data ratio between any two target categories is 1:1.

[0062] Step 3.2: If the ratio is balanced, compress the image data of each target category to the same classification number Nmax; where Nmax is the number of images contained in the target category with the largest proportion after data compression.

[0063] If the data ratio is balanced, that is, the image data ratio between any two target categories is 1:1, then the image data of each target category is directly compressed. During the compression process, the number of images contained in the target category with the largest proportion after compression (called the classification number) Nmax is used as the benchmark, and the number of images of the remaining target categories is also compressed to Nmax.

[0064] The data compression method is to calculate the cosine similarity based on the feature vectors in the image, retaining the images with higher similarity and giving priority to images with higher model thresholds, thereby reducing data duplication and promoting the improvement of data quality. The formula for calculating cosine similarity is as follows:

[0065] (1);

[0066] in, The image feature vector is extracted by the CLIP model's image encoder (such as ViT or ResNet). The CLIP model converts the input image into a high-dimensional vector of fixed dimension. For example, the output of CLIP-ViT is a 512-dimensional or 768-dimensional vector. This vector encodes the global semantic information of the image, such as object category, scene, and texture. This is a text feature vector extracted by the CLIP model's text encoder (e.g., Transformer). The text encoder maps input text (e.g., "tiger") into a vector of the same dimension as the image features (e.g., 512 dimensions). This text feature vector encodes the semantic information of the label. For example, "tiger" is associated with concepts such as "animal," "mane," "forehead stripes," and "grassland."

[0067] It means calculating the square root of the sum of the squares of the elements of the vector, that is:

[0068] (2);

[0069] in, Represents a vector No. Dimensional elements, is the vector dimension. For example, for a 512-dimensional feature vector, the formula =512. Formula (2) is used to eliminate the effect of vector length on similarity in non-normalized scenarios, retaining only directional information. In the CLIP model, L2 normalization is performed on image and text features, allowing cosine similarity to be calculated directly via dot product. This allows the denominator in Formula (1) to be omitted in actual calculations, eliminating the need to calculate Formula (2).

[0070] That is, the denominator in formula (1) is the L2 modulus multiplication of the two vectors, the purpose of which is to normalize the dot product to the interval [-1, 1]. The CLIP model used in this application performs L2 normalization on image and text features during training, which means that it is forced to Therefore, in the inference phase, the encoded feature vector automatically meets the normalization conditions, and the denominator value is: At this time, the cosine similarity is simplified to:

[0071] (3);

[0072] This means that you can directly use the dot product to get cosine similarity without explicitly dividing by the denominator. The normalized dot product is equivalent to cosine similarity, which is computationally more efficient and consistent with the model training objectives.

[0073] The following is an example of classifying a tiger image. Image encoding: Use the CLIP model to encode the image into a 512-dimensional vector. Text encoding: Encode the label "tiger" into a vector of the same dimension . Similarity calculation: So when there are multiple images with similarities to the label "tiger" between 0.84 and 0.85 (the interval [0.84, 0.85] here is a small interval), data compression will retain the images with an image similarity calculated as 0.85.

[0074] Of course, regardless of the calculated similarity value, the data compression method is similar to that used for images within the interval [0.84, 0.85]. Furthermore, when dividing the similarity interval within the target category (tiger) images into smaller intervals, the size of the intervals is determined based on the actual situation. Assuming that all images of the target category "tiger" have similarities within the interval [0.86, 0.96], the span of the small intervals is set to 0.005 during data compression. This means that images within the interval [0.86, 0.865] in one of the small intervals are retained with a similarity of 0.865. Assuming that all images of the target category "tiger" have similarities within the interval [0.87, 0.90], meaning the similarity interval is relatively small, the span of the small intervals is set to 0.001. This means that images within the interval [0.87, 0.871] in one of the small intervals are retained with a similarity of 0.871. By analogy, the number of pictures can be compressed.

[0075] Step 3.3: If the ratio is unbalanced, perform data enhancement on the image data of the target category with a smaller proportion, and perform data compression on the image data of the target category with a larger proportion, and enhance or compress the image data of each target category to the same classification number Nmax.

[0076] If the data is unevenly distributed, data compression is performed on the images that make up the largest proportion. The compression process uses Nmax, the number of categories with the largest proportion after compression, as the benchmark. The remaining categories with the largest proportions are also compressed to Nmax. During data compression, the cosine similarity calculation for each image may result in a different range, which means that the number of retained images will vary. Nmax is obtained only after all image data is compressed, and the maximum value is taken. This is done to preserve the diversity of scarce data.

[0077] Conversely, if data augmentation is performed on image data that accounts for a smaller proportion, the number of enhanced images is also Nmax. Data augmentation can be performed using image rotation, cropping, translation, flipping, and other processing methods.

[0078] Step 4: Optimize the feature dimension of each image in the balanced dataset to obtain the optimized feature vector.

[0079] The step 4 specifically includes the following steps 4.1 to 4.3.

[0080] Step 4.1: Divide each image data in the balanced data set into blocks, dividing each image data into multiple image blocks of M rows × N columns.

[0081] The image is segmented into blocks to subsequently identify whether each block contains the target object. Furthermore, each block can be flattened into a vector, linearly projected to obtain its embedded representation, and positional encoding is added to preserve spatial information. Flattening each block into a vector and obtaining its embedded representation facilitates model computation and recognition. The image information seen by the eye needs to be processed into a special form of digital representation that can be processed by a computer. Positional encoding is added to preserve the spatial information between the blocks into which the original image is segmented. After segmentation, the original image has an overall dimension of M×N; M and N are both integers greater than 1. The original image dimension is height multiplied by width. For convenience, h represents height and w represents width, i.e., h×w. The pixel size of the segmented blocks can be adjusted according to actual conditions, for example, 16×16 pixels.

[0082] The following uses a specific example to illustrate. Figure 2 For the “tiger” category picture shown in , ViT is used to segment the original picture into image blocks of a fixed size (such as 16×16 pixels). Figure 3 As shown, the image is divided into 4 rows × 3 columns, a total of 12 image blocks. Figure 2 The original overall dimensions of the tiger image in the figure are (4×16)×(3×16). After segmentation, Figure 3 The overall dimension of is M×N=4×3.

[0083] Furthermore, each image block can be flattened into a vector. Assuming the original image size is 256×256 pixels and is a three-channel RGB image, its actual shape is 3×256×256. The block size is set to 16×16 pixels, which means the actual shape of each image block is 3×16×16. The image block flattening operation means treating each 16×16 image block as an independent small image and sequentially arranging the 16×16 pixel values ​​of the three R / G / B channels of the image block into a one-dimensional vector. The length of the vector for each image block is 3×16×16 = 768.

[0084] Assuming a 16×16 three-channel image block, the implementation code example is as follows:

[0085] block = torch.randn(3, 16, 16) # shape [3, 16, 16]

[0086] flattened_block = block.flatten() # shape

[768]

[0087] torch.randn() is a PyTorch function used to generate random tensors. block represents an image block. block.flatten() flattens the image block. flattened_block is the flattened vector.

[0088] Furthermore, the flattened 768-dimensional vectors are mapped to the uniform dimension required by ViT (e.g., 512 dimensions) as follows. A learnable fully connected layer (linear layer) is used with a weight matrix of shape 768×512. Each flattened vector (768 dimensions) is multiplied by this matrix to output a 512-dimensional embedding vector. This process is called "linear projection," and its essence is to compress and abstract the features of the original pixel information. The implementation code example is as follows:

[0089] linear_layer = torch.nn.Linear(768, 512) # Define linear layer

[0090] embedded_vector = linear_layer(flattened_block) # Output shape

[512]

[0091] torch.nn.Linear() is a very important module in PyTorch, used to implement fully connected layers (also called linear layers). Fully connected layers are a common layer type in neural networks, used to transform input data into output data, and are typically used to process one-dimensional data. linear_layer refers to the generated fully connected layer. embedded_vector is the resulting embedding vector.

[0092] Furthermore, position encoding is added to the embedding vector to preserve spatial information. The implementation code of the relevant methods is already built into the Transformer library and can be implemented by calling the relevant function code.

[0093] Step 4.2: Perform block-level classification on each image block, mark the image block containing the target object as 1, and mark the image block not containing the target object as 0, to obtain the labeled image block.

[0094] This application fine-tunes the ViT architecture to perform block-level classification for each image block. Specifically, this is achieved by adding a classification head (fully connected layer + softmax) to the output embedding of each block after the ViT Transformer encoder.

[0095] The main modules of ViT include: Patch Embedding, Positional Encoding, Transformer Encoder (multiple layers) and classification head (usually a fully connected layer). This application adds a classification head (fully connected layer + Softmax) to the output embedding of each block after the Transformer encoder of ViT. The function of the fully connected layer is to map the high-dimensional embedding of each block output by the Transformer to the category space (such as two-dimensional space, corresponding to 0 and 1). Softmax converts the linear output into a probability distribution, indicating the probability that each block belongs to each category. The input of the fully connected layer is the embedding vector of each block (embedding_dim, such as 512 dimensions), and the output is the dimension of the number of categories (such as 0 and 1, two-dimensional). The input of Softmax is this two-dimensional vector, and the output is a normalized probability distribution.

[0096] Each image block is input into the improved ViT structure, and after block-level classification, the image block containing the target object is marked as "1", called the target block; the image block that does not contain the target object is marked as "0". "0" and "1" are the category labels of the image block. For example Figure 4 As shown in , the image block containing the target object "tiger" is marked as "1", and the image block not containing the target object "tiger" is marked as "0".

[0097] Step 4.3: Based on the label of each image block, filter and reorganize the multiple image blocks into which each image data is divided to obtain the optimized feature vector.

[0098] Specifically, the algorithm traverses the M rows × N columns of image blocks that each image data is partitioned into. For each row, the image blocks labeled "1" (i.e., target blocks) are retained, and the image blocks labeled "0" are discarded. Furthermore, all image blocks labeled "1" are reorganized into a one-dimensional feature matrix of 1 × X dimensions. After each row of image blocks is successfully reorganized, a sequence start position marker [b] is added, and after the last row of image blocks is reorganized, a sequence end position marker [e] is added. This results in an optimized feature vector, which is a sequence consisting of the target block and markers [b] and [e]. X is the number of image blocks labeled "1" in each image data. [b] stands for beginning, indicating the starting position of the sequence. [e] stands for ending, indicating the ending position of the sequence.

[0099] like Figure 4 and Figure 5As shown in the figure, after the tiger image is segmented and identified, the image blocks in the first and second rows both contain some information about the tiger and are therefore marked as 1. Therefore, these two rows are retained during reassembly. During reassembly, [b] is added to the end of the first row of image blocks, followed by the second row of image blocks, and [b] is added to the end of the second row of image blocks. The first image block in the third row does not contain any information about the tiger and is therefore marked as 0. This image block with a label of 0 is discarded, and the previous [b] is added to the two remaining image blocks marked as 1 and a [b] in the third row. The first image block in the fourth row also does not contain any information about the tiger and is therefore marked as 0. This image block with a label of 0 is discarded, and the previous [b] is added to the two remaining image blocks marked as 1 and a [e] in the fourth row. Since this is the last image block in the image, the sequence ends.

[0100] Step 5: Store the optimized feature vector into the vector database for use in target detection model training.

[0101] After the feature dimension optimization in step 4, the image data is now represented as a feature vector, which can be used directly for model training. The optimized feature vector is stored as a feature dataset in the Milvus vector database. Specifically, a collection is created in Milvus to store the image feature vectors after feature dimension optimization in step 4. The collection contains the following fields: image ID, image label, image path, and feature vector. Other fields can be added as needed. During model training, the image feature vector data can be directly obtained from Milvus for training.

[0102] Milvus is a high-performance, scalable vector database suitable for large-scale AI applications, particularly in scenarios such as NLP (Natural Language Processing), computer vision, and recommender systems. Milvus is compatible with PyTorch, TensorFlow, and Hugging Face, making it easy to integrate into the AI ​​ecosystem. It supports distributed deployment, is suitable for cloud applications, and can be managed using Kubernetes. As a high-performance vector database, Milvus efficiently stores and retrieves large-scale image feature vectors and supports fast similarity searches. Image feature vectors retrieved from Milvus can be directly used for model training, improving training efficiency and model performance.

[0103] When training the object detection model, directly call the corresponding feature dataset from Milvus and divide it into a training set, a validation set, and a test set for training, validating, and testing the object detection model. The training set is used to train the model and adjust model parameters to minimize the loss function. The validation set is used to evaluate model performance and adjust hyperparameters during training. The test set is used to finally evaluate the model's generalization ability and is typically used after model training is complete.

[0104] The object detection model in this application can be a deep learning model such as a convolutional neural network (CNN) model, a recurrent neural network (RNN) model, or a Transformer model. Convolutional neural network models are used for image processing. Recurrent neural network models are used for processing sequence data (such as time series and text). Transformer models are used for natural language processing and multimodal tasks. For example, the input to a Transformer model is a sequence embedding in the form [Batch_size, X, D]. Batch_size is the batch size, which refers to the number of data samples used in each iteration (or "step") when training a machine learning or deep learning model. X is a variable-length sequence, and D is the embedding dimension, which consists of the following two parts: Block content embedding: Features extracted from image blocks, such as a 1×X dimensional sequence, i.e., the optimized feature vector generated in step 4. Position encoding: Position information added to each image block, such as the positions of the [b] and [e] markers. [b] and [e] act as delimiters to help the model identify sequence boundaries.

[0105] The task requirements of an object detection model determine the model's output format. For example, the output can be sequence-to-sequence (Seq2Seq): outputting a sequence of the same length as the input, as in machine translation. Sequence-to-label (Seq2Label): outputting a single classification result, as in image classification. Autoregressive generation: generating subsequent labels one by one, as in text generation. For the object detection task in this application, the following format is used.

[0106] Input: [b] + target block sequence + [e], shape is [Batch_size, X+2, D].

[0107] Output: Designed according to specific task requirements, for example:

[0108] Predict the target category (classification task) → Output [Batch_size, num_classes]. num_classes represents the target category;

[0109] Generate target locations (detection task) → Output [Batch_size, X, 4], where 4 is the number of bounding box coordinates.

[0110] A trained object detection model can be used to implement the following functions. For example, object detection: locating the location of a specific object in an image. Image captioning: generating a natural language description based on a sequence of object blocks. Image inpainting: reconstructing a complete image based on the retained object blocks. Image-text retrieval: matching an image block sequence with a text description.

[0111] The following are several common application scenarios of target detection models.

[0112] Application Scenario 1: Medical Image Analysis: Tumor Region Localization. The goal is to quickly locate tumor regions in CT / MRI images to assist physicians in diagnosis. The trained object detection model takes as input a sequence of medical image blocks preprocessed using the method described in this application. Only target blocks classified as "tumor" (referred to as "blocks") are retained, for example, in the form [b, block 1, block 2, ..., block X, e]. Each block contains local tissue features. The trained object detection model outputs a heat map of tumor locations (marking the coordinates of all image blocks classified as "1" in the original image) or a probability distribution map. This heat map or probability distribution map can be used to assist in diagnosis, reducing physician image review time and improving the efficiency of early cancer screening.

[0113] Application scenario 2, autonomous driving: real-time detection of road obstacles. Its task goal is to identify obstacles such as pedestrians, vehicles, and objects on the road ahead of the vehicle. The image or video stream captured by the on-board camera is preprocessed by the method of this application to generate a corresponding feature vector, that is, a target block sequence. The input of the trained target detection model is a sequence of image blocks, and only the target blocks containing obstacles are retained, such as saved in the form of [b, block A, block B, ..., block X, e]. The output of the trained target detection model is the coordinates of the obstacle bounding box (reconstructed based on the position of the image blocks) and the category (such as pedestrians, vehicles, and objects). Real-time detection of road obstacles based on the trained target detection model can effectively improve the real-time response capability of the autonomous driving system and avoid collisions.

[0114] Application scenario three, industrial quality inspection: product defect detection. The task goal is to detect surface defects of products on the production line, such as cracks, scratches, bubbles, spots, etc. The images or video streams taken by the industrial camera on the production line are preprocessed by the method of this application to generate a corresponding target block sequence. The input of the trained target detection model is a sequence of image blocks of the product, and only the defect blocks are retained, such as saved in the form of [b, defect block 1, ..., defect block X, e]. The output of the trained target detection model is the defect type classification (such as crack = 1, scratch = 2, bubble = 3, spot = 4) and defect location marking. Product surface defect detection based on the trained target detection model can replace manual quality inspection and improve detection efficiency and consistency.

[0115] Application Scenario 4: Security Monitoring: Abnormal Object Identification. The objective is to identify suspicious objects in surveillance video, such as left-behind packages and dangerous goods. The input to the trained object detection model is a sequence of target blocks, obtained by preprocessing video frames using the method described in this application. Only abnormal blocks are retained, for example, in the form [b, abnormal block 1, ..., abnormal block X, e]. The output of the trained object detection model is an abnormal object category alarm (e.g., "package," "knife") and its location coordinates. Abnormal object identification based on the trained object detection model can enhance public safety and reduce manual monitoring costs.

[0116] Application Scenario 5: Agricultural Monitoring: Pest and Disease Area Identification. The objective is to locate crop areas affected by pests and diseases using drone-captured farmland images. The trained object detection model takes as input a sequence of farmland image blocks preprocessed using the method described in this application. Only target blocks containing pests and diseases are retained, referred to as diseased blocks, and stored as [b, diseased block 1, ..., diseased block X, e]. The output of the trained object detection model is a pest and disease severity classification (e.g., mild / severe) and a distribution heat map. Identifying pest and disease areas based on the trained object detection model can aid in precise pesticide application, reduce pesticide waste, and increase crop yields.

[0117] The key points of the image data preprocessing method of the present application are that, on the one hand, the image data is automatically screened and the category ratio is balanced according to the task objectives and target categories, and on the other hand, the image data feature dimensionality reduction is performed. The feature engineering in the image data preprocessing stage processes the image data into feature vector data with a lower overall dimension, so that fewer hardware resources are required in the model training stage, that is, the space complexity is reduced, and the time complexity of training under the same model and parameter conditions is also lower. At the same time, the processed feature vector data can be reused by models composed of other algorithms, reducing resource consumption and development and maintenance costs, and improving resource utilization efficiency and model development efficiency. Among them, time complexity refers to the rate at which the algorithm running time grows with the input scale, usually expressed in big O symbols, such as O(n), O(n 2 ), etc. Space complexity refers to the amount of storage space occupied by the algorithm during its execution, and is also expressed in Big O notation.

[0118] In order to verify the effect of this application, the feature vector obtained by the method of this application is used to train the Transformer model. The advantages of this application in time complexity and space complexity of the target detection task can be demonstrated through the following mathematical derivation and comparative analysis. According to the method of this application, the dimension of the feature vector after each image processing is 1×X, where X varies due to the number of target blocks in the image. Although traditional neural networks (such as fully connected networks, CNNs) usually require a fixed input size, the design of Transformer naturally supports variable-length sequence input. The core of Transformer is the self-attention mechanism, and its calculation process does not depend on the sequence length, so it can process input sequences of any length.

[0119] First, the Transformer's computational complexity primarily stems from its self-attention mechanism, whose time and space complexities are both proportional to the square of the input sequence length n. The time complexity is expressed as O(n²d), where d is the feature dimension. The space complexity is O(n²), stored via the attention matrix. Below is a comparison of the input sequence lengths of traditional methods and our proposed method.

[0120] Traditional methods use raw pixel input. Assuming the original image size is h×w, the input sequence length n1 = h×w. For example, when h = 224 and w = 224, n1 = 224×224 = 50,176.

[0121] In the present method, during the block segmentation phase, the image is divided into p × p pixel blocks, where the number of blocks is (h / p) × (w / p). For example, if p = 16, the number of blocks = (224 / 16) × (224 / 16) = 14 × 14 = 196. During the block classification and reassembly phase, only target blocks marked with "1" are retained. Assuming k target blocks are retained (k < the total number of blocks), if the target coverage area accounts for 10%, then k ≈ 196 × 10% = 19.6 ≈ 20. During the sequence construction phase, special markers [b] and [e] are added, resulting in a final sequence length n² = k + m, where m is the number of special markers. Because the time and space complexity required for special marker processing is not on the same order of magnitude as that of the target blocks, they can be ignored in the calculation, assuming that n² = k.

[0122] The time complexity comparison is as follows.

[0123] The time complexity of the traditional method is: T1 = O(n1²d) = O((h×w)²d). Numerical example: when h = 224, w = 224, and d = 768, T1 ≈ 50,176² × 768 ≈ 1.93×10¹² operations.

[0124] The time complexity of the proposed method is: T² = O(n²d) = O(k²d). Numerical example: when k = 20 and d = 768, T² ≈ 20² × 768 operations.

[0125] Then the speedup ratio is: T1 / T2≈ 1.93×10¹² / 3.07×10 5 ≈ 6.28×10 6 times.

[0126] The space complexity comparison is as follows.

[0127] The space complexity of the traditional method is: S1 = O(n1²) = O((h×w)²). Numerical example: S1≈50,176²≈2.52×10 9 storage units.

[0128] The space complexity of the proposed method is: S² = O(n²²) = O(k²²). Numerical example: when k = 20, S² ≈ 20²² = 400 storage units.

[0129] Then the compression ratio is: S1 / S2≈2.52×10 9 / 400≈6.3×10 6 times.

[0130] From the above comparison, it can be seen that the method of this application can achieve exponential compression of the sequence length. Through block division and classification filtering, the input sequence length is reduced from O(h×w) to O(k), and k << h×w. Mathematical relationship: k ≈ α×h×w / p², where α is the target coverage rate and α << 1. On the other hand, the method of this application can also achieve a quadratic reduction in time and space complexity. Since the complexity is related to n², the compression of the sequence length can directly cause the computational amount and storage requirements to decrease quadratically. In addition, the influence of the feature dimension d is that if the feature dimension d increases after block division (such as d = 768 in ViT), its influence is much smaller than the quadratic compression of the sequence length. Mathematical verification: Assume that d increases to 3 times, but n² decreases to 1 / 10 6 , and the total complexity is still reduced to 1 / 3×10 6 . For the generalization of the actual scenario, the target coverage rate α is usually small (such as α < 20%), then the optimization effect of this application on the complexity will be very significant in most tasks.

[0131] The method of this application compresses the input sequence length from O(h×w) to O(k) (k << h×w) through block classification and sequence recombination, reducing the time complexity and space complexity of the Transformer model to 1 / α²×(p² / (h×w))² of the traditional method, and can achieve a million-fold optimization under typical parameters. This feature is especially suitable for high-resolution image processing tasks.

[0132] In an exemplary embodiment, this application also provides an image data preprocessing device, including:

[0133] An image data set acquisition module, configured to acquire a corresponding image data set according to the task target of the object detection model; the task targets of the object detection model include human object detection, animal object detection, vehicle object detection, lesion area detection, obstacle detection, product defect detection, abnormal item detection, and pest and disease area detection; the corresponding image data sets include human image data sets, animal image data sets, vehicle image data sets, lesion image data sets, obstacle image data sets, product image data sets, item image data sets, and crop image data sets;

[0134] An image classification and screening module, configured to classify and screen various image data in the image data set according to the target types to be detected, and obtain a target type data set;

[0135] A data balance processing module, configured to perform data balance processing on the image data of each target type in the target type data set to obtain a balanced data set;

[0136] A feature dimension optimization module, configured to perform feature dimension optimization processing on each image data in the balanced data set to obtain an optimized feature vector;

[0137] The feature vector storage module is used to store the optimized feature vectors into the vector database for use in target detection model training.

[0138] In an exemplary embodiment, the present application also provides a computer device, which may be a server or a terminal. The computer device includes a processor, a memory, an input / output interface, and a communication interface. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the image data preprocessing method is implemented.

[0139] In an exemplary embodiment, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the image data preprocessing method when executed by a processor.

[0140] In an exemplary embodiment, the present application also provides a computer program product, including a computer program, which implements the image data preprocessing method when executed by a processor.

[0141] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by hardware associated with computer program instructions. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described method embodiments. Any reference to memory or other media in the various embodiments provided herein may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, and the like. Volatile memory may include random access memory (RAM) or external cache memory, and the like. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0142] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0143] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0144] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for preprocessing image data, characterized in that: include: Obtain the corresponding image dataset according to the task objectives of the target detection model; The task objectives of the target detection model include human object detection, animal object detection, vehicle object detection, lesion area detection, obstacle detection, product defect detection, abnormal object detection and pest area detection; the corresponding image datasets include human image datasets, animal image datasets, vehicle image datasets, lesion image datasets, obstacle image datasets, product image datasets, object image datasets and crop image datasets; Classify and filter various types of image data in the image dataset according to the target type to be detected to obtain a target type dataset; Perform data balancing on the image data of each target category in the target category dataset to obtain a balanced dataset; Perform feature dimension optimization on each image data in the balanced dataset to obtain the optimized feature vector; The feature dimension optimization process is performed on each image data in the balanced data set to obtain an optimized feature vector, specifically including: Each image data in the balanced dataset is divided into blocks, each of which is divided into multiple blocks of M rows × N columns; M and N are both integers greater than 1; Perform block-level classification on each image block, mark the image block containing the target object as 1, and mark the image block not containing the target object as 0, to obtain the labeled image block; Based on the labels of each image block, the multiple image blocks divided into each image data are screened and reorganized to obtain the optimized feature vector; The process of filtering and reorganizing the multiple image blocks into which each image data is segmented based on the labels of each image block to obtain an optimized feature vector specifically includes: Traverse each image data into multiple image blocks of M rows × N columns, and keep the image blocks with label 1 and discard the image blocks with label 0 in each row; Reorganize all image blocks with label 1 into a 1×X dimension feature matrix. After each row of image blocks is successfully reorganized, add the sequence start position marker [b]. After the last row of image blocks is reorganized, add the sequence end position marker [e] to obtain the optimized feature vector. Where X is the number of image blocks with label 1 in each image data. The optimized feature vector is stored in the vector database for use in target detection model training.

2. The image data preprocessing method according to claim 1, characterized in that: The method of classifying and screening various types of image data in the image dataset according to the types of targets to be detected to obtain the target type dataset specifically includes: The CLIP model is used to calculate the similarity between each image data in the image dataset and each target category, and the image data with similarity greater than the similarity threshold is retained to form the target category dataset.

3. The image data preprocessing method according to claim 2, characterized in that: The data balancing process is performed on the image data of each target category in the target category dataset to obtain a balanced dataset, specifically including: Determine whether the image data ratio between any two target categories in the target category dataset is balanced; If the ratio is balanced, the image data of each target category is compressed to the same category number Nmax; where Nmax is the number of images contained in the target category with the largest proportion after data compression; If the proportion is unbalanced, the image data of the target category with a smaller proportion will be enhanced, and the image data of the target category with a larger proportion will be compressed, and the image data of each target category will be enhanced or compressed to the same classification number Nmax.

4. The image data preprocessing method according to claim 3, characterized in that: The optimized feature vector is stored in the vector database for use in target detection model training, specifically including: The optimized feature vector is used as a feature data set and stored in the vector database; When training the target detection model, the feature dataset is divided into a training set, a validation set, and a test set for training, validating, and testing the target detection model.

5. A picture data preprocessing device, characterized in that: For implementing the image data preprocessing method according to any one of claims 1 to 4, the image data preprocessing device comprises: An image dataset acquisition module is used to acquire corresponding image datasets according to the task objectives of the target detection model; the task objectives of the target detection model include human object detection, animal object detection, vehicle object detection, lesion area detection, obstacle detection, product defect detection, abnormal object detection and pest area detection; the corresponding image datasets include human image datasets, animal image datasets, vehicle image datasets, lesion image datasets, obstacle image datasets, product image datasets, object image datasets and crop image datasets; The image classification and screening module is used to classify and screen various types of image data in the image dataset according to the target type to be detected, and obtain the target type dataset; A data balancing processing module is used to perform data balancing on the image data of each target category in the target category data set to obtain a balanced data set; The feature dimension optimization module is used to optimize the feature dimension of each image in the balanced data set to obtain the optimized feature vector; The feature vector storage module is used to store the optimized feature vectors into the vector database for use in target detection model training.

6. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image data preprocessing method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image data preprocessing method according to any one of claims 1 to 4 is implemented.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the image data preprocessing method according to any one of claims 1 to 4 is implemented.

Images