Image interpretation model training method, image interpretation method, electronic equipment and computer readable storage medium
By using sample sets and knowledge sets in the image interpretation model training process, the image interpretation model is trained, and used to update the knowledge set when difficult images are encountered, the problem of resource consumption and efficiency reduction caused by frequent training of models in the prior art is solved, and efficient and accurate image interpretation and model training are achieved.
Patent Information
- Application Number
- CN202411775227.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-05
- Publication Date
- 2025-04-18
AI Technical Summary
The prior art requires frequent training of models in image interpretation, resulting in increased resource consumption, increased model parameters, increased interpretation cost and reduced efficiency.
By obtaining the sample set and knowledge set, the image interpretation model is trained, the actual image is interpreted using the initial version model, the interpretation results are evaluated, and the difficult images are added to the knowledge set, and the training is cycled until the preset conditions are met, and a new version of the model is generated.
It realizes that without immediately retraining the model, it can accurately interpret difficult images, reduce model training and interpretation costs, and improve model interpretation ability and training efficiency.
Smart Images
Figure CN119851056A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing and artificial intelligence technology, and in particular to an image interpretation model training method, an image interpretation method, an electronic device and a computer-readable storage medium. Background Art
[0002] Remote sensing image interpretation usually uses an image interpretation model trained based on a training data set. In practical applications, whenever an image is encountered that the model is not good at processing, the model needs to be trained once with the corresponding sample image so that the model can learn the relevant knowledge in the image that it is not good at processing and optimize the model parameters before it can have the ability to interpret this type of image. Training the model once consumes resources, and frequent model training leads to an increase in the number of model parameters. In addition, the model cannot be used during training, which increases the cost of image interpretation and reduces efficiency. Summary of the invention
[0003] Therefore, it is necessary to provide an image interpretation model training method, an image interpretation method, an electronic device and a computer-readable storage medium to address the above technical problems.
[0004] In a first aspect, an embodiment of the present application provides an image interpretation model training method, the method comprising:
[0005] S1: Acquire a sample set and a knowledge set, wherein the sample set includes a plurality of sample images and a plurality of truth maps, the truth maps include truth labels indicating the categories of objects in the sample images, and the knowledge set includes a plurality of feature vector pairs, the feature vector pairs include positive feature vectors and negative feature vectors;
[0006] S2: using the sample set and the knowledge set to train the image interpretation model to obtain a preliminary version of the image interpretation model, wherein the image interpretation model includes an encoding unit, a matching unit, an enhancement unit, and a decoding unit;
[0007] S3: Acquire an actual image, and use the preliminary version of the image interpretation model to interpret the actual image under the guidance of the knowledge set to obtain an interpretation result, wherein the actual image is an image to be interpreted in an actual application;
[0008] S4: performing quality evaluation on the interpretation result to obtain a difficult image;
[0009] S5: using the difficult image to create a feature vector pair to be added to the knowledge set, and storing the difficult image in a difficult example set;
[0010] S6: looping through steps S3-S5 until a first preset condition is met, and then retraining the initial version of the image interpretation model using the difficult example set and the knowledge set to obtain a new version of the image interpretation model.
[0011] Further, the S3 includes:
[0012] S31: interpreting the actual image using the initial version image interpretation model and the knowledge set to obtain m1 actual prediction probability maps;
[0013] S32: Perform visualization processing on the m1 actual prediction probability maps to obtain m1 visualization results.
[0014] The S4 includes:
[0015] S41: performing a preliminary evaluation on the m1 visualization results to obtain a first evaluation result, and obtaining a candidate difficult image according to the first evaluation result;
[0016] S42: annotating and encoding the candidate difficult images to obtain m2 candidate truth maps, wherein the m2 candidate truth maps correspond to m2 types of ground feature categories, and the candidate truth maps contain candidate truth labels;
[0017] S43: converting the m1 actual prediction probability maps to obtain m1 actual prediction true value maps, wherein the m1 actual prediction true value maps correspond to m1 types of land feature categories, and the actual prediction true value maps contain actual prediction labels;
[0018] S44: Calculate quality evaluation scores according to the m1 types of ground object categories, the m2 types of ground object categories, the m1 actual predicted true value images, and the m2 candidate true value images, and obtain the difficult image according to the quality evaluation scores.
[0019] Furthermore, the quality evaluation score includes a first evaluation score and a second evaluation score, wherein the calculation formula of the first evaluation score is:
[0020]
[0021] Among them, m sim is the number of feature categories in the intersection of the m1 feature categories and the m2 feature categories, m sum is the number of the m1 type of land feature categories and the m2 type of land feature categories combined;
[0022] The second evaluation score calculation formula is:
[0023]
[0024] Among them, w x,j is the jth candidate truth label of the xth feature category, is the jth actual predicted label of the xth object category, M 3 is the intersection of the m1 type of ground object category and the m2 type of ground object category, j is an integer between 1 and n, and n is the number of pixels in the sample image;
[0025] The quality evaluation score calculation formula is:
[0026] S q =λ 1 S 1 +λ 2 S 2
[0027] Among them, λ 1 , 2 is the weighting coefficient, λ 1 , 2 ∈(0,1), and λ 1 +λ 2 =1.
[0028] The S2 includes:
[0029] S21: inputting the sample image into the image interpretation model, and the encoding unit performs feature encoding and feature fusion on the sample image to obtain a sample feature vector, a first fusion feature, and a second fusion feature;
[0030] S22: the matching unit performs vector matching using the sample feature vector and the knowledge set to obtain a prompt vector, wherein the prompt vector includes a plurality of feature vector pairs;
[0031] S23: the enhancement unit uses the first fusion feature to enhance the prompt vector to obtain an enhanced prompt vector;
[0032] S24: the decoding unit decodes the second fusion feature and the enhanced hint vector to obtain a plurality of probability maps for the sample image, wherein the probability maps include the predicted probabilities of the ground object categories;
[0033] S25: performing label smoothing processing on the true value label to obtain a smoothed label;
[0034] S26: Using a comprehensive loss function to calculate a comprehensive loss value according to the smoothed label and the predicted probability, and adjusting the parameters of the image interpretation model based on the comprehensive loss value, stopping the training when a preset end condition is reached, and obtaining the initial version of the image interpretation model.
[0035] Furthermore, the comprehensive loss function includes a category loss function and a region loss function, wherein the category loss function is:
[0036]
[0037] The regional loss function is:
[0038]
[0039] Among them, y i,j is the smoothed label of the i-th object category at the j-th pixel in the sample image, is the predicted probability of the i-th object category at the j-th pixel in the sample image, m is the total number of object categories, n is the number of pixels in the sample image, and β is the preservation coefficient;
[0040] The comprehensive loss function is:
[0041] L all =λL 1 +(1-λ)L 2
[0042] Among them, λ is the weighting coefficient, λ∈[0,1].
[0043] Furthermore, the first preset condition includes: the number of difficult images in the difficult example set reaches a certain number and / or the time interval from the last time the preliminary image interpretation model was obtained reaches a certain length.
[0044] In a second aspect, an embodiment of the present application provides an image interpretation method, the method comprising:
[0045] Get the target image;
[0046] Acquire knowledge sets;
[0047] Using the new version of the image interpretation model to interpret the target image under the guidance of the knowledge set, and obtaining an interpretation result for the target image;
[0048] Among them, the new version of the image interpretation model is trained according to the image interpretation model training method provided in the first aspect.
[0049] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it can execute the method of the first aspect.
[0050] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method of the first aspect can be executed.
[0051] The beneficial effects of the present invention include at least:
[0052] (1) By using the knowledge set to guide the initial image interpretation model to interpret the actual image, and when encountering a difficult image, the difficult image is used to create a feature vector pair to be added to the knowledge set to guide the initial interpretation model to continue interpreting the actual image, so that the initial image interpretation model does not need to be retrained immediately, and the accurate interpretation of the difficult image can be achieved, thereby reducing the model training cost and interpretation cost;
[0053] (2) By storing difficult images in a difficult example set and, after the first preset condition is met, using the difficult example set and the knowledge set to retrain the initial version of the image interpretation model, it is possible to reduce the number of model training times and reduce the training cost while accurately improving the model's interpretation ability and improving the model training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0055] Figure 1 A flowchart of the image interpretation model training method provided in an embodiment of the present application;
[0056] Figure 2 A schematic block diagram of the structure of an image interpretation model provided in an embodiment of the present application;
[0057] Figure 3 A schematic block diagram of the workflow of the encoding unit provided in the embodiment of the present application;
[0058] Figure 4 A flowchart of using the preliminary image interpretation model provided in an embodiment of the present application to interpret an actual image to obtain a difficult image;
[0059] Figure 5 The present invention provides a flowchart of an image interpretation method according to an embodiment of the present application. DETAILED DESCRIPTION
[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.
[0061] Figure 1 A flowchart of the image interpretation model training method provided in an embodiment of the present application is shown. Figure 1 As shown, the method includes:
[0062] S1: Acquire a sample set and a knowledge set, wherein the sample set includes a plurality of sample images and a plurality of truth maps, the truth maps include truth labels indicating the categories of objects in the sample images, and the knowledge set includes a plurality of feature vector pairs, the feature vector pairs include positive feature vectors and negative feature vectors;
[0063] S2: using the sample set and the knowledge set to train the image interpretation model to obtain a preliminary version of the image interpretation model, wherein the image interpretation model includes an encoding unit, a matching unit, an enhancement unit, and a decoding unit;
[0064] S3: Acquire an actual image, and use the preliminary version of the image interpretation model to interpret the actual image under the guidance of the knowledge set to obtain an interpretation result, wherein the actual image is an image to be interpreted in an actual application;
[0065] S4: performing quality evaluation on the interpretation result to obtain a difficult image;
[0066] S5: using the difficult image to create a feature vector pair to be added to the knowledge set, and storing the difficult image in a difficult example set;
[0067] S6: looping through steps S3-S5 until a first preset condition is met, and then retraining the initial version of the image interpretation model using the difficult example set and the knowledge set to obtain a new version of the image interpretation model.
[0068] The above steps are described in detail below in conjunction with the embodiments. First, step S1 is described in detail in conjunction with the embodiments.
[0069] According to an embodiment of the present invention, in step S1, a sample set is first obtained, the sample set includes multiple sample images and multiple true value maps, each of which contains multiple real labels indicating each ground object category in the sample image.
[0070] The sample set is pre-made using sample images, and the sample images are pre-processed remote sensing images. The remote sensing images can be three-band remote sensing images or four-band remote sensing images, including one or more types of ground objects.
[0071] The process of making a sample set includes: for a sample image, the multiple types of objects contained in it are respectively marked with masks, and a corresponding category identifier is set for each type of object category to obtain a labeling file, wherein the category identifier of the object category is represented by a positive integer such as 1, 2, ..., m, for example, category identifier 1 represents water area, and category identifier 2 represents forest land. Then, the true value map of the sample image is made using the sample image and the corresponding labeling file. Specifically, one-hot encoding is performed independently for each type of object category in the sample image to obtain multiple true value maps for the sample image, wherein one true value map corresponds to one type of object category, and the pixel points in the true value map are the true value labels of the object category. In the present invention, it is assumed that there are m types of total object categories, and m true value maps corresponding to m types of object categories are obtained for each sample image. It should be noted that if a certain object category does not exist in the sample image, the true value labels in the true value map corresponding to the object category are all 0.
[0072] In step S1, a knowledge set is also obtained, the knowledge set includes a plurality of feature vector pairs, the feature vector pairs include positive feature vectors and negative feature vectors. The knowledge set is pre-made according to the knowledge image, the knowledge image adopts a pre-processed remote sensing image, and the remote sensing image can be a three-band remote sensing image or a four-band remote sensing image.
[0073] The process of making a knowledge set includes: marking different object categories contained in the knowledge image with masks respectively, and setting different category identifiers for the mask areas corresponding to different object categories to obtain the marked knowledge image. Then, the marked knowledge image is divided into multiple image blocks, multiple image blocks within the mask range are selected according to the annotation information of a certain object category, and the Cls Token of these image blocks is extracted using the pre-trained Vision Transformer (ViT) network to obtain the positive feature vector corresponding to the certain object category. According to the annotation information of a certain object category, all image blocks not within the mask range are selected, and the Cls Token is extracted using the ViT network to obtain the corresponding negative feature vector. This pair of positive feature vectors and negative feature vectors is a feature vector pair, corresponding to the certain object category. Other object categories in the knowledge image and other knowledge images are processed in the same way to obtain multiple positive feature vectors and multiple negative feature vectors. It should be noted that in the knowledge set, the feature vector pair is associated with the category identifier of the corresponding object category and other attribute information and stored. In the initial stage, sample images can be used to make the knowledge image of the knowledge set. In one embodiment of the present invention, it is assumed that the total number of ground object categories of all sample images in the sample set and the total number of ground object categories in the knowledge set are both m.
[0074] The present invention constructs a knowledge set including multiple positive eigenvectors and multiple negative eigenvectors to provide the most relevant eigenvector as a visual cue vector to guide the interpretation when the image interpretation model is interpreting, so that the interpretation result is more accurate.
[0075] Next, the above step S2 is described in detail in conjunction with an embodiment.
[0076] Before step S2, an image interpretation model is first constructed. Figure 2 The image interpretation model provided in the embodiment of the present application is shown in a schematic block diagram of the structure, wherein the image interpretation model includes an encoding unit, a matching unit, an enhancement unit and a decoding unit.
[0077] In step S2, the image interpretation model is trained using the sample set and knowledge set obtained in step S1 to obtain a preliminary version of the image interpretation model. The specific steps include:
[0078] S21: Input the sample image into the image interpretation model, and the encoding unit performs feature encoding and feature fusion on the sample image to obtain a sample feature vector, a first fusion feature, and a second fusion feature.
[0079] According to one embodiment of the present invention, the encoding unit includes a dual-path feature extraction network consisting of a CNN network and a ViT network. The CNN network uses the ResNeSt50d network pre-trained by ImageNet. The ViT network can use the ViT network in the pre-trained self-distillation with no labels (DINO) model.
[0080] Figure 3 FIG. 4 shows a schematic block diagram of the working process of the encoding unit. Figure 3 As shown, the feature extraction and feature fusion processing of the sample image using the dual-path feature extraction network includes:
[0081] (1) Use the ImageNet pre-trained ResNeSt50d network to perform multi-scale feature extraction on the sample image to obtain the first sub-feature, the second sub-feature, the third sub-feature and the fourth sub-feature, and use the pre-trained ViT network to extract features from the sample image to obtain the sample feature vector and the fifth sub-feature.
[0082] Specifically, the sample image is input into the ResNeSt50d network, and after being processed by the STEM layer, it is input into the cascaded residual block (Residual Block). The residual block adopts the split-attention block. The four cascaded residual blocks extract features with scales of 1 / 4, 1 / 8, 1 / 16 and 1 / 32 times of the sample image respectively, and correspondingly, the first sub-feature c1, the second sub-feature c2, the third sub-feature c3 and the fourth sub-feature c4 are obtained.
[0083] At the same time, the sample image is processed by the Patch Embedding layer and added to the initial cls Token, which is input into the pre-trained ViT network. Multiple repeatedly stacked encoder blocks are used for feature extraction, for example 12. The final output scale is 1 / 16 times of the sample image, and the fifth sub-feature is obtained. At the same time, the Cls Token output by the ViT network is the sample feature vector fc.
[0084] (2) Using a feature fusion network, the fourth sub-feature c4, the third sub-feature c3 and the fifth sub-feature c5 are feature-fused to obtain a first fused feature f1, and the first fused feature f1, the second sub-feature c2 and the first sub-feature c1 are feature-fused to obtain the second fused feature f2.
[0085] Specifically, the feature fusion network (FFN) includes three sampling layers. The fourth sub-feature c4 is upsampled by 2 times and then fused with the third sub-feature c3 to obtain the fused third sub-feature, and the fused third sub-feature is transformed in channel dimension and then added to the fifth sub-feature c5 to obtain the first fused feature f1, wherein the scale of the first fused feature f1 is 1 / 16 of the sample image.
[0086] The first fused feature f1 is upsampled by 2 times and fused with the second sub-feature c2 to obtain the fused sub-feature fc, and the fused sub-feature fc is upsampled by 2 times and fused with the first sub-feature c1 to obtain the second fused feature f2, wherein the scale of the second fused feature f2 is 1 / 4 of the sample image.
[0087] The dual feature extraction network in the present invention extracts and fuses the multi-scale features of the sample image by using the pre-trained ResNeSt50d network and the ViT network in the DINO model, effectively extracts the depth features of the sample image and makes full use of the scale features, which can improve the segmentation accuracy of the ground object category in the interpretation process.
[0088] S22: The matching unit uses the sample feature vector to perform vector matching from the knowledge set to obtain a prompt vector, where the prompt vector includes a plurality of feature vector pairs.
[0089] According to an embodiment of the present invention, the matching unit performs vector matching by calculating the similarity between the sample feature vector and the positive feature vector in the knowledge set, and obtains the corresponding prompt vector P t . To achieve full-factor interpretation, all feature categories in the knowledge set are matched at the same time, that is, m feature categories are matched at the same time, and the top k positive feature vectors with the highest similarity are selected for each feature category. Specifically, for each feature category, the knowledge set is searched with the sample feature vector and the category identifier of the feature category as parallel conditions to obtain the top k positive feature vectors with the highest similarity, and k negative feature vectors are obtained by association. For m feature categories, the obtained m×k positive feature vectors and m×k negative feature vectors are used as prompt vectors.
[0090] It should be noted that for a certain feature category, if the k positive feature vectors and k negative feature vectors are not matched in the knowledge set, the missing positive feature vectors and negative feature vectors are filled with zero. In particular, if a certain feature category does not match any positive feature vector and negative feature vector, the corresponding k positive feature vectors and k negative feature vectors are defaulted to zero.
[0091] In this step, by using the sample feature vector extracted from the sample image, the prompt vector that best matches each feature category of the sample image can be matched from the knowledge set, thereby achieving the effect of accurately segmenting each feature category in the sample image.
[0092] S23: The enhancement unit uses the first fusion feature to enhance the prompt vector to obtain an enhanced prompt vector.
[0093] Specifically, the enhancement unit uses a cross attention network. The prompt vector is mapped to the query vector (Query) of the cross attention network, the first fusion feature f1 is mapped to the key vector (Key) and the value vector (Value), and the Query is continuously updated using the cross attention mechanism, while the Key and Value remain unchanged, to obtain an enhanced prompt vector.
[0094] S24: The decoding unit decodes the second fusion feature and the enhanced hint vector to obtain a plurality of probability maps corresponding to the sample image, wherein the plurality of probability maps contain prediction probabilities corresponding to the plurality of ground object categories respectively.
[0095] Specifically, the decoding unit performs the second fusion feature f2 and the enhanced prompt vector P' tPerform matrix multiplication in the feature dimension to obtain m×k positive feature maps and m×k negative feature maps, where each feature category corresponds to k positive feature maps and k negative feature maps. Average the k positive feature maps and k negative feature maps of each feature category to obtain the corresponding positive final feature map and negative final feature map. Then use the softmax function to normalize the positive final feature map and the negative final feature map to obtain m probability maps corresponding to m feature categories, and the probability map contains the predicted probability. In addition, the predicted label can be further obtained based on the predicted probability, including the positive predicted label and the negative predicted label.
[0096] It should be noted that if a certain feature category does not match any positive eigenvector or negative eigenvector in the knowledge set, the feature category will be ignored and will not participate in the probability map and subsequent loss calculation.
[0097] S25: Perform label smoothing on the true value label to obtain a smoothed label.
[0098] Label smoothing helps prevent the model from overfitting during training.
[0099] S26: Using a comprehensive loss function to calculate the comprehensive loss value of the image interpretation model according to the smoothed label and the predicted probability, adjusting the parameters of the image interpretation model based on the comprehensive loss value, and stopping the training when a preset end condition is reached to obtain the initial version of the image interpretation model.
[0100] The comprehensive loss function includes the category loss function and the region loss function. The category loss function is:
[0101]
[0102] Among them, y i,j is the smoothed label of the i-th object category at the j-th pixel in a sample image, is the predicted probability of the i-th object category at the j-th pixel in the sample image, m is the number of object categories, and n is the number of pixels in the sample image.
[0103] The category loss function can be used to measure the accuracy of the model's prediction of the object category.
[0104] The regional loss function calculation formula is:
[0105]
[0106] Among them, β is the preservation coefficient, which has a very small value and is used to prevent the denominator from being zero.
[0107] The regional loss function can improve the model's prediction accuracy of the boundary position, making the interpretation result closer to the actual labeled area.
[0108] According to the above formula, the comprehensive loss function is:
[0109] L all =λL 1 +(1-λ)L 2
[0110] Wherein, λ is a weighting coefficient, λ∈[0,1], and its value can be set as needed, for example, set to 0.8.
[0111] The present invention can take into account both the accuracy of the model's prediction of the ground object category and the accuracy of the model's prediction of the boundary position by using the category loss function and the area loss function.
[0112] During the training process, the model parameters are adjusted through optimization algorithms such as gradient descent. When the preset end condition is reached, the training is stopped to obtain the initial version of the image interpretation model. The preset end condition includes but is not limited to the model loss value being less than or equal to the preset threshold, the number of iterations reaching the preset number, etc.
[0113] Next, combine Figure 4 The flowchart shown in the figure for interpreting the actual image using the preliminary image interpretation model to obtain the difficult image describes the above steps S3-S5 in detail.
[0114] like Figure 4 As shown, step S3 includes:
[0115] S31: Use the initial version of the image interpretation model and the knowledge set to interpret the actual image to obtain m1 actual prediction probability maps.
[0116] Specifically, the actual image is input into the initial version of the image interpretation model for interpretation, and m1 actual prediction probability maps are output. The pixel points in the actual prediction probability map are the actual prediction probability values of the actual image at the corresponding pixel points, where m1 is not greater than m. Considering that some ground object categories may not be interpreted, the actual ground object category m1 interpreted does not exceed the total number of ground object categories m.
[0117] S32: Perform visualization processing on the m1 actual prediction probability maps to obtain m1 visualization results.
[0118] The visualization result is obtained by running the visualization processing script. The visualization result is an image in which the segmentation mask area and the ground object category identification are marked on the actual image. The corresponding m1 visualization results are obtained from the m1 actual prediction probability maps.
[0119] In step S4, the quality evaluation of the interpretation result includes:
[0120] S41: Preliminarily evaluate the m1 visualization results to obtain a first evaluation result, and obtain a candidate difficult image according to the first evaluation result.
[0121] The visualization results are manually checked and the quality of the visualization results is evaluated, including whether the classification of each feature category in the m1 visualization results is correct, whether the location area division of each feature category is accurate, whether the boundaries between the masks of different feature categories are clear, etc., to obtain the first evaluation result. The first evaluation result includes good quality and poor quality. The actual image corresponding to the visualization result with poor quality in the first evaluation result is used as a candidate difficult image. If the first evaluation result is good quality, the corresponding actual image can be considered as one that the model is good at processing, and no further judgment is made.
[0122] S42: annotating and encoding the candidate difficult images to obtain m2 candidate truth maps, wherein the m2 candidate truth maps correspond to m2 types of land feature categories, and the candidate truth maps contain candidate truth labels.
[0123] All the object categories in the candidate difficult image are marked with masks and each object category is one-hot encoded to obtain m2 candidate truth maps corresponding to m2 object categories, where the candidate truth maps contain candidate truth labels, and m2 is less than or equal to m.
[0124] S43: converting the m1 actual prediction probability maps to obtain m1 actual prediction true value maps, wherein the m1 actual prediction true value maps correspond to m1 types of land feature categories, and the actual prediction true value maps contain actual prediction labels.
[0125] The conversion process converts the actual predicted probability value into 0 or 1, for example, converting the actual predicted probability value less than 0.5 into 0 or 1, and converting the actual predicted probability value greater than or equal to 0.5 into 1.
[0126] S44: Calculate quality evaluation scores according to the m1 types of ground object categories, the m2 types of ground object categories, the m1 actual predicted true value images, and the m2 candidate true value images, and obtain the difficult image according to the quality evaluation scores.
[0127] Specifically, calculate the first evaluation score:
[0128]
[0129] Among them, m sim is the number of feature categories in the intersection of m1 feature categories and m2 feature categories, m sumIt is the number of m1 types of land feature categories and m2 types of land feature categories combined.
[0130] Calculate the second evaluation score:
[0131]
[0132] Among them, w x,j is the jth candidate truth label of the xth feature category, is the jth actual predicted label of the xth object category, M 3 is the intersection of the m1 type of ground object category and the m2 type of ground object category, j is an integer between 1 and n, and n is the number of pixels in the sample image.
[0133] The quality evaluation score is obtained by the first evaluation score and the second evaluation score:
[0134] S q =λ 1 S 1 +λ 2 S 2
[0135] Among them, λ 1 , 2 is the weighting coefficient, λ 1 , 2 ∈(0,1), and λ 1 +λ 2 =1, its value can be set according to actual needs, for example, 1 Set to 0.4, λ 2 Set to 0.6.
[0136] It is determined whether the quality evaluation score is less than a threshold quality score. If it is less than the threshold quality score, the quality of the interpretation result of the corresponding candidate difficult image is considered to be poor, and the corresponding candidate difficult image is determined to be a difficult image. The threshold quality score can be obtained by statistically analyzing the quality evaluation scores of a large number of high-quality interpretation results in actual applications.
[0137] In an embodiment of the present invention, the quality of the interpretation results is evaluated by combining manual evaluation with evaluation using quality evaluation scores. The dual evaluation is more efficient and accurate. The difficult images selected using the quality evaluation scores are more difficult and more typical. Regardless of whether the difficult images are used for knowledge concentration or for subsequent retraining of the model, the interpretation ability of the model can be better optimized.
[0138] Continue to see Figure 4 , in step S5, a new knowledge vector pair is created based on the difficult image obtained in steps S3-S4 and added to the knowledge set, and the difficult image is stored in the difficult example set.
[0139] The method of constructing a knowledge vector pair using a difficult image is the same as the method in the embodiment of constructing a knowledge set, including annotating the difficult image with a mask and using the pre-trained ViT to obtain positive feature vectors and negative feature vectors. For details, please refer to the relevant steps of the embodiment of constructing a knowledge set, which will not be repeated here. The knowledge vector pair is added to the knowledge set, and the knowledge set at this time contains the knowledge vector pair obtained from the difficult image. The difficult example set is used to store multiple difficult images obtained in a cycle.
[0140] Based on an embodiment of the present invention, in step S6, steps S3-S5 are executed cyclically, which means that the primary version interpretation model is continuously used to interpret multiple actual images over a period of time, and difficult images are continuously collected during this process, and new feature vector pairs are made from the difficult images in real time and added to the knowledge base to guide the primary version interpretation model to interpret the actual images, so that the model can have good interpretation capabilities without retraining the model within a certain period of time. The first preset condition includes but is not limited to a certain number of difficult images in the difficult example set and / or a certain time interval from the last training of the primary version image interpretation model.
[0141] The method of retraining the initial version of the image interpretation model using the latest difficult example set and the latest knowledge set is the same as the method of training the image interpretation model using the sample set and the knowledge set. The relevant steps in the embodiment of training the image interpretation model can be referred to, and will not be repeated here. It should be noted that the new version of the image interpretation model refers to the initial version of the image interpretation model that is considered to be trained at the current stage.
[0142] In an embodiment of the present invention, a preliminary image interpretation model is first obtained by training an image interpretation model, and the model is applied in practice. When the preliminary image interpretation model is used to interpret an actual image, a knowledge set is used to guide the preliminary image interpretation model to interpret the actual image. When a difficult image is encountered, the difficult image is used to make a feature vector pair and added to the knowledge set to guide the preliminary interpretation model to continue interpreting the actual image, so that accurate interpretation of difficult images can be achieved without immediately retraining the preliminary image interpretation model, thereby reducing model training costs and interpretation costs.
[0143] In addition, by storing difficult images in a difficult example set, when the model needs to be retrained, the difficult example set and the knowledge set are used to retrain the initial version of the image interpretation model. This can reduce the number of model training times and reduce training costs while accurately improving the model's interpretation capabilities and improving model training efficiency.
[0144] Figure 5 Flow chart of the image interpretation method provided in the embodiment of the present application. Figure 5 As shown in , the method may include the following steps:
[0145] S501: Acquire a target image;
[0146] S502: Acquire a knowledge set;
[0147] S503: using the new version of the image interpretation model to interpret the target image under the guidance of the knowledge set, and obtaining a target interpretation result for the target image;
[0148] The new version of the image interpretation model is trained according to any of the methods described above.
[0149] An embodiment of the present invention further provides an electronic device, the electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of any one of the methods in the aforementioned method embodiments when executing the program. Further, the electronic device further comprises: at least one input device; at least one output device. The above-mentioned memory, processor, input device and output device are connected via a bus. Among them, the input device may specifically be a camera, a touch panel, a physical button or a mouse, etc. The output device may specifically be a display screen. The memory may be a high-speed random access memory (RAM) memory, or a non-volatile memory (non-volatile memory), such as a disk memory. The memory is used to store a set of executable program codes, and the processor is coupled to the memory.
[0150] The embodiment of the present invention also provides a computer-readable storage medium, which may be disposed in the electronic device in each of the above embodiments, and the computer-readable storage medium may be the electronic device in the aforementioned embodiments. The computer-readable storage medium stores a computer program, and when the program is executed by the processor, the steps of the method described in any one of the aforementioned method embodiments are implemented. Furthermore, the computer-storable medium may also be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes.
[0151] It should be noted that the functional modules in the various embodiments of the present invention can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product.
[0152] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.
[0153] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0154] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for training an image interpretation model, characterized in that: The method comprises: S1: Acquire a sample set and a knowledge set, wherein the sample set includes a plurality of sample images and a plurality of truth maps, the truth maps include truth labels indicating the categories of objects in the sample images, and the knowledge set includes a plurality of feature vector pairs, the feature vector pairs include positive feature vectors and negative feature vectors; S2: using the sample set and the knowledge set to train the image interpretation model to obtain a preliminary version of the image interpretation model, wherein the image interpretation model includes an encoding unit, a matching unit, an enhancement unit, and a decoding unit; S3: Acquire an actual image, and use the preliminary version of the image interpretation model to interpret the actual image under the guidance of the knowledge set to obtain an interpretation result, wherein the actual image is an image to be interpreted in an actual application; S4: performing quality evaluation on the interpretation result to obtain a difficult image; S5: using the difficult image to create a feature vector pair to be added to the knowledge set, and storing the difficult image in a difficult example set; S6: looping through steps S3-S5 until a first preset condition is met, and then retraining the initial version of the image interpretation model using the difficult example set and the knowledge set to obtain a new version of the image interpretation model.
2. The method according to claim 1, characterized in that The S3 includes: S31: interpreting the actual image using the initial version image interpretation model and the knowledge set to obtain m1 actual prediction probability maps; S32: Perform visualization processing on the m1 actual prediction probability maps to obtain m1 visualization results.
3. The method according to claim 2, characterized in that The S4 includes: S41: performing a preliminary evaluation on the m1 visualization results to obtain a first evaluation result, and obtaining a candidate difficult image according to the first evaluation result; S42: labeling and encoding the candidate difficult images to obtain m2 candidate truth maps, wherein the m2 candidate truth maps correspond to m2 types of ground feature categories, and the candidate truth maps include candidate truth labels; S43: converting the m1 actual prediction probability maps to obtain m1 actual prediction true value maps, wherein the m1 actual prediction true value maps correspond to m1 types of land feature categories, and the actual prediction true value maps contain actual prediction labels; S44: Calculate quality evaluation scores according to the m1 types of ground object categories, the m2 types of ground object categories, the m1 actual predicted true value images, and the m2 candidate true value images, and obtain the difficult image according to the quality evaluation scores.
4. The method according to claim 3, characterized in that The quality evaluation score includes a first evaluation score and a second evaluation score, wherein the calculation formula of the first evaluation score is: Among them, m sim is the number of feature categories in the intersection of the m1 feature categories and the m2 feature categories, m sum is the number of the m1 type of land feature categories and the m2 type of land feature categories combined; The second evaluation score calculation formula is: Among them, w x,j is the jth candidate truth label of the xth feature category, is the jth actual predicted label of the xth ground object category, M3 is the intersection of the m1th ground object category and the m2th ground object category, j is an integer between 1 and n, and n is the number of pixels in the sample image; The quality evaluation score calculation formula is: S q =λ1S1+λ2S2 Among them, λ1 and λ2 are weighting coefficients, λ1, λ2∈(0,1), and λ1+λ2=1.
5. The method according to claim 1, characterized in that The S2 includes: S21: inputting the sample image into the image interpretation model, and the encoding unit performs feature encoding and feature fusion on the sample image to obtain a sample feature vector, a first fusion feature, and a second fusion feature; S22: the matching unit performs vector matching using the sample feature vector and the knowledge set to obtain a prompt vector, wherein the prompt vector includes a plurality of feature vector pairs; S23: the enhancement unit uses the first fusion feature to enhance the prompt vector to obtain an enhanced prompt vector; S24: the decoding unit decodes the second fusion feature and the enhanced hint vector to obtain a plurality of probability maps for the sample image, wherein the probability maps include the predicted probabilities of the ground object categories; S25: performing label smoothing processing on the true value label to obtain a smoothed label; S26: Using a comprehensive loss function to calculate a comprehensive loss value according to the smoothed label and the predicted probability, and adjusting the parameters of the image interpretation model based on the comprehensive loss value, stopping the training when a preset end condition is reached, and obtaining the initial version of the image interpretation model.
6. The method according to claim 5, characterized in that The comprehensive loss function includes a category loss function and a region loss function, wherein the category loss function is: The regional loss function is: Among them, y i,j is the smoothed label of the i-th object category at the j-th pixel in the sample image, is the predicted probability of the i-th object category at the j-th pixel in the sample image, m is the total number of object categories, n is the number of pixels in the sample image, and β is the preservation coefficient; The comprehensive loss function is: L all =λL1+(1-λ)L2 Among them, λ is the weighting coefficient, λ∈[0,1].
7. The method according to claim 1, characterized in that The first preset condition includes: the number of difficult images in the difficult example set reaches a certain number and / or the time interval from the last time the initial version of the image interpretation model was obtained reaches a certain length.
8. An image interpretation method, characterized in that: The method comprises: Get the target image; Acquire knowledge sets; Using the new version of the image interpretation model to interpret the target image under the guidance of the knowledge set, and obtaining an interpretation result for the target image; The new version of the image interpretation model is trained according to the method described in any one of claims 1 to 7.
9. An electronic device, characterized in that: The electronic device comprises: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.