Model training method and device, computer equipment and computer readable storage medium

Abstract

The application relates to a model training method and device, computer equipment and a computer readable storage medium. The method comprises: obtaining multiple training images and multiple candidate images; inputting the training images into an initial semantic segmentation model to obtain a first prediction matrix; inputting the candidate images into the initial semantic segmentation model to obtain a second prediction matrix; determining the prediction difference degree of the candidate images based on the second prediction matrix corresponding to the candidate images; determining the feature similarity between the candidate images and the multiple training images based on the second prediction matrix corresponding to the candidate images and the first prediction matrix corresponding to the training images; determining a target image based on the prediction difference degree and the feature similarity of each candidate image; and training the initial semantic segmentation model based on the target image to obtain a target semantic segmentation model. The method can improve the model training efficiency.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model training method, apparatus, computer device, and computer-readable storage medium. Background Technology

[0002] With the development of computer technology, neural network models are widely used for semantic segmentation of images. Semantic segmentation refers to classifying pixels in an image according to image categories. In order to make the output results of the neural network model more accurate, the neural network model needs to be trained before use.

[0003] In traditional techniques, a large number of images are manually labeled to obtain training images, which are then used to train a neural network model to obtain a neural network model with high accuracy. However, since manual labeling takes a lot of time, the training efficiency of the neural network model is low. Summary of the Invention

[0004] Therefore, it is necessary to provide a model training method, apparatus, computer equipment, computer-readable storage medium, and computer program product to address the aforementioned technical problems and improve model training efficiency.

[0005] Firstly, this application provides a model training method, including:

[0006] Acquire multiple training images and multiple candidate images;

[0007] The training image is input into the initial semantic segmentation model to obtain the first prediction matrix; the candidate image is input into the initial semantic segmentation model to obtain the second prediction matrix.

[0008] Based on the second prediction matrix corresponding to the candidate image, the prediction difference of the candidate image is determined;

[0009] Based on the second prediction matrix corresponding to the candidate image and the first prediction matrix corresponding to the training image, determine the feature similarity between the candidate image and multiple training images.

[0010] The target image is determined based on the predicted difference and feature similarity of each candidate image;

[0011] Based on the target image, the initial semantic segmentation model is trained to obtain the target semantic segmentation model.

[0012] Secondly, this application also provides a model training apparatus, comprising:

[0013] The acquisition module is used to acquire multiple training images and multiple candidate images;

[0014] The input module is used to input training images into the initial semantic segmentation model to obtain the first prediction matrix; and to input candidate images into the initial semantic segmentation model to obtain the second prediction matrix.

[0015] The first determining module is used to determine the prediction difference of the candidate image based on the second prediction matrix corresponding to the candidate image.

[0016] The second determining module is used to determine the feature similarity between a candidate image and multiple training images based on the second prediction matrix corresponding to the candidate image and the first prediction matrix corresponding to the training image.

[0017] The selection module is used to determine the target image based on the predicted difference and feature similarity of each candidate image;

[0018] The training module is used to train the initial semantic segmentation model based on the target image to obtain the target semantic segmentation model.

[0019] Thirdly, this application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described model training method.

[0020] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described model training method.

[0021] Fifthly, this application also provides a computer program product comprising a computer program that, when executed by a processor, implements the steps of the above-described model training method.

[0022] The aforementioned model training method, apparatus, computer equipment, and computer-readable storage medium acquire multiple training images and multiple candidate images. The labeled training images are input into an initial semantic segmentation model to obtain a first prediction matrix. Unlabeled candidate images are input into the initial semantic segmentation model to obtain a second prediction matrix. Based on the second prediction matrix corresponding to the candidate images, the prediction difference degree of the initial semantic segmentation model for the candidate images is determined, i.e., the accuracy of the initial semantic segmentation model's predictions for the candidate images is determined. A larger prediction difference degree indicates lower accuracy of the initial semantic segmentation model's predictions for the candidate images, and the greater the uncertainty of the candidate image for the initial semantic segmentation model. The stronger the similarity, the more the candidate image and multiple training images are compared. Based on the second prediction matrix corresponding to the candidate image and the first prediction matrix corresponding to the training image, the feature similarity between the candidate image and multiple training images is determined. The lower the feature similarity, the greater the difference between the candidate image and multiple training images. Based on the prediction difference and feature similarity of each candidate image, the target image is determined. This can be understood as selecting the target image with low prediction accuracy of the initial semantic segmentation model and large difference from multiple training images. Using a small number of target images to train the initial semantic segmentation model can achieve the effect of using all candidate images to train the initial semantic segmentation model, thereby improving the training efficiency of the model. Attached Figure Description

[0023] Figure 1 An application environment diagram of a model training method provided in this application embodiment;

[0024] Figure 2 A schematic flowchart illustrating a model training method provided in an embodiment of this application;

[0025] Figure 3 A flowchart illustrating a feature similarity determination step provided in an embodiment of this application;

[0026] Figure 4 A flowchart illustrating a feature matrix determination step provided in an embodiment of this application;

[0027] Figure 5 A schematic diagram of a framework for determining a feature matrix provided in an embodiment of this application;

[0028] Figure 6 A flowchart illustrating a target image determination step provided in an embodiment of this application;

[0029] Figure 7 A structural block diagram of a model training device provided in an embodiment of this application;

[0030] Figure 8 An internal structural diagram of a computer device provided in an embodiment of this application;

[0031] Figure 9 This is an internal structural diagram of a computer-readable storage medium provided in an embodiment of this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0033] The model training method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on the cloud or other network servers. Both the terminal and the server can be used independently to execute the model training method provided in this embodiment. The terminal and server can also work together to execute the model training method provided in this embodiment. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0034] In some embodiments, such as Figure 2 As shown, a model training method is provided, which can be applied to a computer device, which can be a terminal or a server. The method includes steps 202 to 212.

[0035] Step 202: Obtain multiple training images and multiple candidate images.

[0036] In this context, training images refer to images where each pixel is labeled with an image category, used to train the semantic segmentation model. In other words, training images are labeled images, and multiple training images can reside in the training set. Candidate images are images without image category labeling. Target images are selected from multiple candidate images to train the initial semantic segmentation model; multiple candidate images can reside in the candidate set.

[0037] For example, a computer device acquires multiple training images from a training set and multiple candidate images from a candidate set.

[0038] Step 204: Input the training image into the initial semantic segmentation model to obtain the first prediction matrix; input the candidate image into the initial semantic segmentation model to obtain the second prediction matrix.

[0039] The initial semantic segmentation model refers to a neural network model trained using multiple training images. It is used to predict the probability of an image category corresponding to a pixel in the image. For example, the image to be predicted is input into the initial semantic segmentation model for processing, and the output is a prediction matrix. This prediction matrix is ​​a three-dimensional matrix, where each element represents the probability of an image category corresponding to a pixel. The first prediction matrix refers to the output result of the initial semantic segmentation model predicting the training image. The second prediction matrix refers to the output result of the initial semantic segmentation model predicting the candidate image.

[0040] Step 206: Determine the prediction difference of the candidate image based on the second prediction matrix corresponding to the candidate image.

[0041] The prediction discrepancy refers to the difference between the probabilities that the initial semantic segmentation model predicts a candidate image as belonging to different image categories. For example, if the initial semantic segmentation model predicts a candidate image as belonging to image category A with a 20% probability and as belonging to image category B with a 35% probability, then the prediction discrepancy of the candidate image is 15%.

[0042] For example, the computer device determines the average probability of each image category based on the probability values ​​of each pixel in the same image category in the second prediction matrix corresponding to the candidate image, and determines the prediction difference of the candidate image based on the average probability of each image category.

[0043] Step 208: Based on the second prediction matrix corresponding to the candidate image and the first prediction matrix corresponding to the training image, determine the feature similarity between the candidate image and multiple training images.

[0044] Feature similarity refers to the minimum similarity between a candidate image and multiple training images.

[0045] For example, the computer device calculates the cosine similarity between the second prediction matrix corresponding to the candidate image and the first prediction matrix corresponding to the training image to obtain the similarity between the candidate image and the training image; based on the multiple similarities between the candidate image and multiple training images, the feature similarity between the candidate image and multiple training images is determined.

[0046] Step 210: Determine the target image based on the predicted difference and feature similarity of each candidate image.

[0047] In this context, the target image refers to an image selected from multiple candidate images, for which the initial semantic segmentation model has low prediction accuracy and low similarity to the multiple training images; the target image is used to further train the initial semantic segmentation model. It can be understood that the target image is a candidate image that satisfies uncertainty and diversity.

[0048] For example, a computer device obtains candidate evaluation values ​​for candidate images based on the predicted difference and feature similarity of candidate images, compares the candidate evaluation values ​​corresponding to multiple candidate images, and determines the target image.

[0049] In some embodiments, the computer device subtracts the feature similarity from the predicted difference of the candidate image to obtain the candidate evaluation value of the candidate image, compares the candidate evaluation value with the preset evaluation value, and determines the candidate image corresponding to the candidate evaluation value that is greater than the preset evaluation value as the target image.

[0050] Step 212: Based on the target image, train the initial semantic segmentation model to obtain the target semantic segmentation model.

[0051] For example, a computer device uses the target image to train an initial semantic segmentation model until a preset training stopping condition is met, thus obtaining the target semantic segmentation model.

[0052] In the above model training method, multiple training images and multiple candidate images are acquired. The labeled training images are input into the initial semantic segmentation model to obtain the first prediction matrix. The unlabeled candidate images are input into the initial semantic segmentation model to obtain the second prediction matrix. Based on the second prediction matrix corresponding to the candidate images, the prediction difference of the initial semantic segmentation model for the candidate images is determined, that is, the accuracy of the initial semantic segmentation model's prediction for the candidate images is determined. The greater the prediction difference, the lower the accuracy of the initial semantic segmentation model's prediction for the candidate images, and the stronger the uncertainty of the candidate images for the initial semantic segmentation model. Based on the second prediction matrix corresponding to the candidate images and the first prediction matrix corresponding to the training images, the feature similarity between the candidate images and multiple training images is determined. The lower the feature similarity, the greater the difference between the candidate images and multiple training images. Based on the prediction difference and feature similarity of each candidate image, the target image is determined. This can be understood as selecting the target images with low prediction accuracy of the initial semantic segmentation model and large differences from multiple training images. Using a small number of target images to train the initial semantic segmentation model can achieve the effect of using all candidate images to train the initial semantic segmentation model, thereby improving the training efficiency of the model.

[0053] In some embodiments, determining the prediction dissimilarity of candidate images based on a second prediction matrix corresponding to the candidate images includes:

[0054] Based on the second prediction matrix corresponding to the candidate image, the average probability corresponding to each image category is determined; the second prediction matrix is ​​a three-dimensional matrix, and the element values ​​in the second prediction matrix represent the probability of the image category corresponding to the pixel in the candidate image;

[0055] The first average probability and the second average probability are selected from the average probabilities corresponding to each image category;

[0056] The difference between the first average probability and the second average probability is determined as the prediction difference of the candidate image.

[0057] In this context, image category refers to the class to which a pixel in an image belongs. For example, a pixel located inside an apple in a candidate image corresponds to the image category "apple." The initial semantic segmentation model predicts the probabilities of multiple image categories corresponding to each pixel, and the output prediction matrix contains the predicted probabilities of each pixel corresponding to each image category. The average probability refers to the average probability of all pixels in the candidate image corresponding to a single image category; it can be understood as the average probability of each pixel in the candidate image corresponding to the same image category. The three-dimensional matrix is ​​a matrix composed of two dimensions: the image category dimension and the pixel position dimension. For example, the three dimensions of a three-dimensional matrix are the image category dimension, the pixel X-axis coordinate dimension, and the pixel Y-axis coordinate dimension.

[0058] For example, the computer device, based on the second prediction matrix corresponding to the candidate image, obtains the probability of each pixel in the candidate image corresponding to the corresponding image category for each image category, calculates the average probability of each pixel corresponding to the corresponding image category, and obtains the average probability corresponding to the image category; compares the average probabilities corresponding to each image category in the second prediction matrix, determines the largest average probability as the first average probability, determines the second largest average probability as the second average probability, subtracts the second average probability from the first average probability to obtain the probability difference, and determines the probability difference as the prediction difference degree of the candidate image.

[0059] In some embodiments, the formula for calculating the predicted dissimilarity is as follows:

[0060] margin=max1(probaility)-max2(probaility) formula (1)

[0061] Here, margin refers to the prediction difference, probabilism refers to the average probability of each image category, max1 refers to the maximum probability value, and max2 refers to the second largest probability value.

[0062] In this embodiment, the prediction difference of the initial semantic segmentation model for the candidate image is determined by the second prediction matrix corresponding to the candidate image. The image category corresponding to the first average probability is the most likely image category of the candidate image predicted by the initial semantic segmentation model, and the image category corresponding to the second average probability is the second most likely image category of the candidate image predicted by the initial semantic segmentation model. The prediction difference determined based on the difference between the first average probability and the second average probability characterizes the uncertainty of the initial semantic segmentation model's prediction of the candidate image. The higher the prediction difference, the stronger the uncertainty of the initial semantic segmentation model's prediction of the candidate image. The target image is determined based on the prediction difference, and the target image is used as the training image. That is, the uncertainty of the training image is considered. Using candidate images with stronger uncertainty to train the initial semantic segmentation model helps to improve the training effect of the initial semantic segmentation model.

[0063] In some embodiments, such as Figure 3 As shown, based on the second prediction matrix corresponding to the candidate image and the first prediction matrix corresponding to the training image, the feature similarity between the candidate image and multiple training images is determined, including:

[0064] Step 302: Based on the prediction matrix, perform feature extraction on the image to obtain the feature matrix corresponding to the image; the prediction matrix includes a first prediction matrix corresponding to the training image and a second prediction matrix corresponding to the candidate image; when the prediction matrix is ​​the first prediction matrix, the first feature matrix corresponding to the training image is obtained; when the prediction matrix is ​​the second prediction matrix, the second feature matrix corresponding to the candidate image is obtained.

[0065] The prediction matrix refers to the output matrix of the initial semantic segmentation model. The feature matrix is ​​a two-dimensional matrix used to characterize image features.

[0066] For example, the computer device performs feature extraction on the training image based on the first prediction matrix to obtain a first feature matrix corresponding to the training image; and performs feature extraction on the candidate image based on the second prediction matrix to obtain a second feature matrix corresponding to the candidate image.

[0067] Step 304: Based on the second feature matrix and the first feature matrix, determine the similarity between the candidate image and the training image.

[0068] Similarity refers to the degree of similarity between two datasets. Similarity metrics can include cosine similarity and Pearson correlation coefficient, among others.

[0069] For example, the computer device calculates the cosine similarity between the second feature matrix and the first feature matrix, and determines the cosine similarity as the similarity between the candidate image and the training image.

[0070] In some embodiments, the similarity calculation method between candidate images and training images is as follows:

[0071]

[0072] Where cosine similarity refers to the cosine similarity, A refers to the first feature matrix, B refers to the second feature matrix, i is the element index, n refers to the total number of element values, and A i B refers to the value of the i-th element in the first characteristic matrix. i It refers to the value of the i-th element in the second characteristic matrix.

[0073] Step 306: The smallest similarity among multiple similarities is determined as the feature similarity between the candidate image and multiple training images, and multiple similarities correspond one-to-one with multiple training images.

[0074] For example, for each candidate image, the computer device compares the similarity between the candidate image and each training image, determines the minimum similarity, and sets the minimum similarity as the feature similarity between the candidate image and multiple training images.

[0075] In this embodiment, the minimum similarity is determined as the feature similarity between the candidate image and multiple training images. The smaller the feature similarity, the smaller the similarity between the candidate image and multiple training images, that is, the greater the difference between the candidate image and multiple training images. The target image is determined based on the feature similarity and used as the training image. This takes into account the diversity of training images. Using diverse candidate images to train the initial semantic segmentation model helps to improve the training effect of the initial semantic segmentation model.

[0076] In some embodiments, such as Figure 4 As shown, based on the prediction matrix, feature extraction is performed on the image to obtain the corresponding feature matrix, including:

[0077] Step 402: Determine the label matrix corresponding to the image based on the prediction matrix.

[0078] The label matrix is ​​a three-dimensional matrix used to represent the image category corresponding to each pixel in the image. For example, in the feature matrix, among the multiple image category elements corresponding to each pixel, only one element value is different from the others; that is, the image category of that pixel is the image category corresponding to the different element value.

[0079] For example, a computer device determines the target image category of a pixel based on the probability of different image categories corresponding to the same pixel in the prediction matrix, and determines the label matrix corresponding to the image based on the target image category of each pixel.

[0080] Step 404: Perform dimensionality reduction transformation on the prediction matrix to obtain the prediction dimensionality reduction matrix; perform dimensionality reduction transformation on the label matrix to obtain the label dimensionality reduction matrix.

[0081] Dimensionality reduction refers to the process of reducing the dimension of a matrix. For example, if the prediction matrix is ​​a three-dimensional matrix H×W×C, where H is the number of pixels in a column of the image, W is the number of pixels in a row of the image, and C is the total number of image categories, then the prediction matrix can be transformed into a two-dimensional prediction matrix (H×W)×C through dimensionality reduction.

[0082] For example, the computer device uses the same preset method to convert the prediction matrix into a prediction dimensionality reduction matrix and the label matrix into a label dimensionality reduction matrix.

[0083] Step 406: Transpose the label dimension reduction matrix to obtain the label transpose matrix.

[0084] Transpose refers to the process of replacing the rows of a matrix with columns of the same ordinal number.

[0085] Step 408: The predicted dimensionality reduction matrix and the label transpose matrix are fused to obtain the feature matrix corresponding to the image.

[0086] Here, fusion refers to the process of multiplication. For example, a computer device multiplies the predicted dimensionality reduction matrix by the label transpose matrix to obtain the feature matrix corresponding to the image.

[0087] In some embodiments, the process of feature extraction from an image is as follows: Figure 5 As shown, the initial semantic segmentation model outputs a prediction matrix, which is then dimensionality-reduced to obtain a prediction-reduced matrix. Simultaneously, the prediction matrix is ​​labeled to obtain a label matrix, which is then dimensionality-reduced to obtain a label-reduced matrix. The label-reduced matrix is ​​then transposed to obtain a label-transposed matrix. Finally, the prediction-reduced matrix is ​​multiplied by the label-transposed matrix to obtain the feature matrix of the image.

[0088] In this embodiment, the feature matrix corresponding to the image is obtained by multiplying the prediction dimensionality reduction matrix by the label transpose matrix. The prediction dimensionality reduction matrix represents the probability prediction of the image category corresponding to each pixel in the image by the initial semantic segmentation model, and the label transpose matrix represents the image category corresponding to each pixel in the image. Multiplying the prediction dimensionality reduction matrix by the label transpose matrix yields a feature matrix that fully reflects the predicted features of the image by the initial semantic segmentation model, providing accurate basic data for calculating the similarity between candidate images and training images.

[0089] In some embodiments, determining the label matrix corresponding to the image based on the prediction matrix includes:

[0090] For each pixel in the image, the image category with the highest probability is determined as the target category corresponding to the pixel. The highest probability is the highest probability among the image categories corresponding to the pixel.

[0091] In the prediction matrix, the probability of the target category corresponding to each pixel is set as the first label, and the probability of the remaining image categories is set as the second label, thus obtaining the label matrix corresponding to the image.

[0092] Here, probability represents the likelihood that a pixel belongs to the image category; a higher probability indicates a greater likelihood that the pixel belongs to that image category, and vice versa. The first identifier is the label for the target category of the pixel, and the second identifier is the label for the non-target category of the pixel. The first and second identifiers can be different numbers. For example, the first identifier can be 1, and the second identifier can be 0.

[0093] For example, for each pixel in an image, the computer device obtains the probability of each image category corresponding to the pixel from the prediction matrix, compares the probabilities of each image category corresponding to the pixel, determines the highest probability, determines the image category corresponding to the highest probability as the target category of the pixel, and the remaining image categories are non-target categories. The probability of the target category corresponding to the pixel in the prediction matrix is ​​set to 1, and the probability of the non-target category corresponding to the pixel is set to 0, thus obtaining the label matrix corresponding to the image.

[0094] In this embodiment, each element in the label matrix represents the target category corresponding to each pixel in the image, that is, the image category with the highest probability corresponding to each pixel. The label matrix reflects the image category characteristics of the image.

[0095] In some embodiments, determining the target image based on the predicted difference and feature similarity of each candidate image includes:

[0096] Based on the predicted difference and feature similarity of candidate images, candidate evaluation values ​​for candidate images are determined.

[0097] The candidate image with the highest candidate evaluation value among all candidate images is determined as the target image.

[0098] The candidate evaluation value refers to the numerical value used to evaluate the selectivity of candidate images. It can be understood that the candidate evaluation value characterizes the uncertainty and diversity of candidate images. The larger the candidate evaluation value, the stronger the uncertainty of the initial semantic segmentation model regarding the candidate images, and the higher the diversity of the candidate images compared to multiple training images. In other words, the larger the candidate evaluation value, the better the training effect of using the candidate images corresponding to the candidate evaluation value to train the initial semantic segmentation model.

[0099] For example, the computer device performs sign-changing processing on the feature similarity of the candidate images, that is, adds a negative sign to the feature similarity to obtain the target feature similarity. Then, it adds the target feature similarity to the predicted difference of the candidate images to obtain the candidate evaluation value of the candidate images. The candidate evaluation values ​​corresponding to each candidate image are compared, and the candidate image corresponding to the largest candidate evaluation value is determined as the target image.

[0100] In this embodiment, the candidate image corresponding to the largest candidate evaluation value is determined as the target image. That is, the candidate image with the highest prediction uncertainty of the initial semantic segmentation model and the largest difference from multiple training images is selected, so as to provide training images for the subsequent training of the initial semantic segmentation model.

[0101] In some embodiments, such as Figure 6 As shown, model training methods also include:

[0102] Step 602: Update the target image count to obtain the current target image count.

[0103] Among them, the number of target images refers to the total number of target images.

[0104] For example, after the computer device determines the target image, it adds the number of target images to the target image count to obtain the current target image count.

[0105] Step 604: If the current number of target images is less than the preset number, use the target images as training images to obtain multiple updated training images and multiple candidate images.

[0106] For example, the computer device compares the current number of target images with a preset number. If the current number of target images is equal to the preset number, the initial semantic segmentation model is trained based on the preset number of target images to obtain a target semantic segmentation model. If the current number of target images is less than the preset number, the target images are added to the training image set to obtain multiple updated training images. The target images are removed from the candidate image set to obtain multiple updated candidate images. Then, step 606 is executed.

[0107] Step 606: Repeat the step of obtaining the target image based on the updated multiple training images and multiple candidate images.

[0108] For example, the computer device repeats steps 204-210.

[0109] Step 608 continues until the current number of target images equals the preset number, thus obtaining the preset number of target images.

[0110] Step 610: Based on the target image, train the initial semantic segmentation model to obtain the target semantic segmentation model, including: based on a preset number of target images, train the initial semantic segmentation model to obtain the target semantic segmentation model.

[0111] For example, a computer device trains an initial semantic segmentation model multiple times using a preset number of target images to obtain a target semantic segmentation model.

[0112] In this embodiment, training the initial semantic segmentation model with a small number of target images can achieve the same effect as training the initial semantic segmentation model with all candidate images, thereby improving the training efficiency of the model.

[0113] In one exemplary embodiment, the method for training the semantic segmentation model is as follows:

[0114] The computer device acquires multiple training images from the training set and multiple candidate images from the candidate set. The training images are labeled images, and the candidate images are unlabeled images. The semantic segmentation model is trained using these multiple training images to obtain an initial semantic segmentation model. The training images are then input into the initial semantic segmentation model for processing, and the first prediction matrix corresponding to the training images is output. Similarly, the candidate images are input into the initial semantic segmentation model for processing, and the second prediction matrix corresponding to the candidate images is output.

[0115] The computer device, based on the second prediction matrix, obtains the probability of each pixel in the candidate image corresponding to the corresponding image category for each image category, calculates the average probability of each pixel corresponding to the corresponding image category, obtains the average probability corresponding to the image category, compares the average probabilities corresponding to each image category in the second prediction matrix, determines the largest average probability as the first average probability, determines the second largest average probability as the second average probability, subtracts the second average probability from the first average probability to obtain the probability difference, and determines the prediction difference of the candidate image.

[0116] The computer device performs the following operations on the training images based on the first prediction matrix: Figure 5 The feature extraction shown yields the first feature matrix corresponding to the training image. Based on the second prediction matrix, the candidate images are then processed as follows: Figure 5 The feature extraction process, as shown, yields the second feature matrix corresponding to the candidate image. The cosine similarity between the second feature matrix and the first feature matrix is ​​calculated, and this cosine similarity is determined as the similarity between the candidate image and the training images. For each candidate image, the computer device compares the similarity between the candidate image and each training image, and the minimum similarity is determined as the feature similarity between the candidate image and multiple training images.

[0117] The computer device subtracts the feature similarity from the predicted difference of the candidate images to obtain the candidate evaluation value of the candidate images. The candidate image corresponding to the candidate with the largest candidate evaluation value is determined as the target image. The number of target images is incremented by one to obtain the current number of target images. The current number of target images is compared with a preset number. If the current number of target images equals the preset number, the initial semantic segmentation model is trained based on the preset number of target images to obtain the target semantic segmentation model. If the current number of target images is less than the preset number, the target image is added to the training image set to obtain multiple updated training images. The target image is removed from the candidate image set to obtain multiple updated candidate images. The above steps are repeated until the number of target images equals the preset number to obtain the preset number of target images.

[0118] The computer equipment uses a preset number of target images to train the initial semantic segmentation model multiple times to obtain the target semantic segmentation model.

[0119] The above model training method acquires multiple training images and multiple candidate images. The labeled training images are input into the initial semantic segmentation model to obtain a first prediction matrix. The unlabeled candidate images are input into the initial semantic segmentation model to obtain a second prediction matrix. Based on the second prediction matrix corresponding to the candidate images, the prediction difference of the initial semantic segmentation model for the candidate images is determined, i.e., the accuracy of the initial semantic segmentation model's prediction of the candidate images is determined. The greater the prediction difference, the lower the accuracy of the initial semantic segmentation model's prediction of the candidate images, and the stronger the uncertainty of the candidate images for the initial semantic segmentation model. Based on the second prediction matrix corresponding to the candidate images and the first prediction matrix corresponding to the training images, the feature similarity between the candidate images and the multiple training images is determined. The lower the feature similarity, the greater the difference between the candidate images and the multiple training images. Based on the prediction difference and feature similarity of each candidate image, the target image is determined. This can be understood as selecting target images with low prediction accuracy of the initial semantic segmentation model and large differences from the multiple training images. Using a small number of target images to train the initial semantic segmentation model can achieve the effect of using all candidate images to train the initial semantic segmentation model, thereby improving the training efficiency of the model.

[0120] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0121] Based on the same inventive concept, this application also provides a model training apparatus. The solution provided by this apparatus is similar to the solution described in the above method. Therefore, the specific limitations of one or more model training apparatus embodiments provided below can be found in the limitations of the model training method above, and will not be repeated here.

[0122] In some embodiments, such as Figure 7 As shown, a model training device is provided, comprising:

[0123] The acquisition module 702 is used to acquire multiple training images and multiple candidate images;

[0124] The input module 704 is used to input training images into the initial semantic segmentation model to obtain a first prediction matrix; and to input candidate images into the initial semantic segmentation model to obtain a second prediction matrix.

[0125] The first determining module 706 is used to determine the prediction difference degree of the candidate image based on the second prediction matrix corresponding to the candidate image.

[0126] The second determining module 708 is used to determine the feature similarity between a candidate image and multiple training images based on the second prediction matrix corresponding to the candidate image and the first prediction matrix corresponding to the training image.

[0127] Selection module 710 is used to determine the target image based on the predicted difference and feature similarity of each candidate image;

[0128] Training module 712 is used to train the initial semantic segmentation model based on the target image to obtain the target semantic segmentation model.

[0129] In some embodiments, in determining the prediction dissimilarity of candidate images based on a second prediction matrix corresponding to the candidate images, the first determining module 706 is specifically configured to:

[0130] Based on the second prediction matrix corresponding to the candidate image, the average probability corresponding to each image category is determined; the second prediction matrix is ​​a three-dimensional matrix, and the element values ​​in the second prediction matrix represent the probability of the image category corresponding to the pixel in the candidate image;

[0131] The first average probability and the second average probability are selected from the average probabilities corresponding to each image category;

[0132] The difference between the first average probability and the second average probability is determined as the prediction difference of the candidate image.

[0133] In some embodiments, in determining the feature similarity between a candidate image and multiple training images based on a second prediction matrix corresponding to a candidate image and a first prediction matrix corresponding to a training image, the second determining module 708 is specifically used for:

[0134] Based on the prediction matrix, feature extraction is performed on the image to obtain the corresponding feature matrix. The prediction matrix includes a first prediction matrix corresponding to the training image and a second prediction matrix corresponding to the candidate image. When the prediction matrix is ​​the first prediction matrix, the first feature matrix corresponding to the training image is obtained. When the prediction matrix is ​​the second prediction matrix, the second feature matrix corresponding to the candidate image is obtained.

[0135] Based on the second feature matrix and the first feature matrix, the similarity between the candidate image and the training image is determined;

[0136] The smallest similarity among multiple similarities is determined as the feature similarity between the candidate image and multiple training images, and multiple similarities correspond one-to-one with multiple training images.

[0137] In some embodiments, in terms of extracting features from an image based on a prediction matrix to obtain a feature matrix corresponding to the image, the second determining module 708 is specifically used for:

[0138] Based on the prediction matrix, determine the label matrix corresponding to the image;

[0139] The prediction matrix is ​​dimensionality reduced to obtain the prediction dimensionality-reduced matrix; the label matrix is ​​dimensionality reduced to obtain the label dimensionality-reduced matrix.

[0140] Transpose the reduced label matrix to obtain the transpose of the label matrix;

[0141] The predicted dimensionality reduction matrix and the label transpose matrix are fused to obtain the feature matrix corresponding to the image.

[0142] In some embodiments, in determining the label matrix corresponding to the image based on the prediction matrix, the second determining module 708 is specifically used for:

[0143] For each pixel in the image, the image category with the highest probability is determined as the target category corresponding to the pixel. The highest probability is the highest probability among the image categories corresponding to the pixel.

[0144] In the prediction matrix, the probability of the target category corresponding to each pixel is set as the first label, and the probability of the remaining image categories is set as the second label, thus obtaining the label matrix corresponding to the image.

[0145] In some embodiments, in determining the target image based on the predicted difference and feature similarity of each candidate image, the selection module 710 is specifically used for:

[0146] Based on the predicted difference and feature similarity of candidate images, candidate evaluation values ​​for candidate images are determined.

[0147] The candidate image with the highest candidate evaluation value among all candidate images is determined as the target image.

[0148] In some embodiments, the selection module 710 is further configured to:

[0149] Update the target image count to obtain the current target image count;

[0150] If the current number of target images is less than the preset number, the target images are used as training images to obtain multiple updated training images and multiple candidate images.

[0151] Repeat the steps of obtaining target images based on updated training images and candidate images until the current number of target images equals the preset number, thus obtaining the preset number of target images;

[0152] The initial semantic segmentation model is trained based on the target image to obtain the target semantic segmentation model, including: training the initial semantic segmentation model based on a preset number of target images to obtain the target semantic segmentation model.

[0153] Each module in the aforementioned model training device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0154] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements the steps of the aforementioned model training method. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0155] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0156] In some embodiments, a computer device is also provided, the computer device including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiments.

[0157] In some embodiments, a computer-readable storage medium 900 is provided, on which a computer program 902 is stored. When executed by a processor, the computer program 902 implements the steps in the above method embodiments. Its internal structure diagram can be as follows: Figure 9 As shown.

[0158] In some embodiments, a computer program product is provided, which includes a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0159] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0160] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0161] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0162] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A model training method, characterized in that, include: Acquire multiple training images and multiple candidate images; The training image is input into the initial semantic segmentation model to obtain the first prediction matrix; The candidate image is input into the initial semantic segmentation model to obtain the second prediction matrix; Based on the second prediction matrix corresponding to the candidate images, the average probability corresponding to each image category is determined. The second prediction matrix is ​​a three-dimensional matrix, and the element values ​​in the second prediction matrix represent the probability of the image category corresponding to the pixel in the candidate image; a first average probability and a second average probability are selected from the average probabilities corresponding to each image category; the difference between the first average probability and the second average probability is determined as the prediction difference degree of the candidate image. Based on the second prediction matrix corresponding to the candidate image and the first prediction matrix corresponding to the training image, the feature similarity between the candidate image and multiple training images is determined. The sign of the feature similarity of the candidate images is changed to obtain the target feature similarity; the sign change refers to adding a negative sign to the feature similarity. The predicted difference of the candidate images is added to the target feature similarity to obtain the candidate evaluation value of the candidate images; the candidate image corresponding to the candidate evaluation value with the largest candidate evaluation value among all the candidate images is determined as the target image. Based on the target image, the initial semantic segmentation model is trained to obtain the target semantic segmentation model.

2. The method according to claim 1, characterized in that, The step of selecting the first average probability and the second average probability from the average probabilities corresponding to each of the image categories includes: The average probabilities corresponding to each image category in the second prediction matrix are compared. The largest of the aforementioned average probabilities is determined as the first average probability; The second largest average probability is determined as the second average probability.

3. The method according to claim 1, characterized in that, The step of determining the feature similarity between the candidate image and multiple training images based on the second prediction matrix corresponding to the candidate image and the first prediction matrix corresponding to the training image includes: Based on the prediction matrix, features are extracted from the image to obtain the feature matrix corresponding to the image; the prediction matrix includes a first prediction matrix corresponding to the training image and a second prediction matrix corresponding to the candidate image; when the prediction matrix is ​​the first prediction matrix, a first feature matrix corresponding to the training image is obtained; when the prediction matrix is ​​the second prediction matrix, a second feature matrix corresponding to the candidate image is obtained. Based on the second feature matrix and the first feature matrix, the similarity between the candidate image and the training image is determined; The smallest similarity among multiple similarities is determined as the feature similarity between the candidate image and multiple training images, and the multiple similarities correspond one-to-one with the multiple training images.

4. The method according to claim 3, characterized in that, The step of extracting features from the image based on the prediction matrix to obtain the feature matrix corresponding to the image includes: Based on the prediction matrix, determine the label matrix corresponding to the image; The prediction matrix is ​​subjected to dimensionality reduction transformation to obtain the prediction dimensionality reduction matrix; the label matrix is ​​subjected to dimensionality reduction transformation to obtain the label dimensionality reduction matrix; The label dimensionality reduction matrix is ​​transposed to obtain the label transpose matrix; The predicted dimensionality reduction matrix and the label transpose matrix are fused to obtain the feature matrix corresponding to the image.

5. The method according to claim 4, characterized in that, The step of determining the label matrix corresponding to the image based on the prediction matrix includes: For each pixel in the image, the image category with the highest probability is determined as the target category corresponding to the pixel, where the highest probability is the highest probability among the image categories corresponding to the pixel; In the prediction matrix, the probability of the target category corresponding to the pixel is set as the first identifier, and the probabilities of the remaining image categories are set as the second identifier, thus obtaining the label matrix corresponding to the image.

6. The method according to claim 1, characterized in that, Determining the difference between the first average probability and the second average probability as the prediction difference degree of the candidate image includes: Subtract the second average probability from the first average probability to obtain the probability difference; The probability difference is determined as the prediction difference of the candidate image.

7. The method according to claim 1, characterized in that, The method further includes: Update the target image count to obtain the current target image count; If the current number of target images is less than a preset number, the target image is used as a training image to obtain multiple updated training images and multiple candidate images; Repeat the step of obtaining the target image based on the updated multiple training images and multiple candidate images; The target images are obtained until the current number of target images equals the preset number. Based on the target image, the initial semantic segmentation model is trained to obtain a target semantic segmentation model, including: Based on the preset number of target images, the initial semantic segmentation model is trained to obtain the target semantic segmentation model.

8. A model training device, characterized in that, include: The acquisition module is used to acquire multiple training images and multiple candidate images; The input module is used to input the training image into the initial semantic segmentation model to obtain the first prediction matrix; The candidate image is input into the initial semantic segmentation model to obtain the second prediction matrix; The first determining module is used to determine the average probability corresponding to each image category based on the second prediction matrix corresponding to the candidate image; The second prediction matrix is ​​a three-dimensional matrix, and the element values ​​in the second prediction matrix represent the probability of the image category corresponding to the pixel in the candidate image; a first average probability and a second average probability are selected from the average probabilities corresponding to each image category; the difference between the first average probability and the second average probability is determined as the prediction difference degree of the candidate image. The second determining module is used to determine the feature similarity between the candidate image and multiple training images based on the second prediction matrix corresponding to the candidate image and the first prediction matrix corresponding to the training image. The selection module is used to perform sign-changing processing on the feature similarity of the candidate images to obtain the target feature similarity; the sign-changing processing refers to adding a negative sign to the feature similarity. The predicted difference of the candidate images is added to the target feature similarity to obtain the candidate evaluation value of the candidate images; the candidate image corresponding to the candidate evaluation value with the largest candidate evaluation value among all the candidate images is determined as the target image. The training module is used to train the initial semantic segmentation model based on the target image to obtain the target semantic segmentation model.

9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images