Image processing method, device and storage medium based on artificial intelligence

Abstract

An embodiment of the present application proposes an artificial intelligence-based image processing method, device and storage medium, the method comprising: obtaining a training sample set, the training sample set comprising a first image sample set with category labels and a second image sample set consisting of triplets; using the training sample set to perform semantic learning and metric learning training on an original model, the original model comprising a first branch network and a second branch network, the first branch network and the second branch network comprising shared network parameters; determining an image feature extraction model based on the trained original model, the image feature extraction model being used to extract feature vectors of an image, enabling the model to achieve metric learning while having semantic extraction capabilities, and the image features extracted based on the model can improve the accuracy of image retrieval.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an image processing method, device and storage medium based on artificial intelligence. Background Art

[0002] With the rapid development of Internet technology, the amount of multimedia data has increased rapidly, especially image data as the most important information carrier. How to accurately query and retrieve image data similar to the retrieved image in these massive images has become a research hotspot.

[0003] Image retrieval capabilities often rely on image features that represent images. In scenarios driven by massive amounts of image data, traditional image features used for image comparison, such as scale-invariant feature transform (SIFT), color histogram, and histogram of oriented gradients (HOG), perform poorly when processing unseen or noisy images. Image features extracted by deep learning models based on metric learning, such as the SimCLR (A Simple Framework for Contrastive Learning of Visual Representations) method and the BYOL (Bootstrap Your Own Latent A New Approach to Self-Supervised Learning) method, often fail to accurately and effectively retrieve image data when processing images that have undergone certain spatial transformation attacks (such as cropping of more than 1 / 3) or have never been seen before. Summary of the Invention

[0004] The embodiments of the present invention provide an artificial intelligence-based image processing method, device, and storage medium, which can enable the model to achieve metric learning while having semantic extraction capabilities. The image features extracted based on the model can improve the accuracy of image retrieval.

[0005] In one aspect, an embodiment of the present invention provides an image processing method based on artificial intelligence, the method comprising:

[0006] Acquire a training sample set, where the training sample set includes a first image sample set with category labels and a second image sample set consisting of triplets;

[0007] Using the training sample set to perform semantic learning training and metric learning training on an original model, the original model includes a first branch network and a second branch network, and the first branch network and the second branch network include common network parameters;

[0008] An image feature extraction model is determined based on the trained original model, where the image feature extraction model is used to extract a feature vector of the image.

[0009] On the other hand, an embodiment of the present invention provides an image processing method and apparatus based on artificial intelligence, the apparatus comprising:

[0010] An acquisition module is used to acquire a training sample set, wherein the training sample set includes a first image sample set with category labels and a second image sample set consisting of triplets;

[0011] a processing module, configured to perform semantic learning training and metric learning training on an original model using the training sample set, wherein the original model includes a first branch network and a second branch network, and the first branch network and the second branch network include common network parameters;

[0012] The determination module is used to determine an image feature extraction model based on the trained original model, where the image feature extraction model is used to extract a feature vector of the image.

[0013] On the other hand, an embodiment of the present invention provides a server, which includes a processor, a communication interface and a memory, wherein the processor, communication interface and memory are interconnected, wherein the memory is used to store a computer program, and the computer program includes program instructions, and the processor is configured to call the program instructions to perform the operations involved in the above-mentioned artificial intelligence-based image processing method.

[0014] Correspondingly, an embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the program involved in the above-mentioned artificial intelligence-based image processing method.

[0015] Accordingly, an embodiment of the present invention further provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned artificial intelligence-based image processing method.

[0016] The embodiment of the present invention obtains a training sample set, uses the training sample set to train the semantic learning model and metric learning model in the initialization model, and determines the image feature extraction model based on the trained original model. At the same time, the image feature extraction model is used to process the input image to be retrieved to obtain a feature vector of the image to be retrieved, and the feature vector of the image to be retrieved and the feature vectors in the image feature library are used to determine similar images to the image to be retrieved. It can be seen that the image feature extraction model proposed in this application can have semantic extraction capabilities while realizing metric learning, so that the feature vector of the image to be retrieved obtained by the image feature extraction model can maintain the semantic information of the image, thereby improving the accuracy of image retrieval for unknown images and images with spatial changes such as cropping. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A schematic diagram of the architecture of an artificial intelligence-based image processing system provided by an embodiment of the present invention;

[0019] Figure 2 A flowchart of an image processing method based on artificial intelligence provided by an embodiment of the present invention;

[0020] Figure 3 A schematic diagram of image enhancement provided by an embodiment of the present invention;

[0021] Figure 4 A schematic diagram of the model structure of an original model provided by an embodiment of the present invention;

[0022] Figure 5 A schematic diagram of the structure of a residual block provided by an embodiment of the present invention;

[0023] Figure 6 A schematic diagram of the network structure of a second branch network in an original model provided by an embodiment of the present invention;

[0024] Figure 7 A schematic diagram of a second branch network training process provided by an embodiment of the present invention;

[0025] Figure 8a A schematic diagram of a flow chart of another image processing method based on artificial intelligence provided by an embodiment of the present invention;

[0026] Figure 8bA schematic diagram of a flow chart of another image processing method based on artificial intelligence provided by an embodiment of the present invention;

[0027] Figure 9 A schematic diagram of a search and sorting effect provided by an embodiment of the present invention;

[0028] Figure 10 A schematic structural diagram of an image processing device based on artificial intelligence provided by an embodiment of the present invention;

[0029] Figure 11 A schematic diagram of the structure of a server provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this invention.

[0031] It should be noted that the terms "first" and "second" in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature designated as "first" or "second" may explicitly or implicitly include at least one such feature.

[0032] The technical terms involved in the embodiments of the present invention are explained as follows:

[0033] Image recognition: This involves category-level recognition, regardless of the specific instance of the object. This task considers only the object's category (e.g., person, dog, cat, bird, etc.) and identifies the object's category. A typical example is the recognition task in ImageNet, a large open-source dataset for general object recognition, which identifies which of 1,000 categories an object belongs to. Image recognition provides semantic information about images.

[0034] Image retrieval: Given an image to be retrieved that contains a specific instance (such as a specific object, scene, building, etc.), image retrieval aims to find images containing the same instance from the database images.

[0035] Image deduplication: The task of image deduplication retrieval is to find the newly added images that are duplicates of the existing images in the existing library.

[0036] ImageNet dataset: A large-scale open-source dataset for general object recognition.

[0037] ImageNet pre-trained model: A deep learning network model is trained based on ImageNet, and the parameter weights of the model are obtained as the ImageNet pre-trained model.

[0038] Semi-supervised learning: Semi-supervised learning (SSL) is a machine learning method that combines a small amount of labeled data with a large amount of unlabeled data during training. Semi-supervised learning lies between unsupervised and supervised learning.

[0039] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, cloud storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0040] Computer vision (CV) is the science of making machines "see." Specifically, it refers to machine vision, where cameras and computers replace the human eye in identifying, tracking, and measuring objects. This involves further processing the images, transforming them into images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0041] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0042] With the research and progress of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields. The solution provided in the embodiment of this application involves artificial intelligence computer vision technology, machine learning technology and other technologies, which are specifically explained through the following embodiments.

[0043] The artificial intelligence-based image processing method in the embodiments of the present application can be applied to some common image retrieval tasks. Some application scenarios are listed below.

[0044] For example, the terminal can send the image to be retrieved to the server. After receiving the retrieval image sent by the terminal, the server can input the image into the established image feature extraction model to extract the image features of the image to be retrieved, and use the feature vector of the image to be retrieved and the feature vector of the image stored in the image library to calculate the similarity. The image deduplication or image retrieval task can be performed according to the similarity. If the similarity is greater than or equal to the preset value, it can be determined that the image to be retrieved is in the image library and is a duplicate image in the image library, or the image to be retrieved is an attack image after adding a watermark to the image in the image library; at the same time, the image with a similarity greater than or equal to the preset value can also be used as a similar image of the image to be retrieved. If the image type in the retrieval image is "lotus", all images containing "lotus" are queried from the image library. It can also be applied to tasks such as content-based video deduplication.

[0045] like Figure 1 As shown, an embodiment of the present application provides an image processing system based on artificial intelligence, which can be specifically integrated into an electronic device, which can be a terminal or a server. For example, the image processing system can be integrated into a terminal. The terminal can be a mobile phone, a tablet computer, a laptop computer, a desktop computer, a personal computer (PC), a smart speaker or a smart watch, etc., but is not limited to this. For another example, the image processing system can be integrated into a server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDNs), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0046] It is understandable that the artificial intelligence-based image processing method of this embodiment can be executed on a terminal, on a server, or jointly by a terminal and a server.

[0047] The image processing system based on artificial intelligence includes a user terminal 101 and a server 102. The user terminal 101 and the server 102 are connected via a network, such as a wireless network connection. Take the case where the terminal and the server jointly perform an image processing method based on artificial intelligence as an example. Before describing the interaction between the terminal 101 and the server 102, it is necessary to first describe the process of the server 102 training an image feature extraction model. The image feature extraction model is used to obtain feature vectors of the image to be retrieved and the image to be registered sent by the terminal 101. For example, the user sends the image to be retrieved on the terminal 101 or an application for image retrieval installed on the terminal 101. The server 102 calculates the distance between the feature vector of the image to be retrieved and the feature vector of the image included in the image feature library, performs a similarity comparison, and uses the image corresponding to the feature vector with the largest similarity as a similar graphic of the image to be retrieved.

[0048] The following is a detailed description of the process of server 102 training the image feature extraction model. The image feature extraction model is based on an SSL-based semantic metric learning framework, which includes a semantic learning network and a metric learning network. Specifically, before training the image feature extraction model, it is necessary to obtain a training set, the training sample set including a first image sample set with category labels and a second image sample set consisting of triplets. Then, the first image sample set is used to obtain a classification loss function and a consistency loss function, and the second image sample set is used to obtain a triplet loss function. At the same time, the first image sample set is input into the first branch network in the original model, and the second image sample set is input into the second branch network in the original model to perform semantic learning training and metric learning training. The first branch network is specifically a semantic learning network, and the second branch network is specifically a metric learning network. The first image sample set and the second image sample set are simultaneously subjected to network forward calculation and the total loss function of the original model is calculated. Finally, the overall weight of the model is updated so that the model can achieve metric learning while having semantic extraction capabilities. The image features extracted based on the model can improve the accuracy of image retrieval.

[0049] It can be understood that the architectural diagram of the system described in the embodiment of the present application is intended to more clearly illustrate the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the system architecture and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0050] In one embodiment, Figure 2 As shown, the embodiment of the present invention is based on Figure 1The image processing system based on artificial intelligence provides an image processing method based on artificial intelligence. Figure 1 The server 102 mentioned in is taken as an example. Figure 2 The method according to the embodiment of the present invention is described.

[0051] S201 : Acquire a training sample set, where the training sample set includes a first image sample set with category labels and a second image sample set consisting of triplets.

[0052] In one embodiment, the first sample set is an image sample set including category labels, such as the ImageNet dataset, the COCO dataset, etc., which is not limited in this application. The second sample set is obtained by sampling from the image dataset to obtain an image sample set including triplets. The sampled image dataset can be an image dataset including category labels or an image dataset not including category labels, which is not limited in this application. For example, the present application randomly selects an image sample from an image dataset without category labels as a reference image sample (anchor), and performs image enhancement processing on the reference image sample, such as adding Gaussian noise, salt and pepper noise, cropping, rotation, watermarking, hue conversion, upside down flipping, left-right flipping, and adding a black frame, such as Figure 3 The figure shows a schematic diagram of image enhancement, in which positive image samples belonging to the same category as the reference image sample are obtained, and image samples other than the reference image sample are selected from the sampled image dataset as negative image samples belonging to a different category than the reference image sample. In this case, the reference image sample, the positive image sample, and the negative image sample constitute a triplet. In some image datasets, for example, image data with the category label "dog" includes "corgi" and "golden retriever." In this embodiment, each sample is treated as a category, so that the second image sample set including the triplet can obtain the category attributes of the image in a more refined manner.

[0053] S202: Perform semantic learning training and metric learning training on an original model using the training sample set, wherein the original model includes a first branch network and a second branch network, and the first branch network and the second branch network include common network parameters.

[0054] In one embodiment, Figure 4The figure shows a schematic diagram of the model structure of the original model. The original model includes a first branch network and a second branch network, wherein the first branch network includes a classification layer, and the second branch network includes a feature measurement layer. At the same time, the first branch network and the second branch network also include shared network parameters, which can keep the network models with shared network parameters in the first branch network and the second branch network synchronized during learning. In the training process of the original model, the first branch network in the original model is trained with the first image sample set, and the second branch network in the original model is trained with the second sample set. The total loss value is obtained according to the prediction results of the first image sample set in the first branch network and the prediction results of the second sample set in the second branch network, as well as the total loss function of the original model. The total loss function includes the loss function of the first branch network: the classification loss function and the consistency loss function, and the loss function of the second branch network: the triplet loss function. The network parameters of the first branch network and the second branch network are adjusted according to the total loss value to complete the semantic learning training and metric learning training of the original model. By Figure 5 It can be seen that the network model before the classification layer and the feature measurement layer in the original model mainly extracts features from the image. Through the network model with shared network parameters in the two branch networks, the original model can adapt to both semantic learning and metric learning tasks at the same time, so that the model can achieve metric learning while having semantic extraction capabilities.

[0055] S203 . Determine an image feature extraction model based on the trained original model, where the image feature extraction model is used to extract feature vectors of the image.

[0056] In one embodiment, after obtaining the trained original model, a feature vector of the image is extracted based on the feature measurement layer in the second branch network in the original model.

[0057] In an embodiment of the present application, a training sample set is first obtained, and then the first branch network is trained for semantic learning using the first image sample set in the training sample set, and the second branch network is trained for metric learning using the second image sample set in the training sample set. Finally, the image feature extraction model is determined based on the trained original model, so that the original model can have semantic extraction capabilities while realizing metric learning, so that the image feature extraction model can obtain the semantic information of the image and improve the accuracy of image retrieval.

[0058] In one embodiment, the training of the original model for semantic learning and metric learning using the training sample set includes: training the first branch network for semantic learning using the first image sample set, and training the second branch network for metric learning using the second image sample set; wherein the first branch network and the second branch network include common model parameters, and each triplet in the second image sample set includes a reference image sample, a positive image sample, and a negative image sample.

[0059] The first and second branch networks in the original model include shared network parameters, and their corresponding network model structures can be VGG, AlexNet, or ResNet-101 networks, etc., which are not limited in this application. Taking the network model structure of ResNet-101 pre-trained on the ImageNet dataset as an example, as shown in Table 1, the model structure of the ResNet-101 network is recorded.

[0060] Table 1

[0061]

[0062] As shown in Table 1 above, ResNet-101 includes 5 convolutional layers. In the first convolutional layer Conv1, 64 convolution kernels with a stride of 2 and a size of 7×7 are used; in the second convolutional layer, a maximum pooling with a size of 3×3 and a stride of 2 is used, as well as 3 residual blocks. The residual block structure is as follows: Figure 5 As shown in the figure, it includes three residual units. The residual units mainly use convolution kernels of size 1×1 and 3×3. Among them, the first convolution kernel of size 1×1 is mainly used to compress the number of feature maps, and the second convolution kernel of size 1×1 is used to restore the number of feature maps. The following (64, 128, 256, 512, 1024 and 2048) are the numbers of convolution kernels; 4 residual blocks are used in the third convolution layer; 23 residual blocks are used in the fourth convolution layer; 3 residual blocks are used in the fifth convolution layer. ResNet-101 uses a total of 34 residual blocks.

[0063] Please refer to Table 2, which shows the network structure of the classification layer in the first branch network in the original model. The classification layer includes a pooling layer using maximum pooling and a fully connected layer. After the network model is used to extract the feature vectors of the image samples in the input first sample set, they are further input into the classification layer. The pooling layer in the classification layer is used to reduce the dimension of the image features output by ResNet-101, and the classification function (such as the softmax function) is used in the fully connected layer to obtain the prediction results of the input image samples. The N_cls output by the fully connected layer is the number of categories in the first sample set. For example, if the first sample set includes five faces of people, the number of categories in the first sample set is five.

[0064] Table 2

[0065] Network layer name Output size network Pooling layer 1×2048 Max pool Fully connected layer 1×N_cls full connection

[0066] Among them, the classification layer of the semantic learning model can use a Gaussian distribution with a variance of 0.01 and a mean of 0 for parameter initialization.

[0067] In one embodiment, Figure 6 As shown, this is a schematic diagram of the model structure of the second branch network in the original model. Each triplet in the second sample group includes a reference image sample, a positive image sample, and a negative image sample. The reference image sample, the positive image sample, and the negative image sample in the triplet are respectively input into the second branch network in the original model, and the feature vectors of the reference image sample, the positive image sample, and the negative image sample are obtained in the feature measurement layer. The network structure of the feature measurement layer is a fully connected layer representing the feature vector of the image, and the size can be 1×2048 or 1×1024, etc., which is not limited in this application.

[0068] In one embodiment, a first branch network in the original model is trained using a first image sample set, and a second branch network in the original model is trained using a second sample set.

[0069] Specifically, the original model is trained based on SSL. In each round of batch training, each bs image is selected from the first image sample set (a total of N images) as a classification batch, and N / bs classification batches need to be executed for each training epoch; each 5*bs triples are selected from the second image sample set as a measurement batch, and a total of N / bs batches are sampled. In each round of batch training, the classification batch is used as the input of the first branch network in the original model, and the classification loss function value is calculated according to the prediction result of the first branch network and the category label corresponding to each image in the classification batch. The consistency loss function value is calculated using the prediction result of each image in the classification batch and the corresponding enhanced image in the first branch network; the measurement batch is used as the input of the second branch network in the original model to obtain the prediction result of the triplet image in the measurement batch, and the triplet loss function value is calculated according to the distance between the prediction results of the triplet image; the total loss value of the original model is determined according to the classification loss function value, the consistency loss function value and the triplet loss function value, and then the SGD (Stochastic Gradient The stochastic gradient descent (SGD) algorithm solves the convolution parameters and bias parameters of the target model, and back-propagates them to the original model to adjust the model parameters of the original model.

[0070] Furthermore, after completing a round of batch training, the server takes another set of classification batches and measurement batches to continue training the original model based on the model parameter adjustment of the original model until the training stop condition is met, such as when the specified number of training times is reached, or when the total loss function converges.

[0071] In an embodiment of the present application, part of the data is used with category labels based on SSL, and part of the data is triples for training the original model. By combining two different labeled data sets based on classification learning and metric learning, the utilization rate of data can be improved under limited resources, and the mutual promotion of multiple tasks can be achieved.

[0072] In one embodiment, before training the original model, it is necessary to determine the total loss function of the original model, including: obtaining the loss function of the first branch network and the loss function of the second branch network, the loss function of the first branch network including the classification loss function and the consistency loss function; determining the weight parameters of each loss function in the loss function of the first branch network and the loss function of the second branch network; determining the total loss function of the original model based on the loss function of the first branch network, the loss function of the second branch network and the weight parameters of each loss function.

[0073] In one embodiment, the classification loss function of the first branch network can be determined based on the prediction results and corresponding category labels of the image samples in the first sample set output by the first branch network in the original model. The classification loss function is shown in the following formula (1):

[0074]

[0075] Among them, y is the category label corresponding to the image sample, is the prediction result of the first branch network in the target model, and N is the number of image samples.

[0076] For example, the first branch network in the original model is similar to a multi-classifier for the image samples in the first image sample set. The output prediction results include the probability that the image sample belongs to each category label, and the sum of the probability values ​​is 1. Assume that the category labels of the first image sample set are A, B, and C, and the prediction results output by the first branch network are [0.3, 0.6, 0.1], indicating that the probability that the image sample belongs to category A is 0.3, the probability that it belongs to category B is 0.6, and the probability that it belongs to category C is 0.1.

[0077] In one embodiment, a consistency loss function for the first branch network is determined based on a first prediction result of an image sample in the first image sample set in the first branch network of the original model and a second prediction result of a corresponding enhanced image sample in the first branch network of the original model. For example, the output prediction result of the original image sample p is [0.7, 0.2, 0.1], indicating that the probability of belonging to category A is 0.7, the probability of belonging to category B is 0.2, and the probability of belonging to category C is 0.1; the output prediction result of the enhanced image sample is [0.6, 0.2, 0.2], indicating that the probability of belonging to category A is 0.6, the probability of belonging to category B is 0.2, and the probability of belonging to category C is 0.2; then the consistency loss function is calculated based on the difference between the output prediction result [0.7, 0.2, 0.1] of the original image sample p and the output prediction result [0.6, 0.2, 0.2] of the enhanced image sample. The consistency loss function is shown in the following formula (2):

[0078]

[0079] Among them, p(x i ) is the first prediction result, q(x i ) is the second prediction result.

[0080] In one embodiment, the second image sample set is used as the input of the second branch network in the original model, and the loss function corresponding to the second branch network is determined based on the distance between the triplet image samples in the second image sample set. For example, the triplet sample includes a reference image sample A, a positive image sample P, and a negative image sample N. When training the target model, it is necessary to determine the loss function corresponding to the second branch network based on the distance between the reference image sample A, the positive image sample P, and the negative image sample N. During the original model training process, the distance between the reference image sample A and the positive image sample P should be as small as possible, and the distance between the reference image sample A and the negative image sample N should be as large as possible. Therefore, the second loss function is shown in the following formula (3):

[0081]

[0082] in, is the prediction result of the reference image sample A based on the feature measurement layer of the original model, is the prediction result of the positive image sample P based on the feature measurement layer of the original model, is the prediction result of the negative image sample N based on the feature measurement layer of the original model, M is the number of triplet samples, and α is any reasonable constant, such as 0.2.

[0083] In one embodiment, the server obtains the classification loss function and consistency loss function of the first branch network, the loss function of the second branch network, and the weight parameters corresponding to each loss function, wherein the larger the weight parameter corresponding to the loss function, the greater the degree of influence when adjusting the model parameters of the original model, and the smaller the weight parameter corresponding to the loss function, the smaller the degree of influence when adjusting the model parameters of the original model. The classification loss function, the consistency loss function, and the loss function of the second branch network can be weighted and summed with the corresponding weight parameters to obtain a total loss function. The total loss function can be found in the following formula (4):

[0084] L total =w1L cls +w2L KL +w3L tri (4)

[0085] Among them, w1 is the weight parameter corresponding to the classification loss function, w2 is the weight parameter corresponding to the consistency loss function, and w3 is the weight parameter corresponding to the second loss function. Among them, w1, w2 and w3 can be set according to empirical values. For example, if w1 is set to 1, w2 and w3 are set to 0.5, the classification loss function has the greatest impact on the adjustment of the model parameters of the original model.

[0086] In one embodiment, the use of the first image sample set to train the first branch network for semantic learning, and the use of the second image sample set to train the second branch network for metric learning, include: inputting the images included in the first image sample set and the corresponding category labels into the first branch network to obtain the prediction results of the first branch network; inputting the triplets included in the second image sample set into the second branch network to obtain the prediction results of the second branch network; according to the prediction results of the first branch network, the prediction results of the second branch network and the total loss function of the original model, adjusting the network parameters of the first branch network and the network parameters of the second branch network to complete the semantic learning training and metric learning training of the original model.

[0087] In one embodiment, the images included in the first image sample set and the corresponding category labels are input into the first branch network to obtain the prediction results of the first branch network. At the same time, the images included in the first image sample set are enhanced to obtain enhanced images, such as one or more of Gaussian noise, salt and pepper noise, cropping, rotation, watermarking, hue conversion, upside down flipping, left-right flipping, and adding a black frame. The enhanced images and the corresponding category labels are input into the first branch network to obtain a second prediction result, and the first prediction result and the second prediction result are used as the prediction results of the first branch network. By performing data enhancement on the first data set and introducing the data-enhanced images into the original model training, the original model has the ability to resist attack images, such as still having good retrieval accuracy after the image is resized, oriented, cropped, and watermarked.

[0088] Specifically, the first branch network in the original model is similar to a multi-classifier, and the prediction results output by the original model for the image samples in the first image sample set include the probability that the image samples belong to each category label, and the sum of the probability values ​​is 1. For example, assuming that the category labels of the first image sample set are A, B, and C, the prediction results output by the first branch network are [0.3, 0.6, 0.1], indicating that the probability that the image sample belongs to category A is 0.3, the probability that it belongs to category B is 0.6, and the probability that it belongs to category C is 0.1.

[0089] In one embodiment, the triplets included in the second image sample set are input into the second branch network to obtain a feature vector of each image sample in the triplet, and the feature vector of each image sample in the triplet is used as a prediction result of the second branch network.

[0090] In one embodiment, during each round of batch training of the original model, the classification loss function value of each round of batch training is determined based on the first prediction result of each image sample in the classification batch of the first image sample set in the first branch network of the original model and the corresponding category label; the consistency loss function value of each round of batch training is determined based on the first prediction result of each image sample in the classification batch of the first image sample set in the original model and the second prediction result of the corresponding enhanced image sample in the first branch network of the original model; the triplet loss function value of each round of batch training is determined based on the distance between the prediction results of each image sample in each triplet in the metric batch of the second image sample set in the second branch network of the original model, that is, the distance between the feature vectors of each image sample in the triplet; the total loss value of the original model is obtained according to the calculation formula of the above-mentioned total loss function, as well as the classification loss function value, the consistency loss function value and the triplet loss function value, so that the network parameters of the first branch network and the network parameters of the second branch network can be adjusted according to the gradient descent algorithm to complete the semantic learning training and metric learning training of the original model.

[0091] In one embodiment, Figure 7 As shown, a flowchart of the second branch network training is provided, wherein the training process includes the following steps:

[0092] S701: Input the triples included in the second image sample set into the second branch network to obtain the distance between the images included in each triplet in the second image sample set;

[0093] S702: determining a target triplet as a difficult sample from the triplet included in the second image sample set according to the distance, and obtaining a predicted feature vector of each image included in the target triplet;

[0094] S703: Use the predicted feature vector of each image included in the target triplet as the prediction result of the second branch network.

[0095] In one embodiment, the distance between images of each image sample in the triplet in the second sample set is calculated based on the predicted feature vector of each image obtained by the second branch network, for example, f(x a ) is the feature vector of the reference image sample A in a triplet, f(x p ) is the feature vector of the positive image sample P in a triplet, f(x n ) is the feature vector of the negative image sample N in a triplet, and calculate ||f(x a)-f(x p )||-||f(x a )-f(x n )||, if||f(x a )-f(x p )||-||f(x a )-f(x n )|| is greater than or equal to the threshold, if it is greater than or equal to 0, this triplet is used as the target triplet of the difficult sample, and the predicted feature vector f(x a )、f(x p ) and f(x n ) as the prediction result of the second branch network. By calculating the triplet loss function value of the second network branch using only the target triplet as the prediction result of the second branch network, it is possible to ensure that the original model has a larger gradient update each time it is trained, thereby speeding up the training of the original model.

[0096] In one possible embodiment, during each round of batch training of the original model, every 5*bs triplets can be selected from the second image sample set as a metric batch, wherein the ratio of the target triplet as the difficult sample to other common triplets in a metric batch can be 1:4.

[0097] In one embodiment, Figure 8a As shown, the embodiment of the present invention is based on Figure 1 Another image processing method based on artificial intelligence is provided by the image processing system based on artificial intelligence. This embodiment mainly applies this method to the above Figure 1 Taking the server 102 in FIG. 1 as an example, the following steps are included:

[0098] S801, obtaining a training sample set, where the training sample set includes a first image sample set with category labels and a second image sample set consisting of triplets;

[0099] S802: Perform semantic learning and metric learning training on an original model using the training sample set, where the original model includes a first branch network and a second branch network, and the first branch network and the second branch network include common network parameters;

[0100] S803, determining an image feature extraction model based on the trained original model, wherein the image feature extraction model is used to extract a feature vector of the image;

[0101] S804, obtaining the image to be retrieved;

[0102] S805: Input the image to be retrieved into the image feature extraction model to obtain a feature vector of the image to be retrieved;

[0103] S806: Determine similar images to the image to be retrieved using the feature vector of the image to be retrieved and an image feature library, where the image feature library includes a feature vector of at least one image.

[0104] In one embodiment, before determining similar images of the image to be retrieved using the feature vector of the image to be retrieved and the image feature library, Figure 8b As shown, the server needs to obtain the image to be registered. The user sends the image to be registered to the server at the terminal. The server inputs the image to be registered into the registration module. In the registration module, based on the image feature extraction model trained in this application, the feature vector of the image to be registered is obtained from the feature measurement layer in the image feature extraction model. The feature vector and image identifier of the image to be registered are added to the image feature library. For example, a seller can upload product-related pictures in the application app. The server obtains the feature vector of the uploaded picture and stores it in the cloud image library, making it easier for subsequent users to obtain product information through image search.

[0105] In one embodiment, S804-S806 include: after receiving the image to be retrieved sent by the terminal, the server uses the image feature extraction model in the retrieval module to process the input image to be retrieved, obtains the feature vector of the image to be retrieved, and determines the similarity between the feature vector of the image to be retrieved and each feature vector in the image feature library, and determines the image corresponding to the feature vector in the image feature library whose similarity with the feature vector of the image to be retrieved is greater than the similarity threshold as the similar image of the image to be retrieved. After determining the similar image of the image to be retrieved, the image information can be sent to the terminal from the cloud image library as a retrieval result based on the correspondence between the image identifier in the image feature library and the image to be registered, or when the terminal searches in the local image library, the image identifier is sent to the terminal as a retrieval result, and the terminal obtains similar images from the local image library based on the image identifier. S801-S803 have been described in detail in the above embodiment and will not be repeated in this embodiment. Figure 9 As shown in the figure, it is a retrieval sorting effect diagram, where the first image in each row is the image to be retrieved, and the subsequent images are arranged according to the similarity.

[0106] The similarity is used to measure the similarity between the feature vector of the image to be retrieved and each feature vector in the image feature library. The similarity can be calculated using a similarity algorithm, which includes but is not limited to the Euclidean distance algorithm, the cosine similarity algorithm, the Pearson correlation coefficient algorithm, the Jaccard similarity coefficient algorithm, and the like.

[0107] In an embodiment of the present application, a feature extraction model is used to obtain a feature vector of the image to be retrieved, so that the feature vector of the image to be retrieved holds semantic information. Even when the image to be retrieved is cropped over a large area or processed to be noisy, good retrieval accuracy is still achieved.

[0108] In one embodiment, different network models (VGG, ResNet-50 or AlexNet, etc.) and different pre-trained model weights (such as pre-trained models on Open-image data instead of ImageNet pre-trained models) can be used as network models with shared network parameters, and this application does not limit this. The first branch network can also use data from other tasks to perform multi-task learning with the second branch network, such as video multi-label recognition tasks (such as labeling the video with: whether there are people, whether it is outdoors, whether it is a happy scene, whether it is a cultural building, whether it is a tourist attraction, etc.), and this application does not limit this. If other tasks are used, it is necessary to use the loss function calculation formula corresponding to the other tasks to replace the loss function of the first branch network.

[0109] like Figure 10 As shown, Figure 10 : is a schematic structural diagram of an artificial intelligence-based image processing device provided in an embodiment of the present application, the device comprising:

[0110] An acquisition module 1001 is configured to acquire a training sample set, wherein the training sample set includes a first image sample set with category labels and a second image sample set consisting of triplets;

[0111] A processing module 1002 is configured to perform semantic learning training and metric learning training on an original model using the training sample set, wherein the original model includes a first branch network and a second branch network, and the first branch network and the second branch network include common network parameters;

[0112] The determination module 1003 is used to determine an image feature extraction model based on the trained original model, where the image feature extraction model is used to extract a feature vector of an image.

[0113] In one embodiment, the processing module 1002 is specifically configured to:

[0114] Using the first image sample set to train the first branch network for semantic learning, and using the second image sample set to train the second branch network for metric learning;

[0115] The first branch network and the second branch network include common model parameters, and each triplet in the second image sample set includes a reference image sample, a positive image sample, and a negative image sample.

[0116] In one embodiment, the processing module 1002 is specifically configured to:

[0117] Inputting the images and corresponding category labels included in the first image sample set into the first branch network to obtain a prediction result of the first branch network;

[0118] Inputting the triples included in the second image sample set into the second branch network to obtain a prediction result of the second branch network;

[0119] According to the prediction results of the first branch network, the prediction results of the second branch network and the total loss function of the original model, the network parameters of the first branch network and the network parameters of the second branch network are adjusted to complete the semantic learning training and metric learning training of the original model.

[0120] In one embodiment, the processing module 1002 is also used to obtain the loss function of the first branch network and the loss function of the second branch network, where the loss function of the first branch network includes a classification loss function and a consistency loss function; determine the weight parameters of each loss function in the loss function of the first branch network and the loss function of the second branch network; and determine the total loss function of the original model based on the loss function of the first branch network, the loss function of the second branch network and the weight parameters of each loss function.

[0121] In one embodiment, the processing module 1002 is specifically configured to:

[0122] Inputting the images and corresponding category labels included in the first image sample set into the first branch network to obtain a first prediction result;

[0123] performing enhancement processing on the images included in the first image sample set to obtain enhanced images;

[0124] Inputting the enhanced image and the corresponding category label into the first branch network to obtain a second prediction result;

[0125] The first prediction result and the second prediction result are used as prediction results of the first branch network.

[0126] In one embodiment, the processing module 1002 is specifically configured to:

[0127] Inputting the triplets included in the second image sample set into the second branch network to obtain the distance between the images included in each triplet in the second image sample set;

[0128] determining a target triplet as a difficult sample from the triplet included in the second image sample set according to the distance, and obtaining a predicted feature vector of each image included in the target triplet;

[0129] The predicted feature vector of each image included in the target triplet is used as the prediction result of the second branch network.

[0130] In one embodiment, the acquisition module 1001 is further used to acquire the image to be retrieved;

[0131] The processing module 1002 is further configured to input the image to be retrieved into the image feature extraction model to obtain a feature vector of the image to be retrieved; and determine similar images of the image to be retrieved using the feature vector of the image to be retrieved and an image feature library, wherein the image feature library includes a feature vector of at least one image.

[0132] In one embodiment, the acquisition module 1001 is further used to acquire the image to be registered;

[0133] The processing module 1002 is further configured to input the image to be registered into the image feature extraction model to obtain a feature vector of the image to be registered; and add the feature vector and image identifier of the image to be registered into the image feature library.

[0134] The embodiment of the present invention obtains a training sample set, uses the training sample set to train the semantic learning model and metric learning model in the initialization model, and determines the image feature extraction model based on the trained original model. At the same time, the image feature extraction model is used to process the input image to be retrieved to obtain a feature vector of the image to be retrieved, and the feature vector of the image to be retrieved and the feature vectors in the image feature library are used to determine similar images to the image to be retrieved. It can be seen that the image feature extraction model proposed in this application can have semantic extraction capabilities while realizing metric learning, so that the feature vector of the image to be retrieved obtained by the image feature extraction model can maintain the semantic information of the image, thereby improving the accuracy of image retrieval for unknown images and images with spatial changes such as cropping.

[0135] like Figure 11 As shown, Figure 11 This is a schematic diagram of the structure of a server provided in an embodiment of the present application. The internal structure of the server is as follows Figure 11 As shown, it includes: one or more processors 1101, memory 1102, and communication interface 1103. The processor 1101, memory 1102, and communication interface 1103 can be connected via a bus 1104 or other means. The embodiment of the present application takes the connection via the bus 1104 as an example.

[0136] Among them, the processor 1101 (also known as the CPU (Central Processing Unit)) is the computing core and control core of the computer device. It can parse various instructions within the computer device and process various data of the computer device. For example, the CPU can be used to parse the power on and off commands sent by the user to the computer device and control the computer device to perform power on and off operations. Another example is that the CPU can transmit various interactive data between the internal structures of the computer device, etc. The communication interface 1103 can optionally include a standard wired interface or a wireless interface (such as Wi-Fi, mobile communication interface, etc.), which is controlled by the processor 1101 to send and receive data. The memory 1102 (Memory) is a memory device in the computer device for storing programs and data. It is understood that the memory 1102 here can include both the built-in memory of the computer device and, of course, the extended memory supported by the computer device. The memory 1102 provides storage space, which stores the operating system of the computer device, which may include but is not limited to: Windows system, Linux system, etc., and this application is not limited to this.

[0137] In one embodiment, the processor 1101 executes the following operations by running the executable program code in the memory 1102:

[0138] Acquire a training sample set, where the training sample set includes a first image sample set with category labels and a second image sample set consisting of triplets;

[0139] Using the training sample set to perform semantic learning training and metric learning training on an original model, the original model includes a first branch network and a second branch network, and the first branch network and the second branch network include common network parameters;

[0140] An image feature extraction model is determined based on the trained original model, where the image feature extraction model is used to extract a feature vector of the image.

[0141] In one embodiment, the processor 1101 is specifically configured to:

[0142] Using the first image sample set to train the first branch network for semantic learning, and using the second image sample set to train the second branch network for metric learning;

[0143] The first branch network and the second branch network include common model parameters, and each triplet in the second image sample set includes a reference image sample, a positive image sample, and a negative image sample.

[0144] In one embodiment, the processor 1101 is specifically configured to:

[0145] Inputting the images and corresponding category labels included in the first image sample set into the first branch network to obtain a prediction result of the first branch network;

[0146] Inputting the triples included in the second image sample set into the second branch network to obtain a prediction result of the second branch network;

[0147] According to the prediction results of the first branch network, the prediction results of the second branch network and the total loss function of the original model, the network parameters of the first branch network and the network parameters of the second branch network are adjusted to complete the semantic learning training and metric learning training of the original model.

[0148] In one embodiment, the processor 1101 is also used to obtain the loss function of the first branch network and the loss function of the second branch network, where the loss function of the first branch network includes a classification loss function and a consistency loss function; determine the weight parameters of each loss function in the loss function of the first branch network and the loss function of the second branch network; and determine the total loss function of the original model based on the loss function of the first branch network, the loss function of the second branch network and the weight parameters of each loss function.

[0149] In one embodiment, the processor 1101 is specifically configured to:

[0150] Inputting the images and corresponding category labels included in the first image sample set into the first branch network to obtain a first prediction result;

[0151] performing enhancement processing on the images included in the first image sample set to obtain enhanced images;

[0152] Inputting the enhanced image and the corresponding category label into the first branch network to obtain a second prediction result;

[0153] The first prediction result and the second prediction result are used as prediction results of the first branch network.

[0154] In one embodiment, the processor 1101 is specifically configured to:

[0155] Inputting the triplets included in the second image sample set into the second branch network to obtain the distance between the images included in each triplet in the second image sample set;

[0156] determining a target triplet as a difficult sample from the triplet included in the second image sample set according to the distance, and obtaining a predicted feature vector of each image included in the target triplet;

[0157] The predicted feature vector of each image included in the target triplet is used as the prediction result of the second branch network.

[0158] In one embodiment, the processor 1101 is further used to obtain an image to be retrieved; input the image to be retrieved into the image feature extraction model to obtain a feature vector of the image to be retrieved; and use the feature vector of the image to be retrieved and an image feature library to determine similar images of the image to be retrieved, wherein the image feature library includes a feature vector of at least one image.

[0159] In one embodiment, the processor 1101 is further used to obtain the image to be registered; input the image to be registered into the image feature extraction model to obtain a feature vector of the image to be registered; and add the feature vector and image identifier of the image to be registered to the image feature library.

[0160] The embodiment of the present invention obtains a training sample set, uses the training sample set to train the semantic learning model and metric learning model in the initialization model, and determines the image feature extraction model based on the trained original model. At the same time, the image feature extraction model is used to process the input image to be retrieved to obtain a feature vector of the image to be retrieved, and the feature vector of the image to be retrieved and the feature vectors in the image feature library are used to determine similar images to the image to be retrieved. It can be seen that the image feature extraction model proposed in this application can have semantic extraction capabilities while realizing metric learning, so that the feature vector of the image to be retrieved obtained by the image feature extraction model can maintain the semantic information of the image, thereby improving the accuracy of image retrieval for unknown images and images with spatial changes such as cropping.

[0161] Those skilled in the art will appreciate that all or part of the processes in the above-described embodiment can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium. When executed, the program can include the processes in the above-described embodiment of the file management method. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0162] One or more embodiments of the present application further provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps performed in the above-described method embodiments.

[0163] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. An image processing method based on artificial intelligence, characterized in that: The method comprises: Acquire a training sample set, where the training sample set includes a first image sample set with category labels and a second image sample set consisting of triplets; The first branch network in the original model is trained for semantic learning using the first image sample set, and the second branch network in the original model is trained for metric learning using the second image sample set; the first branch network and the second branch network include common network parameters; the semantic learning includes: reducing the difference between the first prediction results of the image samples in the first image sample set in the first branch network and the corresponding category labels, and reducing the difference between the first prediction results of the image samples in the first image sample set in the first branch network and the second prediction results of the corresponding enhanced image samples in the first branch network; the metric learning is used to reduce the distance between the reference image sample and the positive image sample in the triplet, and increase the distance between the reference image sample and the negative image sample in the triplet; An image feature extraction model is determined based on the trained original model, where the image feature extraction model is used to extract a feature vector of the image.

2. The method according to claim 1, characterized in that The using the first image sample set to train the first branch network for semantic learning, and the using the second image sample set to train the second branch network for metric learning, includes: Inputting the images and corresponding category labels included in the first image sample set into the first branch network to obtain a prediction result of the first branch network; Inputting the triples included in the second image sample set into the second branch network to obtain a prediction result of the second branch network; According to the prediction results of the first branch network, the prediction results of the second branch network and the total loss function of the original model, the network parameters of the first branch network and the network parameters of the second branch network are adjusted to complete the semantic learning training and metric learning training of the original model.

3. The method according to claim 2, characterized in that Before adjusting the network parameters of the first branch network and the second branch network according to the prediction result of the first branch network, the prediction result of the second branch network, and the total loss function of the original model to complete the semantic learning training and metric learning training of the original model, the method further includes: Obtaining a loss function of the first branch network and a loss function of the second branch network, where the loss function of the first branch network includes a classification loss function and a consistency loss function; Determining a weight parameter of each loss function in the loss function of the first branch network and the loss function of the second branch network; The total loss function of the original model is determined according to the loss function of the first branch network, the loss function of the second branch network, and the weight parameters of the respective loss functions.

4. The method according to claim 2 or 3, characterized in that Inputting the images and corresponding category labels included in the first image sample set into the first branch network to obtain a prediction result of the first branch network includes: Inputting the images and corresponding category labels included in the first image sample set into the first branch network to obtain a first prediction result; performing enhancement processing on the images included in the first image sample set to obtain enhanced images; Inputting the enhanced image and the corresponding category label into the first branch network to obtain a second prediction result; The first prediction result and the second prediction result are used as prediction results of the first branch network.

5. The method according to claim 2 or 3, characterized in that Inputting the triples included in the second image sample set into the second branch network to obtain a prediction result of the second branch network includes: Inputting the triplets included in the second image sample set into the second branch network to obtain the distance between the images included in each triplet in the second image sample set; determining a target triplet as a difficult sample from the triplet included in the second image sample set according to the distance, and obtaining a predicted feature vector of each image included in the target triplet; The predicted feature vector of each image included in the target triplet is used as the prediction result of the second branch network.

6. The method according to claim 1, characterized in that After determining the image feature extraction model based on the trained original model, the method further includes: Get the image to be retrieved; Inputting the image to be retrieved into the image feature extraction model to obtain a feature vector of the image to be retrieved; Similar images of the image to be retrieved are determined using the feature vector of the image to be retrieved and an image feature library, wherein the image feature library includes the feature vector of at least one image.

7. The method according to claim 6, characterized in that Before determining similar images of the image to be retrieved using the feature vector of the image to be retrieved and the image feature library, the method further includes: Get the image to be registered; Inputting the image to be registered into the image feature extraction model to obtain a feature vector of the image to be registered; The feature vector and image identifier of the image to be registered are added to the image feature library.

8. An image processing device based on artificial intelligence, characterized in that: The device comprises: An acquisition module is used to acquire a training sample set, wherein the training sample set includes a first image sample set with category labels and a second image sample set consisting of triplets; A processing module, configured to perform semantic learning training on a first branch network in an original model using the first image sample set, and to perform metric learning training on a second branch network in the original model using the second image sample set; the original model comprises a first branch network and a second branch network, the first branch network and the second branch network comprising shared network parameters; the semantic learning comprises: reducing the difference between a first prediction result of an image sample in the first image sample set in the first branch network and a corresponding category label, and reducing the difference between a first prediction result of an image sample in the first image sample set in the first branch network and a second prediction result of a corresponding enhanced image sample in the first branch network; the metric learning is configured to reduce the distance between a reference image sample and a positive image sample in the triplet, and to increase the distance between a reference image sample and a negative image sample in the triplet; The determination module is used to determine an image feature extraction model based on the trained original model, where the image feature extraction model is used to extract a feature vector of the image.

9. A computer-readable storage medium, characterized in that A computer program is stored, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images