Model training, image classification and image retrieval method based on MPEG-7

Abstract

The embodiment of the invention provides an MPEG-7-based model training, image classification and image retrieval method, and relates to the technical field of image processing. According to the specific implementation scheme, multiple different types of descriptors of a sample image are obtained; adjusting the numerical range of each obtained descriptor according to a preset numerical range corresponding to each descriptor, and splicing the adjusted descriptors to obtain a first feature vector of the sample image; inputting the first feature vector into an initial SVM (Support Vector Machine) to obtain a prediction category of the sample image; calculating a loss value based on the difference between the obtained prediction category and the real category of the sample image; and adjusting model parameters of the SVM based on the loss value. Therefore, through the scheme, the classification precision of the model obtained through training can be improved, and the accuracy of image classification can be improved by using the model obtained through training to carry out image classification.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a model training, image classification and image retrieval method based on MPEG-7. Background Art

[0002] Image classification is widely used in the field of image processing and can be applied to industrial quality inspection, defect detection, target detection and other scenarios. For example, in the defect detection scenario, the defects included in the workpiece image can be classified to determine the workpiece with defects.

[0003] Since image classification helps to manage and utilize large amounts of image data more efficiently, how to improve the accuracy of image classification is particularly important. Summary of the invention

[0004] The purpose of the embodiment of the present application is to provide a model training method based on MPEG-7 to improve the classification accuracy of the trained model; and to provide an image classification method based on MPEG-7 to improve the accuracy of image classification; and to provide a retrieval method based on MPEG-7 to improve the retrieval efficiency on the basis of improving the accuracy of image classification, and to provide an image classification and retrieval system based on MPEG-7 to improve the accuracy of image classification and the retrieval efficiency. The specific technical solutions are as follows:

[0005] In a first aspect, an embodiment of the present application provides a model training method based on MPEG-7, the method comprising:

[0006] Acquire descriptors of multiple different categories of sample images; wherein the descriptors are extracted using a multimedia content description interface MPEG-7 descriptor extraction algorithm, and the multiple different categories of descriptors include at least two of the following: a first category of descriptors representing color features, a second category of descriptors representing shape features, and a third category of descriptors representing texture features;

[0007] According to the preset numerical ranges corresponding to the descriptors of each category, the obtained numerical ranges of the descriptors are adjusted, and the adjusted descriptors are spliced ​​to obtain a first feature vector of the sample image; wherein the length of the preset numerical range corresponding to the first category descriptor is greater than the length of the preset numerical range corresponding to the second category descriptor, and the length of the preset numerical range corresponding to the second category descriptor is greater than the length of the preset numerical range corresponding to the third category descriptor;

[0008] Inputting the first feature vector into an initial support vector machine SVM to obtain a predicted category of the sample image;

[0009] Calculating a loss value based on the difference between the obtained predicted category and the true category of the sample image;

[0010] Based on the loss value, the model parameters of the SVM are adjusted.

[0011] In a second aspect, an embodiment of the present application provides an image classification method based on MPEG-7, the method comprising:

[0012] Acquire descriptors of multiple different categories of the image to be classified; wherein the descriptors are extracted using a multimedia content description interface MPEG-7 descriptor extraction algorithm, and the multiple different categories of descriptors include at least two of the following: a first category descriptor representing color features, a second category descriptor representing shape features, and a third category descriptor representing texture features;

[0013] According to the preset numerical ranges corresponding to the descriptors of each category, the obtained numerical ranges of the descriptors are adjusted, and the adjusted descriptors are spliced ​​to obtain a second feature vector of the image to be classified; wherein the length of the preset numerical range corresponding to the first category descriptor is greater than the length of the preset numerical range corresponding to the second category descriptor, and the length of the preset numerical range corresponding to the second category descriptor is greater than the length of the preset numerical range corresponding to the third category descriptor;

[0014] Inputting the second feature vector into at least one pre-trained SVM respectively to obtain the prediction probability corresponding to the output result of each SVM; wherein the prediction probability corresponding to the output result of each SVM represents the probability that the image to be classified belongs to the corresponding prediction category of the SVM, and each SVM is trained based on any one of the above-mentioned MPEG-7-based model training methods;

[0015] The obtained prediction probabilities are normalized, and the prediction category corresponding to the normalized maximum value is determined as the category to which the image to be classified belongs.

[0016] In a third aspect, an embodiment of the present application provides an image retrieval method based on MPEG-7, the method comprising:

[0017] Obtaining a descriptor of the image to be detected as a descriptor to be matched; wherein the descriptor to be matched is extracted using a multimedia content description interface MPEG-7 descriptor extraction algorithm;

[0018] Using any one of the above-mentioned MPEG-7-based image classification methods, the image to be detected is classified to obtain the category to which the image to be detected belongs;

[0019] Obtaining candidate descriptors of candidate images belonging to the same category as the image to be detected; wherein the candidate descriptors are descriptors corresponding to the descriptors to be matched, and the category of each candidate image is obtained by classifying the image based on any one of the above-mentioned MPEG-7-based image classification methods;

[0020] Calculating the similarity between the to-be-matched descriptor and the candidate descriptor of each candidate image respectively;

[0021] A preset number of candidate images with the highest corresponding similarities are determined as retrieval results for the image to be detected.

[0022] In a fourth aspect, an embodiment of the present application provides an image classification and retrieval system based on MPEG-7, the system comprising a feature extraction module, a classification recognition module and an image retrieval module;

[0023] The feature extraction module is used to respectively extract descriptors of multiple different categories of images to be classified using the multimedia content description interface MPEG-7 descriptor extraction algorithm, and to extract descriptors of images to be detected using the MPEG-7 descriptor extraction algorithm;

[0024] The classification recognition module is used to obtain descriptors of multiple different categories of the image to be classified from the feature extraction module, and execute any one of the above-mentioned image classification methods based on MPEG-7;

[0025] The image retrieval module is used to obtain the descriptor of the image to be detected from the feature extraction module and execute any one of the above-mentioned image retrieval methods based on MPEG-7.

[0026] In a fifth aspect, an embodiment of the present application provides a model training device based on MPEG-7, the device comprising:

[0027] A first acquisition module is used to acquire descriptors of various categories of sample images; wherein the descriptors are extracted using a multimedia content description interface MPEG-7 descriptor extraction algorithm, and the various descriptors of various categories include at least two of the following: a first descriptor representing a color feature, a second descriptor representing a shape feature, and a third descriptor representing a texture feature;

[0028] A first adjustment module, configured to adjust the obtained value ranges of the descriptors according to the preset value ranges corresponding to the descriptors of each category, and to splice the adjusted descriptors to obtain a first feature vector of the sample image; wherein the length of the preset value range corresponding to the first category descriptor is greater than the length of the preset value range corresponding to the second category descriptor, and the length of the preset value range corresponding to the second category descriptor is greater than the length of the preset value range corresponding to the third category descriptor;

[0029] A first prediction module, used for inputting the first feature vector into an initial support vector machine SVM to obtain a predicted category of the sample image;

[0030] A loss calculation module, used for calculating a loss value based on the difference between the obtained predicted category and the true category of the sample image;

[0031] A model parameter adjustment module is used to adjust the model parameters of the SVM based on the loss value.

[0032] In a sixth aspect, an embodiment of the present application provides an image classification device based on MPEG-7, the device comprising:

[0033] A second acquisition module is used to acquire descriptors of multiple different categories of the image to be classified; wherein the descriptors are extracted using an MPEG-7 descriptor extraction algorithm, and the multiple different categories of descriptors include at least two of the following: a first category of descriptors representing color features, a second category of descriptors representing shape features, and a third category of descriptors representing texture features;

[0034] A second adjustment module is used to adjust the obtained value ranges of the descriptors according to the preset value ranges corresponding to the descriptors of each category, and splice the adjusted descriptors to obtain a second feature vector of the image to be classified; wherein the length of the preset value range corresponding to the first category descriptor is greater than the length of the preset value range corresponding to the second category descriptor, and the length of the preset value range corresponding to the second category descriptor is greater than the length of the preset value range corresponding to the third category descriptor;

[0035] A second prediction module is used to input the second feature vector into at least one pre-trained SVM to obtain a prediction probability corresponding to the output result of each SVM; wherein the prediction probability corresponding to the output result of each SVM represents the probability that the image to be classified belongs to the prediction category corresponding to the SVM, and each SVM is trained based on any one of the above-mentioned MPEG-7-based model training methods;

[0036] The normalization module is used to normalize the obtained prediction probabilities and determine the prediction category corresponding to the normalized maximum value as the category to which the image to be classified belongs.

[0037] In a seventh aspect, an embodiment of the present application provides an image retrieval device based on MPEG-7, the device comprising:

[0038] A third acquisition module is used to acquire a descriptor of the image to be detected as a descriptor to be matched; wherein the descriptor to be matched is extracted using an MPEG-7 descriptor extraction algorithm;

[0039] A category determination module, used to classify the image to be detected by using any of the above-mentioned MPEG-7-based image classification methods to obtain the category to which the image to be detected belongs;

[0040] a fourth acquisition module, configured to acquire candidate descriptors of candidate images belonging to the same category as the image to be detected; wherein the candidate descriptors are descriptors corresponding to the descriptors to be matched, and the category of each candidate image is obtained by classifying the image based on any one of the above-mentioned MPEG-7-based image classification methods;

[0041] A similarity calculation module, used to calculate the similarity between the descriptor to be matched and the candidate descriptor of each candidate image respectively;

[0042] The retrieval result determination module is used to determine the first preset number of candidate images with the highest corresponding similarity as the retrieval results for the image to be detected.

[0043] In an eighth aspect, an embodiment of the present application provides an electronic device, including:

[0044] Memory, used to store computer programs;

[0045] The processor is used to implement any of the above-mentioned model training methods based on MPEG-7, image classification methods based on MPEG-7 or image retrieval methods based on MPEG-7 when executing the program stored in the memory.

[0046] In the ninth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements any of the above-mentioned MPEG-7-based model training methods, MPEG-7-based image classification methods, or MPEG-7-based image retrieval methods.

[0047] In the tenth aspect, an embodiment of the present application provides a computer program product, which includes executable instructions. When the executable instructions are executed on a computer, the computer executes any one of the above-mentioned MPEG-7-based model training methods, MPEG-7-based image classification methods, or MPEG-7-based image retrieval methods.

[0048] Beneficial effects of the embodiments of the present application: The solution provided by the embodiments of the present application, since the first feature vector is obtained by splicing the adjusted descriptors, the numerical ranges of the adjusted descriptors satisfy: the length of the numerical range of the first type of descriptors is greater than the length of the numerical range of the second type of descriptors, and the length of the numerical range of the second type of descriptors is greater than the length of the numerical range of the third type of descriptors; and the feature with a large numerical range has a greater impact on the classification of the SVM than the feature with a small numerical range. Therefore, the first feature vector is input into the SVM to train the SVM, so that the trained SVM pays more attention to color features, shape features, and texture features during recognition and classification. Then the SVM can recognize and classify images based on the progressive relationship of color-shape-texture. Since the progressive relationship of color-shape-texture conforms to the logical process of human eye recognition of objects, the SVM can recognize and classify objects by imitating the logical process of human eye recognition of objects. Then, the model parameters of the SVM are adjusted by using the difference between the predicted category and the real category output by the SVM, which can improve the classification accuracy of the trained model. Further, using the trained SVM for image classification can improve the accuracy of image classification.

[0049] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0051] Figure 1 A flowchart of a model training method based on MPEG-7 provided in an embodiment of the present application;

[0052] Figure 2 A schematic diagram of an information upload interface provided in an embodiment of the present application;

[0053] Figure 3 A schematic diagram of a vector diagram construction interface provided in an embodiment of the present application;

[0054] Figure 4 A flowchart of an image classification method based on MPEG-7 provided in an embodiment of the present application;

[0055] Figure 5 A flowchart of an image retrieval method based on MPEG-7 provided in an embodiment of the present application;

[0056] Figure 6 A schematic diagram of a search mode selection interface provided in an embodiment of the present application;

[0057] Figure 7 A schematic diagram of the structure of an MPEG-7-based image classification and retrieval system provided in an embodiment of the present application;

[0058] Figure 8 A schematic diagram of the interaction between the modules included in the system provided in the embodiment of the present application;

[0059] Fig. 9 A workflow diagram for each module implemented by the system provided in the embodiment of the present application;

[0060] Fig.10 A schematic diagram of the structure of a model training device based on MPEG-7 provided in an embodiment of the present application;

[0061] Fig.11 A schematic diagram of the structure of an image classification device based on MPEG-7 provided in an embodiment of the present application;

[0062] Fig.12 A schematic diagram of the structure of an image retrieval device based on MPEG-7 provided in an embodiment of the present application;

[0063] Fig.13 A block diagram of an electronic device for implementing the method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0065] Below, the professional terms involved in the embodiments of the present application are first introduced:

[0066] SVM (Support Vector Machine): A supervised learning model commonly used for classification and regression analysis. SVM achieves classification by finding the optimal hyperplane to maximize the interval between data points.

[0067] MPEG-7 (Multimedia Content Description Interface): used to describe the characteristics of multimedia content. This standard defines a set of descriptors for describing multimedia content such as images, audio, and video. These descriptors can describe the color, texture, shape and other characteristics of the image, and can extract specific numerical descriptions from the multimedia content through corresponding extraction algorithms.

[0068] The model training method based on MPEG-7, the image classification method based on MPEG-7 and the image retrieval method based on MPEG-7 provided in the embodiments of the present application can be applied to various electronic devices, such as personal computers, servers, and other devices with data processing capabilities. In addition, it can be understood that the model training method based on MPEG-7, the image classification method based on MPEG-7 and the image retrieval method based on MPEG-7 provided in the embodiments of the present application can be implemented by software, hardware or a combination of software and hardware.

[0069] Among them, an embodiment of the present application provides a model training method based on MPEG-7, including:

[0070] Acquire multiple different types of descriptors of the sample image; wherein the descriptors are extracted using an MPEG-7 descriptor extraction algorithm, and the multiple different types of descriptors include at least two of the following: a first type of descriptor representing color features, a second type of descriptor representing shape features, and a third type of descriptor representing texture features;

[0071] According to the preset numerical ranges corresponding to the descriptors of each category, the obtained numerical ranges of the descriptors are adjusted, and the adjusted descriptors are spliced ​​to obtain a first feature vector of the sample image; wherein the length of the preset numerical range corresponding to the first category descriptor is greater than the length of the preset numerical range corresponding to the second category descriptor, and the length of the preset numerical range corresponding to the second category descriptor is greater than the length of the preset numerical range corresponding to the third category descriptor;

[0072] Input the first feature vector into the initial SVM to obtain the predicted category of the sample image;

[0073] Calculate the loss value based on the difference between the predicted category and the true category of the sample image;

[0074] Based on the loss value, the model parameters of the SVM are adjusted.

[0075] In the solution provided by the embodiment of the present application, since the first feature vector is obtained by splicing the adjusted descriptors, the numerical ranges of the adjusted descriptors satisfy: the length of the numerical range of the first type of descriptors is greater than the length of the numerical range of the second type of descriptors, and the length of the numerical range of the second type of descriptors is greater than the length of the numerical range of the third type of descriptors; and the feature with a large numerical range has a greater impact on the classification of the SVM than the feature with a small numerical range. Therefore, the first feature vector is input into the SVM to train the SVM, so that the trained SVM pays more attention to color features during recognition and classification, and then pays attention to shape features, and finally pays attention to texture features. Then the SVM can recognize and classify images based on the progressive relationship of color-shape-texture. Since the progressive relationship of color-shape-texture conforms to the logical process of human eye recognition of objects, the SVM can recognize and classify objects by imitating the logical process of human eye recognition of objects. Then, the model parameters of the SVM are adjusted by using the difference between the predicted category and the real category output by the SVM, which can improve the classification accuracy of the trained model. Further, using the trained SVM for image classification can improve the accuracy of image classification.

[0076] Combine the following Figure 1 An MPEG-7 based model training method provided in an embodiment of the present application is introduced. Figure 1 As shown, the method includes steps S101-S105:

[0077] S101, obtaining descriptors of multiple different categories of a sample image; wherein the descriptors of multiple different categories are extracted using an MPEG-7 descriptor extraction algorithm, and the descriptors of multiple different categories include at least two of the following: a first type of descriptor representing a color feature, a second type of descriptor representing a shape feature, and a third type of descriptor representing a texture feature;

[0078] In this embodiment, the sample image can be obtained from an open source image dataset, or pre-collected and stored in the local memory of the electronic device, or collected in real time by the image acquisition module of the electronic device, which is reasonable. Exemplarily, the sample image can come from CIFAR-10 (a dataset containing 10 categories of images), CIFAR-100 (a dataset containing 100 categories of images), etc.

[0079] It is understandable that MPEG-7 provides a variety of descriptors, including Color Layout Descriptor (CLD), Scalable Color Descriptor (SCD), Edge Histogram Descriptor (EHD), Color Structure Descriptor (CSD), Texture Browsing Descriptor (TBD), Homogeneous Texture Descriptor (HTD), Contour Shape Descriptor (CSD), Region Shape Descriptor (RSD), etc. In addition, MPEG-7 provides a corresponding extraction algorithm for each descriptor. For example, the extraction algorithm of CLD includes steps such as image segmentation, representative color selection, DCT (Discrete Cosine Transform) and zigzag scanning. That is, the CLD extraction algorithm divides the sample image into multiple blocks, calculates the average pixel color of each block, then performs DCT transformation on these colors, and groups the low-frequency coefficients through zigzag scanning, and finally forms the color layout descriptor of the sample image. It should be noted that each descriptor of the extracted sample image is a numerical sequence, and the numerical sequence includes a predetermined number of digits, which can be set by relevant technical personnel.

[0080] It can be understood that the color layout descriptor performs discrete cosine transform on the color sub-block to extract the spatial distribution characteristics of the color mode; the scalable color descriptor describes the color characteristics of the image based on hue statistics and is scalable; the edge histogram descriptor describes the distribution of edges in the image and is obtained by statistical edge direction histogram; the color structure descriptor counts the occurrence of each color box in the window; the texture layout descriptor calculates the grayscale co-occurrence matrix in each block to obtain the corresponding result; the homogeneous texture descriptor captures the texture information by analyzing the spectrum of the image; the contour shape descriptor simplifies the number of contour points through the Douglas Peucker algorithm and obtains the Fourier transform coefficients; the region shape descriptor calculates the moments of each order and the central moment of the contour to describe the shape characteristics of the region. Then, the first type of descriptors representing color characteristics may include color layout descriptors, scalable color descriptors, etc., the second type of descriptors representing shape characteristics may include edge histogram descriptors, contour shape descriptors, region shape descriptors, etc., and the third type of descriptors representing texture characteristics may include texture layout descriptors, homogeneous texture descriptors, etc.

[0081] In this embodiment, the MPEG-7 descriptor extraction algorithm can be used in advance to extract descriptors of various different categories of the sample image, that is, the descriptors of the sample image can be extracted using the extraction algorithms of the descriptors provided by MPEG-7 to obtain descriptors of various different categories. For example, the color layout descriptor of the sample image is extracted using the CLD extraction algorithm provided by MPEG-7, and the texture layout descriptor of the sample image is extracted using the TBD extraction algorithm provided by MPEG-7. Then, the extracted descriptors include a first-category descriptor representing color features and a third-category descriptor representing texture features. It can be understood that after extracting descriptors of various different categories of the sample image, the descriptors of various different categories of the sample image can be stored in an XML file, so that when executing this solution, the descriptors of various different categories of the sample image can be directly obtained from the XML file.

[0082] It should be noted that, in practical applications, the descriptors of various different categories that need to be extracted for sample images during model training can be determined based on the features of the objects of interest in the actual classification scene. For example, if a model for classifying images containing kittens needs to be trained, since the color features and texture features displayed by different kittens in the images are quite different, then the features of the objects of interest in the actual classification scene are color features and texture features. At this time, the descriptors of various different categories may include a first type of descriptor representing color features and a third type of descriptor representing texture features. For another example, if a model for classifying images containing water cups needs to be trained, since the color features and shape features displayed by different water cups in the images are quite different, then the features of the objects of interest in the actual classification scene are color features and shape features. At this time, the descriptors of various different categories may include a first type of descriptor representing color features and a second type of descriptor representing shape features.

[0083] S102, adjusting the obtained value ranges of the descriptors according to the preset value ranges corresponding to the descriptors of each category, and splicing the adjusted descriptors to obtain a first feature vector of the sample image; wherein the length of the preset value range corresponding to the first category descriptor is greater than the length of the preset value range corresponding to the second category descriptor, and the length of the preset value range corresponding to the second category descriptor is greater than the length of the preset value range corresponding to the third category descriptor;

[0084] In this embodiment, each category of descriptors corresponds to a preset numerical range, and the length of the preset numerical range corresponding to the first category of descriptors is greater than the length of the preset numerical range corresponding to the second category of descriptors, and the length of the preset numerical range corresponding to the second category of descriptors is greater than the length of the preset numerical range corresponding to the third category of descriptors. It should be noted that the preset numerical range corresponding to each category of descriptors is an absolute value range, that is, both endpoints of the preset numerical range are non-negative numbers. The absolute values ​​of the values ​​contained in each adjusted descriptor are within the preset numerical range corresponding to the category to which the descriptor belongs.

[0085] For example, in one implementation, the preset numerical range corresponding to the first type of descriptor may be (0, 100), the preset numerical range corresponding to the second type of descriptor may be (0, 50), and the preset numerical range corresponding to the third type of descriptor may be (0, 10). It should be noted that, in actual applications, the specific upper and lower limits of the preset numerical ranges corresponding to the descriptors of each category may be set by relevant technicians based on experience, and the embodiments of the present application do not limit this.

[0086] It can be understood that, for SVM, the longer the length of the numerical range, the greater the influence of the feature on the classification hyperplane. By setting the length of the numerical range of the first type of descriptor to be greater than the length of the numerical range of the second type of descriptor, and the length of the numerical range of the second type of descriptor to be greater than the length of the numerical range of the third type of descriptor, the color feature has a greater influence on the classification hyperplane than the shape feature, and the shape feature has a greater influence on the classification hyperplane than the texture feature. Then, the numerical ranges of the obtained descriptors are adjusted and spliced ​​to obtain the first feature vector. When the first feature vector is subsequently input into the SVM for training, the trained SVM can pay more attention to the color feature during classification, then the shape feature, and finally the texture feature, so that the trained SVM can recognize and classify images in accordance with the logical process of human eye recognition of objects, that is, in the order of color-shape-texture, first observe the color of the object in the image, then consider the shape of the object, and then compare the surface texture of the object, so that the recognition and classification process is more scientific and effective.

[0087] Optionally, in one implementation, according to preset value ranges corresponding to descriptors of each category, the obtained value ranges of the descriptors are adjusted, and the adjusted descriptors are spliced ​​to obtain a first feature vector of the sample image, including steps A1-A2:

[0088] A1, for each descriptor obtained, mapping each value contained in the descriptor into a specified value range to obtain the adjusted descriptor; wherein the maximum absolute value and the minimum absolute value of the value in the specified value range are respectively the two endpoints of the preset value range corresponding to the category to which the descriptor belongs;

[0089] It can be understood that, since for each descriptor obtained, the maximum absolute value and the minimum absolute value of the value within the specified numerical range are respectively the two endpoints of the preset numerical range corresponding to the category to which the descriptor belongs, by mapping each numerical value contained in the descriptor to the specified numerical range, the absolute value of each numerical value contained in each adjusted descriptor can be within the preset numerical range corresponding to the category to which the descriptor belongs. In addition, since the preset numerical ranges corresponding to descriptors of different categories are different, the specified numerical ranges to which descriptors of different categories are mapped are different.

[0090] Exemplarily, if a regional shape descriptor of a sample image is extracted, the regional shape descriptor belongs to the second category of descriptors. If the preset numerical range corresponding to the second category of descriptors is (0, 50), then the specified numerical range may be (-50, 50), (-20, 50), etc. At this time, the maximum absolute value of the numerical value in the specified numerical range is 50, and the minimum absolute value is 0, satisfying that the maximum absolute value and the minimum absolute value of the numerical value in the specified numerical range are the two endpoints of the preset numerical range respectively.

[0091] Exemplarily, a method for mapping each numerical value contained in a descriptor to a specified numerical range may be: first, using a normalization algorithm to map each numerical value contained in each descriptor to a range of (-1, 1), and then, based on the linear relationship between the specified numerical range and (-1, 1), mapping each numerical value mapped to the range of (-1, 1) to a preset numerical range corresponding to the category to which the descriptor belongs.

[0092] A2, concatenating the adjusted descriptors to obtain the first feature vector of the sample image.

[0093] In this implementation, the splicing order of each descriptor can be preset, and then each descriptor is spliced ​​according to the splicing order to obtain the first feature vector of the sample image. Exemplarily, if the extracted descriptors include a color layout descriptor, a scalable color descriptor, and an edge histogram descriptor, and the preset splicing order is: the scalable color descriptor is spliced ​​after the color layout descriptor, and the edge histogram descriptor is spliced ​​after the scalable color descriptor, then the descriptors are spliced ​​according to the splicing order to obtain the first feature vector. It can be understood that each element in the first feature vector is a value contained in a descriptor.

[0094] It should be noted that since SVM does not care about the splicing order between the descriptors that constitute the input feature vector, that is, the splicing order between the descriptors does not affect the SVM classification of features, the splicing order of the descriptors can be set by relevant technical personnel based on experience, and the embodiment of the present application does not limit the splicing order of the descriptors.

[0095] In addition, it should be noted that in actual applications, after extracting multiple descriptors for each sample image, the descriptors are concatenated in a certain order, and the first feature vector obtained can be stored in an XML (Extensible Markup Language) file as a description scheme. When training the SVM, the stored description schemes can be directly obtained from the XML file to construct a training data set and a verification data set to train and verify the SVM. For example, 80% of the description schemes stored in the XML file are constructed as a training data set, and the remaining 20% ​​of the description schemes are constructed as a verification data set.

[0096] S103, inputting the first feature vector into an initial support vector machine to obtain a predicted category of the sample image;

[0097] It can be understood that SVM is a two-classification model, and its basic principle is to find an optimal classification hyperplane to separate data points of different categories and maximize the interval. The first eigenvector is input into the initial SVM, and the SVM outputs the predicted category of the sample image. For example, if the SVM is used to classify objects of category A and non-category A, 1 represents category A, and -1 represents non-category A. If the SVM outputs 1, it means that the predicted category of the sample image is category A. In addition, since the training goal of the SVM is to find an optimal decision boundary that can separate data points of different categories, the first eigenvectors of multiple sample images need to be input when training the SVM. At this time, the predicted category of each sample image is obtained.

[0098] In this embodiment, the initial SVM uses a radial basis function (RBF) as a kernel function. The radial basis function is expressed as:

[0099]

[0100] in, is the first eigenvector of the i-th sample image, is the first eigenvector of the jth sample image, represents the similarity between the first eigenvector of the i-th sample image and the first eigenvector of the j-th sample image, is a parameter that controls the influence range of the RBF kernel, and ||·|| represents the norm.

[0101] It can be understood that the kernel function of SVM is used to map the low-dimensional input space to the high-dimensional feature space, so that the nonlinear problem becomes linearly separable in the high-dimensional space. By using the kernel function, SVM can find a suitable classification hyperplane in the high-dimensional space, thereby effectively solving the nonlinear classification problem.

[0102] S104, calculating a loss value based on the difference between the obtained predicted category and the true category of the sample image;

[0103] In this embodiment, the difference between the predicted category of the sample image and the true category of the sample image can be calculated using the L1 (absolute value) loss function or the L2 (mean square error) loss function to obtain a loss value. It is understandable that in practical applications, relevant technical personnel can mark the true category of the sample image in advance, and after obtaining the predicted category of each sample image, the loss value is obtained by calculating the total difference between the predicted category and the true category of each sample image. It should be noted that the embodiment of the present application does not limit the method for calculating the difference between the predicted category and the true category of the sample image.

[0104] S105: Adjust the model parameters of the support vector machine based on the loss value.

[0105] In this embodiment, after the loss value is obtained, the model parameters can be adjusted by minimizing the back propagation of the loss value until the preset training end condition is reached. It can be understood that adjusting the model parameters includes adjusting the radial basis function. Parameters are adjusted. Exemplarily, the training end condition may be reaching a preset number of training times, or SVM convergence, which are both reasonable.

[0106] In the solution provided by the embodiment of the present application, since the first feature vector is obtained by splicing the adjusted descriptors, the numerical ranges of the adjusted descriptors satisfy: the length of the numerical range of the first type of descriptors is greater than the length of the numerical range of the second type of descriptors, and the length of the numerical range of the second type of descriptors is greater than the length of the numerical range of the third type of descriptors; and the feature with a large numerical range has a greater impact on the classification of the SVM than the feature with a small numerical range. Therefore, the first feature vector is input into the SVM to train the SVM, so that the trained SVM pays more attention to color features during recognition and classification, and then pays attention to shape features, and finally pays attention to texture features. Then the SVM can recognize and classify images based on the progressive relationship of color-shape-texture. Since the progressive relationship of color-shape-texture conforms to the logical process of human eye recognition of objects, the SVM can recognize and classify objects by imitating the logical process of human eye recognition of objects. Then, the model parameters of the SVM are adjusted by using the difference between the predicted category and the real category output by the SVM, which can improve the classification accuracy of the trained model. Further, using the trained SVM for image classification can improve the accuracy of image classification.

[0107] Optionally, in another embodiment of the present application, the multiple different categories of descriptors include at least two of the following: a first category of descriptors representing color features, a second category of descriptors representing shape features, a third category of descriptors representing texture features, and a fourth category of descriptors representing associations between preset objects in an image, and the length of a preset numerical range corresponding to the third category of descriptors is greater than the length of a preset numerical range corresponding to the fourth category of descriptors.

[0108] In this embodiment, the association relationship between preset objects in the image can also be extracted to obtain a fourth type of descriptor. Exemplarily, in one implementation, the preset numerical range corresponding to the first type of descriptor can be (0, 100), the preset numerical range corresponding to the second type of descriptor can be (0, 50), the preset numerical range corresponding to the third type of descriptor can be (0, 10), and the preset numerical range corresponding to the fourth type of descriptor can be (0, 1). It should be noted that after extracting the fourth type of descriptor of the sample image, the method described in step A1 above can be referred to to map the various numerical values ​​contained in the fourth type of descriptor to the specified numerical range, which will not be repeated here.

[0109] It is understandable that for color images, color is often the most important feature for distinguishing categories, followed by shape, texture, etc. For images containing the same or similar objects, the association relationship of the objects in the image is an important way to distinguish. Since the logical process of human eyes recognizing objects follows the order of color-shape-texture-association relationship, that is, first observe the color of the object in the image, then consider the shape of the object, then compare the surface texture of the object, and finally compare the association relationship between objects from the global comparison map to complete the recognition and classification of the image content, therefore, by setting the length of the preset numerical range corresponding to the third type of descriptor to be greater than the length of the preset numerical range corresponding to the fourth type of descriptor, the influence of the texture feature on the classification hyperplane is greater than the influence of the association relationship between the preset objects in the image on the classification hyperplane, so that the trained SVM can imitate the logical process of human eyes to recognize objects for classification, that is, recognize and classify in the order of color-shape-texture-association relationship, so that the SVM adopts a more scientific and effective way to realize the classification of images, which can improve the classification accuracy.

[0110] Exemplarily, the preset objects in the sample image can be marked in advance in the sample image by relevant technicians, for example, people, events, time, place, etc. in the sample image can be marked, and the marked people and events are the preset objects. The association relationship between the preset objects may include subordinate relationship, interpersonal relationship, position relationship, etc.

[0111] Optionally, in one implementation, the fourth type of descriptor of the sample image is obtained through the following steps B1-B2:

[0112] B1, obtaining a pre-built relationship vector diagram for the sample image; wherein the relationship vector diagram includes a plurality of nodes and association relationships between the nodes, a node represents a preset object in the sample image, and the association relationships between the nodes represent association relationships between the corresponding preset objects;

[0113] B2, use the region descriptor extraction algorithm to extract features from the relationship vector map and obtain the fourth type of descriptor of the sample image.

[0114] In this implementation, the relevant technicians can construct a relationship vector map for the sample image in advance, that is, determine the preset objects in the sample image, take each preset object as a node, and set the association relationship between the nodes to obtain the relationship vector map of the sample image. For example, if the relevant technicians determine that the characters A, B, and C contained in a sample image are preset objects, the relationship between characters A and B is a kinship relationship, the relationship between characters A and C is a friend relationship, and characters B and C are friends, then a node A representing character A, a node B representing character B, and a node C representing character C can be created, and the directed line segment from node A to node B is marked as a kinship relationship, the directed line segment from node B to node C is marked as a friend relationship, and the directed line segment from node C to node A is marked as a friend relationship, then a topological map consisting of nodes A, nodes B, nodes C, and each directed line segment can be obtained, and the topological map is the relationship vector map of the sample image.

[0115] In this implementation, the region descriptor extraction algorithm is an extraction algorithm for extracting region descriptors provided by MPEG-7. It can be understood that since the region descriptor is used to describe the geometric and topological structural features of the region in the image, and the relationship vector map constructed based on the association relationship between the preset objects is in the form of a topological map, after obtaining the relationship vector map of the sample image, the region descriptor extraction algorithm can be used to extract features from the relationship vector map to obtain the fourth type of descriptor of the sample image. The fourth type of descriptor can also be called a relationship graph descriptor (RGD, Relation Graph Descriptor).

[0116] Optionally, in an implementation manner, the relationship vector diagram is obtained by following the steps C1-C3:

[0117] C1, displaying a sample image, a text input box, and embedded information of the sample image in the information upload interface;

[0118] C2, obtaining a description text input by the user in the text input box for describing the association relationship between preset objects in the sample image;

[0119] C3, obtains the relationship vector graph for the sample image created by the user in the vector graph construction interface based on the description text and embedded information.

[0120] In this implementation, the user can enter and save the description text used to describe the association relationship between preset objects in the sample image in the information upload interface of the client for annotating the image. Then, the user can open the vector map construction interface in the client and create the relationship vector map of the sample image in the interface based on the description text and embedded information of the sample image. The created relationship vector map can be stored in a preset storage address, so that the relationship vector map of the sample image can be obtained from the preset storage address later.

[0121] Exemplarily, the description text used to describe the association relationship between preset objects in the sample image may include: structured information such as the object, time, place, event, action included in the image, or other image content information filled in according to the visual content expressed by the image, etc. Exemplarily, the embedded information of the sample image includes the shooting time, photographer, shooting location, image format, image size and other information of the sample image. It can be understood that the embedded information of the image is the attribute information carried in the image when the image is collected. In this implementation, the client used to annotate the image can directly extract part or all of the attribute information from the attribute information carried by the sample image as the embedded information of the sample image.

[0122] For example, the information upload interface is as follows: Figure 2 As shown in the figure, when the user opens the interface, the rectangular box under the "File Directory" displays the storage addresses of multiple sample images in a list form. The user can select a storage address of a sample image from the multiple storage addresses. At this time, the sample image stored in the selected storage address will be displayed in Figure 2 In addition, the interface displays a "creation, exhibition and technical information" control for automatically extracting embedded information of the sample image, including the shooting time, image format and image size of the sample image, and displays it in a rectangular box corresponding to the "creation, exhibition and technical information" control. Figure 2 Various text input boxes are also displayed, including text input boxes corresponding to "structured text description" and text input boxes corresponding to "free text description". The text input boxes corresponding to "structured text description" include text input boxes for entering "place", "time", "object" and "action", and users can fill in each text input box according to the visual content expressed by the sample image.

[0123] For example, if the visual content expressed by a sample image is that person A is drinking an iced drink on a lounge chair on the beach, then based on the sample image, you can enter "beach" in the text input box corresponding to "place", enter "summer" in the text input box corresponding to "time", enter "person A" in the text input box corresponding to "object", and enter "drinking an iced drink" in the text input box corresponding to "action". In addition, you can also fill in other visual content expressed by the sample image in the text input box corresponding to "free text description", for example, fill in the detailed description of the posture and clothing of person A in the text input box corresponding to "free text description".

[0124] like Figure 2 As shown, a control corresponding to "image quality rating" is also displayed below each type of text input box. The control is used to adjust the image quality. The image quality includes three levels, namely "excellent", "average" and "poor". The user can adjust the image quality rating according to the fidelity and intelligibility of the sample image. Below the control corresponding to "image quality rating", various text input boxes corresponding to "metadata description" are also displayed, including a text input box for inputting "metadata creator" and a text input box for inputting "comments". The text input box corresponding to "metadata creator" is used to input the ID of the user who created the description text, and the text input box corresponding to "comments" is used to input the user's comments on the image quality of the sample image.

[0125] For example, the vector graphics interface is as follows: Figure 3 As shown, Figure 3 On the right side of the screen, there are display boxes corresponding to "Person", "Event", and "Time, Place". The display box corresponding to "Person" displays the persons entered by the user in the information upload interface, the display box corresponding to "Event" displays the events entered by the user in the information upload interface, and the display box corresponding to "Time, Place" displays the times and places entered by the user in the information upload interface. There is also an add button ( Figure 3 A rectangle containing a "+" in the middle) and a delete button ( Figure 3 ), you can click the add button set below any display box to add preset objects such as objects, events, and places. After adding, the nodes corresponding to the added preset objects will be displayed in the left area. The left area is used to construct a relationship vector map. Figure 3As shown, "chat" is a node representing an event, "summer" is a node representing time, and "performance" is a node representing an event. By clicking on each node and moving to other nodes, a directed line segment is formed and an association relationship selection is provided, including subordinate relationship, interpersonal relationship, and position relationship, etc. For example, a subordinate relationship is selected between the "chat" node and the "summer" node, and a subordinate relationship is selected between the "summer" and "performance" nodes.

[0126] It can be understood that the relationship vector diagram constructed in the above manner can express the semantic structure of the image, and by establishing association relationships between nodes, a relationship vector diagram expressing the semantic relationship of each preset object in the image can be formed. The relationship vector diagram created for each sample image can be stored together with the description scheme of the sample image in the XML file of the sample image.

[0127] It can be seen that through this scheme, by constructing a fourth type of descriptor representing the association relationship between preset objects in the sample image, richer semantic features in the image can be extracted, so that the classification model trained using the first feature vector constructed using the fourth type of descriptor can further improve the classification accuracy.

[0128] Optionally, in another embodiment of the present application, an image classification method based on MPEG-7 is provided, such as Figure 4 As shown, steps S401-S404 are included:

[0129] S401, obtaining descriptors of multiple different categories of the image to be classified; wherein the descriptors are extracted using an MPEG-7 descriptor extraction algorithm, and the multiple different categories of descriptors include at least two of the following: a first category descriptor representing a color feature, a second category descriptor representing a shape feature, and a third category descriptor representing a texture feature;

[0130] In this embodiment, the images to be classified can be images contained in a database to be searched for images, or images collected in real time by an image acquisition module of an electronic device, which is reasonable. The relevant contents of step S401 can refer to the relevant description of step S101 above, and will not be repeated here.

[0131] Optionally, in one implementation, the multiple different categories of descriptors include: at least two of a first category of descriptors representing color features, a second category of descriptors representing shape features, a third category of descriptors representing texture features, and a fourth category of descriptors representing associations between preset objects in an image, and the length of a preset numerical range corresponding to the third category of descriptors is greater than the length of a preset numerical range corresponding to the fourth category of descriptors.

[0132] Optionally, the fourth type of descriptor of the image to be classified is obtained by following the steps D1-D2:

[0133] D1, obtaining a pre-built relationship vector diagram for the image to be classified; wherein the relationship vector diagram includes multiple nodes and the association relationship between each node, a node represents a preset object in the image to be classified, and the association relationship between each node represents the association relationship between the corresponding preset objects;

[0134] D2, use the region descriptor extraction algorithm to extract features from the relationship vector map and obtain the fourth type of descriptor of the image to be classified.

[0135] It should be noted that the specific implementation of steps D1-D2 can refer to the relevant description of steps B1-B2, which will not be repeated here.

[0136] Optionally, the relationship vector diagram is obtained by following the steps E1-E3:

[0137] E1, displaying the image to be classified, the text input box, and the embedded information of the image to be classified in the information upload interface;

[0138] E2, obtaining a description text input by the user in the text input box for describing the association relationship between preset objects in the image to be classified;

[0139] E3, obtaining a relationship vector graph for the image to be classified, which is created by the user in the vector graph construction interface based on the description text and the embedded information.

[0140] It should be noted that the specific implementation of steps E1-E3 can refer to the relevant description of steps C1-C3, which will not be repeated here.

[0141] S402, adjusting the obtained value ranges of the descriptors according to the preset value ranges corresponding to the descriptors of each category, and splicing the adjusted descriptors to obtain a second feature vector of the image to be classified; wherein the length of the preset value range corresponding to the first category descriptor is greater than the length of the preset value range corresponding to the second category descriptor, and the length of the preset value range corresponding to the second category descriptor is greater than the length of the preset value range corresponding to the third category descriptor;

[0142] The relevant contents of step S402 can refer to the relevant description of the above step S102, which will not be repeated here.

[0143] Optionally, in one implementation, the obtained value ranges of the descriptors are adjusted according to the preset value ranges corresponding to the descriptors of each category, and the adjusted descriptors are spliced ​​to obtain the second feature vector of the image to be classified, including steps F1-F2:

[0144] F1, for each descriptor obtained, map the values ​​contained in the descriptor into a specified value range to obtain the adjusted descriptor; wherein the maximum absolute value and the minimum absolute value of the values ​​in the specified value range are respectively the two endpoints of the preset value range corresponding to the category to which the descriptor belongs; F2, concatenate the adjusted descriptors to obtain the second feature vector of the image to be classified. It should be noted that the specific implementation of steps F1-F2 can refer to the relevant description of steps A1-A2, which will not be repeated here.

[0145] S403, inputting the second feature vector into at least one pre-trained support vector machine to obtain a prediction probability corresponding to an output result of each support vector machine; wherein the prediction probability corresponding to the output result of each support vector machine represents a probability that the image to be classified belongs to a prediction category corresponding to the support vector machine, and each support vector machine is trained based on any one of the above-mentioned MPEG-7-based model training methods;

[0146] In this embodiment, each SVM is trained according to the above-mentioned MPEG-7-based model training method, and each trained SVM corresponds to a fixed classification hyperplane and decision function. The output result of each SVM is the decision function value, which can be converted into a prediction probability. For example, the feature vector Input an SVM, the decision function value output by the SVM is calibrated to a probability using the Sigmoid (activation) function:

[0147] ;

[0148] Among them, the probability after calibration The feature vector representing the input The image belongs to the category The probability of is the decision function value output by the SVM for classifying category i.

[0149] It can be understood that each SVM can achieve binary classification, and multiple SVMs can be used to achieve multi-classification. Exemplarily, if there are four categories to be classified, ABCD, then four SVMs can be established, including: SVM1 for classifying A category and non-A category, SVM2 for classifying B category and non-B category, SVM3 for classifying C category and non-C category, and SVM4 for classifying D category and non-D category. At this time, the prediction category corresponding to SVM1 is category A, and the prediction probability obtained by converting the decision function value output by SVM1 represents the probability that the image to be classified belongs to category A; the prediction category corresponding to SVM2 is category B, and the prediction probability obtained by converting the decision function value output by SVM2 represents the probability that the image to be classified belongs to category B; the prediction category corresponding to SVM3 is category C, and the prediction probability obtained by converting the decision function value output by SVM3 represents the probability that the image to be classified belongs to category C; the prediction category corresponding to SVM4 is category D, and the prediction probability obtained by converting the decision function value output by SVM4 represents the probability that the image to be classified belongs to category D.

[0150] S404, normalizing the obtained prediction probabilities, and determining the prediction category corresponding to the normalized maximum value as the category to which the image to be classified belongs.

[0151] Exemplarily, normalization can be performed by a Softmax (normalized exponential) function, which is expressed as:

[0152]

[0153] in, The feature vector representing the input The probability that the image belongs to category i, is the decision function value output by the SVM for classifying category i, and k is the number of all categories to be classified.

[0154] It can be understood that, since each predicted probability obtained is a predicted probability for each different category, in order to determine the category to which the image to be classified ultimately belongs, each predicted probability can be normalized to ensure that the sum of the probabilities of all categories is 1, and then the predicted category corresponding to the normalized maximum value can be determined as the category to which the image to be classified belongs. Exemplarily, if the predicted probability obtained by converting the decision function value output by SVM1 is normalized to 0.5, the predicted probability obtained by converting the decision function value output by SVM2 is normalized to 0.1, the predicted probability obtained by converting the decision function value output by SVM3 is normalized to 0.2, and the predicted probability obtained by converting the decision function value output by SVM4 is normalized to 0.2, then category A is determined as the category to which the image to be classified belongs.

[0155] It can be seen that, through this solution, since the SVM used for classification is trained based on the MPEG-7 model training method, and the above model training method can improve the classification accuracy of the trained model, then, using the trained SVM for image classification can improve the accuracy of image classification. In addition, in the image classification process, the image privacy classification can be achieved by combining the information such as the association relationship between the preset objects in the image.

[0156] Optionally, the present application embodiment also provides an image retrieval method based on MPEG-7, such as Figure 5 As shown, steps S501-S505 are included:

[0157] S501, obtaining a descriptor of an image to be detected as a descriptor to be matched; wherein the descriptor to be matched is extracted using an MPEG-7 descriptor extraction algorithm;

[0158] In this embodiment, the image to be detected is a reference image for image retrieval, that is, the image to be detected is used to retrieve an image similar to the image to be detected from multiple candidate images. In this embodiment, one or more descriptors of the image to be detected can be extracted in advance, and then the extracted descriptors are stored in a specified XML file, so that when this solution is executed, the descriptors of the image to be detected can be directly obtained from the specified XML file as the descriptors to be matched. Among them, the method of extracting the descriptors of the image to be detected can refer to the relevant description of extracting the descriptors in the above step S101, which will not be repeated here.

[0159] Optionally, in an implementation manner, before obtaining the descriptor of the image to be detected as the descriptor to be matched, the method further includes: displaying a plurality of different retrieval modes in a retrieval mode selection interface.

[0160] In this implementation, a user can select a desired retrieval mode in a retrieval mode selection interface of a client for retrieving images. A plurality of different retrieval modes are displayed in the retrieval mode selection interface, each retrieval mode indicates retrieval using a description scheme, and each description scheme may include one or more descriptors. Exemplarily, each retrieval mode may include: mode 1 indicating retrieval using a color layout descriptor, mode 2 indicating retrieval using a scalable color descriptor, mode 3 indicating retrieval using a color layout descriptor + a scalable color descriptor, mode 4 indicating retrieval using a color layout descriptor + a scalable color descriptor + an edge histogram descriptor, and so on.

[0161] Exemplarily, the search mode selection interface is as follows Figure 6As shown, after opening the retrieval mode selection interface, you can click the "Path" button in the interface to select a storage address from the storage addresses of each image, and the image in the storage address is the image to be detected. The selected storage address is displayed in the rectangular box below "Select Image", and the image in the storage address is displayed in the rectangular box corresponding to "Image Preview". The "Select Retrieval Mode" control provides a variety of selectable retrieval modes, including "Use Color Layout Descriptor", "Use Scalable Color Descriptor", "Use Edge Histogram Descriptor", "Use Color Layout Descriptor + Scalable Color Descriptor", "Use Color Layout Descriptor + Edge Histogram Descriptor", "Use Scalable Color Descriptor + Edge Histogram Descriptor", "Use Color Layout Descriptor + Scalable Color Descriptor + Edge Histogram Descriptor". When "Use Color Layout Descriptor" is selected, the "Color Layout Preview" control in the upper left corner will give the image represented by CLD after the user selects the image. After selecting the image and retrieval mode, you can click the "Search" button to retrieve images similar to the selected image from the database to be retrieved. In addition, it should be noted that if the description scheme includes multiple descriptors, each descriptor may have a default weight assignment, and the weight of each descriptor may be modified by relevant technical personnel based on experience or actual scenario requirements.

[0162] Accordingly, in the present implementation, the method for extracting the descriptor to be matched includes: when a mode selection instruction is received, the descriptor of the image to be detected is extracted as the descriptor to be matched using the MPEG-7 descriptor extraction algorithm according to the features indicated by the retrieval mode selected by the mode selection instruction.

[0163] It is understandable that after displaying multiple different retrieval modes in the retrieval mode selection interface, the user can select a retrieval mode for retrieval, and at this time, the mode selection instruction is received. According to the features indicated by the retrieval mode selected by the mode selection instruction, the descriptors of the image to be detected are extracted using the extraction algorithm corresponding to each descriptor provided by MPEG-7, which is used to extract the features indicated by the selected retrieval model, to obtain the descriptors to be matched. It is understandable that since the features indicated by the selected retrieval mode can be one or more, the descriptors to be matched can also be one or more.

[0164] Exemplarily, the feature indicated by the retrieval pattern may be a feature expressed by a color layout descriptor, a feature expressed by a scalable color descriptor, a feature expressed by a color layout descriptor and a scalable color descriptor, etc. Exemplarily, if the feature indicated by the retrieval pattern is a feature expressed by a color layout descriptor and a scalable color descriptor, then the color layout descriptor of the image to be detected is extracted using the extraction algorithm of CLD, and the scalable color descriptor of the image to be detected is extracted using the extraction algorithm of SCD, and the color layout descriptor and the scalable color descriptor of the image to be detected are used as descriptors to be matched.

[0165] S502, using any of the above-mentioned image classification methods based on the multimedia content description interface to classify the image to be detected, and obtain the category to which the image to be detected belongs;

[0166] The process of classifying the image to be detected according to the above-mentioned image classification method based on MPEG-7 can refer to the relevant description of the above-mentioned steps S401-S404, which will not be repeated here.

[0167] S503, obtaining candidate descriptors of candidate images belonging to the same category as the image to be detected; wherein the candidate descriptors are descriptors corresponding to the descriptors to be matched, and the category of each candidate image is obtained by classifying the image based on any one of the above-mentioned MPEG-7-based image classification methods;

[0168] In this embodiment, the candidate descriptor is a descriptor corresponding to the descriptor to be matched. For example, if the descriptor to be matched is the color layout descriptor of the image to be detected, the candidate descriptor is the color layout descriptor of the candidate image; if the descriptor to be matched is the color layout descriptor and scalable color descriptor of the image to be detected, the candidate descriptor is the color layout descriptor and scalable color descriptor of the candidate image.

[0169] It can be understood that the descriptors corresponding to the descriptors to be matched are extracted from the candidate images belonging to the same category as the image to be detected as candidate descriptors, so that subsequent retrieval can be directly performed from the candidate images belonging to the same category as the image to be detected based on the similarity between each candidate descriptor and the descriptors to be matched, thereby improving the retrieval efficiency.

[0170] It is understandable that, in practical applications, if it is necessary to retrieve an image similar to the image to be detected in an image library, before the retrieval, each image in the image library can be classified using the above-mentioned MPEG-7-based image classification method, and the MPEG-7 descriptor extraction algorithm can be used to extract multiple descriptors from each candidate image belonging to the same category as the image to be detected and save them in the XML file corresponding to each candidate image. Therefore, during the retrieval, the descriptor corresponding to the descriptor to be matched can be directly obtained from the XML file corresponding to each candidate image as the candidate descriptor of the candidate image.

[0171] Exemplarily, before performing a search, the MPEG-7 descriptor extraction algorithm may be used to extract features from each candidate image, and multiple descriptors of each candidate image may be obtained and stored in an XML file corresponding to the candidate image. During a search, if the descriptors to be matched include a color layout descriptor and a scalable color descriptor, the color layout descriptor and the scalable color descriptor of each candidate image may be obtained from the descriptors stored in the XML file corresponding to each candidate image as candidate descriptors of the candidate image.

[0172] S504, respectively calculating the similarity between the descriptor to be matched and the candidate descriptor of each candidate image;

[0173] Exemplarily, the similarity between the descriptor to be matched and the candidate descriptor of each candidate image can be obtained by calculating the Harmanton distance, the Euclidean distance, etc. between the descriptor to be matched and the candidate descriptor of each candidate image. If the descriptor to be matched includes multiple descriptors, then for each candidate image, when calculating the similarity between the descriptor to be matched and the candidate descriptor, the distance between the descriptor to be matched and two corresponding descriptors in the candidate descriptors is calculated respectively, and then the similarity between the descriptor to be matched and the candidate descriptor is determined according to the distances between the multiple groups of corresponding descriptors.

[0174] Optionally, in one implementation, calculating the similarity between the descriptor to be matched and the candidate descriptor of each candidate image includes: for each candidate image, using a Euclidean distance calculation formula to calculate the similarity between the descriptor to be matched and the candidate descriptor of the candidate image, wherein the Euclidean distance calculation formula is:

[0175]

[0176] in, is the similarity between the descriptor to be matched and the candidate descriptor of a candidate image, is the i-th descriptor in the descriptor to be matched, is the i-th descriptor among the candidate descriptors of the candidate image, is the preset weight of the ith descriptor, and n is the number of descriptors to be matched. The ith descriptor in the descriptors to be matched is the descriptor corresponding to the ith descriptor in the candidate descriptors.

[0177] It is understandable that in actual applications, the weight of each descriptor can be set by relevant technicians according to the needs and experience of the actual search scene, and the embodiments of the present application are not limited to this. For example, if a search scene focuses on searching according to the texture features of objects in the image, the weight of each descriptor included in the third category of descriptors can be increased.

[0178] S505: Determine the first preset number of candidate images with the highest corresponding similarities as the retrieval results for the image to be detected.

[0179] It can be understood that by calculating the similarity between the descriptor to be matched and each candidate descriptor, and determining the first preset number of candidate images with the highest corresponding similarity as the retrieval result for the image to be detected, it is possible to retrieve the features represented by the descriptor to be matched as retrieval features, and the first preset number of candidate images retrieved are images with features closest to the retrieval features in the image to be detected. Therefore, by determining the first preset number of candidate images as the retrieval result for the image to be detected, an image with similar retrieval features to the image to be detected can be determined from each candidate image.

[0180] In this implementation, the preset number can be 5, 10, etc. In practical applications, the preset number can be set by relevant technical personnel according to needs, and the embodiments of this application do not limit this. It can be seen that through this solution, according to the similarity between each candidate descriptor and the descriptor to be matched, the search is directly performed from each candidate image belonging to the same category as the image to be detected, which can improve the search efficiency; the image can be searched according to different descriptor combinations to achieve flexible search.

[0181] Corresponding to the embodiments of the above methods, the present application also provides an image classification and retrieval system based on MPEG-7, such as Figure 7 As shown, the system includes a feature extraction module 710, a classification recognition module 720 and an image retrieval module 730;

[0182] The feature extraction module 710 is used to respectively extract descriptors of multiple different categories of the image to be classified using the MPEG-7 descriptor extraction algorithm, and to extract the descriptor of the image to be detected using the MPEG-7 descriptor extraction algorithm;

[0183] The classification identification module 720 is used to obtain descriptors of various different categories of the image to be classified from the feature extraction module, and execute any one of the above-mentioned image classification methods based on MPEG-7.

[0184] The image retrieval module 730 is used to obtain the descriptor of the image to be detected from the feature extraction module and execute any one of the above-mentioned image retrieval methods based on MPEG-7.

[0185] Optionally, in one implementation, the system further includes a classification training module;

[0186] The feature extraction module is further used to extract descriptors of various different categories of the sample image using an MPEG-7 descriptor extraction algorithm;

[0187] The classification training module is used to obtain descriptors of various different categories of sample images from the feature extraction module and execute any of the above model training methods.

[0188] It should be noted that the functions implemented by the feature extraction module, classification recognition module, image retrieval module and classification training module included in the system have been introduced in the above method embodiments and will not be repeated here.

[0189] Optionally, in one implementation, the system further includes a marking module;

[0190] The marking module is used to display the image to be marked, the text input box, and the embedded information of the image to be marked selected by the user in the information upload interface; obtain the description text entered by the user in the text input box to describe the association relationship between preset objects in the image to be marked; and obtain the relationship vector map for the image to be marked created by the user in the vector map construction interface based on the description text and the embedded information.

[0191] The feature extraction module is specifically used to obtain the relationship vector map of the image to be marked from the marking module, and use the regional descriptor extraction algorithm to extract features from the relationship vector map to obtain the fourth type of descriptor of the image to be marked.

[0192] It should be noted that the functions implemented by the marking module included in the embodiment of the present application can refer to the relevant description of the above steps C1-C3, which will not be repeated here.

[0193] Optionally, in one implementation, the system further includes a result display module;

[0194] The result display module is used to obtain the search results from the image search module and display them in a preset display interface.

[0195] In this implementation, the user can search in the client for searching images, and the search results are displayed in the preset display interface of the client. The preset display interface can also display the descriptors to be matched selected by the user during the search, and the relationship vector diagram constructed for the image to be detected, etc.

[0196] Optionally, in one implementation, the system further includes a data processing module;

[0197] The data processing module is used to obtain the image currently used for training and preprocess it to obtain a sample image; obtain the image currently used for classification and preprocess it to obtain an image to be classified; obtain the image currently used for retrieval and preprocess it to obtain an image to be detected;

[0198] The feature extraction module is also used to obtain sample images, images to be classified and images to be detected from the data processing module.

[0199] In this implementation, the image currently used for training, the image currently used for classification, and the image currently used for retrieval may be input by a user or may be pre-stored in a specified database. Preprocessing the image includes adjusting the clarity, size, resolution, etc. of the image. The feature extraction module may obtain the sample image, the image to be classified, and the image to be detected obtained after preprocessing from the data processing module, and extract descriptors.

[0200] In order to better understand the system provided by the embodiment of the present application, a specific example is described below. Figure 8 As shown, a schematic diagram of a system provided for implementing an embodiment of the application is provided, the system including a data processing module, a feature extraction module, a classification training module, a marking module, a retrieval and recognition module (corresponding to the classification and recognition module and the image retrieval module mentioned above) and a result display module.

[0201] The data processing module is used to load image data from a data source and generate an original extensible markup language file after preprocessing and store it in a local database; the XML file is generated according to the content information of the image data.

[0202] The feature extraction module is used to extract the image features of each image from the local database, generate the MPEG-7 descriptor of each image and store it locally, including extracting the color features, texture features, and shape features of the images stored in the local database to obtain color descriptors (corresponding to the first type of descriptors mentioned above), texture descriptors (corresponding to the third type of descriptors mentioned above) and shape descriptors (corresponding to the second type of descriptors mentioned above). Based on the descriptors extracted from each image, a description scheme for the image can be generated and stored in the XML file corresponding to the image. The descriptors of each image are adjusted in value range and then spliced ​​to generate the feature vector of the image.

[0203] The classification training module is used to train the classification model (corresponding to the SVM mentioned above) using the extracted feature vectors, and to perform probability conversion on the output of the classification model, that is, to convert the output into a probability distribution.

[0204] The marking module is used to allow users to freely mark the selected image, including object marking and relationship marking, so as to establish the association relationship between preset objects in the image.

[0205] The retrieval and recognition module is used to classify and recognize each image in the local image library, and read the candidate descriptor of each candidate image from the specified file according to the recognition result, and perform image retrieval according to the descriptor of the specified image (corresponding to the descriptor to be matched in the above), that is, perform retrieval based on the similarity between the candidate descriptors of all candidate images and the descriptor of the specified image, and output the retrieval results.

[0206] The result display module is used to display the search results, and to display the descriptors (i.e., the description scheme used for the search) and the relationship between the objects in the image.

[0207] Figure 8 The arrows in the figure are used to identify data flows, where: 1 represents the transmission of media data; 2 represents the transmission of image data; 3 represents data conversion, i.e., a description scheme is constructed based on the descriptors extracted from the image data and stored in an extensible markup language file to realize the conversion of image data to an XML file; 4 represents information extraction, i.e., automatically extracting embedded information from an image; 5 represents the transmission of an XML file; 6 represents the transmission of control information (used to control the execution of each module); 7 represents the transmission of a feature vector; 8 represents the transmission of a descriptor; 9 represents the transmission of model information, i.e., the transmission of model parameters of a trained classification model; 10 represents the transmission of classification detection results; and 11 represents the transmission of tag information.

[0208] Figure 8 The workflow of each module in the system shown is as follows: Fig. 9 As shown, steps S901-S906 are included:

[0209] S901, image preprocessing, automatic or manual generation of tags to add additional information;

[0210] Image preprocessing includes adjusting image clarity, resolution, size, etc. Automatic tag generation includes automatically extracting embedded information from images, and manual tag generation includes tagging objects and relationships in images to obtain additional information.

[0211] S902, extracting multimedia content description interface descriptors and generating diversified description schemes;

[0212] For each preprocessed image, multiple different categories of MPEG-7 descriptors can be extracted, and different combinations of the extracted descriptors can form multiple different description schemes. For example, among the extracted descriptors, the texture layout descriptor and the contour shape descriptor form one description scheme, and the color layout descriptor and the contour shape descriptor form another description scheme. After the description scheme of each image is generated, it can be stored in the XML file corresponding to the image.

[0213] S903, select a description scheme according to the requirements and build a training data set or a verification data set;

[0214] Select a description scheme based on the requirements of the classification scenario. For example, if the features of the object of interest in the classification scenario are color features and texture features, then select a description scheme consisting of the first type of descriptor representing color features and the third type of descriptor representing texture features. 80% of the description schemes corresponding to each image are constructed as training data sets, and 20% of the description schemes are constructed as training data sets.

[0215] S904, establishing a classification model and training the model;

[0216] SVM is used for classification training, and radial basis function is used as kernel function. The model training process refers to the relevant description of the above steps S101-S105.

[0217] S905, calculating the category distribution of the samples and obtaining a description scheme of the samples;

[0218] That is, by classifying each sample (the image in the image library to be retrieved) to obtain the category distribution of each sample, the description scheme of each sample is obtained from the XML file corresponding to each sample.

[0219] S906, calculating the similarity between the image to be detected and samples of the same category and generating a ranking.

[0220] That is, the similarity between the descriptor of the image to be detected and the descriptor of the samples of the same category (corresponding to the candidate images mentioned above) is calculated and sorted from high to low to obtain the sorting result. According to the sorting result, the first preset number of images can be determined as the retrieval results. In addition, this example provides the results of comparing the classification method of this scheme with the classification method based on convolutional neural networks on data sets of different sizes. Among them, Table 1 shows the comparison of the classification method in this scheme with the model of 3-layer convolutional neural network and 2-layer fully connected layers after 15 epochs (iterations) training on the CIFAR-10 data set:

[0221] Table 1

[0222]

[0223] Table 2 shows the comparison of the classification method of this scheme with the model training results of 15 epochs of 3-layer convolutional neural network and 2-layer fully connected layer on the CIFAR-100 dataset:

[0224] Table 2

[0225]

[0226] It can be seen that the effect of this scheme on medium-sized data sets is similar to that of convolutional neural networks. Compared with convolutional neural networks, this scheme can extract different descriptor combinations according to needs and construct feature vectors for training SVM. Then, the obtained SVM can realize classification according to color features, classification according to texture features, or classification according to the combination of color and texture features, etc. Compared with convolutional neural networks, the classification results of this scheme are more interpretable.

[0227] It can be seen that this scheme builds a scientific and efficient human-eye-like description scheme in the progressive order of color-shape-texture-association, that is, first observe the color of the object in the image, then consider the shape of the object, then compare the surface texture of the object, and finally complete the recognition and classification of the image content from a single object to the overall relationship structure from the association relationship between objects in the global comparison map. It can recognize the image in a way similar to the human eye, has high interpretability and scalability, and can add new descriptors and change descriptor combinations according to user needs to flexibly respond to various classification scenarios. In addition, this scheme can generate embedded information description based on image embedded information, and can also generate image content description based on image content. At the same time, it can create object nodes in the image and establish rich semantic relationships between objects and generate relationship vector diagrams. The relationship vector diagram represents the semantic structure between each object in the image. Both embedded information and relationship vector diagrams can be integrated into the description scheme by adding new descriptors to provide a more targeted image description scheme.

[0228] In the retrieval process, the descriptor and weight combination can also be adjusted according to user requirements to make the retrieval results more targeted, so that multimedia content can be efficiently retrieved and indexed. For example, users can search for similar images based on features such as color and texture of the image. In addition, this solution can make the management and archiving of multimedia content more systematic and orderly. By analyzing user preferences and interests, an intelligent content recommendation system can be built.

[0229] It can be seen that this solution is highly interpretable, and users can intuitively understand and use image information and its classification results. It is scalable and can adjust the selection and weight of descriptors according to the specific needs of users to adapt to different application scenarios. Efficient image classification and retrieval are achieved on medium-sized data sets, and the performance is not inferior to general convolutional neural networks. The system functions cover a series of important steps in multimedia data retrieval and analysis, such as the analysis and utilization of information from various sources in the image, the generation of image descriptors and automatic or manual labeling, image classification and retrieval, etc., providing a complete multimedia data processing solution.

[0230] Corresponding to the above-mentioned embodiment of the model training method based on MPEG-7, the embodiment of the present application also provides a model training device based on MPEG-7, such as Fig.10 As shown, the device comprises:

[0231] The first acquisition module 1010 is used to acquire descriptors of multiple different categories of the sample image; wherein the descriptors are extracted using an MPEG-7 descriptor extraction algorithm, and the multiple different categories of descriptors include at least two of the following: a first category of descriptors representing color features, a second category of descriptors representing shape features, and a third category of descriptors representing texture features;

[0232] The first adjustment module 1020 is used to adjust the obtained value ranges of the descriptors according to the preset value ranges corresponding to the descriptors of each category, and splice the adjusted descriptors to obtain the first feature vector of the sample image; wherein the length of the preset value range corresponding to the first category descriptor is greater than the length of the preset value range corresponding to the second category descriptor, and the length of the preset value range corresponding to the second category descriptor is greater than the length of the preset value range corresponding to the third category descriptor;

[0233] A first prediction module 1030, configured to input the first feature vector into an initial SVM to obtain a predicted category of the sample image;

[0234] A loss calculation module 1040, configured to calculate a loss value based on a difference between the obtained predicted category and the true category of the sample image;

[0235] The model parameter adjustment module 1050 is used to adjust the model parameters of the SVM based on the loss value.

[0236] Optionally, the multiple different categories of descriptors include at least two of the following: a first category of descriptors representing color features, a second category of descriptors representing shape features, a third category of descriptors representing texture features, and a fourth category of descriptors representing associations between preset objects in an image, and the length of the preset numerical range corresponding to the third category of descriptors is greater than the length of the preset numerical range corresponding to the fourth category of descriptors.

[0237] Optionally, the fourth type of descriptor of the sample image is obtained by following steps:

[0238] Obtaining a pre-constructed relationship vector map for the sample image; wherein the relationship vector map includes a plurality of nodes and association relationships between the nodes, a node represents a preset object in the sample image, and the association relationships between the nodes represent association relationships between corresponding preset objects;

[0239] A regional descriptor extraction algorithm is used to extract features from the relationship vector map to obtain a fourth type of descriptor of the sample image.

[0240] Optionally, the relationship vector diagram is obtained by following steps:

[0241] Displaying the sample image, a text input box, and embedded information of the sample image in the information upload interface;

[0242] Acquire a description text input by a user in the text input box for describing the association relationship between preset objects in the sample image;

[0243] A relationship vector map for the sample image created by the user in a vector map construction interface based on the description text and the embedded information is obtained.

[0244] Optionally, the first adjustment module 1020 includes:

[0245] A first mapping submodule is used to map each value contained in each acquired descriptor into a specified value range to obtain an adjusted descriptor; wherein the maximum absolute value and the minimum absolute value of the values ​​in the specified value range are respectively two endpoints of a preset value range corresponding to the category to which the descriptor belongs;

[0246] The first splicing submodule is used to splice the adjusted descriptors to obtain a first feature vector of the sample image.

[0247] Corresponding to the above-mentioned embodiment of the image classification method based on MPEG-7, the embodiment of the present application also provides an image classification device based on MPEG-7, such as Fig.11 As shown, the device comprises:

[0248] The second acquisition module 1110 is used to acquire descriptors of multiple different categories of the image to be classified; wherein the descriptors are extracted using an MPEG-7 descriptor extraction algorithm, and the multiple different categories of descriptors include at least two of the following: a first category of descriptors representing color features, a second category of descriptors representing shape features, and a third category of descriptors representing texture features;

[0249] The second adjustment module 1120 is used to adjust the obtained value ranges of the descriptors according to the preset value ranges corresponding to the descriptors of each category, and splice the adjusted descriptors to obtain the second feature vector of the image to be classified; wherein the length of the preset value range corresponding to the first category descriptor is greater than the length of the preset value range corresponding to the second category descriptor, and the length of the preset value range corresponding to the second category descriptor is greater than the length of the preset value range corresponding to the third category descriptor;

[0250] A second prediction module 1130 is used to input the second feature vector into at least one pre-trained SVM to obtain a prediction probability corresponding to the output result of each SVM; wherein the prediction probability corresponding to the output result of each SVM represents the probability that the image to be classified belongs to the prediction category corresponding to the SVM, and each SVM is trained based on any one of the above-mentioned MPEG-7-based model training methods;

[0251] The normalization module 1140 is used to normalize the obtained prediction probabilities, and determine the prediction category corresponding to the normalized maximum value as the category to which the image to be classified belongs.

[0252] Optionally, the multiple different categories of descriptors include at least two of the following: a first category of descriptors representing color features, a second category of descriptors representing shape features, a third category of descriptors representing texture features, and a fourth category of descriptors representing associations between preset objects in an image, and the length of the preset numerical range corresponding to the third category of descriptors is greater than the length of the preset numerical range corresponding to the fourth category of descriptors.

[0253] Optionally, the fourth type of descriptor of the image to be classified is obtained by the following steps:

[0254] Obtaining a pre-constructed relationship vector diagram for the image to be classified; wherein the relationship vector diagram includes a plurality of nodes and association relationships between the nodes, a node represents a preset object in the image to be classified, and the association relationships between the nodes represent association relationships between corresponding preset objects;

[0255] A regional descriptor extraction algorithm is used to extract features from the relationship vector map to obtain a fourth type of descriptor of the image to be classified.

[0256] Optionally, the relationship vector diagram is obtained by following steps:

[0257] Displaying the image to be classified, a text input box, and embedded information of the image to be classified in the information upload interface;

[0258] Acquire a description text input by a user in the text input box for describing the association relationship between preset objects in the image to be classified;

[0259] A relationship vector map for the image to be classified is obtained, which is created by the user in a vector map construction interface based on the description text and the embedded information.

[0260] Optionally, the second adjustment module 1120 includes:

[0261] A second mapping submodule is used to map each value contained in each acquired descriptor into a specified value range to obtain an adjusted descriptor; wherein the maximum absolute value and the minimum absolute value of the value in the specified value range are respectively two endpoints of a preset value range corresponding to the category to which the descriptor belongs;

[0262] The second splicing submodule is used to splice the adjusted descriptors to obtain a second feature vector of the image to be classified.

[0263] Corresponding to the above-mentioned embodiment of the image retrieval method based on MPEG-7, the embodiment of the present application also provides an image retrieval device based on MPEG-7, such as Fig.12 As shown, the device comprises:

[0264] The third acquisition module 1210 is used to acquire the descriptor of the image to be detected as the descriptor to be matched; wherein the descriptor to be matched is extracted using the MPEG-7 descriptor extraction algorithm;

[0265] A category determination module 1220, configured to classify the image to be detected by using any of the above-mentioned MPEG-7-based image classification methods to obtain the category to which the image to be detected belongs;

[0266] The fourth acquisition module 1230 is used to acquire candidate descriptors of each candidate image that belongs to the same category as the image to be detected; wherein the candidate descriptors are descriptors corresponding to the descriptors to be matched, and the category of each candidate image is obtained by classifying the image based on any one of the above-mentioned MPEG-7-based image classification methods;

[0267] A similarity calculation module 1240 is used to calculate the similarity between the descriptor to be matched and the candidate descriptor of each candidate image respectively;

[0268] The retrieval result determination module 1250 is used to determine the first preset number of candidate images with the highest corresponding similarities as the retrieval results for the image to be detected.

[0269] Optionally, the device further comprises:

[0270] A display module, used for displaying a plurality of different retrieval modes in a retrieval mode selection interface before extracting the descriptor of the image to be detected as the descriptor to be matched by using the MPEG-7 descriptor extraction algorithm;

[0271] The method of extracting the descriptor to be matched includes:

[0272] When a mode selection instruction is received, the descriptor of the image to be detected is extracted as the descriptor to be matched by using an MPEG-7 descriptor extraction algorithm according to the features indicated by the retrieval mode selected by the mode selection instruction.

[0273] In the technical solution of the present application, operations such as acquisition, storage, use, processing, transmission, provision and disclosure of images involved are all carried out with the user's authorization.

[0274] The present application also provides an electronic device, such as Fig.13 As shown, including:

[0275] Memory 1301, used for storing computer programs;

[0276] The processor 1302 is used to implement the steps of any one of the above-mentioned model training method based on MPEG-7, image classification method based on MPEG-7, or image retrieval method based on MPEG-7 when executing the program stored in the memory 1301;

[0277] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 1302, the communication interface, and the memory 1301 communicate with each other via the communication bus.

[0278] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0279] The communication interface is used for communication between the above electronic device and other devices.

[0280] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0281] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0282] In another embodiment provided in the present application, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned MPEG-7-based model training methods, MPEG-7-based image classification methods, or MPEG-7-based image retrieval methods are implemented.

[0283] In another embodiment provided by the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any one of the MPEG-7-based model training methods, MPEG-7-based image classification methods, or MPEG-7-based image retrieval methods in the above-mentioned embodiments.

[0284] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0285] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0286] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, and computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0287] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A model training method based on MPEG-7, characterized in that: The method comprises: Acquire descriptors of multiple different categories of sample images; wherein the descriptors are extracted using a multimedia content description interface MPEG-7 descriptor extraction algorithm, and the multiple different categories of descriptors include at least two of the following: a first category of descriptors representing color features, a second category of descriptors representing shape features, and a third category of descriptors representing texture features; According to the preset numerical ranges corresponding to the descriptors of each category, the obtained numerical ranges of the descriptors are adjusted, and the adjusted descriptors are spliced ​​to obtain a first feature vector of the sample image; wherein the length of the preset numerical range corresponding to the first category descriptor is greater than the length of the preset numerical range corresponding to the second category descriptor, and the length of the preset numerical range corresponding to the second category descriptor is greater than the length of the preset numerical range corresponding to the third category descriptor; Inputting the first feature vector into an initial support vector machine SVM to obtain a predicted category of the sample image; Calculating a loss value based on the difference between the obtained predicted category and the true category of the sample image; Based on the loss value, the model parameters of the SVM are adjusted.

2. The method according to claim 1, characterized in that The multiple different categories of descriptors include at least two of the following: a first category of descriptors representing color features, a second category of descriptors representing shape features, a third category of descriptors representing texture features, and a fourth category of descriptors representing associations between preset objects in an image, wherein the length of a preset numerical range corresponding to the third category of descriptors is greater than the length of a preset numerical range corresponding to the fourth category of descriptors.

3. The method according to claim 2, characterized in that The fourth type of descriptor of the sample image is obtained by following the steps: Obtaining a pre-constructed relationship vector map for the sample image; wherein the relationship vector map includes a plurality of nodes and association relationships between the nodes, a node represents a preset object in the sample image, and the association relationships between the nodes represent association relationships between corresponding preset objects; A regional descriptor extraction algorithm is used to extract features from the relationship vector map to obtain a fourth type of descriptor of the sample image.

4. The method according to claim 3, characterized in that The relationship vector diagram is obtained by the following steps: Displaying the sample image, a text input box, and embedded information of the sample image in the information upload interface; Acquire a description text input by a user in the text input box for describing the association relationship between preset objects in the sample image; A relationship vector map for the sample image created by the user in a vector map construction interface based on the description text and the embedded information is obtained.

5. The method according to any one of claims 1 to 4, characterized in that: The method of adjusting the obtained value ranges of the descriptors according to the preset value ranges corresponding to the descriptors of each category, and splicing the adjusted descriptors to obtain the first feature vector of the sample image includes: For each descriptor obtained, each value contained in the descriptor is mapped into a specified value range to obtain the adjusted descriptor; wherein the maximum absolute value and the minimum absolute value of the value in the specified value range are respectively two endpoints of the preset value range corresponding to the category to which the descriptor belongs; The adjusted descriptors are concatenated to obtain a first feature vector of the sample image.

6. An image classification method based on MPEG-7, characterized in that: The method comprises: Acquire descriptors of multiple different categories of the image to be classified; wherein the descriptors are extracted using a multimedia content description interface MPEG-7 descriptor extraction algorithm, and the multiple different categories of descriptors include at least two of the following: a first category descriptor representing color features, a second category descriptor representing shape features, and a third category descriptor representing texture features; According to the preset numerical ranges corresponding to the descriptors of each category, the obtained numerical ranges of the descriptors are adjusted, and the adjusted descriptors are spliced ​​to obtain a second feature vector of the image to be classified; wherein the length of the preset numerical range corresponding to the first category descriptor is greater than the length of the preset numerical range corresponding to the second category descriptor, and the length of the preset numerical range corresponding to the second category descriptor is greater than the length of the preset numerical range corresponding to the third category descriptor; Inputting the second feature vector into at least one pre-trained SVM respectively to obtain the prediction probability corresponding to the output result of each SVM; wherein the prediction probability corresponding to the output result of each SVM represents the probability that the image to be classified belongs to the corresponding prediction category of the SVM, and each SVM is trained based on the method according to any one of claims 1 to 5; The obtained prediction probabilities are normalized, and the prediction category corresponding to the normalized maximum value is determined as the category to which the image to be classified belongs.

7. The method according to claim 6, characterized in that The multiple different categories of descriptors include at least two of the following: a first category of descriptors representing color features, a second category of descriptors representing shape features, a third category of descriptors representing texture features, and a fourth category of descriptors representing associations between preset objects in an image, wherein the length of a preset numerical range corresponding to the third category of descriptors is greater than the length of a preset numerical range corresponding to the fourth category of descriptors.

8. The method according to claim 7, characterized in that The fourth type of descriptor of the image to be classified is obtained by the following steps: Obtaining a pre-constructed relationship vector diagram for the image to be classified; wherein the relationship vector diagram includes a plurality of nodes and association relationships between the nodes, a node represents a preset object in the image to be classified, and the association relationships between the nodes represent association relationships between corresponding preset objects; A regional descriptor extraction algorithm is used to extract features from the relationship vector map to obtain a fourth type of descriptor of the image to be classified.

9. The method according to claim 8, characterized in that The relationship vector diagram is obtained by the following steps: Displaying the image to be classified, a text input box, and embedded information of the image to be classified in the information upload interface; Acquire a description text input by a user in the text input box for describing the association relationship between preset objects in the image to be classified; A relationship vector map for the image to be classified is obtained, which is created by the user in a vector map construction interface based on the description text and the embedded information.

10. The method according to any one of claims 6 to 9, characterized in that: The method of adjusting the obtained value ranges of the descriptors according to the preset value ranges corresponding to the descriptors of each category, and splicing the adjusted descriptors to obtain the second feature vector of the image to be classified includes: For each descriptor obtained, each value contained in the descriptor is mapped into a specified value range to obtain the adjusted descriptor; wherein the maximum absolute value and the minimum absolute value of the value in the specified value range are respectively two endpoints of the preset value range corresponding to the category to which the descriptor belongs; The adjusted descriptors are concatenated to obtain a second feature vector of the image to be classified.

11. An image retrieval method based on MPEG-7, characterized in that: The method comprises: Obtaining a descriptor of the image to be detected as a descriptor to be matched; wherein the descriptor to be matched is extracted using a multimedia content description interface MPEG-7 descriptor extraction algorithm; Using the MPEG-7 based image classification method according to any one of claims 6 to 10, classify the image to be detected to obtain the category to which the image to be detected belongs; Obtaining candidate descriptors of candidate images belonging to the same category as the image to be detected; wherein the candidate descriptors are descriptors corresponding to the descriptors to be matched, and the category of each candidate image is obtained by classifying the image based on the MPEG-7-based image classification method according to any one of claims 6 to 10; Calculating the similarity between the to-be-matched descriptor and the candidate descriptor of each candidate image respectively; A preset number of candidate images with the highest corresponding similarities are determined as retrieval results for the image to be detected.

12. The method according to claim 11, characterized in that Before obtaining the descriptor of the image to be detected as the descriptor to be matched, the method further includes: Displaying multiple different search modes in the search mode selection interface; The method of extracting the descriptor to be matched includes: When a mode selection instruction is received, the descriptor of the image to be detected is extracted as the descriptor to be matched by using an MPEG-7 descriptor extraction algorithm according to the features indicated by the retrieval mode selected by the mode selection instruction.

13. An image classification and retrieval system based on MPEG-7, characterized in that: The system includes a feature extraction module, a classification and recognition module and an image retrieval module; The feature extraction module is used to respectively extract descriptors of multiple different categories of images to be classified using the multimedia content description interface MPEG-7 descriptor extraction algorithm, and to extract descriptors of images to be detected using the MPEG-7 descriptor extraction algorithm; The classification recognition module is used to obtain descriptors of multiple different categories of the image to be classified from the feature extraction module, and execute the MPEG-7 based image classification method according to any one of claims 6 to 10; The image retrieval module is used to obtain the descriptor of the image to be detected from the feature extraction module and execute the MPEG-7 based image retrieval method according to any one of claims 11-12.

14. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-12 when executing a program stored in a memory.

15. A computer program product, characterized in that The computer program product comprises executable instructions, and when the executable instructions are executed on a computer, the computer is enabled to execute the method according to any one of claims 1 to 12.

Images