Image content search and matching system and method based on machine learning

Abstract

The present invention relates to the field of data processing technology, and discloses an image content search and matching system and method based on machine learning, comprising: inputting an image to be searched into a landscape type detection model to identify the landscape type to which the image to be searched belongs; retrieving an image data cluster of the corresponding landscape type from a preconfigured image database according to the landscape type; obtaining a high-level semantic feature set of the image to be searched and each landscape image, calculating a first similarity coefficient according to the high-level semantic feature set, and screening out candidate images from the corresponding image data cluster according to the first similarity coefficient; obtaining a low-level visual feature set of the image to be searched and each candidate image, calculating a second similarity coefficient according to the low-level visual feature set; screening out similar images from the candidate images, and outputting and feeding back introduction information associated with the similar images; the present invention is conducive to improving the image search speed while ensuring the accuracy of the image search results.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically, to an image content search and matching system and method based on machine learning. Background Art

[0002] With the popularization of the Internet and mobile devices, travel platforms are becoming an important way for users to obtain travel information and book travel services. In order to provide more intuitive and personalized services, many travel platforms have introduced image content-based search and matching functions, allowing users to search for travel destinations or attractions by uploading images. However, the existing image content search and matching technology has certain technical bottlenecks and cannot meet the efficient and accurate query requirements of large-scale image databases.

[0003] Current image content search mainly relies on comparing the uploaded image with all the images in the database one by one, which often leads to a large number of invalid comparisons and unnecessary computational overhead; due to the lack of an effective screening mechanism, the system will perform a global search, resulting in high computational complexity and low efficiency, and greatly increasing the server load, thereby reducing the system's response speed; at the same time, due to the lack of structured organizational form for image storage and distribution, it not only affects the search efficiency, but also has an adverse impact on the accuracy of search results; the system cannot accurately locate the similarity between the user's uploaded image and the target image, resulting in a large deviation in image search results, and is not suitable for image search scenarios with large-scale concurrent access by users.

[0004] Therefore, how to design an efficient and accurate image content search and matching system to reduce invalid comparisons and improve search speed and accuracy has become a technical problem that current tourism platforms urgently need to solve. Summary of the invention

[0005] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides an image content search and matching system and method based on machine learning.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A machine learning-based image content search and matching system, the system comprising:

[0008] An image recognition module is used to obtain an image to be searched and input the image to be searched into a pre-trained landscape type detection model to identify the landscape type to which the image to be searched belongs;

[0009] A data retrieval module is used to retrieve an image data cluster corresponding to a landscape type from a pre-configured image database according to the landscape type; the image data cluster includes a plurality of landscape images and introduction information associated with each landscape scene image;

[0010] A preliminary screening module, used to obtain a high-level semantic feature set of the image to be searched and each landscape image, calculate a first similarity coefficient between the image to be searched and each landscape image according to the high-level semantic feature set, and screen out a plurality of candidate images from the corresponding image data cluster according to the first similarity coefficient;

[0011] A similarity calculation module, used for obtaining a low-level visual feature set of the image to be searched and each candidate image, and calculating a second similarity coefficient between the image to be searched and each candidate image according to the low-level visual feature set;

[0012] The image search module is used to screen out similar images that match the image to be searched from multiple candidate images according to the first similarity coefficient and the second similarity coefficient, and output the introduction information associated with the similar images as feedback.

[0013] Furthermore, before obtaining the image to be searched, the process includes:

[0014] Obtaining the captured image uploaded by the user and extracting image quality data of the captured image;

[0015] The image quality data includes the size of the image, the brightness value of each pixel, the noise level of the image, and the maximum and minimum brightness values ​​of the image;

[0016] Inputting the image quality data into a pre-built mathematical calculation model to obtain a quality coefficient of the captured image;

[0017] The mathematical calculation model is expressed as follows:

[0018] ;

[0019] Where: is the quality coefficient, and are the width and height respectively, is the Laplace operator, For the image at pixel points The brightness value at is the maximum brightness value in the image, is the minimum brightness value in the image, is the noise level of the image; and is a correction factor greater than zero, ;

[0020] The quality coefficient of the captured image is compared with the preset quality coefficient threshold. If the quality coefficient is greater than or equal to the quality coefficient threshold, the corresponding captured image is judged to meet the preset quality output standard and the corresponding captured image is marked as an image to be searched; if the quality coefficient is less than the quality coefficient threshold, the corresponding captured image is judged to not meet the preset quality output standard.

[0021] Further, when it is determined that the corresponding captured image does not meet the preset quality output standard, a preset display content is output for feedback; wherein the preset display content is the words "please re-upload the image";

[0022] The generation logic of the landscape type detection model is as follows:

[0023] Acquire historical scenery type recognition training data, and divide the historical scenery type recognition training data into a scenery type recognition training set and a scenery type recognition test set; the historical scenery type recognition training data includes a plurality of images to be searched and annotated labels of their corresponding scenery types;

[0024] Constructing a first classification network, taking the image to be searched in the scenery type recognition training set as the input of the first classification network, taking the annotated label of the scenery type in the scenery type recognition training set as the output of the first classification network, training the first classification network, and obtaining an initial scenery type detection network;

[0025] The initial scenery type detection network is model verified using the scenery type recognition test set, and an initial scenery type detection network having a value greater than or equal to a preset first test accuracy threshold is output as a trained scenery type detection model.

[0026] Further, the high-level semantic feature set includes a plurality of high-level semantic features, and the high-level semantic features specifically refer to the object type of the main area in the image;

[0027] The step of obtaining a high-level semantic feature set of the image to be searched includes:

[0028] The K-means clustering algorithm is used to distinguish pixels of the search image, and the area in the form of pixel clusters is used as the cluster area, and R cluster areas are obtained, where R is an integer greater than zero;

[0029] Input each cluster region into the pre-trained semantic feature extraction model one by one to obtain the object type of each cluster region;

[0030] The generation logic of the semantic feature extraction model is as follows:

[0031] Acquire historical semantic feature extraction training data, and divide the historical semantic feature extraction training data into a semantic feature extraction training set and a semantic feature extraction test set, wherein the historical semantic feature extraction training data includes a plurality of clustered regions in the form of images and annotation labels of their corresponding object types;

[0032] Constructing a second classification network, taking the clustered regions in the form of images in the semantic feature extraction training set as inputs of the second classification network, taking the annotation labels of the object types in the semantic feature extraction training set as outputs of the second classification network, training the second classification network, and obtaining an initial semantic feature extraction network;

[0033] Using the semantic feature extraction test set to perform model verification on the initial semantic feature extraction network, outputting an initial semantic feature extraction network that is greater than or equal to a preset second test accuracy threshold as a trained semantic feature extraction model;

[0034] Determine whether the object type in the cluster area belongs to the non-target object type. If so, remove the corresponding cluster area; if not, mark the corresponding cluster area as the main area;

[0035] Count all the main regions in the R cluster regions, extract the object type of each main region, and use the object type of each main region as a high-level semantic feature;

[0036] The high-level semantic features of all main regions are counted to obtain a high-level semantic feature set of the image to be searched.

[0037] Furthermore, the calculating of the first similarity coefficient between the image to be searched and each landscape image includes:

[0038] Using a pre-built first similarity measurement model, similarity measurement is performed on the high-level semantic feature set of the image to be searched and the landscape image to obtain a first similarity coefficient between the image to be searched and the landscape image;

[0039] The expression of the first similarity measurement model is as follows:

[0040] ;

[0041] Where: represents the first similarity coefficient, Represents the high-level semantic feature set of the image to be searched, A set of high-level semantic features representing landscape images, represents the number of high-level semantic features shared by the search image and the landscape image, represents the number of features in the high-level semantic feature set of the image to be searched that do not appear in the high-level semantic feature set of the landscape image. The number of features in the high-level semantic feature set of the landscape image that do not appear in the high-level semantic feature set of the image to be searched;

[0042] The first similarity coefficients between the image to be searched and all the landscape images are calculated to obtain the first similarity coefficient between the image to be searched and each landscape image.

[0043] Furthermore, the step of selecting a plurality of candidate images from the corresponding image data cluster includes:

[0044] Extracting a first similarity coefficient between the image to be searched and each landscape image, and comparing the first similarity coefficient with a preset first similarity coefficient threshold;

[0045] If the first similarity coefficient is greater than or equal to the first similarity coefficient threshold, marking the corresponding landscape image as a candidate image;

[0046] If the first similarity coefficient is less than the first similarity coefficient threshold, the corresponding landscape image is not marked as a candidate image.

[0047] Furthermore, the calculating of the second similarity coefficient between the image to be searched and each candidate image includes:

[0048] Using a pre-built second similarity measurement model, similarity measurement is performed on the low-level visual feature set of the image to be searched and the candidate image to obtain a second similarity coefficient between the image to be searched and the candidate image;

[0049] The expression of the second similarity measurement model is as follows:

[0050] ;

[0051] Where: represents the second similarity coefficient, represents the feature value of the kth low-level visual feature in the low-level visual feature set of the image to be searched, represents the feature value of the kth low-level visual feature in the low-level visual feature set of the candidate image, Represents the number of low-level visual features;

[0052] The second similarity coefficients of the image to be searched and all candidate images are counted to obtain the second similarity coefficient of the image to be searched and each candidate image.

[0053] Furthermore, the low-level visual feature set includes a plurality of feature values ​​of low-level visual features, and the feature values ​​of the low-level visual features include boundary moments, aspect ratios, color moments, energy and entropy of a main area in an image;

[0054] The step of selecting a similar image that matches the image to be searched from multiple candidate images includes:

[0055] Extracting a first similarity coefficient and a second similarity coefficient between each candidate image and the image to be searched;

[0056] Performing a comprehensive calculation based on the first similarity coefficient and the second similarity coefficient to obtain a comprehensive similarity coefficient between each candidate image and the image to be searched;

[0057] The mathematical expression of the comprehensive calculation is as follows:

[0058] ;

[0059] Where: represents the comprehensive similarity coefficient, and is a correction factor greater than zero, ;

[0060] The comprehensive similarity coefficients of each candidate image and the image to be searched are sorted from large to small, and the candidate image corresponding to the first ranking of the comprehensive similarity coefficient is marked as a similar image that matches the image to be searched.

[0061] A method for searching and matching image content based on machine learning, the method comprising:

[0062] Obtaining an image to be searched, and inputting the image to be searched into a pre-trained landscape type detection model to identify the landscape type to which the image to be searched belongs;

[0063] Retrieving an image data cluster corresponding to the scenery type from a preconfigured image database according to the scenery type; the image data cluster includes a plurality of scenery images and introduction information associated with each scenery scene image;

[0064] Obtaining a high-level semantic feature set of the image to be searched and each landscape image, calculating a first similarity coefficient between the image to be searched and each landscape image according to the high-level semantic feature set, and selecting a plurality of candidate images from the corresponding image data clusters according to the first similarity coefficients;

[0065] Obtaining a low-level visual feature set of the image to be searched and each candidate image, and calculating a second similarity coefficient between the image to be searched and each candidate image according to the low-level visual feature set;

[0066] A similar image that matches the image to be searched is screened out from a plurality of candidate images according to the first similarity coefficient and the second similarity coefficient, and the introduction information associated with the similar image is output as feedback.

[0067] Compared with the prior art, the present invention has the following beneficial effects:

[0068] The present application discloses an image content search and matching system and method based on machine learning, comprising: inputting an image to be searched into a landscape type detection model to identify the landscape type to which the image to be searched belongs; retrieving an image data cluster of the corresponding landscape type from a preconfigured image database according to the landscape type; obtaining a high-level semantic feature set of the image to be searched and each landscape image, calculating a first similarity coefficient according to the high-level semantic feature set, and screening out candidate images from the corresponding image data cluster according to the first similarity coefficient; obtaining a low-level visual feature set of the image to be searched and each candidate image, calculating a second similarity coefficient according to the low-level visual feature set; screening out similar images from the candidate images, and outputting and feeding back the introduction information associated with the similar images; based on the above technical features, the present invention reduces invalid global searches, shortens the search time, and reduces the time required for searching. By reducing the search scope, it is possible to effectively avoid comparing all images in a large-scale image database one by one, thus greatly reducing computational overhead. Furthermore, by quickly matching low-level visual features, subsequent similarity calculations are made more efficient, thereby further improving the search response speed. This is beneficial for improving the image search speed while ensuring the accuracy of image search results, avoiding large deviations in image search results, and meeting the needs of users to query tourist destinations or attractions by uploading images on travel platforms. This is suitable for image search scenarios with concurrent access by large numbers of users. In addition, by setting up an automatic feedback mechanism, it is beneficial for automatic evaluation of image quality, ensuring that the input image meets the preset standards, and ensuring the validity and high quality of image data from the source, thus providing a reliable foundation for subsequent image search and matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A schematic diagram of the module structure of an image content search and matching system based on machine learning provided by the present invention;

[0070] Figure 2 A flowchart of an image content search and matching method based on machine learning provided by the present invention. DETAILED DESCRIPTION

[0071] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0072] Example 1

[0073] See also Figure 1 As shown, this embodiment discloses a system for searching and matching image content based on machine learning, and the system includes:

[0074] The image recognition module 101 is used to obtain an image to be searched and input the image to be searched into a pre-trained landscape type detection model to identify the landscape type to which the image to be searched belongs;

[0075] The scenery types include, but are not limited to, natural scenery types and urban scenery types, wherein the natural scenery types include, but are not limited to, forest types, lake types, and grassland types, etc.; the urban scenery types include, but are not limited to, building and landmark types, park types, and block types, etc.;

[0076] Specifically, before obtaining the image to be searched, the following steps are included:

[0077] Obtaining the captured image uploaded by the user and extracting image quality data of the captured image;

[0078] The image quality data includes the size of the image, the brightness value of each pixel, the noise level of the image, and the maximum and minimum brightness values ​​of the image;

[0079] Inputting the image quality data into a pre-built mathematical calculation model to obtain a quality coefficient of the captured image;

[0080] The mathematical calculation model is expressed as follows:

[0081] ;

[0082] Where: is the quality coefficient, and are the width and height (i.e. the size of the image), is the Laplace operator, which is used to represent the image at position The second-order derivative at , For the image at pixel points The brightness value at is the maximum brightness value in the image (i.e. the brightest pixel), is the minimum brightness value in the image (i.e. the darkest pixel), is the noise level of the image; and is a correction factor greater than zero, ; Wherein, the noise level is obtained by calculating the ratio of the noise intensity (i.e., noise energy) in the image to the effective signal intensity (i.e., signal energy) in the image;

[0083] Wherein, the Laplace operator can be expressed as: ; Where: It represents the second-order derivative of the image in the x-axis direction, reflecting the change of the brightness value of the image in the horizontal direction; It represents the second-order derivative of the image in the y-axis direction, reflecting the change of the brightness value of the image in the vertical direction;

[0084] The quality coefficient of the captured image is compared with a preset quality coefficient threshold. If the quality coefficient is greater than or equal to the quality coefficient threshold, it is determined that the corresponding captured image meets the preset quality output standard, and the corresponding captured image is marked as a to-be-searched image; if the quality coefficient is less than the quality coefficient threshold, it is determined that the corresponding captured image does not meet the preset quality output standard.

[0085] In a specific implementation, when it is determined that the corresponding captured image does not meet the preset quality output standard, a preset display content is output for feedback; wherein the preset display content is the words "please re-upload the image";

[0086] It can be understood that: by calculating the quality coefficient of the captured image uploaded by the user, and performing quality judgment or resampling reminder on the captured image according to the quality coefficient, it is helpful to determine whether the captured image uploaded by the user meets the preset quality output standard, and then it is helpful to ensure the accuracy of subsequent subject area acquisition, and further, it is helpful to improve the accuracy of subsequent image content search;

[0087] Specifically, the generation logic of the landscape type detection model is as follows:

[0088] Acquire historical scenery type recognition training data, and divide the historical scenery type recognition training data into a scenery type recognition training set and a scenery type recognition test set; the historical scenery type recognition training data includes a plurality of images to be searched and annotated labels of their corresponding scenery types;

[0089] It should be noted that: the images to be searched in the historical landscape type recognition training data are collected by technicians according to the above-mentioned quality standards, and the annotation labels of the landscape types in the historical landscape type recognition training data are manually annotated by technicians or AI data annotators; for example; assuming that the image to be searched is a photo of the Temple of Heaven in Beijing, the technicians or AI data annotators will annotate the image to be searched as a building and landmark type in the urban landscape type;

[0090] Constructing a first classification network, taking the image to be searched in the scenery type recognition training set as the input of the first classification network, taking the annotated label of the scenery type in the scenery type recognition training set as the output of the first classification network, training the first classification network, and obtaining an initial scenery type detection network;

[0091] Using the landscape type recognition test set to perform model verification on the initial landscape type detection network, outputting an initial landscape type detection network with a precision greater than or equal to a preset first test accuracy threshold as a trained landscape type detection model;

[0092] Among them, the first classification network is specifically one of the classification algorithm models such as naive Bayes, convolutional neural network or recurrent neural network.

[0093] The data retrieval module 102 is used to retrieve an image data cluster corresponding to a landscape type from a pre-configured image database according to the landscape type; the image data cluster includes a plurality of landscape images and introduction information associated with each landscape scene image;

[0094] It should be noted that: the preconfigured image database is manually constructed in advance by technical personnel, and the image database includes several image data clusters, each of which stores multiple landscape images representing a certain landscape type; each image data cluster is formed after automatic identification and clustering by a pre-trained landscape type detection model; wherein each landscape image is manually associated and bound with corresponding introduction information in advance by technical personnel, and the introduction information includes but is not limited to the name of the landscape, the location of the landscape, the price of the landscape and the history of the landscape.

[0095] A preliminary screening module 103 is used to obtain a high-level semantic feature set of the image to be searched and each landscape image, calculate a first similarity coefficient between the image to be searched and each landscape image according to the high-level semantic feature set, and screen multiple candidate images from the corresponding image data cluster according to the first similarity coefficient;

[0096] The high-level semantic feature set includes a plurality of high-level semantic features, and the high-level semantic features specifically refer to the object type of the main area in the image;

[0097] In implementation, the step of obtaining a high-level semantic feature set of an image to be searched includes:

[0098] The K-means clustering algorithm is used to distinguish pixels of the search image, and the area in the form of pixel clusters is used as the cluster area, and R cluster areas are obtained, where R is an integer greater than zero;

[0099] Input each cluster region into the pre-trained semantic feature extraction model one by one to obtain the object type of each cluster region;

[0100] Specifically, the generation logic of the semantic feature extraction model is as follows:

[0101] Acquire historical semantic feature extraction training data, and divide the historical semantic feature extraction training data into a semantic feature extraction training set and a semantic feature extraction test set, wherein the historical semantic feature extraction training data includes a plurality of clustered regions in the form of images and annotation labels of their corresponding object types;

[0102] It should be noted that: the clustering regions in the historical semantic feature extraction training data are collected and processed in advance by the technicians according to the above-mentioned image segmentation process; the annotation labels of the object types in the historical semantic feature extraction training data are obtained by manual annotation by the technicians or AI data annotators;

[0103] Constructing a second classification network, taking the clustered regions in the form of images in the semantic feature extraction training set as inputs of the second classification network, taking the annotation labels of the object types in the semantic feature extraction training set as outputs of the second classification network, training the second classification network, and obtaining an initial semantic feature extraction network;

[0104] Using the semantic feature extraction test set to perform model verification on the initial semantic feature extraction network, outputting an initial semantic feature extraction network that is greater than or equal to a preset second test accuracy threshold as a trained semantic feature extraction model;

[0105] Wherein, like the above-mentioned first classification network, the second classification network is specifically one of the classification algorithm models such as naive Bayes, convolutional neural network or recurrent neural network;

[0106] Determine whether the object type in the cluster area belongs to the non-target object type. If so, remove the corresponding cluster area; if not, mark the corresponding cluster area as the main area;

[0107] It should be noted that the object type refers to the object type contained in the cluster area. For example, for an image of a forest type in a natural scenery type, if there are two cluster areas in the image, and the two cluster areas respectively contain a specific tree type, such as pine trees and cypress trees, then the pine trees and cypress trees are called the object types of the corresponding cluster areas; for another example, for an image of a building and landmark type in an urban scenery type, if there are two cluster areas in the image, and the two cluster areas respectively contain a specific building type, such as a stone tablet and a stone base (i.e., a base made of stone), then the stone tablet and the stone base are called the object types of the corresponding cluster areas;

[0108] It should also be noted that non-target object types refer to those object types that are not related to the scenery, such as cars, people or the sky;

[0109] Count all the main regions in the R cluster regions, extract the object type of each main region, and use the object type of each main region as a high-level semantic feature;

[0110] Count the high-level semantic features of all main regions to obtain a high-level semantic feature set of the image to be searched;

[0111] It should be noted that the logic for obtaining the high-level semantic feature set of each of the landscape images is the same as the logic for the high-level semantic feature set of the above-mentioned image to be searched. For details, please refer to the above-mentioned description process and will not be repeated here. The difference is that when obtaining the high-level semantic feature set of each landscape image, there is no need to perform non-target object type judgment and elimination operations. This is because each landscape image stored in the image database is obtained without non-target object types. To further explain, there is no clustering area of ​​non-target object types such as cars, people or sky in each landscape image.

[0112] In implementation, the step of calculating the first similarity coefficient between the image to be searched and each landscape image includes:

[0113] Using a pre-built first similarity measurement model, similarity measurement is performed on the high-level semantic feature set of the image to be searched and the landscape image to obtain a first similarity coefficient between the image to be searched and the landscape image;

[0114] The expression of the first similarity measurement model is as follows:

[0115] ;

[0116] Where: represents the first similarity coefficient, Represents the high-level semantic feature set of the image to be searched, A set of high-level semantic features representing landscape images, represents the number of high-level semantic features shared by the search image and the landscape image, represents the number of features in the high-level semantic feature set of the image to be searched that do not appear in the high-level semantic feature set of the landscape image. The number of features in the high-level semantic feature set of the landscape image that do not appear in the high-level semantic feature set of the image to be searched;

[0117] Counting the first similarity coefficients between the image to be searched and all the landscape images, and obtaining the first similarity coefficient between the image to be searched and each landscape image;

[0118] In implementation, the step of selecting a plurality of candidate images from the corresponding image data cluster includes:

[0119] Extracting a first similarity coefficient between the image to be searched and each landscape image, and comparing the first similarity coefficient with a preset first similarity coefficient threshold;

[0120] If the first similarity coefficient is greater than or equal to the first similarity coefficient threshold, marking the corresponding landscape image as a candidate image;

[0121] If the first similarity coefficient is less than the first similarity coefficient threshold, the corresponding landscape image is not marked as a candidate image;

[0122] For example, it is assumed that the scenery type of the image to be searched is a forest type in the natural scenery type, and the image to be searched is identified to contain multiple high-level semantic feature sets, including pine trees, cypress trees, and fir trees. Therefore, ={pine, cypress, fir}, and it is also assumed that there is a landscape image in the corresponding image data cluster, and the landscape image is also identified to contain multiple high-level semantic feature sets, including pine, oak and maple, respectively. Therefore, ={pine, oak, maple}. Further, by comparison, it can be seen that the high-level semantic feature shared by the search image and the landscape image is pine. Therefore, , and in the high-level semantic feature set of the image to be searched, the features that do not appear in the high-level semantic feature set of the landscape image are cypress and fir, so , and in the high-level semantic feature set of landscape images, the features that do not appear in the high-level semantic feature set of the image to be searched are oak and maple. Therefore, Finally, substitute the above data into the expression of the first similarity coefficient to obtain If the first similarity coefficient threshold is 0.5, it means that the landscape image is not similar to the image to be searched, and therefore, it is not considered as a candidate image.

[0123] A similarity calculation module 104 is used to obtain a low-level visual feature set of the image to be searched and each candidate image, and calculate a second similarity coefficient between the image to be searched and each candidate image according to the low-level visual feature set;

[0124] The low-level visual feature set includes multiple feature values ​​of low-level visual features, and the feature values ​​of the low-level visual features are not limited to boundary moments (including but not limited to perimeter, area and roundness, etc.), aspect ratios, color moments (including but not limited to mean, variance and skewness of colors, etc.), energy and entropy, etc. of the main area in the image;

[0125] The low-level visual feature set of the image to be searched or the candidate image is collected and calculated using edge detection methods (such as Canny edge detection), minimum enclosing rectangle method, Numpy library and gray level co-occurrence matrix (GLCM), etc.;

[0126] In implementation, the step of calculating the second similarity coefficient between the image to be searched and each candidate image includes:

[0127] Using a pre-built second similarity measurement model, similarity measurement is performed on the low-level visual feature set of the image to be searched and the candidate image to obtain a second similarity coefficient between the image to be searched and the candidate image;

[0128] The expression of the second similarity measurement model is as follows:

[0129] ;

[0130] Where: represents the second similarity coefficient, represents the feature value of the kth low-level visual feature in the low-level visual feature set of the image to be searched, represents the feature value of the kth low-level visual feature in the low-level visual feature set of the candidate image, Represents the number of low-level visual features;

[0131] The second similarity coefficients of the image to be searched and all candidate images are counted to obtain the second similarity coefficient of the image to be searched and each candidate image.

[0132] The image search module 105 is used to select similar images that match the image to be searched from multiple candidate images according to the first similarity coefficient and the second similarity coefficient, and output the introduction information associated with the similar images as feedback;

[0133] In implementation, the step of selecting a similar image that matches the image to be searched from multiple candidate images includes:

[0134] Extracting a first similarity coefficient and a second similarity coefficient between each candidate image and the image to be searched;

[0135] Performing a comprehensive calculation based on the first similarity coefficient and the second similarity coefficient to obtain a comprehensive similarity coefficient between each candidate image and the image to be searched;

[0136] The mathematical expression of the comprehensive calculation is as follows:

[0137] ;

[0138] Where: represents the comprehensive similarity coefficient, and is a correction factor greater than zero, ;

[0139] Sort the comprehensive similarity coefficients of each candidate image and the image to be searched from large to small, and mark the candidate image with the first comprehensive similarity coefficient as a similar image that matches the image to be searched;

[0140] By setting up a hierarchical search mechanism, high-level semantic features are first used to perform a preliminary screening of images, which reduces invalid global searches, narrows the search scope, and can effectively avoid one-by-one comparison of all images in a large-scale image database, greatly reducing computational overhead; further, low-level visual features are used for rapid matching, so that subsequent similarity calculations are more efficient, thereby further improving the search response speed. As a result, the present invention is conducive to improving the image search speed while ensuring the accuracy of image search results, avoiding large deviations in image search results, and meeting the needs of users on a travel platform to query travel destinations or attractions by uploading images. It is suitable for image search scenarios with concurrent access by large numbers of users.

[0141] Example 2

[0142] See also Figure 2 As shown, based on the same inventive concept, this embodiment discloses a method for image content search and matching based on machine learning. For details not provided in this embodiment, please refer to the description of the relevant parts in Embodiment 1. The method includes:

[0143] S201: Acquire an image to be searched, and input the image to be searched into a pre-trained landscape type detection model to identify the landscape type to which the image to be searched belongs;

[0144] S202: Retrieving an image data cluster corresponding to a landscape type from a pre-configured image database according to the landscape type; the image data cluster includes a plurality of landscape images and introduction information associated with each landscape scene image;

[0145] S203: obtaining a high-level semantic feature set of the image to be searched and each landscape image, calculating a first similarity coefficient between the image to be searched and each landscape image according to the high-level semantic feature set, and selecting a plurality of candidate images from the corresponding image data cluster according to the first similarity coefficient;

[0146] S204: obtaining a low-level visual feature set of the image to be searched and each candidate image, and calculating a second similarity coefficient between the image to be searched and each candidate image according to the low-level visual feature set;

[0147] S205: Screening out similar images that match the image to be searched from multiple candidate images according to the first similarity coefficient and the second similarity coefficient, and outputting the introduction information associated with the similar images as feedback.

[0148] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters, weights and thresholds in the formula are set by technicians in this field according to actual conditions.

[0149] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired network or a wireless network. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media sets. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD) or a semiconductor medium. The semiconductor medium may be a solid-state hard disk.

[0150] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0151] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0152] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only one, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0153] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0154] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0155] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

[0156] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An image content search and matching system based on machine learning, characterized in that: The system comprises: An image recognition module is used to obtain an image to be searched and input the image to be searched into a pre-trained landscape type detection model to identify the landscape type to which the image to be searched belongs; The generation logic of the landscape type detection model is as follows: Acquire historical scenery type recognition training data, and divide the historical scenery type recognition training data into a scenery type recognition training set and a scenery type recognition test set; the historical scenery type recognition training data includes a plurality of images to be searched and annotated labels of their corresponding scenery types; Constructing a first classification network, taking the image to be searched in the scenery type recognition training set as the input of the first classification network, taking the annotated label of the scenery type in the scenery type recognition training set as the output of the first classification network, training the first classification network, and obtaining an initial scenery type detection network; Using the landscape type recognition test set to perform model verification on the initial landscape type detection network, outputting an initial landscape type detection network with a precision greater than or equal to a preset first test accuracy threshold as a trained landscape type detection model; A data retrieval module is used to retrieve an image data cluster corresponding to a landscape type from a pre-configured image database according to the landscape type; the image data cluster includes a plurality of landscape images and introduction information associated with each landscape scene image; A preliminary screening module, used to obtain a high-level semantic feature set of the image to be searched and each landscape image, calculate a first similarity coefficient between the image to be searched and each landscape image according to the high-level semantic feature set, and screen multiple candidate images from the corresponding image data cluster according to the first similarity coefficient; the high-level semantic feature set includes multiple high-level semantic features, and the high-level semantic features specifically refer to the object type in the main area of ​​the image; A similarity calculation module is used to obtain a low-level visual feature set of the image to be searched and each candidate image, and calculate a second similarity coefficient between the image to be searched and each candidate image according to the low-level visual feature set; the low-level visual feature set includes a plurality of feature values ​​of low-level visual features, and the feature values ​​of the low-level visual features include a boundary moment, an aspect ratio, a color moment, energy and entropy of a main area in the image; The image search module is used to screen out similar images that match the image to be searched from multiple candidate images according to the first similarity coefficient and the second similarity coefficient, and output the introduction information associated with the similar images as feedback.

2. The image content search and matching system based on machine learning according to claim 1, characterized in that: Before obtaining the image to be searched, include: Obtaining the captured image uploaded by the user and extracting image quality data of the captured image; The image quality data includes the size of the image, the brightness value of each pixel, the noise level of the image, and the maximum and minimum brightness values ​​of the image; Inputting the image quality data into a pre-built mathematical calculation model to obtain a quality coefficient of the captured image; The mathematical calculation model is expressed as follows: ； Where: is the quality coefficient, and are the width and height respectively, is the Laplace operator, For the image at pixel points The brightness value at is the maximum brightness value in the image, is the minimum brightness value in the image, is the noise level of the image; and is a correction factor greater than zero, ; The quality coefficient of the captured image is compared with the preset quality coefficient threshold. If the quality coefficient is greater than or equal to the quality coefficient threshold, the corresponding captured image is judged to meet the preset quality output standard and the corresponding captured image is marked as an image to be searched; if the quality coefficient is less than the quality coefficient threshold, the corresponding captured image is judged to not meet the preset quality output standard.

3. The image content search and matching system based on machine learning according to claim 2, characterized in that: When it is determined that the corresponding captured image does not meet the preset quality output standard, the preset display content is output for feedback; wherein, the preset display content is the words "Please re-upload the image".

4. The image content search and matching system based on machine learning according to claim 1, characterized in that: The step of obtaining a high-level semantic feature set of the image to be searched includes: The K-means clustering algorithm is used to distinguish pixels of the search image, and the area in the form of pixel clusters is used as the cluster area, and R cluster areas are obtained, where R is an integer greater than zero; Input each cluster region into the pre-trained semantic feature extraction model one by one to obtain the object type of each cluster region; The generation logic of the semantic feature extraction model is as follows: Acquire historical semantic feature extraction training data, and divide the historical semantic feature extraction training data into a semantic feature extraction training set and a semantic feature extraction test set, wherein the historical semantic feature extraction training data includes a plurality of clustered regions in the form of images and annotation labels of their corresponding object types; Constructing a second classification network, taking the clustered regions in the form of images in the semantic feature extraction training set as inputs of the second classification network, taking the annotation labels of the object types in the semantic feature extraction training set as outputs of the second classification network, training the second classification network, and obtaining an initial semantic feature extraction network; Using the semantic feature extraction test set to perform model verification on the initial semantic feature extraction network, outputting an initial semantic feature extraction network that is greater than or equal to a preset second test accuracy threshold as a trained semantic feature extraction model; Determine whether the object type in the cluster area belongs to the non-target object type. If so, remove the corresponding cluster area; if not, mark the corresponding cluster area as the main area; Count all the main regions in the R cluster regions, extract the object type of each main region, and use the object type of each main region as a high-level semantic feature; The high-level semantic features of all main regions are counted to obtain a high-level semantic feature set of the image to be searched.

5. The image content search and matching system based on machine learning according to claim 1, characterized in that: The calculating of the first similarity coefficient between the image to be searched and each landscape image comprises: Using a pre-built first similarity measurement model, similarity measurement is performed on the high-level semantic feature set of the image to be searched and the landscape image to obtain a first similarity coefficient between the image to be searched and the landscape image; The expression of the first similarity measurement model is as follows: ； Where: represents the first similarity coefficient, Represents the high-level semantic feature set of the image to be searched, A set of high-level semantic features representing landscape images, represents the number of high-level semantic features shared by the search image and the landscape image, represents the number of features in the high-level semantic feature set of the image to be searched that do not appear in the high-level semantic feature set of the landscape image. The number of features in the high-level semantic feature set of the landscape image that do not appear in the high-level semantic feature set of the image to be searched; The first similarity coefficients between the image to be searched and all the landscape images are calculated to obtain the first similarity coefficient between the image to be searched and each landscape image.

6. The image content search and matching system based on machine learning according to claim 5, characterized in that: The step of selecting a plurality of candidate images from the corresponding image data cluster comprises: Extracting a first similarity coefficient between the image to be searched and each landscape image, and comparing the first similarity coefficient with a preset first similarity coefficient threshold; If the first similarity coefficient is greater than or equal to the first similarity coefficient threshold, marking the corresponding landscape image as a candidate image; If the first similarity coefficient is less than the first similarity coefficient threshold, the corresponding landscape image is not marked as a candidate image.

7. The image content search and matching system based on machine learning according to claim 6, characterized in that: The calculating the second similarity coefficient between the image to be searched and each candidate image comprises: Using a pre-built second similarity measurement model, similarity measurement is performed on the low-level visual feature set of the image to be searched and the candidate image to obtain a second similarity coefficient between the image to be searched and the candidate image; The expression of the second similarity measurement model is as follows: ； Where: represents the second similarity coefficient, represents the feature value of the kth low-level visual feature in the low-level visual feature set of the image to be searched, represents the feature value of the kth low-level visual feature in the low-level visual feature set of the candidate image, Represents the number of low-level visual features; The second similarity coefficients of the image to be searched and all candidate images are counted to obtain the second similarity coefficient of the image to be searched and each candidate image.

8. The image content search and matching system based on machine learning according to claim 7, characterized in that: The step of selecting a similar image that matches the image to be searched from a plurality of candidate images includes: Extracting a first similarity coefficient and a second similarity coefficient between each candidate image and the image to be searched; Performing a comprehensive calculation based on the first similarity coefficient and the second similarity coefficient to obtain a comprehensive similarity coefficient between each candidate image and the image to be searched; The mathematical expression of the comprehensive calculation is as follows: ； Where: represents the comprehensive similarity coefficient, and is a correction factor greater than zero, ; The comprehensive similarity coefficients of each candidate image and the image to be searched are sorted from large to small, and the candidate image corresponding to the first ranking of the comprehensive similarity coefficient is marked as a similar image that matches the image to be searched.

9. A method for searching and matching image content based on machine learning, characterized in that: The method is implemented based on the image content search and matching system based on machine learning according to any one of claims 1 to 8, and the method comprises: Obtaining an image to be searched, and inputting the image to be searched into a pre-trained landscape type detection model to identify the landscape type to which the image to be searched belongs; Retrieving an image data cluster corresponding to the scenery type from a preconfigured image database according to the scenery type; the image data cluster includes a plurality of scenery images and introduction information associated with each scenery scene image; Obtaining a high-level semantic feature set of the image to be searched and each landscape image, calculating a first similarity coefficient between the image to be searched and each landscape image according to the high-level semantic feature set, and selecting a plurality of candidate images from the corresponding image data clusters according to the first similarity coefficients; Obtaining a low-level visual feature set of the image to be searched and each candidate image, and calculating a second similarity coefficient between the image to be searched and each candidate image according to the low-level visual feature set; A similar image that matches the image to be searched is screened out from a plurality of candidate images according to the first similarity coefficient and the second similarity coefficient, and the introduction information associated with the similar image is output as feedback.

Images