Picture retrieval optimization method based on street view retrieval

Abstract

The application discloses a picture retrieval optimization algorithm based on street view retrieval and belongs to the technical field of picture retrieval methods, which comprises the following steps: extracting image features of street view pictures based on the relationship five-tuple of joint texture features, local features and picture information; performing preliminary retrieval on the street view pictures by using similarity detection; and performing accurate retrieval on the street view pictures by combining five-tuple information extraction and local feature description. The application can construct the relationship five-tuple of street view pictures based on BERT and prior knowledge features, construct a street view picture knowledge graph, establish the subject, predicate, object, time and space relationship five-tuple of the street view picture knowledge graph, fully utilize the prior knowledge base information of the street view pictures, deeply mine the semantic information of the picture knowledge graph, realize high-precision retrieval and high-efficiency retrieval of the street view pictures, and further enrich the features of the street view picture retrieval by extracting the time and space attributes of the pictures.

Description

TECHNICAL FIELD

[0001] The application relates to a retrieval method, in particular to a picture retrieval optimization method based on street view retrieval, and belongs to the technical field of picture retrieval methods. BACKGROUND

[0002] The picture retrieval technology based on the picture retrieval technology is a hot direction in the current picture retrieval technology research, because it does not retrieve the picture itself, but retrieves the color, texture and shape and other features of the picture, greatly improves the efficiency and speed of the picture retrieval, reduces the calculation amount of the picture retrieval, and can guarantee the effectiveness of the retrieval. The current mature picture retrieval system mainly depends on picture similarity for retrieval. The classification algorithm used in this content-based picture retrieval is perceptual hashing algorithm. It can assign an identification feature vector to each picture. When retrieving the picture, the identification feature vector is used for retrieval, and the result with the shortest distance between the feature vectors can be returned to realize the picture retrieval. Therefore, the perceptual hashing algorithm has high picture retrieval efficiency, but the retrieval does not have rotation invariance, and the description and capture of details are poor. The similarity of the street view pictures in the geographical pictures is high, and the street view pictures need to be distinguished and retrieved by relying on the detailed description, which cannot meet the requirements of the street view picture retrieval. Another picture retrieval method is a picture retrieval method relying on global features. The picture retrieval system of the global feature retrieval is mainly used for the picture retrieval of the theme with obvious color, texture and shape features. The color of the street view picture is related to the light angle, and the global color is similar. The color as the global feature cannot well complete the street view picture retrieval work, and the texture feature retrieval accuracy is poor and cannot meet the accuracy requirements of the street view picture registration. Therefore, the picture retrieval algorithm based on the global feature is not suitable for the street view picture retrieval. There is no knowledge graph constructed for the street view picture in the current picture retrieval algorithm, and the relationship five-tuple of the picture knowledge graph cannot be extracted to realize the improvement of the street view picture retrieval accuracy and efficiency. Therefore, the application provides a picture retrieval optimization method based on street view retrieval. The algorithm can construct the relationship five-tuple of the street view picture based on BERT and prior knowledge features, construct the street view picture knowledge graph, establish the subject, predicate, object, time and space relationship five-tuple of the street view picture knowledge graph, fully utilize the prior knowledge base information of the street view picture, deeply mine the semantic information of the picture knowledge graph, realize the high-precision retrieval and high-efficiency retrieval of the street view picture, and further enrich the features of the street view picture retrieval. Combined with the content-based image retrieval technology, the accuracy and efficiency of the street view picture retrieval are further improved. SUMMARY

[0003] The main purpose of the present application is to solve the problem that the current picture retrieval algorithm does not construct a knowledge graph for street view pictures, and cannot realize the improvement of street view picture retrieval accuracy and efficiency according to the relationship five tuple extraction of the picture knowledge graph, and provide a picture retrieval optimization method based on street view retrieval.

[0004] The purpose of the present application can be achieved by adopting the following technical solutions:

[0005] A picture retrieval optimization method based on street view retrieval includes the following steps:

[0006] S1. Image feature extraction: the texture features of the street view picture are extracted by using the gray level co-occurrence matrix to form the texture feature vector of the street view picture, and the local invariant features in the picture are accurately extracted as the local feature vector by using the scale invariant algorithm, and the relationship five tuple based on BERT and prior knowledge features is created for the street view picture, and the relationship five tuple feature vector of the street view picture is extracted;

[0007] S2. Street view picture preliminary retrieval: for the texture feature vector of the picture, the feature similarity distance is measured to form the picture index sorting, and the street view picture is preliminarily retrieved;

[0008] S3. Street view picture accurate retrieval: the picture index sorting formed by the preliminary retrieval is combined with the local feature vector of the street view picture to realize the accurate retrieval of the street view picture, and the relationship five tuple of the street view picture is retrieved to realize the high-precision matching of the space-time information of the street view picture.

[0009] As a further method of the present application, the street view image local feature vector construction method in S1 includes the following steps:

[0010] A1. Street view picture spatial extreme value detection: the convolution of the Gaussian function and the image function is used as the scale function of the scale invariant feature conversion to create a scale space, and the local extreme points on the space are found; A2. Street view picture key point positioning: the corner points and edge points which are not changed by the rotation angle, illumination transformation and affine transformation are selected as the suspected key points, the key points are detected by using the Taylor series expansion, the points not meeting the threshold are proposed, the principal curvatures are calculated by using the hessian matrix, and the edge response is removed;

[0011] A3. Street view picture key point direction determination: the field of the positioned key point is selected, and the main direction and multiple auxiliary directions of the key point are determined by using the 360-degree gradient histogram of the key point field;

[0012] A4. Street view picture key point description: the neighborhood of the key point is selected, the window formed by the neighborhood is divided into multiple sub-blocks, each sub-block forms a seed point, the information of the four seed points forms a feature point, the gradient histogram of each sub-block in eight directions is calculated, and the local feature vector is formed.

[0013] As a further method of the application, the S1 street view picture is based on the relationship five tuple extraction method of BERT and prior knowledge features:

[0014] B1. Create a street view picture relationship five tuple: preprocess the extracted street view picture information text to obtain a text sequence at the word level, take each element of each text sequence as a token, input the text sequence into the BERT model for encoding to obtain a semantic feature vector h of each token of the sequence, use the AC automatic calculation method to search for street view picture knowledge keywords in the prior knowledge base to obtain prior knowledge features pre, label the prior knowledge features pre, add time and space features at the beginning and end of each token, fuse the retrieved matching prior knowledge features pre with the semantic feature vector to form a fusion feature vector h+pre;

[0015] B2. Feature vector extraction: input the formed fusion vector into two semi-pointer-semi-label structures with sigmoid activation function full connection layer, predict the beginning and end positions of the subject entity, take the feature vectors at the beginning and end positions of the subject entity for vector average operation, and then add them to the token sequence text to obtain a feature vector h+pre+s. Then, connect the obtained feature vector h+pre+s with n semi-pointer-semi-label structures to predict the type of predicate and the beginning and end positions of object, encode them into the feature vector of the token sequence text to obtain a feature vector h+pre+s+p+o. Connect the feature vector h+pre+s+p+o with two semi-pointer-edge label structures to simultaneously predict the beginning and end positions of time and location, and encode them into the feature vector of the token sequence text to obtain a relationship five tuple feature vector.

[0016] As a further method of the application, the algorithm uses the picture matching failure rate and the matching point failure rate to evaluate the quality of local feature retrieval for local feature vector matching retrieval, wherein the picture matching failure rate is the ratio of the number of matching failed pictures to the total number of sample pictures, and the matching point failure rate is the ratio of the number of incorrect matching point pairs in matching success to the number of matching point pairs. The evaluation formula of the picture matching failure rate and the matching point pair failure rate is:

[0017] ;

[0018] In the formula: is the picture matching failure rate of the street view picture, is the number of all street view pictures participating in retrieval, is the number of matching failed pictures;

[0019] ;

[0020] In the formula: is the matching point pair failure rate of the street view picture, is the matching point pair of the street view picture participating in the search, is the number of matching failed point pairs.

[0021] As a further method of the present application, the search method of the street view picture relationship five tuple in S3 includes the following steps:

[0022] C1. Head entity, relationship type and tail entity triple search: keyword search and semantic search are performed on the head entity, relationship type and tail entity of the street view picture information to form a primary search triple ranking;

[0023] C2. Time attribute and spatial attribute search: the time attribute and spatial attribute of the street view picture information in the formed primary search triple ranking are matched to realize accurate search of the five tuple attribute.

[0024] As a further method of the present application, the model in S1 based on BERT and prior knowledge features utilizes the Transformer model to jointly train the binary classification cross loss function.

[0025] The beneficial technical effects of the present application are: according to the picture search optimization method based on street view search of the present application, the relationship five tuple of the street view picture can be constructed based on BERT and prior knowledge features, the street view picture knowledge graph can be constructed, the subject, predicate, object, time and spatial relationship five tuple of the street view picture knowledge graph can be established, the prior knowledge base information of the street view picture can be fully utilized, the semantic information of the picture knowledge graph can be deeply mined, high-precision search and high-efficiency search of the street view picture can be realized, the temporal and spatial attributes of the picture can be extracted, the features of the street view picture search are further enriched, and the precision and efficiency of the street view picture search are further improved in combination with the content-based image search technology. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is a flowchart of the picture search optimization method based on street view search according to the present application. DETAILED DESCRIPTION

[0027] In order to make the technical solution of the present application clearer and more explicit to those skilled in the art, the present application will be described in further detail below in conjunction with the embodiments and drawings, but the embodiments of the present application are not limited thereto.

[0028] As shown in Figure 1 , the picture search optimization method based on street view search provided by the present embodiment includes the following steps:

[0029] S1. Image feature extraction: the texture features of the street view picture are extracted by using a gray level co-occurrence matrix to form a street view picture texture feature vector, and the local invariant features in the picture are accurately extracted by using a scale invariant algorithm as a local feature vector, and the relationship five-tuple based on BERT and prior knowledge features is created for the street view picture, and the relationship five-tuple feature vector of the street view picture is extracted;

[0030] S2. Street view picture preliminary retrieval: for the texture feature vector of the picture, the feature similarity distance is measured to form a picture index order, and the street view picture is preliminarily retrieved;

[0031] S3. Street view picture accurate retrieval: the picture index order formed by the preliminary retrieval is combined with the local feature vector of the street view picture to realize accurate retrieval of the street view picture, and the relationship five-tuple of the street view picture is retrieved to realize high-precision matching of the space-time information of the street view picture.

[0032] The picture retrieval optimization method based on street view retrieval provided by the application can construct the relationship five-tuple of the street view picture based on BERT and prior knowledge features, construct the knowledge graph of the street view picture, establish the subject, predicate, object, time and space relationship five-tuple of the knowledge graph of the street view picture, fully utilize the prior knowledge base information of the street view picture, deeply mine the semantic information of the picture knowledge graph, realize high-precision retrieval and high-efficiency retrieval of the street view picture, and further enrich the features of the street view picture retrieval, combined with the content-based image retrieval technology, further improve the accuracy and efficiency of the street view picture retrieval.

[0033] The street view image local feature vector construction method in S1 includes the following steps:

[0034] A1. Street view picture space extreme value detection: a scale space is created by using the convolution of a Gaussian function and an image function as a scale function of scale invariant feature conversion, and local extreme points on the space are searched;

[0035] A2. Street view picture key point positioning: selecting corner points and edge points which are not changed by rotation angle, illumination transformation and affine transformation as suspected key points, detecting the key points by using Taylor series expansion, removing points not meeting the threshold, calculating the principal curvature by using a hessian matrix, and removing edge responses;

[0036] A3. Street view picture key point direction determination: selecting the field of the positioned key points, determining a main direction and multiple auxiliary directions of the key points by using a 360-degree gradient histogram of the key point field;

[0037] A4. Street view picture key point description: select the key point neighborhood, divide the window formed into multiple sub-blocks, each sub-block forms a seed point, the information of four seed points forms a feature point, calculate the gradient histogram of eight directions of each sub-block to form a local feature vector.

[0038] Through the construction of the local feature vector of the street view image, the scale invariant features in the street view image can be captured, the factors that change with light, color, etc. in the street view image are excluded, the capture of the scale invariant features of the street view image is accurately realized, and the accuracy of the street view picture feature matching is improved.

[0039] The S1 street view picture relationship five tuple extraction method based on BERT and prior knowledge features is:

[0040] B1. Create a street view picture relationship five tuple: preprocess the extracted street view picture information text to obtain a text sequence at the word level, take each element of each text sequence as a token, input the text sequence into a BERT model to obtain a semantic feature vector h of each token of the sequence, use an AC automatic calculation method to search for a prior knowledge feature pre in a prior knowledge base, label the prior knowledge feature pre, add time and space features at the beginning and end of each token, fuse the retrieved matching prior knowledge feature pre with the semantic feature vector to form a fusion feature vector h+pre;

[0041] B2. Feature vector extraction: input the formed fusion vector into two semi-pointer-semi-label structures with sigmoid activation function full connection layer, predict the beginning and end positions of the subject entity, take the feature vectors of the beginning and end positions of the subject entity for vector average operation, and then add the token sequence text to obtain a feature vector h+pre+s, then connect the obtained feature vector h+pre+s with n semi-pointer-semi-label structures to predict the type of predicate and the beginning and end positions of object, encode them into the feature vector of the token sequence text to obtain a feature vector h+pre+s+p+o, connect two semi-pointer-edge label structures behind the feature vector h+pre+s+p+o, simultaneously predict the beginning and end positions of time and location, and encode them into the feature vector of the token sequence text to obtain a relationship five tuple feature vector.

[0042] Through the creation of the street view picture information relationship five tuple, the information description and feature record of the street view picture can be enriched, and the content-based picture retrieval can be assisted to improve the retrieval accuracy and speed.

[0043] The algorithm uses picture matching failure rate and matching point failure rate to evaluate the quality of local feature retrieval for the evaluation of local feature vector matching retrieval, wherein the picture matching failure rate is the ratio of the number of matching failed pictures to the total number of sample pictures, the matching point failure rate is the ratio of the number of error matching point pairs in matching success to the number of matching point pairs, and the evaluation formula of the picture matching failure rate and the matching point pair failure rate is:

[0044] ;

[0045] In the formula: is the picture matching failure rate of the street view picture, is the number of street view pictures participating in retrieval, is the number of matching failed pictures;

[0046] ;

[0047] In the formula: is the matching point pair failure rate of the street view picture, is the matching point pair of the street view picture participating in retrieval, is the number of matching failed point pairs.

[0048] The content-based retrieval results of the street view pictures are evaluated by the matching failure rate and the matching point failure rate, so as to facilitate the observation of the retrieval accuracy and effect of the street view image retrieval technology.

[0049] The retrieval method of the street view picture relationship five tuple in S3 includes the following steps:

[0050] C1. Head entity, relationship type and tail entity triple retrieval: keyword retrieval and semantic retrieval are performed on the head entity, relationship type and tail entity of the street view picture information to form a primary retrieval triple sorting;

[0051] C2. Time attribute and space attribute retrieval: The time attribute and space attribute of the street view picture information in the formed primary retrieval triple sorting are matched to realize accurate retrieval of the five tuple attribute.

[0052] Retrieving the relationship five tuple of the street view picture text information can add the information characteristics of the picture, enrich the feature description during the street view picture retrieval, and facilitate to improve the accuracy and efficiency of the street view picture retrieval

[0053] The model in S1 based on BERT and prior knowledge features uses the Transformer model to jointly train the binary cross-entropy loss function.

[0054] To sum up, in the embodiment, the picture retrieval optimization method based on street view retrieval according to the embodiment can capture the scale invariant features in the street view image, eliminate the factors that change with light and color, accurately capture the scale invariant features of the street view image, and improve the accuracy of the street view picture feature matching. Through the creation of the street view picture information relationship five tuple, the information description and feature record of the street view picture can be enriched, and the precision and speed of the content-based picture retrieval can be improved. The matching failure rate and the matching point failure rate are used to evaluate the content-based retrieval result of the street view picture, so that the retrieval accuracy and effect of the street view image retrieval technology can be observed. The retrieval of the relationship five tuple of the street view picture text information can add the information features of the picture, enrich the feature description during the street view picture retrieval, and improve the accuracy and efficiency of the street view picture retrieval

[0055] The above is only a further embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can make equivalent replacement or change according to the technical solution and concept of the present application within the scope disclosed by the present application, which belongs to the protection scope of the present application.

Claims

1. A method for optimizing picture retrieval based on street view retrieval, characterized in that, Comprise the following steps: S1. Image feature extraction: extract the texture features of the street view picture by using the gray level co-occurrence matrix, form the texture feature vector of the street view picture, accurately extract the local invariant features in the picture as the local feature vector by using the scale invariant algorithm, and create a relationship five tuple based on BERT and prior knowledge features for the street view picture; extract the relationship five tuple feature vector of the street view picture; S2. Street view picture preliminary retrieval: for the texture feature vector of the picture, measure the feature similarity distance to form a picture index order, and preliminarily retrieve the street view picture; S3. Street view picture accurate retrieval: combine the picture index order formed by the preliminary retrieval with the local feature vector of the street view picture to realize accurate retrieval of the street view picture, and retrieve the relationship five tuple of the street view picture to realize high-precision matching of the space-time information of the street view picture; The street view image local feature vector construction method in S1 comprises the following steps: A1. Street view picture spatial extreme value detection: use the convolution of the Gaussian function and the image function as the scale function of the scale invariant feature conversion to create a scale space, and find the local extreme points on the space; A2. Street view picture key point positioning: select the corner points and edge points that are not changed by rotation angle, illumination transformation and affine transformation as suspected key points, detect the key points by using Taylor series expansion, remove the points that do not meet the threshold, and remove the edge response by using hessian matrix to calculate the principal curvature; A3. Street view picture key point direction determination: select the field of the positioned key point, and determine the main direction and multiple auxiliary directions of the key point by using the 360-degree gradient histogram of the key point field; A4. Street view picture key point description: select the neighborhood of the key point, divide the formed window into multiple sub-blocks, form seed points for each sub-block, and form a feature point by combining the information of the four seed points; calculate the gradient histogram of eight directions of each sub-block to form a local feature vector; The street view picture relationship five tuple extraction method based on BERT and prior knowledge features in S1 is: B1. Create street view picture relationship five tuple: preprocess the extracted street view picture information text to obtain a text sequence at the word level, take each element of each text sequence as a token, input the text sequence into the BERT model for coding to obtain the semantic feature vector h of each token of the sequence, and use the AC automatic calculation method to retrieve the prior knowledge features pre of the street view picture in the prior knowledge base; label the prior knowledge features pre, add time and space features at the beginning and end of each token, fuse the retrieved matching prior knowledge features pre and the semantic feature vector to form a fusion feature vector h+pre; B2. Feature vector extraction: the formed fusion vector is input into two semi-pointer-semi-annotation structures with sigmoid activation function full connection layer, the start and end positions of the subject entity are predicted, the feature vectors of the start and end positions of the subject entity are averaged, and then added to the token sequence text to obtain the feature vector h+pre+s, then the obtained feature vector h+pre+s is connected with n semi-pointer-semi-annotation structures to predict the type of predicate and the start and end positions of object, and the feature vector of token sequence text is encoded into the feature vector of token sequence text to obtain the feature vector h+pre+s+p+o, then two semi-pointer-edge annotation structures are connected behind the feature vector h+pre+s+p+o to simultaneously predict the start and end positions of time and location, and the feature vector of token sequence text is encoded into the feature vector of token sequence text to obtain the feature vector of relationship five tuple.

2. The method of claim 1, wherein the method further comprises: The algorithm uses picture matching failure rate and matching point failure rate to evaluate the quality of local feature retrieval for local feature vector matching retrieval, wherein the picture matching failure rate is the ratio of the number of matching failure pictures to the total number of sample pictures, and the matching point failure rate is the ratio of the number of error matching point pairs in matching success to the number of matching point pairs, and the evaluation formula of picture matching failure rate and matching point pair failure rate is: ； In the formula: is a picture matching failure rate of the street view picture, is a number of all street view pictures participating in the search, is a number of pictures that failed to match. ； wherein: is the matching point pair failure rate for street view pictures, is the total number of matching point pairs for street view pictures participating in the search, is the number of matching point pairs that failed.

3. The method of claim 1, wherein the method further comprises: The retrieval method of the relationship five tuple of the street view picture in S3 includes the following steps: C1. Head entity, relationship type and tail entity triple retrieval: keyword retrieval and semantic retrieval are performed on the head entity, relationship type and tail entity formed by the street view picture information to form a primary retrieval triple ranking; C2. Time attribute and space attribute retrieval: the time attribute and space attribute of the street view picture information in the formed primary retrieval triple ranking are matched to realize accurate retrieval of the five tuple attribute.

4. The method of claim 1, wherein the method further comprises: The model based on BERT and prior knowledge features in S1 utilizes the Transformer model to jointly train the binary classification cross loss function.

Images