Image-text retrieval method and system based on big data

Abstract

The invention discloses an image-text retrieval method and system based on big data, relates to the technical field of data processing, and solves the technical problems that the capability of dynamically adjusting screening conditions according to data characteristics and user requirements is lacked, the internal relation between images and texts cannot be fully mined, and the user requirements are difficult to accurately match. According to the method, multi-dimensional features such as colors, textures and shapes are extracted from image information, features are extracted from text information through a bag-of-word model, deep mining of image-text information is achieved, essential features of images and texts can be represented more comprehensively, the retrieval accuracy is improved, single matching, correlation calculation and screening and secondary matching are carried out, and the retrieval efficiency is improved. The text feature and image feature matching results are further combined and screened, finally, the combined retrieval results are matched and screened again, irrelevant information can be effectively eliminated through a multi-level screening mechanism, images and texts meeting requirements are accurately positioned from mass data, and the quality of the retrieval results is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a graphic and text retrieval method and system based on big data. Background Art

[0002] In today's digital age, the amount of graphic and text information on the Internet has exploded. Whether it is in search engines, e-commerce platforms, or various professional databases, there are massive amounts of graphic and text materials.

[0003] The patent application with publication number CN117972133A provides a method and system for image and text retrieval based on big data, which includes: obtaining an image to be annotated and a text to be annotated, and performing image area detection on the image to be annotated to obtain an image detection rule corresponding to the image to be annotated and performing text area detection on the text to be annotated to obtain a text detection rule corresponding to the text to be annotated; performing information fusion on the image detection rule and the text detection rule to obtain a bag-of-words model; obtaining a search image and a search text, and calculating a fusion result corresponding to the search image and the search text according to the bag-of-words model; preprocessing the search text according to the fusion result to obtain a processed text corresponding to the search text; performing keyword weight calculation on the processed text to obtain a text weight corresponding to the processed text; determining a first similarity value between the search text and the text to be annotated according to the text weight; and determining a search result corresponding to the search image and the search text according to the first similarity value.

[0004] When faced with massive amounts of data, existing search systems lack the ability to dynamically adjust screening conditions based on data characteristics and user needs, and are unable to fully explore the intrinsic connections between images and texts, making it difficult to quickly and accurately locate the content that best meets user needs among numerous search results. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a graphic and text retrieval method and system based on big data, which solves the problems of lack of ability to dynamically adjust screening conditions according to data characteristics and user needs, inability to fully explore the intrinsic connection between graphics and texts, and difficulty in accurately matching user needs.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a picture and text retrieval method based on big data, the method specifically comprises the following steps: Step 1: Obtain the image and text to be searched, and at the same time obtain the image and text information to be searched, and classify the image and text information to be searched to obtain image information and text information; Step 2: extract corresponding features from the obtained image information and text information to generate image features and text features; Step 3: Match the image features to obtain feature matching results, calculate the correlation between the image features and the image and text information, filter the image features according to the correlation to obtain filtering features, filter the feature matching results according to the filtering features to obtain secondary matching results, and comprehensively filter all image features to obtain image matching results; Step 4: Match the text features with the search database to obtain the text feature matching results, and combine the image matching results to perform comprehensive search and matching judgment to generate the image and text search results.

[0007] As a further solution of the present invention, the specific method of generating image features in step 2 is: The method of extracting color features is to divide the color space of the image into several bins (intervals), which are recorded as i, and i=1, 2, ..., j, where j represents the number of intervals. Then, the number of pixels in each interval i is counted and recorded as Li, and the corresponding image color distribution histogram is established according to the obtained number of pixels. At the same time, the first-order moments corresponding to different colors are calculated according to the obtained color distribution histogram, and the calculated first-order moments are used as the color features of the image. The method of extracting texture features is to compare each pixel in the image with its neighboring pixels, mark the pixel as 0 or 1 according to the comparison result, obtain a binary image, and then count the occurrence frequency of different binary patterns as the texture feature of the image; The method of extracting shape features is to extract the contour of the object in the image, calculate the perimeter, area, and eccentricity of the contour, so as to describe the shape of the object and generate shape features; The obtained color features, texture features and shape features are combined to obtain the corresponding image features.

[0008] As a further solution of the present invention, the specific method of obtaining the text features in step 2 is: Obtain text information and substitute it into the bag-of-words model for analysis to obtain text features.

[0009] As a further solution of the present invention, the specific method of obtaining the screening features in step 3 is: The image feature is obtained and recorded as i, where i=1, 2, ..., j, where j represents the number of image features. Then, the image feature i is matched with the retrieval library to obtain the feature matching result. At the same time, the correlation value between the image feature i and the graphic information is calculated, and the calculated correlation value is compared with the threshold. The image features corresponding to the correlation value greater than the threshold are screened and recorded as the screened features.

[0010] As a further solution of the present invention, the specific method of obtaining the image matching result in step 3 is: The obtained screening features are matched and screened with the feature matching results for a second time to obtain the secondary matching results. At the same time, all image features i are matched and screened as a whole with the secondary matching results. The secondary matching results corresponding to all image features i in the secondary matching results are obtained and recorded as image matching results.

[0011] As a further solution of the present invention, the specific method of generating the image and text search results in step 4 is: The obtained text features are matched with the retrieval library to obtain the text feature matching results, and are labeled as a, where a=1, 2, ..., b, where b represents the number of text feature matching results. At the same time, all image matching results are obtained and labeled as n, where n=1, 2, ..., m, where m represents the number of image matching results. Then, the similarity values ​​of the text feature matching result a and the image matching result n are calculated in sequence according to the label order and are recorded as X. an , and set the corresponding comparison value, and the similarity value X an Match with comparison value; Filter similarity value X an The image matching results corresponding to values ​​greater than the comparison value are recorded as the results to be analyzed, and it is determined whether there are image matching results with a similarity value of 1 in the results to be analyzed. If so, the corresponding image matching results are selected and the image and text retrieval results are generated at the same time. If not, a comprehensive analysis signal is generated and processed.

[0012] As a further solution of the present invention, the specific method of processing the comprehensive analysis signal in step 4 is: Combine text features with image features to obtain combined features, and search the combined features with the search library to obtain combined search results, and then match and screen the combined search results with the results to be analyzed; The combined search results are labeled as k, and k=1, 2, ..., o, where o represents the number of combined search results, and the results to be analyzed are labeled as y, and y=1, 2, ..., r, where r represents the number of results to be analyzed. At the same time, the similarity between the combined search results and the results to be analyzed is calculated in the order of the labels, and the combined search results and the results to be analyzed corresponding to the similarity greater than the comparison value are selected and recorded as the pre-selected image and text results h, and h=1, 2, ..., g, where g represents the number of pre-selected image and text results; Obtain all pre-selected image and text results h, and obtain the pre-selected image features and pre-selected text features corresponding to the pre-selected image and text results h, and match the obtained pre-selected image features and pre-selected text features with the image features and text features of the image and text to be retrieved, calculate the proportion of the overlapping features of the two, and then compare the calculated proportion value with the preset proportion value, and at the same time filter out the pre-selected image and text results corresponding to the proportion value greater than the preset proportion value, and use it as the retrieval standard to generate image and text retrieval results, and then sort them from large to small according to the proportion value.

[0013] The image and text retrieval system based on big data includes an image and text information acquisition module, an image and text feature extraction module, a feature matching analysis module, and a retrieval result output module; A graphic information acquisition module, which is used to transmit the acquired graphic information to be retrieved to the graphic feature extraction module; The image and text feature extraction module is used to classify the acquired image and text information to be retrieved to obtain image information and text information, and to extract corresponding features of the obtained image information and text information to generate image features and text features, and transmit the two to the feature matching analysis module at the same time; A feature matching analysis module, which matches image features to obtain feature matching results, calculates the correlation between image features and image-text information, screens image features according to the correlation to obtain screening features, and screens feature matching results according to the screening features to obtain secondary matching results, and comprehensively screens all image features to obtain image matching results, matches text features with the retrieval library to obtain text feature matching results, and performs comprehensive retrieval matching judgment in combination with the image matching results to generate image-text retrieval results, and transmits the image-text retrieval results to the retrieval result output module; The search result output module is used to display the acquired image and text search results to the corresponding search users.

[0014] Beneficial Effects The present invention provides a method and system for image and text retrieval based on big data. Compared with the prior art, it has the following beneficial effects: The present invention extracts multi-dimensional features such as color, texture, and shape from image information, and uses a bag-of-words model to extract features from text information, thereby achieving in-depth mining of image and text information, representing the essential features of images and texts more comprehensively, and improving the accuracy of retrieval. Single matching and correlation calculation and screening are performed, and then secondary matching is performed. The matching results of text features and image features are further combined and screened, and finally the combined retrieval results are matched and screened again. The multi-level screening mechanism can effectively exclude irrelevant information, accurately locate images and texts that meet the requirements from massive data, and greatly improve the quality of retrieval results. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a step method diagram of the present invention; Figure 2 This is a block diagram of the system principle of the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] For example, see Figure 1 , the present application provides a method for image and text retrieval based on big data, which specifically includes the following steps: Step 1: Obtain the image and text to be retrieved, and at the same time obtain the image and text information to be retrieved, and classify the image and text information to be retrieved to obtain image information and text information.

[0018] Step 2: Perform corresponding feature extraction on the obtained image information and text information to generate image features and text features.

[0019] S1. Extracting features of image information, and the specifically extracted features include color features, texture features and shape features; The method for extracting color features is to divide the color space of the image into several bins (intervals), which are recorded as i, and i=1, 2, ..., j, where j represents the number of intervals, and then count the number of pixels in each interval i and record it as Li, and establish the corresponding image color distribution histogram based on the obtained number of pixels. For example, for the RGB color space, each color channel can be divided into several levels, and then the number of pixels under each level combination is counted, and the first-order moment corresponding to different colors is calculated based on the obtained color distribution histogram, and the first-order moment here is expressed as the mean, and the calculated first-order moment is used as the color feature of the image; Assume that it is divided into j intervals, and the center value of each interval is c i (For example, when 0-255 is divided into 8 intervals, the center value of each interval can be obtained by calculation), the number of pixels in each interval is Li, then the first-order moment (mean) of the channel is The calculation formula is .

[0020] The method of extracting texture features is to compare each pixel in the image with its neighboring pixels, mark the pixel as 0 or 1 according to the comparison result, so as to obtain a binary image, and then count the occurrence frequency of different binary patterns as the texture feature of the image, or describe the texture feature of the image by counting the occurrence frequency of pixel pairs with specific gray values ​​in the image in different directions and distances; The way to extract shape features is to extract the contour of the object in the image and calculate the geometric features of the contour, such as the perimeter, area, eccentricity, etc., to describe the shape of the object and generate shape features. For example, the perimeter can reflect the boundary length of the object, the area can represent the size of the object, and the eccentricity can describe whether the shape of the object is close to a circle or an ellipse.

[0021] The obtained color features, texture features and shape features are combined to obtain corresponding image features; S2. Extract the text features of the image, specifically obtain the text information, and substitute the text information into the bag-of-words model for analysis to obtain the text features.

[0022] Step 3: Match the image features to obtain feature matching results, and calculate the correlation between the image features and the image and text information. According to the correlation, the image features are screened to obtain the screening features. At the same time, the feature matching results are screened according to the screening features to obtain the secondary matching results, and all image features are comprehensively screened to obtain the image matching results.

[0023] Obtain image features and record them as i, and i=1, 2, ..., j, where j represents the number of image features, then match the image feature i with the retrieval library to obtain a feature matching result, and here the content corresponding to the image feature in the retrieval library is obtained, and it is a single image feature match. For example, if feature i=1 represents a certain color distribution pattern in the image, then search for image data with a similar color distribution pattern in the retrieval library, and calculate the association value between the image feature i and the image and text information, and here the corresponding vectors are established and the association value is calculated by cosine similarity. For example, the color histogram feature of the image is converted into a vector A, and the corresponding color histogram feature of an image in the retrieval library is converted into a vector B, and their association values ​​are calculated by the cosine similarity formula, and the calculated association value is compared with the threshold, and the value of the threshold is determined by summing all the association values ​​and calculating the mean, for example, if the calculated threshold is 0.6, then the image features with an association value greater than 0.6 are regarded as screening features, and then the image features corresponding to the association value greater than the threshold are screened and recorded as screening features; The obtained screening features are matched and screened with the feature matching results to obtain the secondary matching results. At the same time, all image features i are matched and screened with the secondary matching results as a whole. Here, the whole means that all image features are matched with the secondary matching results as a whole. The secondary matching results corresponding to all image features i in the secondary matching results are obtained and recorded as image matching results. For example, after the first matching and correlation calculation screening, some images that initially meet the conditions are obtained. In the secondary matching, these images are further checked to see if they contain all the initially extracted image features. Only images containing all image features will be determined as the final image matching results.

[0024] Step 4: Match the text features with the search database to obtain the text feature matching results, and combine the image matching results to perform comprehensive search and matching judgment to generate the image and text search results.

[0025] S41, matching the obtained text features with the search library to obtain text feature matching results, and here the similarity between the text features and the search library is calculated for screening and matching, and the labels are denoted by a, and a=1, 2, ..., b, where b represents the number of text feature matching results, and all image matching results are obtained at the same time, and the labels are denoted by n, and n=1, 2, ..., m, where m represents the number of image matching results, and then the similarity values ​​of the text feature matching result a and the image matching result n are calculated in sequence according to the labeling order and denoted by X an , and the similarity value calculated here is the similarity value between the text feature matching result and the corresponding text information in the image matching result. For example, when a=1, the text feature matching result corresponding to a=1 is calculated with the image matching results corresponding to n=1, n=2, n=3, ..., respectively, and the obtained similarity value is recorded as X 11 , X 12 …, and set the corresponding comparison value at the same time, and the value range of the comparison value here is [0.8, 1], and the similarity value X an Match with comparison value; S42, screening similarity value X an The image matching result corresponding to the value greater than the comparison value is recorded as the result to be analyzed, and it is determined whether there is an image matching result with a similarity value of 1 in the result to be analyzed. If so, the corresponding image matching result is selected and the image and text retrieval result is generated at the same time. If not, a comprehensive analysis signal is generated and the comprehensive analysis signal is processed; S43, obtaining the generated comprehensive analysis signal, and combining the text feature with the image feature to obtain a combined feature, and searching the combined feature with the search library to obtain a combined search result, and then matching and screening the combined search result with the result to be analyzed, and the specific matching and screening method is as follows: The combined search results are labeled as k, and k=1, 2, ..., o, where o represents the number of combined search results, and the results to be analyzed are labeled as y, and y=1, 2, ..., r, where r represents the number of results to be analyzed. At the same time, the similarity between the combined search results and the results to be analyzed is calculated in the order of the labels, and the combined search results and the results to be analyzed corresponding to the similarity greater than the comparison value are selected, and the comparison value here is the same as the above comparison value. The specific similarity calculation method is the same as the similarity calculation method in the above S41, and is recorded as the pre-selected image and text result h, and h=1, 2, ..., g, where g represents the number of pre-selected image and text results; Obtain all pre-selected image and text results h, and obtain the pre-selected image features and pre-selected text features corresponding to the pre-selected image and text results h, and match the obtained pre-selected image features and pre-selected text features with the image features and text features of the image and text to be retrieved, calculate the proportion of the overlapping features of the two, and the overlapping features here include image features and text features, specifically the total number of value features, then compare the calculated proportion value with the proportion preset value, and the value of the proportion preset value is determined by the total features of the corresponding image and text to be retrieved, and the specific proportion preset value is two-thirds of the total feature value, and at the same time, filter the pre-selected image and text results corresponding to the proportion value greater than the proportion preset value, and use it as the retrieval standard to generate image and text retrieval results, and then sort them from large to small according to the proportion value.

[0026] For example, a search user is conducting a product search combining images and texts. The image and text to be searched is an introduction to a smart phone, which includes image and text features of appearance, performance, camera, screen, operating system, etc. The previous combined search results are screened to obtain some pre-selected image and text results. These results contain information of different mobile phones. By calculating the proportion of overlapping features, it is found that some of the pre-selected image and text results not only contain similar appearance and performance descriptions, but also highly overlap with the image and text to be searched in terms of camera and operating system, and the proportion of their overlapping features exceeds the set two-thirds. These results are sorted from high to low according to the degree of overlap, and the information that best meets user needs is placed in front, making it convenient for users to find the most similar product information.

[0027] For example 2, please refer to Figure 2 The present application provides a graphic and text retrieval system based on big data, including: a graphic and text information acquisition module, a graphic and text feature extraction module, a feature matching analysis module and a retrieval result output module, and the above functional modules are unidirectionally electrically connected.

[0028] A graphic information acquisition module, which is used to transmit the acquired graphic information to be retrieved to the graphic feature extraction module; A graphic feature extraction module, which is used to classify the acquired graphic information to be retrieved to obtain image information and text information, and perform corresponding feature extraction on the obtained image information and text information to generate image features and text features, and transmit the two to the feature matching analysis module at the same time, and the specific processing method is the same as the processing process of step 2 in embodiment 1; A feature matching analysis module, which matches image features to obtain feature matching results, calculates the correlation between image features and graphic information, screens image features according to the correlation to obtain screening features, and screens feature matching results according to the screening features to obtain secondary matching results, and comprehensively screens all image features to obtain image matching results, and the specific processing method is the same as the processing process of step three in embodiment one, matches text features with a search library to obtain text feature matching results, and performs comprehensive search and matching judgment in combination with the image matching results to generate graphic and text search results, and transmits the graphic and text search results to the search result output module, and the specific processing method is the same as the processing process of step four in embodiment one; The search result output module is used to display the acquired image and text search results to the corresponding search users.

[0029] Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0030] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A picture and text retrieval method based on big data, characterized in that: The method specifically comprises the following steps: Step 1: Obtain the image and text to be searched, and at the same time obtain the image and text information to be searched, and classify the image and text information to be searched to obtain image information and text information; Step 2: extract corresponding features from the obtained image information and text information to generate image features and text features; Step 3: Match the image features to obtain feature matching results, calculate the correlation between the image features and the image and text information, filter the image features according to the correlation to obtain filtering features, filter the feature matching results according to the filtering features to obtain secondary matching results, and comprehensively filter all image features to obtain image matching results; Step 4: Match the text features with the search database to obtain the text feature matching results, and combine the image matching results to perform comprehensive search and matching judgment to generate the image and text search results.

2. The image and text retrieval method based on big data according to claim 1 is characterized in that: The specific method of generating image features in step 2 is: The method for extracting color features is to divide the color space of the image into i intervals, where i=1, 2, ..., j, where j represents the number of intervals, and then count the number of pixels in each interval i as Li, and establish the corresponding image color distribution histogram based on the obtained number of pixels, and calculate the first-order moment corresponding to different colors based on the obtained color distribution histogram, and use the calculated first-order moment as the color feature of the image; The method of extracting texture features is to compare each pixel in the image with its neighboring pixels, mark the pixel as 0 or 1 according to the comparison result, obtain a binary image, and then count the occurrence frequency of different binary patterns as the texture feature of the image; The method of extracting shape features is to extract the contour of the object in the image, calculate the perimeter, area, and eccentricity of the contour, so as to describe the shape of the object and generate shape features; The obtained color features, texture features and shape features are combined to obtain the corresponding image features.

3. The image and text retrieval method based on big data according to claim 1 is characterized in that: The specific method of obtaining the text features in step 2 is: Obtain text information and substitute it into the bag-of-words model for analysis to obtain text features.

4. The image and text retrieval method based on big data according to claim 1 is characterized in that: The specific method of obtaining the screening features in step 3 is: The image feature is obtained and recorded as i, where i=1, 2, ..., j, where j represents the number of image features. Then, the image feature i is matched with the retrieval library to obtain the feature matching result. At the same time, the correlation value between the image feature i and the graphic information is calculated, and the calculated correlation value is compared with the threshold. The image features corresponding to the correlation value greater than the threshold are screened and recorded as the screened features.

5. The image and text retrieval method based on big data according to claim 1 is characterized in that: The specific method of obtaining the image matching result in step 3 is: The obtained screening features are matched and screened with the feature matching results for a second time to obtain the secondary matching results. At the same time, all image features i are matched and screened as a whole with the secondary matching results. The secondary matching results corresponding to all image features i in the secondary matching results are obtained and recorded as image matching results.

6. The image and text retrieval method based on big data according to claim 1, characterized in that: The specific method of generating the image and text search results in step 4 is: The obtained text features are matched with the retrieval library to obtain the text feature matching results, and are labeled as a, where a=1, 2, ..., b, where b represents the number of text feature matching results. At the same time, all image matching results are obtained and labeled as n, where n=1, 2, ..., m, where m represents the number of image matching results. Then, the similarity values ​​of the text feature matching result a and the image matching result n are calculated in sequence according to the label order and are recorded as X. an , and set the corresponding comparison value, and the similarity value X an Match with comparison value; Filter similarity value X an The image matching results corresponding to values ​​greater than the comparison value are recorded as the results to be analyzed, and it is determined whether there are image matching results with a similarity value of 1 in the results to be analyzed. If so, the corresponding image matching results are selected and the image and text retrieval results are generated at the same time. If not, a comprehensive analysis signal is generated and processed.

7. The image and text retrieval method based on big data according to claim 6 is characterized in that: The specific method of processing the comprehensive analysis signal in step 4 is: Combine text features with image features to obtain combined features, and search the combined features with the search library to obtain combined search results, and then match and screen the combined search results with the results to be analyzed; The combined search results are labeled as k, and k=1, 2, ..., o, where o represents the number of combined search results, and the results to be analyzed are labeled as y, and y=1, 2, ..., r, where r represents the number of results to be analyzed. At the same time, the similarity between the combined search results and the results to be analyzed is calculated in the order of the labels, and the combined search results and the results to be analyzed corresponding to the similarity greater than the comparison value are selected and recorded as the pre-selected image and text results h, and h=1, 2, ..., g, where g represents the number of pre-selected image and text results; Obtain all pre-selected image and text results h, and obtain the pre-selected image features and pre-selected text features corresponding to the pre-selected image and text results h, and match the obtained pre-selected image features and pre-selected text features with the image features and text features of the image and text to be retrieved, calculate the proportion of the overlapping features of the two, and then compare the calculated proportion value with the preset proportion value, and at the same time filter out the pre-selected image and text results corresponding to the proportion value greater than the preset proportion value, and use it as the retrieval standard to generate image and text retrieval results, and then sort them from large to small according to the proportion value.

8. A big data-based image and text retrieval system, used to execute the big data-based image and text retrieval method according to any one of claims 1 to 7, characterized in that: It includes a picture and text information acquisition module, a picture and text feature extraction module, a feature matching analysis module and a retrieval result output module; A graphic information acquisition module, which is used to transmit the acquired graphic information to be retrieved to the graphic feature extraction module; The image and text feature extraction module is used to classify the acquired image and text information to be retrieved to obtain image information and text information, and to extract corresponding features of the obtained image information and text information to generate image features and text features, and transmit the two to the feature matching analysis module at the same time; A feature matching analysis module, which matches image features to obtain feature matching results, calculates the correlation between image features and image-text information, screens image features according to the correlation to obtain screening features, and screens feature matching results according to the screening features to obtain secondary matching results, and comprehensively screens all image features to obtain image matching results, matches text features with the retrieval library to obtain text feature matching results, and performs comprehensive retrieval matching judgment in combination with the image matching results to generate image-text retrieval results, and transmits the image-text retrieval results to the retrieval result output module; The search result output module is used to display the acquired image and text search results to the corresponding search users.

Images