A topic-oriented semantic image retrieval method

Abstract

The present invention discloses a method for image retrieval oriented to topic semantics. First, the topic distribution hyperparameters and entity distribution hyperparameters are calculated respectively according to open source image samples, and a large sample entity detection model and a small sample entity detection template based on YOLO are constructed respectively for large sample entities and small sample entities. Then, image topic retrieval is generated for topic semantics and entity semantics, and then entity detection is performed on images in a database based on the large sample entity detection model and the small sample entity detection template to determine whether the topic semantics and entity semantics of the image meet the retrieval requirements, and finally a database image set that meets the retrieval requirements is output. The present invention is oriented to high-level semantic topic retrieval, and semantic detection and entity detection are both interrelated and independent of each other, which is conducive to continuously optimizing and improving entity retrieval technology according to topic semantic retrieval requirements, and has a high degree of flexibility.

Description

Technical Field

[0001] The present invention relates to the technical field of image retrieval, and in particular to an image retrieval method oriented to subject semantics. Background Art

[0002] Traditional image retrieval technology is content-based image retrieval technology (CBIR). The basic idea is to extract low-level visual features from images, such as color, texture, and shape, and then measure the similarity between the user's query image and the image in the database based on these features, and retrieve images that are similar to the user's query image in visual features. However, users need to retrieve images from the semantic level and thematic level, such as "man and nature" and "sports event", and the same image theme may not have similar low-level visual features. Therefore, traditional content-based image retrieval technology is not sufficient to support theme-oriented image retrieval. In contrast, the present invention is aimed at theme-oriented semantics image retrieval and innovatively proposes an image theme retrieval method based on image entity detection. Summary of the invention

[0003] Purpose of the invention: The technical problem to be solved by the present invention is to provide an image retrieval method oriented to subject semantics in view of the deficiencies in the prior art.

[0004] In order to solve the above technical problems, the present invention discloses a subject semantics-oriented image retrieval method, comprising:

[0005] Step 1: Calculate the topic distribution hyperparameters and entity distribution hyperparameters based on the open source image samples;

[0006] Step 2: construct a large sample entity detection model and a small sample entity detection template;

[0007] Step 3, generate image topic retrieval based on topic semantics and entity semantics;

[0008] Step 4: Perform entity detection to determine whether the subject semantics and entity semantics of the image meet the retrieval requirements;

[0009] Step 5: Repeat step 4 to obtain the image retrieval set Rel.

[0010] Furthermore, assuming that the subject of each image belongs to one of K subjects and the probability distribution of the subject in the K subject categories belongs to the Dirichlet distribution, , represents the topic distribution hyperparameter, K ≥ 1; further assuming that each topic k corresponds to an entity distribution and the entity distribution belongs to the Dirichlet distribution, , represents the entity distribution hyperparameter, where N represents the number of entity categories, 1 ≤ k ≤ K. In addition, it is assumed that a large number of open source image samples such as Baidu Encyclopedia can be obtained, and each image can be represented as an entity set after entity detection. Under the above assumptions, step 1 includes the following steps:

[0011] Step 1.1 Topic distribution hyperparameter calculation: Based on open source image samples such as Baidu Encyclopedia, build a set of image training samples for K topics, and record the distribution ratio of training samples on K topics as , ,……, , calculate the Dirichlet topic distribution according to formula (1) The topic distribution hyperparameters in :

[0012] (1);

[0013] Step 1.2 Calculation of entity distribution hyperparameters: According to the entity distribution in each image training sample on each topic k, sort the N entities from high to low according to the frequency of entity categories in the topic, denoted as , ,……, , assuming that Class is the number of large sample entity categories with sample size not less than Ns, Class is the number of small sample entity categories with sample size less than Ns. , Ns represents the sample quantity threshold, and the distribution ratio of N entities on each topic k is recorded as , ,……, , then the Dirichlet entity distribution corresponding to each topic k is calculated according to formula (2): The entity distribution hyperparameters :

[0014] (2);

[0015] Furthermore, the large sample entity detection model constructed in step 2 is based on the YOLO model (reference: Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection, CVPR2016) to construct a large sample entity detection model, and its input is For each image theme k, the entity categories are sorted from high to low according to their frequency of occurrence in the theme. Assuming that all samples have been labeled with positions and categories, select the samples with labeled positions and categories. High frequency entity ( ) samples to train the k-th YOLO model, , p represents the high-frequency entity index; if the same large sample entity category corresponds to multiple YOLO models, the YOLO model with the best average overlap (IOU, Intersection over Union) is selected to detect its category confidence and position. The optimal YOLO model is selected for each large sample entity category to build a large sample entity detection model, and then the large sample entity detection model is used to detect and obtain the training sample. Entity detection position coordinates (X coordinate, Y coordinate), detection width (W), detection height (H), and category confidence of large sample entity categories.

[0016] Furthermore, in step 2, constructing a small sample entity detection template includes: Small sample entity categories, based on M Hu invariant moments (reference: Research on image shape feature extraction based on Hu invariant moments, "Network Security Technology and Applications", 2017) to build a small sample entity detection template, that is, the number of rows is , Hu invariant moment matrix with M columns , where the matrix unit represents the average value of the mth Hu invariant moment of the small sample entity category q on the entity category samples, where , , ;

[0017] Furthermore, step 3 includes setting of image retrieval conditions for subject semantics and initialization of image retrieval, wherein the setting of image retrieval conditions for subject semantics sets database image retrieval for subject semantics from two dimensions, namely subject retrieval scope and entity retrieval scope, including:

[0018] Theme semantic setting: The subject scope of database retrieval is set according to the subject of the image to be retrieved. The subject scope is represented as a set T. ;

[0019] Entity semantic setting: Set the entity scope of database retrieval according to the entity of the image to be retrieved. The entity scope is represented as a set E. , where R represents the number of entities set, ;

[0020] Image retrieval initialization includes: initializing the image retrieval set Rel to an empty set, that is, , initialize the number of images to be retrieved Num to the number of images in the database, and initialize the index g of the image to be retrieved to 0, that is, g=0;

[0021] Furthermore, step 4 includes:

[0022] Step 4-1, judging the number of images to be retrieved: judging whether the number Num of images to be retrieved in the database is zero, if yes, skip to step 5, if no, continue to step 4-2;

[0023] Step 4-2, image index update: update the current image index to g=g+1, and update the number of images to be retrieved in the database Num=Num-1;

[0024] Step 4-3, detecting large sample entities;

[0025] Step 4-4, detecting small sample entities;

[0026] Step 4-5, construct entity detection matrix;

[0027] Step 4-6, determine the image theme and entity category to obtain the retrieval results.

[0028] Further, step 4-3 includes: detecting the large sample entity category based on the large sample entity detection model, and the detection result includes Entities belonging to large sample entity categories X, Y coordinates, width and height of the detection window, The highest confidence and corresponding category in the large sample entity category are expressed as: , , , , and , the highest confidence and the corresponding category satisfy:

[0029] (3)

[0030] (4)

[0031] in, Represents a large sample of entities The confidence on the u-th category, ;

[0032] Furthermore, step 4-4 includes: after the large sample entity is detected, the remaining part of the image is divided into several regions based on the GrabCut algorithm, each region corresponds to a candidate small sample entity , and its corresponding M Hu invariant moment features are recorded as , , according to formula (5) calculate the candidate small sample entity Confidence of belonging to the qth small sample entity category , :

[0033] (5)

[0034] Then we get the candidate small sample entity Affiliate The confidence of the small sample entity category. The small sample entity detection result includes the X and Y coordinates of the small sample entity, the detection window width W, the detection window height H, The highest confidence and corresponding category in the small sample entity category are expressed as: , , , , and , corresponding to category satisfy:

[0035] (6)

[0036] (7)

[0037] in, Represents a small sample entity The confidence score on the qth category.

[0038] Further, the step 4-5 includes: based on the large sample entity detection results and the small sample entity detection results, comprehensively generating The entity detection matrix DArray, Indicates that the category confidence in large sample entity detection and small sample entity detection is greater than the threshold The number of entities, that is

[0039] (8)

[0040] The 6 columns represent the corresponding entities In the image g, the X coordinate, Y coordinate, width W, height H, entity category cls and category confidence conf, if entity e belongs to the large sample entity, , corresponding to the large sample entity detection results in step 9, that is, respectively , , , , and If it is a small sample entity, it corresponds to the small sample entity detection result in step 10, that is, , , , , and ;

[0041] Further, the steps 4-6 include:

[0042] Step 4-6-1, image topic determination: determine whether the topic of image g meets the query requirements, and determine the topic of image g according to formula (9):

[0043] (9)

[0044] in, represents the probability that image g belongs to topic k, represents the probability that entity e belongs to topic k, and conf(e) represents the category confidence of entity e; if the topic of image g belongs to the retrieval topic range T, that is, , if yes, continue to step 4-6-2, otherwise jump to step 4-1;

[0045] Step 4-6-2, Entity Category Judgment: Judgment Entity Detection Matrix Whether the entity collection in contains the entity range E retrieved by the database, that is, , if yes, continue with step 4-6-3, otherwise jump to step 4-1;

[0046] Step 4-6-3, image output: add image g to the retrieval set, that is , skip to step 4-1.

[0047] Beneficial effects: (1) It overcomes the limitation of traditional content-based image retrieval technology that performs image retrieval based on the similarity of low-level visual features such as color, texture and shape while ignoring high-level semantics, and innovatively proposes a new technology for high-level semantic topic retrieval; (2) Semantic detection and entity detection are both interrelated and independent of each other, which is conducive to the continuous optimization and improvement of entity retrieval technology according to the needs of topic semantic retrieval and has high flexibility; (3) The Latent Dirichlet Allocation (LDA) model is conducive to the description of complex topic semantics and entity semantics. Image retrieval can refine the description granularity of image topic semantics. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and other advantages of the present invention will become more clear.

[0049] Figure 1 This is a process of a subject semantics-oriented image retrieval method provided by an embodiment of the present invention. Figure 1 .

[0050] Figure 2 This is a process of a subject semantics-oriented image retrieval method provided by an embodiment of the present invention. Figure 2 . DETAILED DESCRIPTION

[0051] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0052] The embodiment of the present invention discloses a topic semantics-oriented image retrieval method, which draws on the Latent Dirichlet Allocation (LDA) model, assuming that each image theme belongs to one of K themes and the probability distribution of the theme in the K theme categories belongs to the Dirichlet distribution; each theme corresponds to an entity distribution and the entity distribution belongs to the Dirichlet distribution; each image can be represented as an entity set after entity detection, such as Figure 1 As shown, the method includes:

[0053] Step 1: Calculate the topic distribution hyperparameters and entity distribution hyperparameters based on the open source image samples;

[0054] Step 2: For large sample entities and small sample entities, a large sample entity detection model and a small sample entity detection template based on the target detection network YOLO (You Only Look Once) are constructed respectively;

[0055] Step 3, generate image topic retrieval based on topic semantics and entity semantics;

[0056] Step 4: Perform entity detection on the images in the database based on the large sample entity detection model and the small sample entity detection template to determine whether the subject semantics and entity semantics of the image meet the retrieval requirements;

[0057] Step 5: Repeat step 4 to output a database image set that meets the retrieval requirements, namely, the image retrieval set Rel.

[0058] The embodiment of the present invention assumes that the subject of each image belongs to one of K subjects and the probability distribution of the subject in the K subject categories belongs to Dirichlet distribution. , represents the topic distribution hyperparameter, K ≥ 1. Further assume that each topic k corresponds to an entity distribution and the entity distribution belongs to the Dirichlet distribution, , represents the entity distribution hyperparameter, where N represents the number of entity categories, 1 ≤ k ≤ K. In addition, it is assumed that a large number of open source image samples such as Baidu Encyclopedia can be obtained, and each image can be represented as an entity set after entity detection. Figure 2 As shown, under the above assumptions, step 1 specifically includes the following steps:

[0059] Step 1.1, topic distribution hyperparameter calculation: Based on open source image samples such as Baidu Encyclopedia, build a set of image training samples for K topics, and record the distribution ratio of training samples on K topics as , ,……, , calculate the Dirichlet topic distribution according to formula (1) The topic distribution hyperparameters in :

[0060] (1);

[0061] Step 1.2, entity distribution hyperparameter calculation: According to the entity distribution in each image training sample on each topic k, the N types of entities are sorted from high to low according to the frequency of entity categories in the topic, recorded as , ,……, , assuming that The class is the number of large sample entity categories with a sample size of no less than 200. The class is the number of small sample entity categories with a sample size of less than 200. , the distribution ratio of N entities on each topic k is recorded as , ,……, , then the Dirichlet entity distribution corresponding to each topic k is calculated according to formula (2): The entity distribution hyperparameters :

[0062] (2);

[0063] Divide N entities into A large sample of entities and A small sample entity, After the entity positions and categories are marked, the samples belonging to the large sample entities can be used to train the YOLO model in step 2. A sufficient number of samples can ensure the accuracy of the YOLO model. The samples to which the small sample entities belong can be used to construct a small sample entity detection template in step 2, which has low requirements on the number of samples.

[0064] In step 2, we build a large-sample entity detection model based on YOLO: We build a large-sample entity detection model based on the YOLO model (reference: Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection, CVPR2016). The input is training samples of large sample entity categories, for each image theme k, sort the entities from high to low according to their frequency of occurrence in the theme, assuming that all samples have been labeled with positions and categories, and select the samples with labeled positions and categories. High frequency entity ( ) samples to train the k-th YOLO model, If the same large sample entity category corresponds to multiple YOLO models, the YOLO model with the best average overlap (IOU, Intersection over Union) is selected to detect its category confidence and position. The optimal YOLO model is selected for each large sample entity category to build a large sample entity detection model, and then the large sample entity detection model is used to detect and obtain the training sample. Entity detection position coordinates (X coordinate, Y coordinate), detection width (W), detection height (H), and category confidence of large sample entity categories;

[0065] Step 2: Construction of small sample entity detection template: Small sample entity categories, based on M=7 Hu invariant moments (reference: Research on image shape feature extraction based on Hu invariant moments, "Network Security Technology and Applications", 2017) to build a small sample entity detection template, that is, the number of rows is , Hu invariant moment matrix with M columns , where the matrix unit represents the average value of the mth Hu invariant moment of the small sample entity category q on the entity category samples, where , ;

[0066] Hu invariant moment is a highly generalized image feature that is invariant to translation, grayscale, scale, and rotation in continuous images and is suitable for describing the shape of entities.

[0067] Step 3 includes setting of image retrieval conditions oriented to subject semantics and initialization of image retrieval. The setting of image retrieval conditions oriented to subject semantics sets database image retrieval oriented to subject semantics from two dimensions, namely subject retrieval scope and entity retrieval scope, including:

[0068] Theme semantic setting: The subject scope of database retrieval is set according to the subject of the image to be retrieved. The subject scope is represented as a set T. ;

[0069] Entity semantic setting: Set the entity scope of database retrieval according to the entity of the image to be retrieved. The entity scope is represented as a set E. , where R represents the number of entities set, ;

[0070] Image retrieval initialization: Initialize the image retrieval set to an empty set, that is, , initialize the number of images to be retrieved Num to the number of images in the database, and initialize the index g of the image to be retrieved to 0, that is, g=0;

[0071] Step 4 includes:

[0072] Step 4-1, judging the number of images to be retrieved: judging whether the number Num of images to be retrieved in the database is zero, if yes, skip to step 5, if no, continue to step 4-2;

[0073] Step 4-2, image index update: update the current image index to g=g+1, and update the number of images to be retrieved in the database Num=Num-1;

[0074] Step 4-3, detecting large sample entities;

[0075] Step 4-4, detecting small sample entities;

[0076] Step 4-5, construct entity detection matrix;

[0077] Step 4-6, determine the image theme and entity category to obtain the retrieval results.

[0078] Step 4-3 includes: detecting the large sample entity category based on the large sample entity detection model, and the detection results include Entities belonging to large sample entity categories X, Y coordinates, width and height of the detection window, The highest confidence and corresponding category in the large sample entity category are expressed as: , , , , and , the highest confidence and the corresponding category satisfy:

[0079] (3)

[0080] (4)

[0081] in, Representing large sample entities The confidence on the u-th category, ;

[0082] Step 4-4 includes: after the large sample entity is detected, the remaining part of the image is divided into several regions based on the image segmentation algorithm GrabCut algorithm, each region corresponds to a candidate small sample entity , and its corresponding M Hu invariant moment features are recorded as , , according to formula (5) calculate the candidate small sample entity Confidence of belonging to the qth small sample entity category , :

[0083] (5)

[0084] Then we get the candidate small sample entity Affiliate The confidence of the small sample entity category. The small sample entity detection result includes the X and Y coordinates of the small sample entity, the detection window width W, the detection window height H, The highest confidence and corresponding category in the small sample entity category are expressed as: , , , , and , corresponding to category satisfy:

[0085] (6)

[0086] (7)

[0087] in, Represents a small sample entity Confidence in the qth category;

[0088] Step 4-5 includes: based on the large sample entity detection results and the small sample entity detection results, comprehensively generate The entity detection matrix DArray, Indicates that the category confidence in large sample entity detection and small sample entity detection is greater than the threshold The number of entities, that is

[0089] (8)

[0090] The 6 columns represent the corresponding entities In the image g, the X coordinate, Y coordinate, width W, height H, entity category cls and category confidence conf, if entity e belongs to the large sample entity, , corresponding to the large sample entity detection results in step 9, that is, respectively , , , , and If it is a small sample entity, it corresponds to the small sample entity detection result in step 10, that is, , , , , and ;

[0091] Steps 4-6 include:

[0092] Step 4-6-1, image topic determination: determine whether the topic of image g meets the query requirements, and determine the topic of image g according to formula (9):

[0093] (9)

[0094] The topic is judged by calculating the expected probability value of all entities appearing in the current image, where: represents the probability that image g belongs to topic k, represents the probability that entity e belongs to topic k, and conf(e) represents the category confidence of entity e; if the topic of image g belongs to the retrieval topic range T, that is, , if yes, continue to step 4-6-2, otherwise jump to step 4-1;

[0095] Step 4-6-2, Entity Category Judgment: Judgment Entity Detection Matrix Whether the entity collection in contains the entity range E retrieved by the database, that is, , if yes, continue with step 4-6-3, otherwise jump to step 4-1;

[0096] Step 4-6-3, image output: add image g to the retrieval set, that is , skip to step 4-1.

[0097] The method of the present invention breaks through the limitation of traditional content-based image retrieval technology that performs image retrieval through the similarity of low-level visual features such as color, texture and shape while ignoring high-level semantics, and innovatively proposes a new technology for high-level semantic topic retrieval; the semantic detection and entity detection in the present invention are both interrelated and independent of each other, which is conducive to the continuous optimization and improvement of entity retrieval technology according to the needs of topic semantic retrieval, and has a high degree of flexibility; the Latent Dirichlet Allocation (LDA) model is conducive to the description of complex topic semantics and entity semantics. Image retrieval, refinement of the description granularity of image topic semantics.

[0098] Example

[0099] Step 1.1, topic distribution hyperparameter calculation

[0100] Assume that the open source image samples are used to construct a set of image training samples with two themes, namely, nature and humanities. The distribution ratio of the training samples on the two themes is =1 / 4, =3 / 4, then the two topic distribution hyperparameters are calculated as follows:

[0101] Calculate the Dirichlet topic distribution according to formula (1) The hyperparameters in :

[0102]

[0103] It can be concluded that =1, =3.

[0104] Step 1.2, Entity Distribution Hyperparameter Calculation

[0105] Assuming that the entity categories with the highest frequency in the two themes of nature and humanities constructed by the open source image samples are "mountains", "rivers", "lakes", "elderly people" and "children" in descending order of frequency, among which "mountains", "rivers", "elderly people" and "children" are large sample entities with a sample size of no less than 200, "lakes" are small sample entities with a sample size of less than 200, and the distribution ratios of the five entities in the theme of "nature" are = 1 / 3, =1 / 3, =1 / 3, =0 and =0, the distribution ratios of the five entities in the “Humanities” theme are =0, =0, =0, =3 / 5 and =2 / 5, then the entity distribution hyperparameters of the two topics are calculated as follows:

[0106] According to the entity distribution in each image training sample on the two topics, the N=5 entities with the highest frequency on the two topics, "mountains", "rivers", "lakes", "elderly people" and "children", are obtained, among which the number of large sample entities =4, number of small sample entities =1, then according to formula (2), the Dirichlet entity distribution corresponding to the natural topic (k=1) and the humanities topic (k=2) is calculated, and the result is: , .

[0107] According to step 2, a large sample entity detection model is constructed: based on =4 large sample entity training samples, build a YOLO model for each of the two image themes, select the YOLO model with the best average overlap (IOU, Intersection over Union) for the same large sample entity category to detect its category confidence and position, the YOLO models corresponding to theme 1 and theme 2 are denoted as Y1 and Y2 respectively, and the entity detection category sets corresponding to YOLO models Y1 and Y2 are { , }and{ , };

[0108] According to the small sample entity detection template constructed in step 2: based on = 1 training sample of a small sample entity, for the small sample entity of topic 1 Construct a small sample entity detection template, that is, the number of rows is =1, Hu invariant moment matrix with M=7 columns ;

[0109] According to step 3, the generated topic range is T={topic 1}, and the entity range is E={ }, the retrieval results for the current image g are as follows:

[0110] Step 4-3, large sample entity detection, detects large sample entity categories based on the large sample entity detection model. The detection results include =Entities belonging to 4 large sample entity categories { , , , The X and Y coordinates of the detection window, the width W and height H of the detection window, the highest confidence in the large sample entity category and the corresponding category are shown in Table 1: (unit: pixel)

[0111] Table 1 Large sample entity detection results

[0112]

[0113] Step 4-4, small sample entity detection: After the large sample entity detection, the remaining part of the image is divided into several regions based on the GrabCut algorithm. Each region corresponds to a candidate small sample entity. The 7 Hu invariant moment features of each region are calculated. The entity category of the candidate small sample entity is calculated according to formula (5). The confidence level is shown in Table 2:

[0114] Table 2 Small sample entity detection results

[0115]

[0116] Step 4-5: Entity detection matrix construction, based on large sample entity detection results and small sample entity detection results, comprehensive generation The entity detection matrix DArray, Indicates that the category confidence in large sample entity detection and small sample entity detection is greater than the threshold The number of entities, ES={ , , }, generate the entity detection matrix DArray, as shown in Table 3:

[0117] Table 3 Entity detection matrix DArray

[0118]

[0119] Step 4-6-1: Image theme determination: According to formula (9), determine the theme of the current image.

[0120]

[0121] in, represents the expected probability that image g belongs to topic k, represents the expected probability value of entity e belonging to topic k, conf(e) represents the category confidence of entity e; it is judged that the current image topic belongs to the retrieval topic range T, that is, , continue with step 4-6-2;

[0122] Step 4-6-2, Entity Category Judgment: Determine whether the entity set Rows(DArray) in the entity detection matrix DArray contains the entity range E={ }, continue with step 4-6-3;

[0123] Step 4-6-3, image output, add the current image to the search set , skip to step 4-1.

[0124] When performing thematic retrieval of image resources in different professional fields such as scientific and technological intelligence, medical treatment, and health, the method of the present invention can be used to mine relevant image themes and materials in various professional fields from the distribution relationship of themes and entity associations, thereby assisting professionals to efficiently obtain relevant information related to the image themes and make more accurate decisions. The method of the present invention overcomes the limitations of traditional content-based image retrieval technology that performs image retrieval through the similarity of low-level visual features such as color, texture, and shape while ignoring high-level semantics, improves the accuracy and efficiency of thematic retrieval, and can continuously optimize and improve entity retrieval technology according to the needs of thematic semantic retrieval, with a high degree of flexibility.

[0125] The present invention provides a method for image retrieval oriented to subject semantics. There are many methods and approaches to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A topic semantics-oriented image retrieval method, characterized in that: include: Step 1: Calculate the topic distribution hyperparameters and entity distribution hyperparameters based on the open source image samples; Step 2: Build a large sample entity detection model and a small sample entity detection template; Step 3, generate image topic retrieval based on topic semantics and entity semantics; Step 4: Perform entity detection to determine whether the subject semantics and entity semantics of the image meet the retrieval requirements; Step 5, repeat step 4 to obtain the image retrieval set Rel; The step 4 comprises: step 4-1, determining whether the number Num of images to be retrieved in the database is zero, if yes, jumping to step 5, if no, continuing to step 4-2; Step 4-2, update the current image index to g=g+1, and update the number of images to be retrieved in the database Num=Num-1; Step 4-3, detecting large sample entities; Step 4-4, detecting small sample entities; Step 4-5, construct entity detection matrix; Step 4-6, determine the image theme and entity category to obtain the retrieval results; The step 4-3 includes: detecting the large sample entity category based on the large sample entity detection model, and the detection result includes Entity to which the large sample entity category belongs X, Y coordinates, width and height of the detection window, The highest confidence in the entity category of a large sample And corresponding categories ,in: ， ， Represents a large sample of entities The confidence on the u-th category, ; The step 4-4 includes: dividing the remaining part of the image into several regions based on the GrabCut algorithm, each region corresponding to a candidate small sample entity , and its corresponding M Hu invariant moment features are recorded as , calculate candidate small sample entities Confidence of belonging to the qth small sample entity category : ， Then we get the candidate small sample entity Affiliate The confidence of the small sample entity category. The small sample entity detection result includes the X and Y coordinates of the small sample entity, the detection window width W, the detection window height H, The highest confidence in the small sample entity category And corresponding categories ,in: ， ， Represents a small sample entity The confidence score on the qth category.

2. The image retrieval method based on subject semantics according to claim 1, characterized in that: Assume that the subject of each image belongs to one of K subjects and the probability distribution of the subject in the K subject categories belongs to the Dirichlet distribution , represents the topic distribution hyperparameter, K ≥ 1; assuming that each topic k corresponds to an entity distribution and the entity distribution belongs to the Dirichlet distribution , represents entity distribution hyperparameters, where N represents the number of entity categories, 1 ≤ k ≤ K; assuming that a large number of open source image samples can be obtained, and each image can be represented as an entity set after entity detection; the step 1 includes the following steps: Step 1.1: Build a set of image training samples with K topics based on open source image samples, obtain the distribution ratio of training samples on K topics, and calculate the Dirichlet topic distribution. The topic distribution hyperparameters in ; Step 1.2: According to the entity distribution in each image training sample on each topic k, sort the N entities from high to low according to the frequency of occurrence of entity categories in the topic, where Class is the number of large sample entity categories with sample size not less than Ns, Class is the number of small sample entity categories with sample size less than Ns. , Ns represents the sample quantity threshold, obtains the distribution ratio of N types of entities on each topic k, and calculates the Dirichlet entity distribution corresponding to each topic k The entity distribution hyperparameters .

3. The method for image retrieval based on subject semantics according to claim 2, characterized in that: The large sample entity detection model constructed in step 2 is based on the YOLO model, and its input is training samples of large sample entity categories, for each image theme k, sort the entity categories from high to low according to their frequency of occurrence in the theme, assuming that all samples have been labeled with positions and categories, and select the first High frequency entity The kth YOLO model is trained with samples of , p represents the index of the high-frequency entity category, If the same large sample entity category is used for the training of more than two YOLO models, the YOLO model with the best average overlap IOU is selected to detect its category confidence and position. The best YOLO model is selected for each large sample entity category to build a large sample entity detection model, and then the training sample is obtained based on the large sample entity detection model. The corresponding entity detection position coordinates, detection width, detection height and category confidence of the large sample entity.

4. The method for image retrieval based on subject semantics according to claim 3, characterized in that: The step 2 of constructing a small sample entity detection template includes: Small sample entity categories, based on M Hu invariant moments to build a small sample entity detection template, that is, the number of rows is , Hu invariant moment matrix with M columns , where the matrix unit represents the average value of the mth Hu invariant moment of the small sample entity category q on the entity category samples, where , , .

5. The method for image retrieval based on subject semantics according to claim 4, characterized in that: Step 3 includes setting image retrieval conditions and initializing image retrieval based on subject semantics. The image retrieval condition setting oriented to the subject semantics includes: Theme semantic setting: The subject scope of database retrieval is set according to the subject of the image to be retrieved. The subject scope is represented as a set T. ; Entity semantic setting: Set the entity scope of database retrieval according to the entity of the image to be retrieved. The entity scope is represented as a set E. , where R represents the number of entities set, ; The image retrieval initialization includes: initializing the image retrieval set Rel to an empty set, initializing the number of images to be retrieved Num to the number of images in the database, and initializing the index of the image to be retrieved g to 0.

6. The method for image retrieval based on subject semantics according to claim 1, characterized in that: The steps 4-5 include: comprehensively generating based on the large sample entity detection results and the small sample entity detection results The entity detection matrix DArray, Indicates that the category confidence in large sample entity detection and small sample entity detection is greater than the threshold The 6 columns represent the corresponding entities. The X coordinate, Y coordinate, width W, height H, entity category cls and category confidence conf in the image g.

7. The method for image retrieval based on subject semantics according to claim 1, characterized in that: The steps 4-6 include: Step 4-6-1, determine the topic to which the image g belongs. If the topic of the image g belongs to the search topic range T, continue with step 4-6-2, otherwise jump to step 4-1; ， in, represents the probability that image g belongs to topic k, represents the probability that entity e belongs to topic k, and conf(e) represents the category confidence of entity e; Step 4-6-2, Entity Category Judgment: Judgment Entity Detection Matrix Whether the entity collection in contains the entity range E retrieved from the database, if yes, continue with step 4-6-3, otherwise jump to step 4-1; Step 4-6-3, image output: add image g to the retrieval set, that is , skip to step 4-1.

Images