Image search method and device, computer device and storage medium

Abstract

This invention relates to the field of image search, and discloses an image search method, apparatus, computer device, and storage medium. The method includes: acquiring an image to be searched and extracting target features of a target object from the image; searching a first search result that matches the target features from an image database; the first search result includes several first images; the target similarity between the first images and the image to be searched is greater than a preset threshold; extracting multi-dimensional features from each first image, and forming a first feature set based on the target features and all multi-dimensional features; fitting the features of each dimension in the first feature set to obtain fitted features of the target object; and obtaining a second search result for the image to be searched based on the fitted features. This invention can improve the recall rate of the second search result.

Description

Technical Field

[0001] This invention relates to the field of image search, and more particularly to an image search method, apparatus, computer device, and storage medium. Background Technology

[0002] In existing technologies, when searching for an image, the target features of the target object in the image are typically extracted, and then these features are compared against a database to obtain the search results. Here, the target object can be a person or other detected object.

[0003] However, the target features extracted from the image to be searched are difficult to represent the characteristics of the target object, resulting in a low recall rate of search results. Summary of the Invention

[0004] Therefore, it is necessary to provide an image search method, apparatus, computer device, and storage medium to address the aforementioned technical problems and improve the recall rate of search results.

[0005] An image search method includes:

[0006] Acquire the image to be searched, and extract the target features of the target object from the image to be searched;

[0007] The image database is searched for a first search result that matches the target feature; the first search result includes a plurality of first images; the target similarity between the first image and the image to be searched is greater than a preset threshold.

[0008] Extract multidimensional features from each first image, and form a first feature set based on the target feature and all the multidimensional features;

[0009] Fit the features of each dimension in the first feature set to obtain the fitted features of the target object;

[0010] The second search result for the image to be searched is obtained based on the fitted features.

[0011] An image search device, comprising:

[0012] The first feature extraction module is used to acquire the image to be searched and extract the target features of the target object from the image to be searched.

[0013] A first search module is used to find a first search result that matches the target feature from an image database; the first search result includes a plurality of first images; the target similarity between the first image and the image to be searched is greater than a preset threshold.

[0014] The second feature extraction module is used to extract multi-dimensional features from each first image and form a first feature set based on the target feature and all the multi-dimensional features.

[0015] The feature fitting module is used to fit the features of each dimension in the first feature set to obtain the fitted features of the target object.

[0016] The second search module is used to obtain the second search result of the image to be searched based on the fitted features.

[0017] A computer device includes a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor implements the image search method described above when executing the computer-readable instructions.

[0018] One or more readable storage media storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the image search method described above.

[0019] The image search method, apparatus, computer equipment, and storage medium described above improve the recall rate of the second search result by performing feature fitting on the confidence portion (first image) of the first search result, transforming the search for a single feature value into a secondary search for a more reliable fitted feature value. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of an application environment for an image search method according to an embodiment of the present invention;

[0022] Figure 2 This is a flowchart illustrating an image search method according to an embodiment of the present invention;

[0023] Figure 3 This is a schematic diagram of the structure of an image search device according to an embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] The image search method provided in this embodiment can be applied to, for example, Figure 1 In this application environment, the client communicates with the server. Clients include, but are not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0027] In one embodiment, such as Figure 2 As shown, an image search method is provided, which is then applied to... Figure 1 Taking the server side as an example, the explanation includes the following steps S10-S50.

[0028] S10. Obtain the image to be searched, and extract the target features of the target object from the image to be searched.

[0029] Understandably, the image to be searched refers to an image containing the target object. The target object can be a person or an object. If the target object is a person, then the target feature can be a facial feature.

[0030] S20. Search the image database for a first search result that matches the target feature; the first search result includes several first images; the target similarity between the first image and the image to be searched is greater than a preset threshold.

[0031] Understandably, after obtaining the target features, the target similarity (such as cosine similarity) between the target features and each image in the image database can be calculated. Here, target similarity refers to the similarity between the target object in the search image and the comparison object in the image database. If the target similarity is greater than a preset threshold, then the image is selected as the first image. The preset threshold can be set according to actual needs. The preset threshold ensures that the object in the first image and the target object are the same object.

[0032] S30. Extract the multidimensional features of each first image, and form a first feature set based on the target feature and all the multidimensional features.

[0033] Understandably, multi-dimensional features of objects in each first image can be extracted and combined with target features of the target object in the image to be searched to form a first feature set. The target features and multi-dimensional features have the same dimension, which can be n. Taking a face as an example, the first feature set can be represented as F = {f1, f2, f3, ..., f...} p},f1,f2,f3,...,f p Let represent the individual facial feature values, and p be the number of images in the first image plus 1. Each facial feature value can be represented as:

[0034] f1 = [d 11 ,d 12 ,...,d 1n ];

[0035] f2=[d 21 ,d 22 ,...,d 2n ];

[0036] …

[0037] f p =[d p1 ,d p2 ,...,d pn ].

[0038] Where, d 11 ,d 12 ,...,d 1n This represents the values ​​of each dimension of the first multidimensional feature f1;

[0039] d 21 ,d 22 ,...,d 2n This represents the values ​​of each dimension of the first multidimensional feature f2;

[0040] d p1 ,d p2 ,...,d pn f represents the p-th multidimensional feature. p The values ​​for each dimension.

[0041] S40. Fit the features of each dimension in the first feature set to obtain the fitted features of the target object.

[0042] Understandably, features in each dimension of the first feature set can be fitted to obtain the fitted features of the target object. In one example, the fitted feature f f It can be represented as: f f =[d f1 ,d f2 ,...,d fn ], where d f1The fitting result is the fit applied to all features in the first dimension of the first feature set; d f2 The fitting result is the fit applied to all features in the second dimension of the first feature set; d fn This is the fitting result for fitting all features in the nth dimension of the first feature set.

[0043] S50. Obtain the second search result of the image to be searched based on the fitted features.

[0044] Understandably, after obtaining the fitted features, a secondary search can be performed on the image database using the fitted features to obtain a second search result.

[0045] This embodiment improves the recall rate of the second search result by performing feature fitting on the confidence portion (first image) of the first search result, transforming the search for a single feature value into a secondary search for a more reliable fitted feature value.

[0046] Optionally, step S40, namely fitting the features of each dimension in the first feature set to obtain the fitted features of the target object, includes:

[0047] S401. Obtain the dimensional feature set of the specified dimension from the first feature set.

[0048] Understandably, the specified dimension can be any dimension in the first feature set, and can be one of 1 to n dimensions. The feature set D of the specified dimension can be represented as D = {d1, d2, d3, ..., dn}. p}

[0049] When determining the weight of each element in D, for any dimension element, if the number of outliers is small and the outliers are far apart, their impact on the fitted value is sufficiently small. However, if a certain number of outliers are close together, this data cannot be ignored and needs to have some impact on the fitted value. For example, in dimension feature set 1: 0.3, 0.31, 0.32, ..., 0.39, 0.6, 0.8, the influence of 0.6 and 0.8 on the fitting result needs to be eliminated as much as possible. Similarly, in dimension feature set 2: 0.3, 0.31, 0.32, ..., 0.39, 0.8, 0.81, 0.82, a sufficient number of closely spaced outliers have formed in dimension feature set 2, and they need to have a significant impact on the fitted value.

[0050] Based on the above requirements, the following weight configuration rules can be set:

[0051] Rule 1: The weight of each element is affected by the weight of all other elements.

[0052] Rule 2: The closer other elements are to the target element, the greater their influence on the target element's weight.

[0053] Rule 3: The higher the weight of other elements, the greater their influence on the weight of the target element.

[0054] Rule 4: The higher the weight of an element, the greater the sum of its influence on the weights of other elements.

[0055] right Let its weight be W(d) k ), element weight matrix d k Let be any element in D.

[0056] Based on rule 1, Where f(d) i ,d k ) is d i For d k The impact of weights.

[0057] Based on rules 2 and 3,

[0058] Based on rule 4, d i The sum of the effects of the weights on all other elements = W(d) i ),so

[0059]

[0060] It needs to be considered that if the feature set of a dimension has two identical elements, such as d i =d k Then f(d) i ,d k )=W(d i ), d k They seized d i The full weight of the contribution. Therefore, a smoothing factor α is added, i.e. Where p is the number of elements in the dimensional feature set.

[0061] W(d k This can be represented as:

[0062]

[0063] S402. Determine the intermediate matrix based on the set of dimensional features.

[0064] Understandably, to simplify the formulas involved in step S401, an intermediate matrix R can be constructed based on the dimensional feature set. p×p The elements of the intermediate matrix can be obtained by processing the dimensional feature set using the intermediate matrix transformation formula.

[0065] The intermediate matrix transformation formulas include:

[0066]

[0067] Where, r ik This refers to the element in the i-th row and k-th column of the intermediate matrix;

[0068] d i It is the i-th element in the set of dimensional features;

[0069] d k It is the k-th element in the set of dimensional features;

[0070] d m It is the m-th element in the set of dimensional features;

[0071] D is the set of dimensional features.

[0072] S403. Obtain the smoothing factor, and determine the element weight matrix of the specified dimension based on the intermediate matrix and the smoothing factor.

[0073] Understandably, after obtaining the intermediate matrix, the element weight matrix W can be expressed as:

[0074]

[0075] Where α is the smoothing factor;

[0076] R is the intermediate matrix;

[0077] p is the number of elements in D;

[0078] E is a p×p dimensional matrix, and all its elements have the value 1.

[0079] Since the sum of the weights is 1, that is: ∑ W(dk)∈W W(dk) = 1.

[0080] Therefore, the element weight matrix can be obtained by processing the intermediate matrix and smoothing factor through the weight matrix transformation formula.

[0081] The formulas for transforming the weight matrix include:

[0082]

[0083] Wherein, W is the element weight matrix;

[0084] J is a p×p identity matrix;

[0085] α is the smoothing factor;

[0086] R is the intermediate matrix, which is a p×p matrix;

[0087] p is the number of elements in the dimensional feature set;

[0088] I is a column vector in which all elements are 1.

[0089] S404. Determine the dimensional fitting features of the specified dimension based on the dimensional feature set and the element weight matrix.

[0090] Understandably, after obtaining the element weight matrix, the dimensional fitting feature for a specified dimension can be calculated based on the dimensional feature set and the element weight matrix. The dimensional fitting feature for a specified dimension can be expressed as:

[0091]

[0092] Where d represents the dimension of the fitted feature;

[0093] d i Let i be the i-th element in the feature set of dimension; i can take values ​​from 1 to p.

[0094] p is the number of elements in the feature set of dimension;

[0095] W(d i ) is d i The weight.

[0096] After obtaining the fitting features in each dimension, they can be combined to form the fitting features.

[0097] In this embodiment, the dimension fitting features are calculated based on the weight configuration rules (rules 1 to 4), which can reduce the impact of outliers on the dimension fitting features.

[0098] Optionally, step S50, namely obtaining the second search result of the image to be searched based on the fitted features, includes:

[0099] S501. Normalize the fitted features to generate normalized fitted features;

[0100] S502. Search the image database for the second search result that matches the normalized fitting features.

[0101] Understandably, after obtaining the fitted features, these features can be normalized to generate normalized fitted features, which can be expressed as:

[0102]

[0103] in, The normalized fitted feature is the i-th dimension;

[0104] d i Fit the feature to the i-th dimension;

[0105] d m This represents the m-th element in the fitted feature f;

[0106] The value of m ranges from 1 to n;

[0107] n is the number of elements in the fitted feature f.

[0108] Then, a second search result matching the normalized fitted features is found in the image database.

[0109] This embodiment uses normalized fitted features to compare with the multidimensional features of images in the image database, which can greatly improve the efficiency of image search.

[0110] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0111] In one embodiment, an image search device is provided, which corresponds one-to-one with the image search methods described in the above embodiments. For example... Figure 3 As shown, the image search device includes a first feature extraction module 10, a first search module 20, a second feature extraction module 30, a feature fitting module 40, and a second search module 50. Detailed descriptions of each functional module are as follows:

[0112] The first feature extraction module 10 is used to acquire the image to be searched and extract the target features of the target object from the image to be searched.

[0113] The first search module 20 is used to search for a first search result that matches the target feature from an image database; the first search result includes a plurality of first images; the target similarity between the first image and the image to be searched is greater than a preset threshold.

[0114] The second feature extraction module 30 is used to extract multi-dimensional features of each first image and form a first feature set based on the target feature and all the multi-dimensional features.

[0115] The feature fitting module 40 is used to fit the features of each dimension in the first feature set to obtain the fitted features of the target object.

[0116] The second search module 50 is used to obtain the second search result of the image to be searched based on the fitted features.

[0117] Optionally, the feature fitting module 40 includes:

[0118] The dimension feature set acquisition unit is used to acquire a dimension feature set of a specified dimension from the first feature set;

[0119] Determine intermediate matrix units, used to determine intermediate matrices based on the set of dimensional features;

[0120] Determine the weight matrix unit to obtain the smoothing factor, and determine the element weight matrix of the specified dimension based on the intermediate matrix and the smoothing factor;

[0121] A dimension fitting feature unit is defined, which is used to determine the dimension fitting feature of the specified dimension based on the dimension feature set and the element weight matrix.

[0122] Optionally, determining intermediate matrix units is further used to process the dimensional feature set through an intermediate matrix transformation formula to obtain the elements of the intermediate matrix; the intermediate matrix transformation formula includes:

[0123]

[0124] Where, r ik This refers to the element in the i-th row and k-th column of the intermediate matrix;

[0125] d i It is the i-th element in the set of dimensional features;

[0126] d k It is the k-th element in the set of dimensional features;

[0127] d m It is the m-th element in the set of dimensional features;

[0128] D is the set of dimensional features.

[0129] Optionally, determining the weight matrix units is further configured to process the intermediate matrix and the smoothing factor using a weight matrix transformation formula to obtain the element weight matrix; the weight matrix transformation formula includes:

[0130]

[0131] Wherein, W is the element weight matrix;

[0132] J is a p×p identity matrix;

[0133] α is the smoothing factor;

[0134] R is the intermediate matrix, which is a p×p matrix;

[0135] p is the number of elements in the dimensional feature set;

[0136] I is a column vector in which all elements are 1.

[0137] Optionally, the second search module 50 includes:

[0138] A normalization unit is used to normalize the fitted features and generate normalized fitted features;

[0139] A secondary search unit is used to find a second search result that matches the normalized fitted features from the image database.

[0140] For specific limitations regarding the image search device, please refer to the limitations on the image search method above, which will not be repeated here. Each module in the aforementioned image search device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0141] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a readable storage medium and internal memory. The readable storage medium stores an operating system, computer-readable instructions, and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The database stores data related to the image search method. The network interface communicates with external terminals via a network connection. When the computer-readable instructions are executed by the processor, an image search method is implemented. The readable storage medium provided in this embodiment includes both non-volatile and volatile readable storage media.

[0142] In one embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor performs the following steps when executing the computer-readable instructions:

[0143] Acquire the image to be searched, and extract the target features of the target object from the image to be searched;

[0144] The image database is searched for a first search result that matches the target feature; the first search result includes a plurality of first images; the target similarity between the first image and the image to be searched is greater than a preset threshold.

[0145] Extract multidimensional features from each first image, and form a first feature set based on the target feature and all the multidimensional features;

[0146] Fit the features of each dimension in the first feature set to obtain the fitted features of the target object;

[0147] The second search result for the image to be searched is obtained based on the fitted features.

[0148] In one embodiment, one or more computer-readable storage media storing computer-readable instructions are provided. The readable storage media provided in this embodiment include non-volatile readable storage media and volatile readable storage media. The readable storage media stores computer-readable instructions, which, when executed by one or more processors, perform the following steps:

[0149] Acquire the image to be searched, and extract the target features of the target object from the image to be searched;

[0150] The image database is searched for a first search result that matches the target feature; the first search result includes a plurality of first images; the target similarity between the first image and the image to be searched is greater than a preset threshold.

[0151] Extract multidimensional features from each first image, and form a first feature set based on the target feature and all the multidimensional features;

[0152] Fit the features of each dimension in the first feature set to obtain the fitted features of the target object;

[0153] The second search result for the image to be searched is obtained based on the fitted features.

[0154] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When executed, these computer-readable instructions can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0155] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0156] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A picture search method characterized by, include: Acquire the image to be searched, and extract the target features of the target object from the image to be searched; The image database is searched for a first search result that matches the target feature; the first search result includes a plurality of first images; the target similarity between the first image and the image to be searched is greater than a preset threshold. Extract multidimensional features from each first image, and form a first feature set based on the target feature and all the multidimensional features; wherein the target feature and the multidimensional features have the same dimension. Fit the features of each dimension in the first feature set to obtain the fitted features of the target object; The second search result of the image to be searched is obtained based on the fitted features; The step of fitting the features of each dimension in the first feature set to obtain the fitted features of the target object includes: Obtain the dimensional feature set of the specified dimension from the first feature set; Determine the intermediate matrix based on the aforementioned dimensional feature set; Obtain the smoothing factor, and determine the element weight matrix of the specified dimension based on the intermediate matrix and the smoothing factor; The dimensional fitting features of the specified dimension are determined based on the dimensional feature set and the element weight matrix.

2. The picture search method of claim 1, wherein, The step of determining the intermediate matrix based on the dimensional feature set includes: The elements of the intermediate matrix are obtained by processing the dimensional feature set using an intermediate matrix transformation formula; the intermediate matrix transformation formula includes: wherein is the element of the intermediate matrix in the i-th row and k-th column; is the i-th element in the set of dimension features; is the kth element in the set of dimension features; It is the m-th element in the set of dimensional features; D is the set of dimensional features.

3. The image search method as described in claim 2, characterized in that, The step of obtaining the smoothing factor, and determining the element weight matrix of the specified dimension based on the intermediate matrix and the smoothing factor, includes: The element weight matrix is ​​obtained by processing the intermediate matrix and the smoothing factor using a weight matrix transformation formula; the weight matrix transformation formula includes: Wherein, W is the element weight matrix; J is a p×p identity matrix; α is the smoothing factor; R is the intermediate matrix, which is a p×p matrix; p is the number of elements in the dimensional feature set; I is a column vector in which all elements are 1.

4. The image search method as described in claim 1, characterized in that, The step of obtaining the second search result of the image to be searched based on the fitted features includes: The fitted features are normalized to generate normalized fitted features; The second search result that matches the normalized fitted features is retrieved from the image database.

5. An image search device, characterized in that, include: The first feature extraction module is used to acquire the image to be searched and extract the target features of the target object from the image to be searched. A first search module is used to find a first search result that matches the target feature from an image database; the first search result includes a plurality of first images; the target similarity between the first image and the image to be searched is greater than a preset threshold. The second feature extraction module is used to extract multidimensional features from each first image and form a first feature set based on the target feature and all the multidimensional features; wherein the target feature and the multidimensional features have the same dimension. The feature fitting module is used to fit the features of each dimension in the first feature set to obtain the fitted features of the target object. The second search module is used to obtain the second search result of the image to be searched based on the fitted features; The feature fitting module includes: The dimension feature set acquisition unit is used to acquire a dimension feature set of a specified dimension from the first feature set; Determine intermediate matrix units, used to determine intermediate matrices based on the set of dimensional features; Determine the weight matrix unit to obtain the smoothing factor, and determine the element weight matrix of the specified dimension based on the intermediate matrix and the smoothing factor; A dimension fitting feature unit is defined, which is used to determine the dimension fitting feature of the specified dimension based on the dimension feature set and the element weight matrix.

6. The image search device as described in claim 5, characterized in that, The second search module includes: A normalization unit is used to normalize the fitted features and generate normalized fitted features; A secondary search unit is used to find a second search result that matches the normalized fitted features from the image database.

7. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, characterized in that, When the processor executes the computer-readable instructions, it implements the image search method as described in any one of claims 1 to 4.

8. One or more readable storage media storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the image search method as described in any one of claims 1 to 4.

Images