Image retrieval apparatus

Abstract

An image retrieval apparatus according to the technology of the present disclosure is an image retrieval apparatus including a feature reduction processing circuit (100) and a retrieval processing unit (200). A matrix calculator (120) included in a feature reduction processing circuit (100) calculates a matrix (Cd) from stored data (Xd-1). A projection conversion unit (130) included in the feature reduction processing circuit (100) multiplies the matrix (Cd) by the sequentially acquired image feature amount {fi}. A second projection conversion unit (210) included in a search processing unit (200) calculates a low-dimensional image feature (gtargetx) from an input image feature (ftarget) of a search target using a matrix (Cd). A search execution unit (220) included in a search processing unit (200) calculates a distance between a vector calculated by a projection conversion unit (130) and a low-dimensional image feature (gtargetx) in a feature space, and performs a search for a possible image feature by using the distance. The vector calculated by the projection conversion unit (130) belongs to a database divided by time windows.

Description

Technical Field

[0001] The present disclosure relates to an image retrieval device. Background Art

[0002] In the field of image search technology, in order to achieve high-speed search, it is known that the dimension of image feature quantity can be reduced using a matrix obtained by statistical analysis. Here, the statistical analysis refers to, for example, principal component analysis and discriminant analysis. In addition, here, image feature quantity is a concept used in the field of machine learning technology, and is a vector in a feature space generated from an image. Image features, as the name implies, represent the characteristics of an image.

[0003] For example, Patent Document 1 discloses a technique for transforming image features by a first linear transformation parameter to facilitate a classification problem, wherein the first linear transformation parameter is a matrix obtained based on a statistical analysis result.

[0004] Reference List

[0005] Patent Literature

[0006] Patent Document 1: WO 2014 / 167880 A1 Summary of the invention

[0007] Technical issues

[0008] The image retrieval device shown in Patent Document 1 can be used as a system for searching for lost children or suspicious persons from video images captured by surveillance cameras at facilities such as airports. In this specification, a system for searching for lost persons or suspicious persons from video images captured by surveillance cameras at facilities such as airports is referred to as a "query system".

[0009] The inventors of the disclosed technology discovered that when the image retrieval device illustrated in Patent Document 1 is applied to a query system, although the images are taken by the same surveillance camera, the matrix suitable for dimensionality reduction gradually changes within a time window of one day.

[0010] One purpose of the disclosed technology is to provide an image retrieval device to solve the problem that a matrix suitable for dimensionality reduction will gradually change.

[0011] Technical means of solving problems

[0012] An image retrieval device according to the disclosed technology is an image retrieval device including a feature reduction processing circuit and a retrieval processing unit.

[0013] The matrix calculator included in the feature reduction processing circuit is based on the stored data (X d-1 )Calculate the matrix (Cd ).

[0014] The projection transformation unit included in the feature reduction processing circuit transforms the matrix (C d ) multiplied by the image feature quantities {f i}.

[0015] The second projection transformation unit included in the retrieval processing unit uses the matrix (C d ) from the input image feature quantity (f target ) Calculate low-dimensional image features (g target_x ).

[0016] The search execution unit included in the retrieval processing unit calculates the vector calculated by the projection transformation unit and the low-dimensional image feature (g target_x ) and perform a search for possible image features by using the distance. The vectors calculated by the projective transformation unit belong to a database divided by time windows.

[0017] Advantageous Effects of the Invention

[0018] Since the image retrieval device according to the disclosed technology has the above-described configuration, it is possible to solve the problem that a matrix suitable for dimensionality reduction gradually changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a block diagram showing the functional configuration of the feature reduction processing circuit 100 according to the disclosed technology.

[0020] Figure 2 1 is a timing chart showing the processing procedure performed in time series by the feature reduction processing circuit 100 according to the disclosed technology.

[0021] Figure 3 is a block diagram showing the functional configuration of the retrieval processing unit 200 . DETAILED DESCRIPTION

[0022] As described above, the image feature quantity, which is the premise of the disclosed technology, is a concept used in the field of machine learning technology, and is a vector quantity in a feature space generated for an image. The disclosed technology is not limited to which learning model the image feature quantity is generated by, nor is it limited to the way the image feature quantity is generated. However, it will be easier to understand if the image feature is understood as an intermediate product of a learning model such as a convolutional neural network (CNN). CNN is a type of artificial neural network, and is considered to be an effective means, particularly in the field of image analysis technology.

[0023] Implementation Method 1

[0024] The image retrieval device according to the disclosed technology includes a feature reduction processing circuit 100 and a retrieval processing unit 200 .

[0025] The feature reduction processing circuit 100 is a processing circuit that acquires image feature quantities and performs feature reduction processing such as dimensionality reduction.

[0026] The retrieval processing unit 200 is a processing circuit that performs possible image feature retrieval from a database divided by time span (hereinafter referred to as "time window") when the image feature to be retrieved is input.

[0027] Figure 1 1 is a block diagram showing the functional configuration of the feature reduction processing circuit 100 according to the disclosed technology. Figure 1 As shown, the feature reduction processing circuit 100 includes a data storage unit 110 , a matrix calculator 120 and a projection transformation unit 130 .

[0028] Figure 2 is a time chart showing the processing flow of the feature reduction processing circuit 100 according to the disclosed technology in a time series manner. Figure 2 , the portion marked as ST110 indicates the time point at which the data storage unit 110 performs processing. Likewise, the portion marked as ST120 indicates the time point at which the matrix calculator 120 performs processing. Also, similarly, the portion marked as ST130 indicates the time point at which the projection transformation unit 130 performs processing.

[0029] (Data Storage Unit 110 Included in Feature Reduction Processing Circuit 100)

[0030] The data storage unit 110 included in the feature reduction processing circuit 100 is a component that stores the sequentially acquired image features as data. Assume that the sequentially acquired image feature quantities are represented by {f i}, where i = 1, 2, ..., m. The data to be stored (X) is given by the following formula:

[0031]

[0032] As shown in formula 1, the image feature quantities {f i} is a real vector of size 1×n. In this specification, the image feature quantity {f i} is defined as a row vector. As shown in Formula 1, image features are usually real vectors with real numbers as components, but the definition can also be extended to complex vectors with complex numbers as components.

[0033] When the image feature quantities {f i} is m, then the data (X) stored in the data storage unit 110 is a matrix of size m×n. As shown in Formula 1, the data (X) is usually a real matrix. However, when the image feature quantity {f i} is extended to a complex vector, data (X) becomes a complex matrix.

[0034] In this specification, when the emphasis is placed on data (X) being data from the (d-1)th time window, a subscript “d-1” is added to the lower right corner and recorded as “X d-1 In addition, the database consisting of data belonging to the d-th time window will be referred to as the "d-th database" below.

[0035] (Matrix calculator 120 included in feature reduction processing circuit 100)

[0036] The matrix calculator 120 included in the feature reduction processing circuit 100 is a matrix calculator that calculates the feature reduction processing according to the data (X) of the (d-1)th time window. d-1 ) to calculate the matrix (C d ) components. Figure 2 As shown, the matrix (C d ) is used for arithmetic processing to sequentially reduce the image feature quantity {f i An important technical feature of the image retrieval device according to the disclosed technology is that the matrix C used for the operation processing related to the d-th time window d is based on the data (X) associated with the (d-1)th time window d-1 ) is calculated. It should be noted that the matrix C used to reduce the feature quantity d The letter "C" in COMCOM comes from the abbreviation of the English word "Compression".

[0037] The matrix (C d ) may be a matrix obtained by statistical analysis such as principal component analysis and discriminant analysis, as shown in Patent Document 1. Specifically, the matrix (C d ) may be a matrix obtained based on singular value decomposition (hereinafter referred to as "SVD"). Assume that the data (X) associated with the (d-1)th time window d-1 ) is shown below.

[0038]

[0039] The "U" in Equation 2 is the left singular vector {u i}(i=1,2,…,r). Moreover, “V” in Equation 2 is a matrix composed of right singular vectors {v i}(i=1,2,…,r). The superscript “T” on the right indicates the transposition operation. “S” in Formula 2 is a diagonal matrix in which the singular values ​​are arranged in order of magnitude. “m” is the data (X) associated with the (d-1)th time window. d-1 ) is the total number of rows, and is equal to the acquired image feature {f i}. "n" is the number of data (X) associated with the (d-1)th time window. d-1 ) is the total number of columns, and is equal to each image feature {f i}. Here we assume that "m" is greater than "n". The right subscript "r" in Formula 2 is the abbreviation of "rank", which means that X d-1 If X d-1 is of full rank, then "r" = "n". d-1 The rank is "r", which is equivalent to the vector {f i The dimension of the space spanned by}(i=1,2,…,m) is “r”.

[0040] The matrices U and V consisting of singular vectors satisfy the following properties.

[0041]

[0042] I(r) shown on the right side of Formula 3 is an r×r unit matrix.

[0043] If X d-1 If it is not a zero matrix, we can use the singular value decomposition (hereinafter referred to as "SVD") result shown in Formula 2 and the properties shown in Formula 3 to define X d-1 According to the definition of Moore-Penrose type generalized inverse, X d-1 The generalized inverse matrix of is shown below.

[0044]

[0045] If X d-1 is a regular matrix, then the generalized inverse matrix shown in Equation 4 is equivalent to X d-1 The inverse matrix of .

[0046] By row vector {f i}(i=1,2,…,m) n The subspace of d-1The row domain is sometimes also called the row space, but in this paper we will use the name “row domain”. i}(i=1,2,…,r) after transposition, it forms X d-1 An orthogonal normalized basis for the row field of .

[0047] According to one aspect of the image retrieval device of the disclosed technology, a method for projecting a vector onto X is introduced into the query system. d-1 The projection matrix (Pv) is given by the following formula.

[0048] R ν =VV T …(5)

[0049] P shown in Formula 5 v is a projection matrix of size n×n and rank r. Note that VV in Equation 5 T Compared with V shown in Equation 3 T V is different.

[0050] In the following, the matrix multiplication from the right side is called "right multiplication". Assume that the image feature quantity of the query image is given, that is, "f target By changing the projection matrix (P v ) multiplied by f target , we can use f target Projection to X d-1 In the query system, the search scope can be limited to X d-1 The row domain can help to shorten the processing time. It should be noted that the letter "P" representing the projection matrix comes from the first letter of the English word "Projection".

[0051] By multiplying the matrix V on both sides of the equation and applying the property shown in equation 3, equation 2 can be transformed into the following equation.

[0052]

[0053] Here, {g i}(i=1,2,…,m) is the image feature after dimensionality reduction (hereinafter referred to as “low-dimensional image feature”).

[0054] The following conclusion can be drawn from Equation 6. If a row vector f x Belong to X d-1 The row domain, that is, f x It can be obtained by {f i}(i=1,2,…,m), then the row vector f x The result vector obtained by right multiplication by V can also be expressed by {gi It is represented by a linear combination of}(i=1,2,…,m).

[0055]

[0056] Here, {g i Each of}(i=1,2,…,m) is a row vector of size 1×r. Moreover, since f x V is {g i}, so f x V is also a row vector of size 1×r. i}(i=1,2,…,m) are the corresponding coefficients respectively.

[0057] Suppose we have another row vector f y . With f x Different, f y Does not belong to X d-1 For f that does not belong to the row domain y , can be obtained by transforming the projection matrix (P v ) is right-multiplied by fy to project it into the row domain. Since the projected vector f y P v belongs to this row domain, so f y P v can be re-expressed as f x The above process can be expressed as follows:

[0058]

[0059] In short, even if f y does not belong to the row domain, by simply multiplying "V" by f y , the resulting vector f y V can always be expressed as {g i}(i=1,2,…,m) (see Formula 8).

[0060] As described above, the matrix (C d ) can be a matrix "V" consisting of singular vectors. If X d-1 If it is a column full rank matrix and "r" is equal to "n", then multiplying by "V" will not help change the size of the vector. In this case, an approximate dimensionality reduction method can be considered, that is, setting the singular values ​​close to 0 to 0. The details of this method are shown in Implementation 2.

[0061] (Projection Transformation Section 130 Included in Feature Reduction Processing Circuit 100)

[0062] The projection transformation unit 130 included in the feature reduction processing circuit 100 is a component that multiplies the matrix (C d ) by the sequentially obtained image feature amounts {f i}. C d is a matrix calculated by the matrix calculator 120. Here, each image feature amount {f i} is defined as a row vector. Therefore, the multiplication of the matrix (C d ) is a right multiplication of the image feature amounts {f i}, and the result is f i C d . When C d is regarded as "V", the size of C d is n×r (r < n). Therefore, the projection transformation unit 130 is a component that converts the image feature amounts {f i} into low-dimensional image features {g i}.

[0063] Although the low-dimensional image features {g i} in Equation 6 are calculated through "V", the technology disclosed in the present disclosure is not limited thereto. Here, as long as they are low-dimensional image features calculated by the projection transformation unit 130, even if they are obtained by multiplying with a matrix (C d ) different from "V", they are also referred to as {g i}.

[0064] As Figure 1 shown, the low-dimensional image features {gi} generated by the projection transformation unit 130 are stored in the storage device. The image retrieval device can access this storage device. The low-dimensional image features {gi} are stored as a database for each time window. The matrix (C d ) used for multiplication by the projection transformation unit 130 is also used by the retrieval processing unit 200 of the image retrieval device. Therefore, this matrix (C d ) is shared between the feature reduction processing circuit 100 and the retrieval processing unit 200.

[0065] Figure 3 is a block diagram showing the functional configuration of the retrieval processing unit 200. The feature reduction processing circuit 100 of the image retrieval device is a component used in the image data accumulation stage. In contrast, the retrieval processing unit 200 of the image retrieval device is a component used in the image search stage. When explaining the processing content of the image retrieval device according to the technology disclosed in the present disclosure, the processing is separately defined as the "image data accumulation stage" and the "image search stage". However, in the case where the image retrieval device is actually put into use, these two stages can be carried out simultaneously. That is, the retrieval processing unit 200 can retrieve images in parallel with the image data accumulation executed by the feature reduction processing circuit 100.

[0066] Figure 3 The "f target " represents the image feature quantity of the image to be retrieved. If the image retrieval device is a query system, then "f target ” is the image feature of the image being queried.

[0067] like Figure 3 As shown, the search processing unit 200 includes a second projection transformation unit 210 (210-0, 210-1, ..., 210-x, ...) and a search execution unit 220 (220-0, 220-1, ..., 220-x, ...). Figure 3 As shown, the function module ending with "-0" represents the function module for processing the database related to the d-th time window; the function module ending with "-1" represents the function module for processing the database related to the (d-1)-th time window; the function module ending with "-x" represents the function module for processing the database related to the (dx)-th time window. How many databases the retrieval processing unit 200 is to search is a design matter of the image retrieval device. This matter can be determined according to the intended use and specifications of the image retrieval device.

[0068] (Second Projection Transformation Unit 210 Included in the Retrieval Processing Unit 200)

[0069] The second projection transformation unit 210 included in the search processing unit 200 is used to transform the input f targe t is a component for calculating low-dimensional image features. It can be considered that the second projection transformation unit 210 corresponds to the projection transformation unit 130 in the feature reduction processing circuit 100. As described above, each matrix (C d , C d-1 , C d-2 ,…,C d-x , …) are shared with the second projection transformation unit 210 in the retrieval processing unit 200.

[0070] The low-dimensional image feature (g target_x ) is given by the following formula.

[0071]

[0072] When the projection transformation unit 130 uses the matrix (C d-x ) is "V", the matrix (C d-x ) is also a "V".

[0073] (Search Execution Section 220 Included in Retrieval Processing Unit 200)

[0074] The search execution unit 220 included in the retrieval processing unit 200 is based on the low-dimensional image features (g target_x ) Search components of possible image features from databases of different time windows. The term "search" used here has the same meaning as the term in the field of computer technology, which refers to the process of retrieving required data or determining its storage location.

[0075] Specifically, the search execution unit 220 calculates the distance in the feature space and extracts possible image features according to the calculated distance. The distance calculated by the search execution unit 220 is an indicator distance for evaluating similarity, such as Euclidean distance, Mahalanobis distance, Bhattacharyya distance, etc.

[0076] As shown in Equation 2 and Equation 6, when SVC is used in ascending order of singular value size, the generated low-dimensional image features (g target_x ) is the most important element, and the second component is the second most important element (the same applies hereinafter). Therefore, the search execution unit 220 may use only the first component, or only the first k components (k is a natural number greater than 1 and less than r) to calculate the distance and extract candidates for possible image features. Alternatively, the search execution unit 220 may, for example, use only the first component or only the first k components to calculate the distance, and perform filtering to separate components that should be included in the search and components that should not be included in the search.

[0077] The idea of ​​using only the first k components to calculate the distance is consistent with the dimensionality reduction method, which is an approximate method in the form of SVD, that is, setting singular values ​​close to 0 to 0. As mentioned above, this approximate dimensionality reduction method of setting singular values ​​close to 0 to 0 will be shown in Implementation 2.

[0078] The search performed by the search execution unit 220 is performed by calculating the low-dimensional image features (g target_x ) and the low-dimensional image features {g i It should be noted that it is not necessary to know all the low-dimensional image features {g i} distance.

[0079] The low-dimensional image features to be searched (g target_x ) and the low-dimensional image features of the database {g i The values ​​of the components of} can be represented by binary numbers. Binary representation helps to narrow the search candidate range (narrowing the candidate range is called "filtering"). Specifically, the search performed by the search execution unit 220 starts from the high bit and checks whether the two data represented by the binary numbers match. When comparing the low-dimensional image features to be searched (g target_x) and a low-dimensional image feature of interest {g i}, if the respective highest bits do not match, then the pair is unlikely to have the shortest distance. Therefore, data whose respective highest bits do not match can be excluded from the search candidates. After filtering by the highest bit, similar filtering processing is performed by the next highest bit in turn. This search method is conceptually similar to a search method called binary search.

[0080] It should be noted that the above method is not limited to binary numbers. The method of checking whether two data represented by a general N-bit representation are matched in order from the higher bits can also be regarded as filtering in the search.

[0081] As mentioned above, the retrieval process performed by the retrieval processing unit 200 can be performed in parallel with the image data accumulation process performed by the feature reduction processing circuit 100. For example, if the current time belongs to the dth time window, the search execution unit 220 can perform a search on the dth database in parallel with the process in which the feature reduction processing circuit 100 accumulates image data in the dth database. Figure 2 As shown, the processing ST120 in the d-th time window can only be performed after all the ST110 processing in the d-th time window is completed. Here, ST110 is the processing of the data storage unit 110, and ST120 is the processing of the matrix calculator 120. Therefore, according to the traditional concept, before the data storage unit 110 completes all the ST110 processing of acquiring image features, the image retrieval device cannot perform the projection transformation processing (ST130) by the projection transformation unit 130. That is, according to the traditional way of thinking, it is impossible to search the database of the d-th time window while the image data is accumulated in the d-th time window.

[0082] It is conceivable to use a fixed matrix (C) to generate low-dimensional image features. However, as mentioned earlier, even for images obtained from the same surveillance camera, when observed in a time window divided into one day, the matrix suitable for use in dimensionality reduction will gradually change. In other words, as time goes by, using a fixed matrix (C) for dimensionality reduction will become less and less suitable for the query system.

[0083] One of the excellent effects of the image retrieval apparatus according to Embodiment 1 is that the projection transformation unit 130 can sequentially perform the projection transformation processing (ST130) even before the data storage unit 110 has completed all the processing of acquiring image features (ST110).

[0084] Another excellent effect of the image retrieval apparatus according to Embodiment 1 is that the search execution section 220 can perform a search on the dth database while the feature reduction processing circuit 100 accumulates image data in the dth database.

[0085] In summary, the excellent effects of the image retrieval device according to embodiment 1 are that it can solve the problem of "matrix changes suitable for dimensionality reduction" and can perform high-speed retrieval on multiple databases related to different time windows.

[0086] These effects are obtained by actively exploiting the property that, for example, when the time window is set to a day, the change of the matrix suitable for use in dimensionality reduction is gradual.

[0087] Focusing on the characteristic that the change is gradual, the image retrieval device according to Embodiment 1 uses the matrix C in the d-th time window d Perform projection processing, where the matrix C d is based on the data (X) associated with the (d-1)th time window d-1 ) is calculated. Due to this technical feature, the image retrieval device according to embodiment 1 exhibits the above-mentioned excellent effect.

[0088] Implementation Method 2

[0089] The image retrieval device according to Embodiment 2 is a modified example of the image retrieval device of the disclosed technology. The image retrieval device according to Embodiment 2 is the same as the image retrieval device described in Embodiment 1 in terms of functional configuration. Therefore, unless otherwise specified, the same reference numerals as those in Embodiment 1 are used in Embodiment 2.

[0090] It can be considered that the image retrieval device according to Embodiment 2 is an embodiment of realizing the matrix calculated by the matrix calculator 120. According to Embodiment 2, the matrix calculated by the matrix calculator 120 is an approximation process based on the SVD form. The approximation in the SVD form refers to an approximation method of replacing singular values ​​close to 0 with 0.

[0091] The SVD shown in Equation 2 can be approximated by:

[0092]

[0093] The right side of formula 10 represents the result after replacing the (k+1)th to rth singular values ​​with 0 (zero). It should be noted that the singular values ​​in "S" are arranged in order of size.

[0094] The diagonal matrix on the right side of Equation 10 is represented by an "S" with a tilde on top. The tilde indicates that the "S" with the tilde is similar to the original "S" on the left side of Equation 10. Equation 10 represents an approximation in the form of SVD.

[0095] As shown in Formula 10, "k" is a positive integer less than the rank "r".

[0096] The right side of equation 10 can be transformed as shown below:

[0097]

[0098] Here, the subscript "trunc" on the right side of Formula 11 comes from the word "truncation". trunc is a matrix of size m×k. trunc is a diagonal matrix of size k×k, V trunc is a matrix of size n×k.

[0099] In the image retrieval device according to Embodiment 2, V of Formula 11 is used trunc As the matrix calculated by the matrix calculator 120. Therefore, in the image retrieval device according to the second embodiment, the projection transformation unit 130 and the second projection transformation unit 210 both use V trunc Projection.

[0100] Using V trunc The effect of projection is similar to that of using only low-dimensional image features (g target_x ) to calculate the distance.

[0101] The image retrieval device according to the second embodiment has the following unique effect: d-1 When it is a column full rank matrix and "r" is equal to "n", the image features (f target ) is reduced from n to k.

[0102] Through this processing, the image retrieval device according to embodiment 2 solves the problem in the manner described in embodiment 1. That is, the image retrieval device according to embodiment 2 solves the problem of "matrix changes suitable for dimensionality reduction" and can perform high-speed retrieval for multiple databases related to different time windows.

[0103] Implementation 3

[0104] The image retrieval device according to Embodiment 3 is a modification of the image retrieval device according to the disclosed technology.

[0105] Unless otherwise specified, the same reference numerals as those in the previous embodiments are used in Embodiment 3. In Embodiment 3, descriptions overlapping with those in the previous embodiments will be omitted as appropriate.

[0106] As described above, the technical feature of the image retrieval apparatus according to Embodiment 1 and Embodiment 2 is that the matrix C used in the arithmetic processing related to the d-th time window d is based on the data (X) associated with the (d-1)th time windowd-1 ) is calculated. Therefore, in the image retrieval apparatus according to the first and second embodiments, the data (X) associated with the (d-1)th time window is used. d-1 ), i.e., the adjacent most recent past time window.

[0107] The image retrieval device according to Embodiment 3 generalizes the part of "the adjacent recent past" to "the past that can be predicted to be similar based on the characteristics of the image data from experience". In other words, when observing in a time window of one day, "the adjacent recent past" is a specific example of "the past that can be predicted to be similar based on experience". From a technical point of view, whether the image data "is adjacent to the immediate past" is not essential, the key is "the characteristics of the image data can be predicted to be similar based on experience".

[0108] The database used by the image retrieval device is not limited to the database of the time window with "1 day" as the unit. For example, it may also be necessary to create a database in the time window with 6 hours as the unit, such as 0:00 to 6:00, 6:00 to 12:00, 12:00 to 18:00, and 18:00 to 24:00. Assume that the current time is included in the time window of "12:00 to 18:00". At this time, if it is known from experience that the characteristics of the image data related to this time window are similar to the characteristics of the image data of the time window of "yesterday from 12:00 to 18:00", then the image retrieval device according to embodiment 3 can use the data of "12:00 to 18:00" yesterday to calculate the matrix C d .

[0109] As shown in the above embodiments, the characteristics of image data related to humans may change periodically according to the life behaviors of humans.

[0110] The characteristics of image data have some typical change cycles, such as a cycle of one day, one week, or one year.

[0111] For example, the characteristics of image data may differ between weekdays and weekends.

[0112] Assume that the current time is included in the time window of "Sunday". At this time, if it is known from experience that the characteristics of the image data related to the time window are similar to the characteristics of the image data of "last Sunday", the image retrieval device according to embodiment 3 can use the data related to "last Sunday" to calculate the matrix C d This weekly changing cycle is also believed to originate from human behavior.

[0113] For another example, the characteristics of image data may differ between summer and winter. Assume that the current time is included in the time window of "summer".

[0114] At this time, if it is known from experience that the characteristics of the image data related to the time window are similar to the characteristics of the image data of "last summer", the image retrieval device according to embodiment 3 can use the data related to "last summer" to calculate the matrix C d This one-year cycle of change is also thought to stem from differences in human behavior, especially in clothing.

[0115] The characteristics of image data may also differ between days with special events and ordinary days. Special events include festivals, concerts of famous artists, international conferences, international sports events, etc.

[0116] For example, the characteristics of image data on Christmas Eve may be different from those of image data on ordinary days.

[0117] Assume that the current time is included in the time window of "Christmas Eve". At this time, if it is known from experience that the characteristics of the image data related to this time window are similar to the characteristics of the image data of "last Christmas Eve", the image retrieval device according to Embodiment 3 can use the data related to "last Christmas Eve" to calculate the matrix C d As shown in this example, changes in specific days with special events also arise from human life behavior.

[0118] It can be seen that the characteristics of human-related image data are closely related to human life behavior. And human life behavior will also be greatly affected by disasters such as war, epidemics, and natural disasters.

[0119] For example, assume that the current time is included in the time window of Christmas Eve 2022. Considering the impact of the spread of the new coronavirus infection, the image retrieval device according to Embodiment 3 can use data related to Christmas Eve to calculate the matrix C d , the data is not from last year, but from 2019 or 2018 before the spread of the new coronavirus pandemic.

[0120] The image retrieval device according to Embodiment 3 calculates the projection matrix (C d ).

[0121] By changing the projection matrix (Cd ) Generalization of calculation-related matters, the image retrieval device according to embodiment 3 can perform high-speed searches on multiple databases related to different time windows under the premise of considering human life behavior.

[0122] Industrial Applicability

[0123] The disclosed technology can be applied to a system for searching for missing persons or suspicious persons from video images captured by surveillance cameras at facilities such as airports, and has industrial applicability.

[0124] Reference numerals list

[0125] 100 feature reduction processing circuit, 110 data storage unit, 120 matrix calculator, 130 projection transformation unit, 200 search processing unit, 210 second projection transformation unit, 220 search execution unit.

Claims

1. An image retrieval device, the image retrieval device include: feature reduction processing circuit; as well as Retrieval processing unit; in, The matrix calculator included in the feature reduction processing circuit is based on the stored data (X d-1 )Calculate the matrix (C d ); The projection transformation unit included in the feature reduction processing circuit transforms the matrix (C d ) multiplied by the image feature quantities {f i }, The second projection transformation unit included in the retrieval processing unit uses the matrix (C d ) from the input image features of the search target (f target ) Calculate low-dimensional image features (g target_x ), The search execution unit included in the retrieval processing unit calculates the vector calculated by the projection transformation unit and the low-dimensional image feature (g target_x ) and by using the distance to search for possible image features, the vector calculated by the projective transformation unit (130) belongs to a database divided by time windows.

2. The image retrieval device according to claim 1, in, The matrix (C d ) is obtained by using the data (X d-1 ) is calculated by the singular value decomposition of .

3. The image retrieval device according to claim 1, in, The matrix (C d ) is obtained by using the data (X d-1 ) is calculated by approximate singular vectors.

4. The image retrieval device according to any one of claims 1, 2 and 3, in, The stored data relates to images taken in a contiguous recent past time window.

5. The image retrieval device according to any one of claims 1, 2 and 3, in, The stored data relates to images taken within a past time window that can be empirically predicted to have similar characteristics to the image data.

Images