Image processing apparatus, image processing method, and machine-readable storage medium

Abstract

The present disclosure relates to an image processing device, an image processing method and a machine readable storage medium. The image processing device comprises: a first extraction unit extracting global features of an image; a second extraction unit detecting key points related to a specific object in the image and extracting local features of the image corresponding to the key points; a first calculation unit calculating an information passing metric for a graph convolution network, nodes of the information passing metric comprising the global features and the local features corresponding to the key points; and a second calculation unit optimizing the local features of the image by the graph convolution network using the information passing metric. The first calculation unit calculates the information passing metric by: defining a mask matrix defining whether information is passed between different nodes; and calculating an adjacency matrix defining an amount of information passed between different nodes. The image processing device can optimize comparison features for re-identification, improving the accuracy of re-identification results.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image processing, and in particular to an image processing apparatus, an image processing method and a machine readable storage medium for re-identification. BACKGROUND

[0002] This section provides background information relating to the present disclosure, which is not necessarily prior art.

[0003] Re-identification is a very practical technology in computer vision. It uses computer vision technology to determine whether a specific object exists in an image or video sequence, and can be widely used in intelligent video monitoring, intelligent security and other fields.

[0004] In specific implementation, the global feature of the image can be extracted by using an existing feature extraction network. The global feature is used to calculate similarity and sort the similarity to obtain the re-identification result. In recent years, research shows that reasonable use of local features of an image can improve the accuracy of the re-identification result. Extracting features near key points as local features of the image is an effective method for obtaining local features. At the same time, the global feature contains more rich and comprehensive feature information. Therefore, how to effectively combine the global feature and the local feature is a key technology to improve the accuracy of the re-identification result. SUMMARY

[0005] This section provides a general overview of the present disclosure, rather than a comprehensive disclosure of all aspects or all features of the present disclosure.

[0006] The purpose of the present disclosure is to provide an image processing apparatus, an image processing method and a machine readable storage medium for optimizing features in re-identification.

[0007] According to an aspect of the present disclosure, an image processing apparatus is provided, comprising: a first extraction unit configured to extract a global feature of an image; a second extraction unit configured to detect a key point related to a specific object in the image and extract a local feature of the image corresponding to the key point; a first calculation unit configured to calculate an information transmission metric for a graph convolution network, the nodes of the information transmission metric comprising the global feature and the local feature corresponding to the key point; and a second calculation unit configured to optimize the local feature of the image by the graph convolution network using the information transmission metric, wherein the first calculation unit calculates the information transmission metric by: defining a mask matrix, the mask matrix defining whether information is transmitted between different nodes; and calculating an adjacency matrix, the adjacency matrix defining the amount of information transmitted between different nodes.

[0008] According to another aspect of the present disclosure, there is provided an image processing method, comprising: extracting global features of an image; detecting key points related to a specific object in the image and extracting local features of the image corresponding to the key points; calculating an information passing metric for a graph convolution network, the nodes of the information passing metric comprising the global features and the local features corresponding to the key points; and optimizing the local features of the image by the graph convolution network using the information passing metric, wherein the calculating the information passing metric comprises: defining a mask matrix, the mask matrix defining whether information is passed between different nodes; and calculating an adjacency matrix, the adjacency matrix defining an amount of information passed between different nodes.

[0009] According to another aspect of the present disclosure, there is provided a machine readable storage medium carrying a program product comprising machine readable instruction codes stored therein, wherein the instruction codes, when read and executed by a computer, can cause the computer to perform the image processing method according to the present disclosure.

[0010] Using the image processing apparatus, the image processing method and the machine readable storage medium according to the present disclosure, the local features can be optimized by the graph convolution network using both the global features and the local features as the nodes of the graph convolution network and reasonably designing the adjacency matrix, so as to provide more optimal comparison features for re-identification, further improving the accuracy of the re-identification result.

[0011] Further areas of applicability will become apparent from the description provided herein. The description and specific examples in this summary are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0012] The drawings described herein are for the purpose of illustrating selected embodiments only and are not intended to limit the scope of the present disclosure. In the drawings:

[0013] Figure 1 a block diagram illustrating a structure of an image processing apparatus according to an embodiment of the present disclosure;

[0014] Figure 2 a block diagram illustrating a structure of an image processing apparatus according to another embodiment of the present disclosure;

[0015] Figure 3 a schematic diagram illustrating an image processing apparatus according to an embodiment of the present disclosure;

[0016] Figure 4 a schematic diagram illustrating an image processing apparatus according to an embodiment of the present disclosure;

[0017] Figure 5 a schematic diagram illustrating an image processing apparatus according to an embodiment of the present disclosure;

[0018] Figure 6 This is a schematic diagram illustrating a mask matrix according to an embodiment of the present disclosure;

[0019] Figure 7 This is a flowchart illustrating an image processing method according to an embodiment of the present disclosure;

[0020] Figure 8 A flowchart illustrating an image processing method according to another embodiment of the present disclosure; and

[0021] Figure 9 This is a block diagram of an exemplary structure of a general-purpose personal computer in which image processing apparatus and methods according to embodiments of the present disclosure can be implemented.

[0022] While this disclosure is readily subject to various modifications and substitutions, specific embodiments thereof have been shown by way of example in the accompanying drawings and are described in detail herein. However, it should be understood that the description of specific embodiments herein is not intended to limit this disclosure to the specific forms disclosed, but rather, this disclosure is intended to cover all modifications, equivalents, and substitutions falling within the spirit and scope of this disclosure. It should be noted that throughout the drawings, corresponding reference numerals indicate corresponding parts. Detailed Implementation

[0023] Examples of this disclosure will now be described more fully with reference to the accompanying drawings. The following description is merely exemplary and is not intended to limit the disclosure, its application, or its uses.

[0024] Example embodiments are provided so that this disclosure will become exhaustive and will fully convey its scope to those skilled in the art. Numerous specific details, such as examples of particular components, apparatus, and methods, are set forth to provide a detailed understanding of embodiments of this disclosure. It will be apparent to those skilled in the art that the specific details are not required, and that the example embodiments may be implemented in many different forms, none of which should be construed as limiting the scope of this disclosure. In some example embodiments, well-known processes, well-known structures, and well-known techniques are not described in detail.

[0025] As described above, the present application aims to effectively utilize global features and local features near key points of an image to perform object re-identification, such as pedestrian re-identification. Although existing human key point prediction can achieve high accuracy, when the object to be identified in the image is occluded, resulting in poor performance of feature extraction, the features can be optimized by a graph convolutional network (GCN). Specifically, the local features near the key points can be regarded as nodes of the graph convolutional network, and the local features are optimized by passing information between the nodes. However, the global features contain more rich and comprehensive feature information, so the present application also uses the global features as input nodes of the graph convolutional network. The image processing apparatus according to the embodiments of the present application will be described below in conjunction with Figure 1 An image processing apparatus according to an embodiment of the present application will be described below in conjunction with

[0026] Figure 1 A block diagram illustrating the structure of an image processing apparatus according to an embodiment of the present application is shown. As Figure 1 shown, the image processing apparatus 100 according to an embodiment of the present application can include a first extraction unit 110, a second extraction unit 120, a first calculation unit 130, and a second calculation unit 140.

[0027] First, the first extraction unit 110 can extract global features of an image. Here, the image can represent a single image or multiple frames of images.

[0028] Further, the second extraction unit 120 can detect key points related to a specific object in the image, and extract local features of the image corresponding to the key points. For example, in pedestrian re-identification, the specific object is a specific person.

[0029] Further, the first calculation unit 130 can calculate an information passing measure for a graph convolutional network. Here, the nodes of the information passing measure include the global features and the local features corresponding to the key points.

[0030] Specifically, the first calculation unit 130 calculates the information passing measure by defining a mask matrix that defines whether information is passed between different nodes, and calculating an adjacency matrix that defines the amount of information passed between different nodes. In addition, the first calculation unit 130 can calculate the elements of the adjacency matrix based on the local features and the global features. It should be noted that the different nodes here include the global features and the local features corresponding to the key points. Details will be described below.

[0031] Further, the second calculation unit 140 can optimize the local features of the image by the graph convolutional network using the information passing measure calculated by the first calculation unit 130.

[0032] Therefore, the image processing apparatus 100 according to the embodiments of the present disclosure can optimize local features by using a graph convolutional network, using both global and local features as nodes of the graph convolutional network and rationally designing the adjacency matrix.

[0033] The following is combined with Figures 2 to 6 The configuration of the image processing device is described in more detail.

[0034] Figure 2 A block diagram illustrating the structure of an image processing apparatus according to another embodiment of the present disclosure is shown. Figure 2 As shown, an image processing apparatus 200 according to another embodiment of the present disclosure may include a first extraction unit 210, a key point detection unit 2201, a local feature extraction unit 2202, a construction unit 250, a first calculation unit 230, a second calculation unit 240, and a matching and sorting unit 260. Figure 2 The first extraction unit 210 shown corresponds to Figure 1 The first extraction unit 110, the key point detection unit 2201, and the local feature extraction unit 2202 correspond to Figure 1 The second extraction unit 120 and the first calculation unit 230 correspond to the second extraction unit 120 and the first calculation unit 230, respectively. Figure 1 The first calculation unit 130 in the middle, and the second calculation unit 240 corresponding to Figure 1 The second calculation unit 140 in the process.

[0035] First, the first extraction unit 210 can extract global features of the image.

[0036] Furthermore, the key point detection unit 2201 can detect key points related to specific objects in the image, such as pedestrians, and the local feature extraction unit 2202 can extract local features of the image corresponding to the key points.

[0037] Furthermore, the construction unit 250 can construct a feature matrix that includes local features and global features.

[0038] Furthermore, the first computation unit 130 can calculate the information transfer metric for the graph convolutional network based on the feature matrix constructed by the construction unit 250. Specifically, the first computation unit 130 calculates the information transfer metric by: defining a mask matrix, which defines whether information is transferred between different nodes; and calculating an adjacency matrix based on the feature matrix, which defines the amount of information transferred between different nodes.

[0039] Specifically, the first calculation unit 130 can calculate the elements of the adjacency matrix based on the local features and the global feature. For example, the first calculation unit 130 calculates the elements of the adjacency matrix by the following way: for each key point, calculating an intermediate feature corresponding to the key point based on the local feature and the global feature; mapping the intermediate features of the respective key points into intermediate values respectively through a fully connected network; and calculating the adjacency matrix based on the intermediate values corresponding to each key point. The intermediate feature of each key point described above is the absolute value of the difference between the local feature of each key point and the corresponding global feature.

[0040] In addition, the first calculation unit 130 normalizes the calculated adjacency matrix.

[0041] Further, the second calculation unit 140 can perform matrix multiplication operation on the adjacency matrix and the feature matrix, and input the operation result into a fully connected network to obtain an optimized feature matrix, so as to obtain optimized local features and global features.

[0042] Thus, the present application designs a graph convolution network. In the network, the global feature and the local feature are both taken as input nodes of the graph convolution network. Moreover, the transmission metrics between nodes (including the transmission metrics between local features and the transmission metrics between local features and global features) are learned by the network according to the relationship between the respective features.

[0043] The operations of the respective units in the above image processing apparatus will be described below respectively in the case of a single image and the case of continuous multiple frames of images in a video. Figure 2

[0044] <The case of a single image>

[0045] In the case of a single image, the mask matrix defines the transmission between the global feature of the image and all the local features, and between the local features corresponding to adjacent key points. The case of a single image will be described below in combination with Figure 3 Figure 3 To illustrate the schematic principle diagram of the image processing apparatus according to the embodiments of the present disclosure.

[0046] As Figure 3 shown in the present application, the global feature is used as one of the input nodes of the graph convolution network. In Figure 3 the lower part of the figure shows an example of a human body, in which case the key point detection unit 2201 can detect 13 key points (smaller round dots in the figure) numbered 0, 1, …, 12 corresponding to the human body, and the local feature extraction unit 2202 can extract local features corresponding to the 13 key points respectively. The global feature (larger circle in the figure) is calculated by the first extraction unit 210. It can be seen that for the image, 13 local features and one global feature can be obtained.​​

[0047] Assuming the dimension of each feature is 2048, the constructing unit 250 can form all the features into a feature matrix X with a size of 14x2048. Among them, the 1st to 13th rows of the matrix are the local features of the image, and the 14th row of the matrix is the global feature of the image. Each row of the feature matrix X is a node in the graph convolution network, the i-th row is the i-th node, and there are 14 nodes in total. It should be noted that 2048 is only an example of the dimension, and other dimensions are also possible, and for the case of a specific object being a person, 13 key points can be detected, and for other objects, the number of object points will change accordingly.

[0048] Further, the first computing unit 230 can generate a mask matrix M with a size of 14x14. In the definition of the example of the present application, if there is information transmission from node j to node i, then M(i,j) = 1, otherwise M(i,j) = 0.

[0049] Figure 3 The arrows in the lower example indicate the direction of information transmission. It should be noted that in order to avoid making the figure unclear, not all information transmissions are necessarily shown. The transmission between local features is determined by the human skeletal structure, so information transmission can occur between the local features of key points at adjacent positions, and no information transmission occurs if the key points are far apart in the human skeletal structure. In the example of the present application, there is information transmission between the global feature and all local features, and between the local features corresponding to adjacent key points. In addition, although only the optimized local features are shown in the example of the present application, as described above, the global feature can also be optimized. Figure 3 Figure 3

[0050] In addition, the first computing unit 230 can calculate the adjacency matrix A. Let f i ,i = 0, 1, 2, … 13 be the feature vector of the i-th row in the feature matrix X. The distance d i between each local feature and the global feature is calculated as follows (1).

[0051] d i = abs(f i -f 13 ), i = 0, 1, 2, … 12 (1)

[0052] As shown in equation (1), d i is the absolute value of the difference between the local feature of each key point and the corresponding global feature.

[0053] The first computing unit 230 maps d i to D i through a fully connected network. d i ​​D i is an example of the intermediate value in the present application. i D i is a real number, and all D i are super-parameters.

[0054] It can be understood that each local feature can be mapped to a score representing its reliability. i The larger D i , the lower the score of node i, i.e., the lower the reliability. The smaller D i , the higher the score of node i, i.e., the higher the reliability. According to the score, nodes with high reliability pass more information to nodes with low reliability. Conversely, nodes with low reliability pass less information to nodes with high reliability. Therefore, information can be passed more efficiently through a graph convolution network.

[0055] If M(i,j) = 0, then A(i,j) = 0, which represents that node j passes no information to node i.

[0056] If M(i,j) = 1 and 0≤i,j≤12, then A(i,j) is calculated by equation (2).

[0057] A(i,j) = ReLU(1 + D j ) × (D i + D j ) (2)

[0058] In equation (2), ReLU represents that when the value behind it is greater than zero, the operation value is the value itself, and when the value behind it is less than or equal to zero, the operation value is zero. Equation (2) represents the information transmission measure between local features. It should be noted that A(i,j) represents the amount of information that node j passes to node i. This is also consistent with the description above regarding reliability. That is, in equation (2), when D i >D j , A(i,j) > A(j,i).

[0059] Since there is information transmission from the global feature to all local features, M(i,13) = 1, 0≤i≤12. The amount of information transmission is defined by equation (3).

[0060] A(i,13) = 2 D i (3)

[0061] Equation (3) represents the information transmission measure from the global feature to the local features. As shown in equation (3), the amount of information transmission from the global feature to the local features corresponding to different key points is defined as the intermediate value Di twice.

[0062] It can be defined that the local features do not transfer information to the global feature, thus M(13,j) = 0, 0≤j≤12, then A(13,j) = 0, 0≤j≤12.

[0063] In addition, in the case of a single image, there is only one global feature, in order to optimize the global feature at the same time, define M(13,13) = 1, A(13,13) = 1.

[0064] Thus, the first calculation unit 230 can obtain the adjacency matrix A. Finally, the first calculation unit 230 normalizes the adjacency matrix A by row, so that the sum of the elements of each row is 1.

[0065] Further, the second calculation unit 240 inputs the feature matrix X into the graph convolution network, and obtains the improved local features and global features through the following formula (4).

[0066] X_improved = ReLU(fc(AX)) + X (4)

[0067] Where fc(·) is a full connection network.

[0068] <The case of continuous multiple frames of images>

[0069] In the case of multiple images which are continuous multiple frames of video from a camera source, the nodes of the information transfer measure include the global feature of each image in the multiple images and the local feature corresponding to the key point of each image in the multiple images. The case of multiple images is described below in combination with Figure 4 and Figure 5 . Figure 4 and Figure 5 are schematic diagrams of an image processing apparatus according to an embodiment of the present disclosure. Although Figure 4 and Figure 5 show the case of three images, the method of the present application can be applied to the case of more images. The operation principle of each unit in Figure 2 is described in detail below.

[0070] In general, as Figure 4 indicated, it is assumed that the sample includes three images frame_1, frame_2 and frame_3. In combination with the above description of Figure 3The description of the key points, local features and global features in the human body example, the number of key points K is 13. Therefore, K local features local 1_1, …, local 1_K and a global feature global_1 can be obtained for the first frame image frame_1, K local features local 2_1, …, local 2_K and a global feature global_2 can be obtained for the second frame image frame_2, and K local features local 3_1, …, local 3_K and a global feature global_3 can be obtained for the third frame image frame_3.

[0071] As shown in Figure 4 , through the information transmission between the nodes of the three frame images, the optimized global features and local features can be obtained for each frame image. For example, global_1', local 1_1', …, local 1_K' can be obtained for the first frame image, global_2', local 2_1', …, local 2_K' can be obtained for the second frame image, and global_3', local 3_1', …, local 3_K' can be obtained for the third frame image.

[0072] Specifically, first, the first extraction unit 210 extracts global features for each frame image in the sample, the key point detection unit 2201 detects the key points of the human in each frame image, and the local feature extraction unit 2202 extracts the local features near the key points in each frame image. As described above, 13 local features and one global feature are obtained for each frame image. Therefore, 39 local features and 3 global features can be obtained for a sample.

[0073] Assuming that the dimension of each feature is 2048, the construction unit 250 can construct all the features into a feature matrix X with a size of 42x2048. The 1st to 13th rows, the 14th to 26th rows, and the 27th to 39th rows of the matrix are the local features of the first, second, and third frame images, respectively, and the 40th, 41st, and 42nd rows of the matrix are the global features of the first, second, and third frame images, respectively. Each row of the feature matrix X is a node in the graph convolution network, the i-th row is the i-th node, and there are 42 nodes in total.

[0074] Further, the first computing unit 230 can generate a mask matrix M with a size of 42x42. In the definition of the example of the present application, if there is information transmission from node j to node i, M(i,j) = 1, otherwise M(i,j) = 0. In the same frame image, the manner of information transmission between local features of key points is determined by the human skeletal structure. Information transmission can be performed between local features of key points in adjacent positions, and not performed if the key points are far away in the human skeletal structure. In addition, information transmission also needs to be performed between global features and local features in the same frame, between global features in different frames, and between local features in different frames.

[0075] Figure 5 The manner of information transmission between nodes of three frames of images is shown. As shown in Figure 5 , in the case of three frames, in a single image, the manner of information transmission is the same as described above in connection with Figure 3 . Between different images, information is transmitted between global features of three frames, for example, information can be transmitted between global_1, global_2 and global_3. In addition, information is transmitted between local features of corresponding key points of different images, for example, transmitted between local 1_1, local 2_1, local 3_1. In addition, information can be transmitted between local features corresponding to adjacent key points of different images, for example, transmitted between local 1_i and local 2_k (i and k represent natural numbers between 1 and K), and transmitted between local 1_i and local 3_k. It should be noted that not all information transmissions are necessarily shown in order to avoid making the figure unclear.

[0076] Thus, for each image, the mask matrix M can define information transmission between global features and all local features of the image, and between local features corresponding to adjacent key points, and for different images, the mask matrix M defines information transmission between local features corresponding to corresponding key points of different images, between global features of different images, and between local features corresponding to adjacent key points of different images.

[0077] Figure 6 A schematic diagram of the mask matrix M according to an embodiment of the present disclosure is shown. In Figure 6 , the numbers in the vertical direction and the numbers in the horizontal direction represent node numbers 0, 1, …, 41, respectively. Figure 6 The black part in the

[0078] As Figure 6As shown, when 0≤i≤38, 0≤j≤38 ( Figure 6 In box A, M(i,j) = M(j,i), meaning the information transfer between the two local features is bidirectional. The diagonal lines marked with dashed lines in box A represent the information transfer within the local features of each of the three frames: the region 0≤i≤12, 0≤j≤12 corresponds to the information transfer within the local features of the first frame; the region 13≤i≤25, 13≤j≤25 corresponds to the information transfer within the local features of the second frame; and the region 26≤i≤38, 26≤j≤38 corresponds to the information transfer within the local features of the third frame. Furthermore, the region 0≤i≤12, 13≤j≤25 corresponds to the information transfer from the local features of the second frame to the local features of the first frame. Similarly, the meaning of the information transfer represented by the other regions in box A can be understood.

[0079] Figure 6 The region defined by the middle frame B determines whether information is passed between global features. For example... Figure 6 As shown, there is information transfer between the three global features. The area above and to the left of box B indicates whether there is information transfer between the global and local features. Figure 6 As shown, the area above the vertical box B indicates that the global features of each frame of the image pass information to all its local features, while the area to the left of the horizontal box B is all zero, which indicates that the local features do not pass information to the global features.

[0080] Although the details of the mask matrix M in this application have been described above, this is only one example of a mask matrix, and the specific form of the mask matrix is ​​not limited to this. Figure 6 The form shown.

[0081] Furthermore, the first computing unit 230 can compute the adjacency matrix A based on the mask matrix M and the feature matrix X. Let f i Let i = 0, 1, 2, ..., 41 be the feature vector of the i-th row in the feature matrix X. The distance d between each local feature and the global feature of the frame containing that local feature is calculated using equations (5) to (7). i d at each key point i It is the absolute value of the difference between the local features of each key point and the corresponding global features.

[0082] d i =abs(f i -f 39 ), i = 0, 1, 2, ... 12 (5)

[0083] d i =abs(f i -f 40 ), i = 13, 14, 15, ... 25 (6)

[0084] d i =abs(f i -f 41 ), i = 26, 28, 29, ... 38 (7)

[0085] The first computing unit 230 will d i Mapped to D via a fully connected network i The d here i As an example of an intermediate feature in this application, D i This is an example of an intermediate value in this application. d i The size is 1×2048, D i Let D be a real number, and all D i The sum of (0≤i≤38) is the hyperparameter.

[0086] If M(i,j)=0, then A(i,j)=0, which means that node j has no information to pass on to node i.

[0087] If M(i,j)=1 and 0≤i,j≤38, then A(i,j) is calculated by equation (8).

[0088] A(i,j)=ReLU(1+D i -D j )×(D i +D j (8)

[0089] Equation (8) represents the information transfer metric between local features.

[0090] If M(i,j)=1 and 0≤i≤38,39≤j≤41, then define:

[0091] A(i,j)=2 D i (9)

[0092] Equation (9) represents the transfer metric between global and local features. It can be seen that, in the adjacency matrix, the amount of information transferred from global features to local features corresponding to different keypoints is defined as the intermediate value D corresponding to the keypoint. i Twice as much.

[0093] If M(i,j)=1 and 39≤i,j≤41, then define:

[0094] A(i,j)=0.5 (10)

[0095] Equation (10) represents the transitivity metric between global features. It should be noted that, for example... Figure 2 As shown in box B, global features do not pass information to themselves.

[0096] Thus, the first computing unit 230 can obtain the adjacency matrix A. Finally, the first computing unit 230 normalizes the adjacency matrix A by row, so that the sum of the elements of each row is 1.

[0097] Further, the second computing unit 240 inputs the feature matrix X into the graph convolution network, and obtains the improved local features and global features by the following formula (11).

[0098] X_improved = ReLU(fc(AX)) + X (11)

[0099] where fc(·) is a fully connected network.

[0100] In the above example, the sample includes three frames of images, while in the case where the sample includes more frames of images (assuming the number of images is N), the constructing unit 250 can construct a feature matrix including global features and local features of N images. In the case where the specific object in the image is a human body, the size of the feature matrix is (14*N) x feature length (e.g. 2048).

[0101] Similarly, the first computing unit 230 can define a mask matrix with a size of (14*N) x (14*N), and calculate the corresponding adjacency matrix A. The difference is that when calculating the information transmission measure between global features, the amount of information transmission between global features of different images is defined as 1 / (N-1). Other calculations are similar to the case of three frames.

[0102] The corresponding features representing the local features and global features of the multi-frame images can be obtained by averaging the features obtained in the case of multi-frame images, etc.

[0103] Therefore, in the present application, processing a single frame of image can be extended to processing multiple frames of images. A graph convolution network capable of effectively transmitting information in the spatial dimension and in the temporal dimension is designed. The transmission between nodes includes the transmission between local features of the same frame of image, the transmission between local features and global features of the same frame of image, the transmission between local features of different key points of different frames of image, the transmission between local features of different key points of different frames of image, the transmission between global features of different frames of image, etc. The transmission measure between nodes is learned by the network according to the relationship between each feature.

[0104] Since there can be different occlusion patterns and more abundant information in multi-frame images, the accuracy of the re-identification result can be effectively improved.

[0105] <Matching and sorting>

[0106] In the above manner, optimized local features and global features can be obtained for a single image or multiple frames of images.

[0107] The above method can be performed on a sample to be identified or a sample to be searched. Here, the sample to be searched refers to a sample in which a specific object to be identified in re-identification is known to exist, and the sample to be identified refers to a sample including a plurality of objects to be identified. Both the sample to be searched and the sample to be identified can be a single image or a plurality of images. The above-described processing is performed on the sample to be searched, thereby obtaining improved local features and global features of the sample to be searched with respect to the specific object to be searched. The above-described processing is performed on the sample to be identified, thereby obtaining improved local features and global features of the sample to be identified with respect to each of the plurality of objects included therein. Similarity matching and ranking between the sample to be identified and the sample to be searched are performed using the improved features, and a re-identification result is obtained, that is, which object in the sample to be identified is the specific object to be searched is found.

[0108] This is achieved by the matching and ranking unit 260 in FIG. 26. That is, the matching and ranking unit 260 compares the optimized local features or the optimized local features and global features of the sample to be identified and the sample to be searched, and ranks the comparison results, thereby determining which object in the sample to be identified is the specific object to be searched in the sample to be searched. Figure 7

[0109] For example, in the case of searching for a single image, it can be considered that the image is a first image and a second image from different camera sources. For example, the first image is a search image in which a specific object to be searched is designated, and the second image is an identification image including a plurality of objects. Optimized feature matrices can be obtained for the specific object in the first image and each of the plurality of objects in the second image, respectively, using the above-described method, thereby obtaining optimized local features and global features. The matching and ranking unit 260 determines which object in the second image is the specific object in the first image by comparing the optimized local features or the optimized local features and global features of the first image and the second image.

[0110] Here, the optimized features of each object can be obtained by obtaining an image related to each of the plurality of objects from the second image (a plurality of second images in which different objects are marked, respectively), and then optimizing the features with respect to the marked object in each second image.

[0111] ​For example, when searching for samples comprising multiple frames of images, the images can be considered to include multiple images (first samples) of consecutive frames from a first video source from a first camera source and multiple images (second samples) of consecutive frames from a second video source from a second camera source. For example, the first sample is a search sample specifying a particular object to be searched, while the second sample is a sample to be identified comprising multiple objects. The optimized feature matrix can be obtained for each object in the first sample and each of the multiple objects in the second sample using the method described above. The matching and ranking unit 260 determines which object in the second sample is the specific object in the first sample by comparing the optimized local features of the first sample and the optimized local features and global features of the second sample.

[0112] Specifically, the object tracking method can be used to obtain samples related to each of the multiple objects from the second samples (multiple second samples of different objects are labeled respectively), and then the features of the labeled objects in each second sample are optimized to obtain the optimized features of each object.

[0113] Furthermore, it should be noted that the specific object to be searched in the sample to be searched can be multiple objects. In this case, it is only necessary to perform the above processing on each of the specific objects in the sample to be searched to determine which object in the sample to be identified is the specific object to be searched. In addition, although the examples described above depict the case where the sample to be searched and the sample to be identified are both single images or multiple frames, it is understood that this application aims to optimize local and global features for single images or multiple frames. Therefore, the application examples are not limited to the above cases. For example, it is also applicable to the case where the sample to be searched is a single image and the sample to be identified is multiple frames.

[0114] Furthermore, without specifying a particular object in the sample to be searched, features can be optimized separately for multiple objects in the sample to be searched and the sample to be identified, and then matched and sorted to determine whether the two include the same object.

[0115] The following is combined with Figure 7 To describe an image processing method according to embodiments of the present disclosure.

[0116] like Figure 8 As shown, the image processing method according to an embodiment of the present disclosure begins at step S110. In step S110, global features of the image are extracted.

[0117] Next, in step S120, key points related to specific objects in the image are detected, and local features of the image corresponding to the key points are extracted.

[0118] Next, in step S130, an information passing measure for the graph convolution network is calculated, the nodes of the information passing measure including global features and local features corresponding to the key points. Calculating the information passing measure includes defining a mask matrix that defines whether information is passed between different nodes, and calculating an adjacency matrix that defines an amount of information passed between different nodes. Elements of the adjacency matrix can be calculated based on the local features and the global features.

[0119] Next, in step S140, the local features of the image are optimized by the graph convolution network using the information passing measure. After this, the process ends.

[0120] According to an embodiment of the present disclosure, calculating the adjacency matrix includes, for each key point, calculating an intermediate feature corresponding to the key point based on the local feature and the global feature, mapping the intermediate feature of each key point to an intermediate value respectively through a fully connected network, and calculating the adjacency matrix based on the intermediate value corresponding to each key point. Specifically, the intermediate feature of each key point is an absolute value of a difference between the local feature of each key point and the corresponding global feature. In the adjacency matrix, an amount of information passing from the global feature to the local feature corresponding to different key points is defined as twice the intermediate value corresponding to the key point.

[0121] According to an embodiment of the present disclosure, the method further includes constructing a feature matrix including the local features and the global features, normalizing the adjacency matrix, and performing a matrix multiplication operation on the adjacency matrix and the feature matrix, and inputting an operation result into the fully connected network to obtain an optimized feature matrix, thereby obtaining the optimized local features and the global features.

[0122] According to an embodiment of the present disclosure, in a case where the image is a single image, the mask matrix defines that information is passed between the global feature of the image and all the local features, and between the local features corresponding to adjacent key points.

[0123] According to an embodiment of the present disclosure, in a case where the image is a plurality of images that are consecutive multiple frames of a first video from a first camera source, the nodes of the information passing measure include the global feature of each of the plurality of images and the local feature corresponding to the key point of each of the plurality of images.

[0124] Specifically, for each image, the mask matrix defines that information is passed between the global feature of the image and all the local features, and between the local features corresponding to adjacent key points, and for different images, the mask matrix defines that information is passed between the local features corresponding to the corresponding key points of different images, between the global features of different images, and between the local features corresponding to adjacent key points of different images.

[0125] Thus, the image processing method according to the embodiments of the present disclosure can optimize the local features and the global features using the graph convolution network by using both the global features and the local features as the nodes of the graph convolution network and reasonably designing the adjacency matrix.

[0126] The image processing method according to another embodiment of the present disclosure will be described below in combination with Figure 8 FIG. 1.

[0127] As Figure 9 shown in FIG. 1, the image processing method according to the embodiments of the present disclosure starts from step S210. In step S210, samples of objects to be recognized are acquired.

[0128] Next, in step S220, the global features of each frame of images in the samples are extracted.

[0129] Next, in step S230, the local features near the key points of each frame of images in the samples are extracted.

[0130] Next, in step S240, all the local features and the global features of the samples are constituted into a feature matrix X, and the feature matrix is input into the graph convolution network to obtain improved global features and local features.

[0131] Next, in step S250, the samples of objects to be searched are subjected to the same processing method as the samples of objects to be recognized to obtain improved global features and local features.

[0132] Next, in step S260, similarity matching and sorting between the samples to be recognized and the samples to be searched are performed using the improved features to obtain a re-identification result. After that, the process ends.

[0133] Thus, the image processing method according to the embodiments of the present disclosure can optimize the local features and the global features using the graph convolution network by using both the global features and the local features as the nodes of the graph convolution network and reasonably designing the adjacency matrix, thereby providing more optimal comparison features for re-identification and further improving the accuracy of the re-identification result.

[0134] The various specific implementations of the above steps of the image processing method according to the embodiments of the present disclosure have been described in detail above, and will not be repeated here.

[0135] Obviously, each operation process of the image processing method according to the present disclosure can be implemented in the form of a computer executable program stored in various machine-readable storage media.

[0136] Furthermore, the objective of this disclosure can also be achieved by providing a storage medium storing the aforementioned executable program code directly or indirectly to a system or device, and having a computer or central processing unit (CPU) in the system or device read and execute the aforementioned program code. In this case, as long as the system or device has the function of executing a program, the implementation of this disclosure is not limited to a program, and the program can be in any form, such as an object program, a program executed by an interpreter, or a script program provided to an operating system.

[0137] The aforementioned machine-readable storage media include, but are not limited to: various memories and storage units, semiconductor devices, disk units such as optical, magnetic and magneto-optical disks, and other media suitable for storing information.

[0138] Alternatively, the technical solution of this disclosure can also be implemented by connecting to a corresponding website on the Internet, downloading and installing the computer program code according to this disclosure onto the computer, and then executing the program.

[0139] Figure 9 This is a block diagram of an exemplary structure of a general-purpose personal computer in which image processing apparatus and methods according to embodiments of the present disclosure can be implemented.

[0140] like Figure 9 As shown, CPU 1301 executes various processes based on programs stored in read-only memory (ROM) 1302 or programs loaded into random access memory (RAM) 1303 from storage section 1308. RAM 1303 also stores data required as needed when CPU 1301 executes various processes, etc. CPU 1301, ROM 1302, and RAM 1303 are interconnected via bus 1304. Input / output interface 1305 is also connected to bus 1304.

[0141] The following components are connected to the input / output interface 1305: input section 1306 (including keyboard, mouse, etc.), output section 1307 (including display, such as cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.), storage section 1308 (including hard disk, etc.), and communication section 1309 (including network interface card, such as LAN card, modem, etc.). The communication section 1309 performs communication processing via a network, such as the Internet. Drive 1310 may also be connected to the input / output interface 1305 as needed. Removable media 1311, such as disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1310 as needed, so that computer programs read from them can be installed into storage section 1308 as needed.

[0142] In a case where the above series of processes are implemented by software, a program constituting the software is installed from a network such as the Internet or a storage medium such as the detachable medium 1311.

[0143] Those skilled in the art will appreciate that such storage media are not limited to the ​ The detachable medium 1311 shown is one in which a program is stored, and is distributed separately from the apparatus to provide the program to the user. Examples of the detachable medium 1311 include a magnetic disk (including a floppy disk (registered trademark)), a compact disc (including a compact disc read only memory (CD-ROM) and a digital versatile disc (DVD)), a magneto-optical disk (including a mini disk (MD) (registered trademark)), and a semiconductor memory. Alternatively, the storage medium can be a ROM 1302, a hard disk included in the storage section 1308, or the like, in which a program is stored, and which is distributed to the user together with the apparatus including them.

[0144] In the system and the method of the present disclosure, it is apparent that each component or each step can be decomposed and / or recombined. These decompositions and / or recombination should be considered as equivalents of the present disclosure. Also, the steps of performing the above series of processes can naturally be executed in a time series in the order of explanation, but do not necessarily have to be executed in a time series. Some steps can be executed in parallel or independently of each other.

[0145] Although the embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, it is to be understood that the implementations described above are merely for the purpose of illustration of the present disclosure and do not constitute a limitation thereon. Various modifications and changes can be made to the implementations described above without departing from the essence and scope of the present disclosure. Therefore, the scope of the present disclosure is to be limited only by the appended claims and their equivalents.

[0146] With respect to the implementations including the above embodiments, the following supplementary notes are also disclosed:

[0147] Supplementary note 1. An image processing apparatus comprising:

[0148] a first extraction unit that extracts a global feature of an image;

[0149] a second extraction unit that detects a key point related to a specific object in the image and extracts a local feature of the image corresponding to the key point;

[0150] a first calculation unit that calculates an information passing measure for a graph convolutional network, nodes of the information passing measure including the global feature and the local feature corresponding to the key point; and

[0151] a second calculation unit that optimizes the local feature of the image by a graph convolutional network using the information passing measure,

[0152] wherein the first computing unit computes the information passing measure by:

[0153] defining a mask matrix defining whether information is passed between different nodes; and

[0154] computing an adjacency matrix defining an amount of information passed between different nodes.

[0155] Note 2. The image processing apparatus according to Note 1, wherein the first computing unit computes elements of the adjacency matrix based on the local features and the global features.

[0156] Note 3. The image processing apparatus according to Note 2, wherein the first computing unit computes the adjacency matrix by:

[0157] computing, for each key point, an intermediate feature corresponding to the key point based on the local feature and the global feature of the key point;

[0158] mapping the intermediate feature of each key point to an intermediate value by a fully connected network; and

[0159] computing the adjacency matrix based on the intermediate value corresponding to each key point.

[0160] Note 4. The image processing apparatus according to Note 3, wherein the intermediate feature of each key point is an absolute value of a difference between the local feature of the key point and the corresponding global feature.

[0161] Note 5. The image processing apparatus according to Note 3, wherein the first computing unit defines, in the adjacency matrix, an amount of information passed from a global feature to local features corresponding to different key points as twice the intermediate value corresponding to the key point.

[0162] Note 6. The image processing apparatus according to Note 3, further comprising a constructing unit configured to construct a feature matrix including the local features and the global features,

[0163] wherein the first computing unit computes the adjacency matrix based on the feature matrix and the mask matrix, and normalizes the computed adjacency matrix, and

[0164] the second computing unit performs a matrix multiplication operation on the adjacency matrix and the feature matrix, and inputs a result of the operation to a fully connected network to obtain an optimized feature matrix, thereby obtaining optimized local features and global features.

[0165] Note 7. The image processing apparatus according to any one of Notes 1 to 6, wherein the image is a single image, and the mask matrix defines information passing between the global feature of the image and all local features, and between local features corresponding to neighboring key points.

[0166] Note 8. The image processing apparatus according to any one of Notes 1 to 6, wherein the image is a plurality of images of consecutive multiple frames of a first video from a first camera source, and the nodes of the information passing measure include a global feature of each of the plurality of images and a local feature of each of the plurality of images corresponding to a key point.

[0167] Note 9. An image processing method comprising:

[0168] extracting a global feature of an image;

[0169] detecting a key point related to a specific object in the image, and extracting a local feature of the image corresponding to the key point;

[0170] computing an information passing measure for a graph convolution network, nodes of the information passing measure including the global feature and the local feature corresponding to the key point; and

[0171] optimizing local features of the image by a graph convolution network with the information passing measure,

[0172] wherein computing the information passing measure comprises:

[0173] defining a mask matrix defining whether information passes between different nodes; and

[0174] computing an adjacency matrix defining an amount of information passing between different nodes.

[0175] Note 10. The method according to Note 9, wherein elements of the adjacency matrix are computed based on the local feature and the global feature.

[0176] Note 11. The method according to Note 10, wherein computing the adjacency matrix comprises:

[0177] for each key point, computing an intermediate feature corresponding to the key point based on the local feature and the global feature;

[0178] mapping the intermediate feature of each key point to an intermediate value respectively by a fully connected network; and

[0179] computing the adjacency matrix based on the intermediate value corresponding to each key point.

[0180] Note 12. The method of note 11, wherein the intermediate feature of each key point is an absolute value of a difference between the local feature of the key point and the corresponding global feature.

[0181] Note 13. The method of note 11, wherein in the adjacency matrix, an amount of information transferred from a global feature to a local feature of a corresponding different key point is defined as twice the intermediate value corresponding to the key point pair.

[0182] Note 14. The method of note 11, further comprising:

[0183] constructing a feature matrix comprising the local features and the global features;

[0184] calculating the adjacency matrix based on the feature matrix and the mask matrix, and normalizing the adjacency matrix; and

[0185] performing a matrix multiplication operation on the adjacency matrix and the feature matrix, and inputting a result of the operation into a fully connected network to obtain an optimized feature matrix, thereby obtaining optimized local features and global features.

[0186] Note 15. The method of any one of notes 9 to 14, wherein the image is a single image, and the mask matrix defines information transfer between a global feature of the image and all local features, and between local features corresponding to neighboring key points.

[0187] Note 16. The method of any one of notes 9 to 14, wherein the image is a plurality of images of consecutive multiple frames of a first video from a first camera source, and the nodes of the information transfer measure include a global feature of each of the plurality of images and a local feature corresponding to a key point of each of the plurality of images.

[0188] Note 17. The method of note 16, wherein,

[0189] for each image, the mask matrix defines information transfer between a global feature of the image and all local features, and between local features corresponding to neighboring key points, and

[0190] for different images, the mask matrix defines information transfer between local features corresponding to corresponding key points of different images, between global features of different images, and between local features corresponding to neighboring key points of different images.

[0191] Note 18. The method of note 16, wherein a number of the plurality of images is N, and in the adjacency matrix, an amount of information transfer between global features of different images is defined as 1 / (N-1).

[0192] Paragraph 19. The method according to paragraph 9, wherein the specific object is partially occluded in the image.

[0193] Paragraph 20. A machine-readable storage medium carrying a program product comprising machine-readable instruction codes stored therein, wherein the instruction codes, when read and executed by a computer, are capable of causing the computer to perform the image processing method according to paragraphs 9-19.

Claims

1. An image processing apparatus, comprising: The first extraction unit extracts global features of the image; The second extraction unit detects key points related to a specific object in the image and extracts local features of the image corresponding to the key points. The first computing unit calculates the information transfer metric for the graph convolutional network, wherein the nodes of the information transfer metric include the global features and the local features corresponding to the key points; as well as The second computational unit utilizes the information transfer metric to optimize local features of the image through a graph convolutional network. The first computing unit calculates the information transmission metric through the following operations: Define a mask matrix, which defines whether information is passed between different nodes; and Calculate the adjacency matrix, which defines the amount of information transferred between different nodes. The first calculation unit calculates the elements of the adjacency matrix based on the local features and the global features.

2. The image processing apparatus according to claim 1, wherein, The first computing unit calculates the adjacency matrix through the following operations: For each key point, calculate the intermediate features corresponding to the key point based on the local features and the global features; The intermediate features of each key point are mapped to intermediate values ​​through a fully connected network; as well as The adjacency matrix is ​​calculated based on the intermediate value corresponding to each key point.

3. The image processing apparatus according to claim 2, wherein, The intermediate features of each key point are the absolute values ​​of the differences between the local features of each key point and the corresponding global features.

4. The image processing apparatus according to claim 2, wherein, In the adjacency matrix, the first calculation unit defines the amount of information transferred from global features to local features corresponding to different key points as twice the intermediate value corresponding to the key point.

5. The image processing apparatus according to claim 2, further comprising: The construction unit constructs a feature matrix including the local features and the global features. The first calculation unit calculates the adjacency matrix based on the feature matrix and the mask matrix, and normalizes the calculated adjacency matrix. The second computing unit performs matrix multiplication on the adjacency matrix and the feature matrix, and inputs the result into the fully connected network to obtain the optimized feature matrix, thereby obtaining optimized local features and global features.

6. The image processing apparatus according to any one of claims 1 to 5, wherein, The image is a single image, and the mask matrix definition information is passed between the global features and all local features of the image, as well as between the local features corresponding to neighboring keypoints.

7. The image processing apparatus according to any one of claims 1 to 5, wherein, The images are multiple images from a first video source with multiple consecutive frames, and the nodes of the information transmission metric include global features of each of the multiple images and local features of each of the multiple images corresponding to key points.

8. An image processing method, comprising: Extract global features from the image; Detect key points related to a specific object in the image, and extract local features of the image corresponding to the key points; Calculate the information transfer metric for the graph convolutional network, wherein the nodes of the information transfer metric include the global features and the local features corresponding to the key points; as well as The information transfer metric is used to optimize local features of the image through a graph convolutional network. The calculation of the information transmission metric includes: Define a mask matrix, which defines whether information is passed between different nodes; and Calculate the adjacency matrix, which defines the amount of information transferred between different nodes. The elements of the adjacency matrix are calculated based on the local features and the global features.

9. A machine-readable storage medium having thereon carrying a program product including machine-readable instruction code stored thereon, wherein, When the instruction code is read and executed by a computer, it enables the computer to perform the image processing method according to claim 8.

Images