Object Location and Retrieval Method Based on End-to-End Network and Related Devices

Abstract

The present application discloses an object positioning and retrieval method and related devices based on an end-to-end network. The method includes: inputting a target image into a first feature extraction layer for feature extraction to obtain a first feature image; inputting the first feature image into a convolutional layer to obtain a second feature image; inputting the second feature image into an object information layer to obtain an object information feature map; inputting the object information feature map into a data analysis layer to parse out the type, position, and object weight feature of the object, constructing the object weight feature into a two-dimensional weight matrix, compressing the ROI image where the object position is located to a fixed size and performing convolution calculation with the two-dimensional weight matrix to obtain an object REID feature map; inputting it into a second feature extraction layer to obtain a third feature image; inputting the third feature image into a first fully connected layer, a second fully connected layer, and a first normalization layer to obtain an object re-identification feature. The embodiments of the present application can achieve efficient object positioning and retrieval.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an object positioning and retrieval method based on an end-to-end network and related equipment. Background Art

[0002] Target retrieval is a kind of image segmentation based on the geometric and statistical features of the target. It combines the segmentation and recognition of the target. Its accuracy and real-time performance are important capabilities of the entire system. Object retrieval, as an application of target retrieval, is also gaining more and more attention in practical applications. For example, common objects include hats, tops, backpacks, logos, pants, handbags, suitcases, shoes, etc. At present, the problem of how to achieve efficient object retrieval needs to be solved urgently. Summary of the invention

[0003] The embodiments of the present application provide an object positioning and retrieval method and related equipment based on an end-to-end network, which can realize efficient object retrieval.

[0004] In a first aspect, an embodiment of the present application provides an object retrieval method, the method comprising:

[0005] Get the target image;

[0006] Inputting the target image into a first feature extraction layer for feature extraction to obtain a first feature image;

[0007] Inputting the first feature image into a convolutional layer to obtain a second feature image;

[0008] Inputting the second feature image into the object information layer to obtain an object information feature map, wherein the object information feature map includes: type, position, and object weight features of the object;

[0009] Input the object information feature map into the data analysis layer, parse out the type, location and object weight features of the object, construct the object weight features into a two-dimensional weight matrix, compress the ROI image where the object is located to a fixed size and perform convolution calculation with the two-dimensional weight matrix to obtain the object REID feature map;

[0010] Inputting the object REID feature map into the second feature extraction layer to obtain a third feature image;

[0011] The third feature image input is sequentially input into the first fully connected layer, the second fully connected layer and the first normalization layer to obtain the object re-identification feature after compressing the feature dimension.

[0012] In a second aspect, an embodiment of the present application provides an object retrieval device, the device comprising: an acquisition unit, a first extraction unit, an analysis unit, a second extraction unit and an output unit, wherein:

[0013] The obtaining unit is configured to obtain a target image;

[0014] The first extraction unit is configured to input the second feature image into the object information layer to obtain an object information feature map, where the object information feature map includes: the type, position, and object weight feature of the object;

[0015] The analysis unit is configured to input the object information feature map into the data analysis layer, parse out the type, position, and object weight feature of the object, construct the object weight feature into a two-dimensional weight matrix, compress the ROI image where the object position is located to a fixed size and perform convolution calculation with the two-dimensional weight matrix to obtain an object REID feature map;

[0016] The second extraction unit is configured to input the object REID feature map into the second feature extraction layer to obtain a third feature image;

[0017] The output unit is configured to sequentially input the third feature image into a first fully connected layer, a second fully connected layer, and a first normalization layer to obtain an object re-identification feature after compressing the feature dimension.

[0018] In a third aspect, an embodiment of the present application provides an electronic device, including a processor, a memory, a communication interface, and one or more programs, where the above one or more programs are stored in the above memory and are configured to be executed by the above processor, and the above programs include instructions for performing the steps in the first aspect of the embodiment of the present application.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, where the above computer-readable storage medium stores a computer program for electronic data exchange, and the above computer program enables a computer to execute some or all of the steps described in the first aspect of the embodiment of the present application.

[0020] In a fifth aspect, an embodiment of the present application provides a computer program product, where the above computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the above computer program is operable to enable a computer to execute some or all of the steps described in the first aspect of the embodiment of the present application. The computer program product may be a software installation package.

[0021] Implementing the embodiments of the present application has the following beneficial effects:

[0022] It can be seen that in the embodiments of the present application, for the object positioning and retrieval method and related devices based on the end-to-end network, a target image is obtained, and the target image is input into the first feature extraction layer for feature extraction to obtain a first feature image. The first feature image is input into the convolutional layer to obtain a second feature image. The second feature image is input into the object information layer to obtain an object information feature map. The object information feature map includes: the type, location, and object weight feature of the object. The object information feature map is input into the data analysis layer to parse out the type, location, and object weight feature of the object. The object weight feature is constructed into a two-dimensional weight matrix. The ROI image where the object is located is compressed to a fixed size and convolved with the two-dimensional weight matrix to obtain an object REID feature map. The object REID feature map is input into the second feature extraction layer to obtain a third feature image. The third feature image is sequentially input into the first fully connected layer, the second fully connected layer, and the first normalization layer to obtain the object re-identification feature after compressing the feature dimension. On the one hand, since a linear model is adopted and the end-to-end network is used in both feature retrieval and object re-identification feature extraction, the object retrieval efficiency can be improved. On the other hand, two fully connected layers are used to achieve dimensionality reduction, which can achieve a faster dimensionality reduction speed and also has higher accuracy. Therefore, the present application can achieve efficient object positioning and retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0024] Figure 1A is a schematic flowchart of an object retrieval method provided by an embodiment of the present application;

[0025] Figure 1B is a schematic structural diagram of an object retrieval model provided by an embodiment of the present application;

[0026] Figure 1C is a schematic structural diagram of another object retrieval model provided by an embodiment of the present application;

[0027] Figure 1D is a schematic structural diagram of a feature extraction network provided by an embodiment of the present application;

[0028] Figure 1E is a schematic structural diagram of another feature extraction network provided by an embodiment of the present application;

[0029] Figure 1FIt is a schematic structural diagram of another feature extraction network provided by an embodiment of the present application;

[0030] Figure 1G It is a schematic structural diagram of another feature extraction network provided by an embodiment of the present application;

[0031] Figure 1H It is a schematic structural diagram of another feature extraction network provided by an embodiment of the present application;

[0032] Figure 1I It is a schematic structural diagram of another feature extraction network provided by an embodiment of the present application;

[0033] Figure 2 It is a schematic flowchart of another object retrieval method provided by an embodiment of the present application;

[0034] Figure 3 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

[0035] Figure 4 It is a block diagram of the functional units of an object retrieval device provided by an embodiment of the present application. Detailed implementation manners

[0036] In order to enable those skilled in the art to better understand the solution of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art without making creative efforts based on the embodiments in the present application belong to the scope of protection of the present application.

[0037] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally further include steps or units not listed, or may optionally further include other steps or units inherent to these processes, methods, products, or devices.

[0038] Referring to "embodiment" in this article means that a specific feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present application. The phrase appears in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein may be combined with other embodiments.

[0039] The electronic devices described in the embodiments of the present application may include smart phones (such as Android phones, iOS phones, Windows Phone phones, etc.), tablet computers, handheld computers, driving recorders, traffic command platforms, servers, laptop computers, mobile Internet devices (MIDs), or wearable devices (such as smart watches, Bluetooth headsets), etc. The above are only examples, not an exhaustive list, including but not limited to the above electronic devices.

[0040] The embodiments of the present application will be introduced in detail below.

[0041] Please refer to Figure 1A , Figure 1A which is a schematic flow chart of an object retrieval method provided by the embodiments of the present application. As shown in the figure, the object retrieval method includes:

[0042] 101. Obtain a target image.

[0043] Among them, in the embodiments of the present application, an object is an item. For example, an object may include at least one of the following: hat, top, backpack, button, logo, pants, handbag, suitcase, shoes, socks, mask, glasses, wallet, and so on. The target image may be one or more images, that is, to realize object retrieval for a single image, or to realize batch object retrieval.

[0044] In a specific implementation, the target image may be a pedestrian image, or may be any image including an object. In the embodiments of the present application, it can be applied to an electronic device, and the electronic device may include an object recognition model, as Figure 1B shown, the object recognition model may include: an input layer, a first feature extraction layer, a convolutional layer, an object information layer, a data analysis layer, a second feature extraction layer, a first fully connected layer, a second fully connected layer, a first normalization layer, and an output layer. Based on the above object recognition model, the object type, object coordinates can be retrieved on the pedestrian image, and at the same time, the object (person re-identification, REID) feature is output, and similar targets are searched according to the combination of the object retrieval probability and the REID feature.

[0045] The network of the above object recognition model can adopt a multi-scale and multi-branch structure. After the network is deepened and widened, the accuracy will be improved, but the performance will decline. Therefore, the training network and the inference network can be separated. As Figure 1C shown, the training network can be a multi-scale and multi-branch network. Through network fusion, the multi-scale and multi-branch network is fused into a linear network to obtain the object recognition model as Figure 1B shown, so as to ensure the accuracy of the network and improve the performance at the same time.

[0046] In the application stage, i.e., the inference process, the network of the object recognition model is an end-to-end network that combines the previous target retrieval network and the REID feature extraction network to simultaneously output the type, location, and REID features of the object. The network structure is as Figure 1B shown.

[0047] Among them, in the first feature extraction layer, the network structures in the training process and the inference process can be different. According to Figures 1D - 1I shown, the corresponding network can be selected during the training and inference processes. The first feature extraction layer can be composed of Figures 1D - 1I three network structures in multiple combinations to adapt to different hardware.

[0048] Furthermore, when inputting a pedestrian image into the input layer, in the location type REID layer in the network, the positions, types, and corresponding preliminary REID features of hats, coats, backpacks, logos, trousers, handbags, suitcases, shoes, etc. existing in the pedestrian image can be output. Then, in the data analysis layer, according to the output type and location, the original image coordinates can be located first, the region of interest roi can be intercepted, and then the image is compressed, convolved with the output preliminary REID features, and an activation function is used to obtain the REID feature map. Then, it is further optimized through the second feature extraction layer, and finally, the network is used to compress the REID feature dimension.

[0049] Furthermore, the network layer can be used to compress the feature length. As Figure 1B shown, the second feature extraction layer is connected to a fully connected layer, and then, another fully connected layer is connected, and the REID feature length is compressed through the network.

[0050] In the training stage, as Figure 1C shown, the training object REID feature network adopts a three-level network loss function. The first level is the preliminary REID features output by the location type REID layer, and the REID features are optimized according to loss function 1; the second level is the REID features output by the first fully connected layer, and they are optimized according to loss function 2; the third level is the REID features output by the second fully connected layer, and they are optimized according to loss function 3. The network structure is as Figure 1C shown. There are two branches behind fc512. One is connected to a bn layer and then to a loss function; the other branch connects from the 512-dimensional layer to a 128-dimensional layer, then to a bn layer and then to another loss function, and only the 128-dimensional features are used for inference.

[0051] The training network can adopt a multi-branch multi-scale network. The inference network fuses the network branches into a linear model. There are two strategies for network fusion. The first is to fuse the conv and bn layers, and the second is to merge the network layers with the same channels and sizes together.

[0052] As shown Figure 1C in the figure, the sample image (pedestrian image) is input into the input layer. In the pedestrian image, the positions and types of the hat, upper garment, backpack, logo, trousers, handbag, suitcase, shoes, etc. are marked. The pedestrian data is enhanced, the image is compressed to a fixed size, and then input into the network. In the position type REID layer, the position, type, and preliminary REID features of each point are output. The loss function of the position and type is calculated to optimize the type and position; the loss function of the preliminary REID features is calculated to optimize the preliminary REID features. Then the network is connected to the data analysis layer. The roi is obtained according to the coordinates, convolved with the preliminary REID features, passed through the activation function, and the REID feature map is output. The feature map is input into the fc512 layer through the feature extraction layer and divided into two branches. One branch is connected to the bn layer and the loss function to optimize the 512-dimensional REID features; the other branch is connected to the fc128 layer, then to the bn layer and the loss function to optimize the 128-dimensional REID features. Only the 128-dimensional features are used for searching during inference.

[0053] 102. Input the target image into the first feature extraction layer for feature extraction to obtain the first feature image.

[0054] Among them, the target image can be input into the first feature extraction layer for feature extraction to obtain the first feature image. The specific structure of the first feature extraction layer can be as Figures 1D - 1I shown. This three-group network structure combines into a large network structure. Generally, there are two strategies for network fusion. The first is to fuse the convolutional layer conv and the normalization layer bn layer; the second is to merge the network layers with the same channels and sizes together. As Figures 1D - 1E shown, it can fuse the convolutional layer conv and the normalization layer bn layer. Figure 1D is the structure before fusion. Figure 1E is the structure after fusion; as Figures 1F - 1G shown, the network layers with the same channels and sizes can be merged together. Figure 1F is the structure before fusion. Figure 1G is the structure after fusion; as Figures 1H - 1I shown, it is to fuse the Figure 1D and Figure 1F structures. Figure 1H is the structure before fusion. Figure 1I is the structure after fusion.

[0055] 103. Input the first feature image into the convolutional layer to obtain the second feature image.

[0056] Among them, the first feature image can be input into the convolutional layer to obtain the second feature image, that is, deeper image features can be obtained.

[0057] 104. Input the second feature image into the object information layer to obtain an object information feature map, which includes the type, location, and object weight feature of the object.

[0058] Among them, the location feature can represent the coordinate position of the object. The object type feature is used to identify the type of the object. The object re-identification feature is the REID feature. In a specific implementation, the second feature image can be input into the object information layer to achieve object re-identification, and an object information feature map is obtained. The object information feature map includes a location feature, an object type feature, and an object re-identification feature.

[0059] In a specific implementation, the last channel of the object information layer can be c*h*w, where c is the channel, h is the height, and w is the width. The value of the h*w feature map of channel c = 0 is the probability value of the object. Set a threshold P. When the value at a certain location is greater than P, it represents the existence of an object. Channels 1, 2, 3, and 4 represent the location of the object (x1, y1, x2, y2, x1, y1 are the upper left coordinates, and x2, y2 are the lower right corner information). Channel c - 4 represents the object weight information.

[0060] 105. Input the object information feature map into the data analysis layer, parse out the type, location, and object weight feature of the object, construct the object weight feature into a two-dimensional weight matrix, compress the ROI image where the object is located to a fixed size, and perform convolution calculation with the two-dimensional weight matrix to obtain an object REID feature map.

[0061] In a specific implementation, the first object re-identification image can be input into the data analysis layer to achieve image compression. For example, only analyze the region of interest in the image, or reduce the dimension of the features to obtain an object REID feature map.

[0062] Optionally, in step 105 above, input the object information feature map into the data analysis layer, parse out the type, location, and weight feature of the object, construct the weight feature into a two-dimensional weight matrix, compress the ROI image where the object is located to a fixed size, and perform convolution calculation with the two-dimensional weight matrix to obtain an object REID feature map, which can include the following steps:

[0063] 51. Parse out the object type, location, and object weight feature from the object information feature map; parse out the location of the object in the image, obtain the object ROI, and compress the object ROI to a fixed size;

[0064] 52. Construct the weight vector corresponding to the weight feature into a two-dimensional weight matrix, and perform convolution calculation on the compressed object ROI and the object weight matrix feature to obtain an n*n network feature map;

[0065] 53. Input the n*n network feature map into an activation function to obtain the n*n object REID feature map.

[0066] In specific implementation, the object information feature map can be parsed to obtain the object type, location, and object weight feature; the location of the object in the image can be parsed to obtain the object ROI, the object ROI is compressed to a fixed size, the ROI is compressed, and then the weight vector corresponding to the object weight feature is constructed into a two-dimensional weight matrix, and the compressed object ROI and the object weight matrix feature are subjected to convolution calculation to obtain the n*n network feature map, which can reduce the amount of computation. Then, the n*n network feature map is input into the activation function to obtain the n*n object REID feature map, so as to achieve the purpose of image compression.

[0067] 106. Input the object REID feature map into a second feature extraction layer to obtain a third feature image.

[0068] In specific implementation, the object REID feature map can be input into the second feature extraction layer to obtain the third feature image, that is, the features are optimized, which helps to improve the recognition accuracy.

[0069] 107. Input the third feature image into a first fully connected layer, a second fully connected layer, and a first normalization layer in sequence to obtain the object re-identification feature after compressing the feature dimension.

[0070] Among them, the dimension of the first fully connected layer can be greater than the dimension of the second fully connected layer. Furthermore, the third feature image can be input into the first fully connected layer, the second fully connected layer, and the first normalization layer in sequence to obtain the object re-identification feature, and further achieve the purpose of dimensionality reduction. Compared with the traditional principal component analysis method PCA, the present application can achieve a faster dimensionality reduction speed and also has higher accuracy.

[0071] In specific implementation, different networks are constructed according to the different requirements of the training network and the inference network; in the training network, the first fully connected layer is connected to the normalization layer, and in the inference network, the first fully connected layer does not have a normalization layer.

[0072] In specific implementation, the object REID feature map can be input into the first fully connected layer and then into the second fully connected layer. The dimensions of the first fully connected layer and the second fully connected layer are different, and finally input into the normalization layer to compress the REID feature dimension through the network. The network directly realizes the positioning of the object and the extraction of the REID feature.

[0073] Optionally, there are P objects in the target image, and when searching, P is a positive integer greater than 1. After the above step 107, the following steps can also be included:

[0074] A1. Determine the respective similarity probabilities of the P objects in their respective base libraries based on the object re-identification features, obtaining Q probabilities, where Q is a positive integer;

[0075] A2. Construct a first-level vector of the query image according to the Q probabilities;

[0076] A3. Obtain the retrieval probabilities of each object in the query image output by the object information layer, and construct a second-level vector with the retrieval probabilities;

[0077] A4. Determine the retrieval probability according to the first-level vector and the second-level vector.

[0078] In specific implementation, when searching for P objects, where P is a positive integer greater than 1, the respective similarity probabilities of the P objects in their respective base libraries can be determined based on the object re-identification features, obtaining Q probabilities, where Q is a positive integer. A first-level vector of the query image is constructed according to the Q probabilities. The query image can be the target image or other images. Obtain the retrieval probabilities of each object in the query image output by the object information layer, and construct a second-level vector with the retrieval probabilities. Determine the retrieval probability according to the first-level vector and the second-level vector. Input the corresponding similar images according to the similarity probability set. In this way, images with more objects to be searched can be preferentially displayed, improving the retrieval efficiency.

[0079] For example, in the application of the object search scenario, when searching for multiple objects, a two-level search method is adopted. In the first-level search, the respective similarity probabilities of multiple pedestrian objects are obtained, and the search probabilities are combined into a new vector, which is the first-level vector. In the second-level search, the retrieval probabilities are combined into a feature vector, that is, the second-level vector. Multiply the second-level vector by the first-level vector to obtain the final similarity probability.

[0080] Taking a backpack and a handbag as an example, the retrieval probabilities of the backpack and the handbag in the query image can be combined into a vector V = [da, db], which is the first-level vector. The retrieval probabilities of the backpack in the library are P1a, P2a, P3a,..., Pma (assuming there are m backpacks in the library images), and the retrieval probabilities of the handbag in the library are P1b, P2b, P3b,..., Pnb (there are n handbags). The search probabilities of multiple objects are combined into a first-level vector, that is, the second-level vector, F1 = [p1a, p1b], F2 = [p2a, p2b], F3 = [p3a, p3b],..., Fn = [pna, pnb],..., Fm = [pma, 0], (m > n). The final similarity probabilities are P1v = V·F1, P2v = V·F2, P3v = V·F3,..., Pmv = V·Fm. Sort these probabilities, and the similar images can be output according to the probability magnitudes.

[0081] Optionally, before step 101, the following steps may further be included:

[0082] In the training stage, the network model of the first feature layer is trained through a first loss function, and the first loss function is used to optimize the type, position, and object weight features of the objects in the image;

[0083] and / or,

[0084] In the training stage, the first fully connected layer is connected to the first normalization layer, and the REID feature network model of the first fully connected layer is optimized and trained through a second loss function, and the second loss function is used to indicate the difference between the sample image and the output result of the second normalization layer;

[0085] and / or,

[0086] In the training stage, the second fully connected layer is connected to the second normalization layer, and the REID feature network model of the second fully connected layer is optimized and trained through a third loss function, and the third loss function is used to indicate the difference between the sample image and the third normalization layer.

[0087] Wherein, the above sample image can be a sample image of an object to be recognized. Each sample image can include at least one object, and each object can correspond to a corresponding object label (object type). The sample image can be one or more.

[0088] In specific implementation, when training an object recognition model as Figure 1B shown, that is, when training the object REID feature network, it can be optimized according to a three-level loss function. As Figure 1C shown, in the training stage, the network model of the first feature layer can be trained through a first loss function. The first loss function is used for the difference between the sample image and the output result of the first feature layer. Through this first loss function, the model parameters of the network model of the first feature layer can be optimized to improve its model performance.

[0089] In addition, in the training stage, the first fully connected layer is connected to the second normalization layer, and the network model of the first fully connected layer is trained through a second loss function. The second loss function is used to indicate the difference between the sample image and the output result of the second normalization layer. Through this first loss function, the model parameters of the network model of the first fully connected layer can be optimized to improve its model performance.

[0090] In addition, in the training stage, the second fully connected layer is connected to the third normalization layer, and the network model of the second fully connected layer is trained through a third loss function. The third loss function is used to indicate the difference between the sample image and the third normalization layer. Through this first loss function, the model parameters of the network model of the second fully connected layer can be optimized to improve its model performance.

[0091] Among them, the second normalization layer may be the same as or different from the first normalization layer, the third normalization layer may be the same as or different from the first normalization layer, and the second normalization layer may be the same as or different from the third normalization layer.

[0092] Among them, the first loss function, the second loss function, and the third loss function may be different loss functions.

[0093] Among them, for optimizing the type, the loss function Lc = loss(cls_out, label); for optimizing the position, the loss function Lp = loss(pos_out, pos_label); for optimizing the weight feature, the loss function Lf = loss(fea_out, fea_label).

[0094] Among them, cls_out, pos_out, and fea_out are the results output by the object information layer; label and pos_label are the marked object types and positions. fea_label is the object ROI region compressed to a fixed size as the weight feature for object marking. Loss is the loss function. The first loss function is L1 = Lc + Lp + Lf. Optimize the L1 loss function alone.

[0095] Among them, the second loss function is L2 = loss(feat1_out, fea1_label), where feat1_out is the feature vector output by the first normalization layer, and fea1_label is the object ROI region compressed to a certain size as the feature for object marking.

[0096] Among them, the third loss function is L3 = loss(feat2_out, fea2_label), where feat2_out is the feature vector output by the second normalization layer, and fea2_label is the object ROI region compressed to a certain size as the feature for object marking.

[0097] Optionally, before step 101 above, the following steps may further be included:

[0098] B1. Obtain the target hardware parameters;

[0099] B2. Determine the target network configuration parameters corresponding to the target hardware parameters;

[0100] B3. Configure the network model of the first feature extraction layer or the second feature extraction layer according to the target network configuration parameters.

[0101] Among them, in the embodiments of the present application, the hardware parameters may include at least one of the following: the number of cores, CPU performance parameters, memory size, GPU performance parameters, NPU performance parameters, etc., which are not limited herein. The network configuration parameters may include at least one of the following: network type identifier, number of convolutional layers, number of normalization layers, activation function type, etc., which are not limited herein. The electronic device may pre-store the mapping relationship between the preset hardware parameters and the network configuration parameters. Furthermore, after obtaining the target hardware parameters, the target network configuration parameters corresponding to the target hardware parameters can be determined according to the mapping relationship, and then the network model of the first feature extraction layer can be configured according to the target network configuration parameters.

[0102] Of course, the network model of the second feature extraction layer and the network model of the first feature extraction layer may be the same or different, and they may construct the network model based on the same or similar methods, which will not be elaborated herein.

[0103] It can be seen that in the object retrieval method described in the embodiments of the present application, a target image is obtained, the target image is input into the first feature extraction layer for feature extraction to obtain a first feature image, the first feature image is input into the convolutional layer to obtain a second feature image, the second feature image is input into the object information layer to obtain an object information feature map. The object information feature map includes: the type, position, and object weight feature of the object. The object information feature map is input into the data analysis layer to parse out the type, position, and object weight feature of the object. The object weight feature is constructed into a two-dimensional weight matrix, and the ROI image where the object position is located is compressed to a fixed size and convolved with the two-dimensional weight matrix to obtain an object REID feature map. The object REID feature map is input into the second feature extraction layer to obtain a third feature image, and the third feature image is sequentially input into the first fully connected layer, the second fully connected layer, and the first normalization layer to obtain the object re-identification feature after compressing the feature dimension. On the one hand, since a linear model is adopted and an end-to-end network is used in both feature retrieval and object re-identification feature extraction, the object retrieval efficiency can be improved. On the other hand, two fully connected layers are used to achieve dimensionality reduction, which can achieve a faster dimensionality reduction speed and also have higher accuracy. Therefore, the present application can achieve efficient object positioning and retrieval.

[0104] Consistent with the above Figure 1A shown embodiment, please refer to Figure 2 , Figure 2 which is a schematic flowchart of an object retrieval method provided by an embodiment of the present application, applied to an electronic device. As shown in the figure, the object retrieval method includes:

[0105] 201. Obtain a target image.

[0106] 202. Input the target image into the first feature extraction layer for feature extraction to obtain a first feature image.

[0107] 203. Input the first feature image into the convolutional layer to obtain a second feature image.

[0108] 204. Input the second feature image into the object information layer to obtain an object information feature map, where the object information feature map includes: the type, location, and object weight feature of the object.

[0109] 205. Input the object information feature map into the data analysis layer, parse out the type, location, and object weight feature of the object, construct the object weight feature into a two-dimensional weight matrix, compress the ROI image where the object location is to a fixed size and perform convolution calculation with the two-dimensional weight matrix to obtain an object REID feature map.

[0110] 206. Input the object REID feature map into the second feature extraction layer to obtain a third feature image.

[0111] 207. Input the third feature image into the first fully connected layer, the second fully connected layer, and the first normalization layer in sequence to obtain the object re-identification feature after compressing the feature dimension.

[0112] 208. When there are P objects in the target image and during the search, where P is a positive integer greater than 1, determine the respective similarity probabilities of the P objects in their respective bottom libraries according to the object re-identification feature to obtain Q probabilities, and Q is a positive integer.

[0113] 209. Construct a first-level vector of the query image according to the Q probabilities.

[0114] 210. Obtain the retrieval probability of each object in the query image output by the object information layer, and construct a second-level vector with the retrieval probability.

[0115] 211. Determine the retrieval probability according to the first-level vector and the second-level vector.

[0116] Among them, the specific descriptions of the above steps 201 - 211 can refer to the corresponding steps of the object retrieval method described above Figure 1A and will not be elaborated here.

[0117] It can be seen that for the object retrieval method described in the embodiments of the present application, on the one hand, since a linear model is adopted and end-to-end networks are used for both feature retrieval and object re-identification feature extraction, the object retrieval efficiency can be improved. On the other hand, two fully connected layers are used to achieve dimensionality reduction, which can achieve a faster dimensionality reduction speed and also have higher accuracy. Therefore, the present application can achieve efficient object retrieval and can preferentially display images with a large number of objects to be searched, thereby improving the retrieval efficiency.

[0118] Consistent with the above embodiments, please refer to Figure 3 , Figure 3 FIG. is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As shown in the figure, the electronic device includes a processor, a memory, a communication interface, and one or more programs. The one or more programs are stored in the memory and are configured to be executed by the processor. In the embodiments of the present application, the program includes instructions for performing the following steps:

[0119] Obtain a target image;

[0120] Input the target image into a first feature extraction layer for feature extraction to obtain a first feature image;

[0121] Input the first feature image into a convolutional layer to obtain a second feature image;

[0122] Input the second feature image into an object information layer to obtain an object information feature map, where the object information feature map includes: the type, position, and object weight feature of the object;

[0123] Input the object information feature map into a data analysis layer, parse out the type, position, and object weight feature of the object, construct the object weight feature into a two-dimensional weight matrix, compress the ROI image where the object position is located to a fixed size, and perform convolutional calculation with the two-dimensional weight matrix to obtain an object REID feature map;

[0124] Input the object REID feature map into a second feature extraction layer to obtain a third feature image;

[0125] Input the third feature image into a first fully connected layer, a second fully connected layer, and a first normalization layer in sequence to obtain an object re-identification feature with a compressed feature dimension.

[0126] Optionally, in the aspect of inputting the object information feature map into a data analysis layer, parsing out the type, position, and weight feature of the object, constructing the weight feature into a two-dimensional weight matrix, compressing the ROI image where the object position is located to a fixed size, and performing convolutional calculation with the two-dimensional weight matrix to obtain an object REID feature map, the program includes instructions for performing the following steps:

[0127] Parse the object type, location, and weight features from the object information feature map; parse the location of the object in the image, obtain the object ROI, and compress the object ROI to a fixed size;

[0128] Construct the weight vector corresponding to the weight feature into a two-dimensional weight matrix, and perform convolution calculation on the compressed object ROI and the object weight matrix feature to obtain an n*n network feature map;

[0129] Input the n*n network feature map into the activation function to obtain the n*n object REID feature map.

[0130] Optionally, in the step of inputting the object information feature map into the data analysis layer, parsing the type, location, and re-identification features of the object, compressing the ROI image where the object is located, and performing convolution calculation with the object re-identification features to obtain the object REID feature map, the above program includes instructions for performing the following steps:

[0131] Compress the object ROI according to the location of the object in the image, and perform convolution calculation on the compressed object ROI and the object re-identification features to obtain an operation result;

[0132] Input the operation result into the activation function to obtain the object REID feature map.

[0133] Optionally, when there are P objects in the target image and a search is performed, where P is a positive integer greater than 1, the above program further includes instructions for performing the following steps:

[0134] Determine the respective similarity probabilities of the P objects in their respective bottom libraries according to the object re-identification features to obtain Q probabilities, where Q is a positive integer;

[0135] Construct a first-level vector of the query image according to the Q probabilities;

[0136] Obtain the retrieval probabilities of each object in the query image output by the object information layer, and construct a second-level vector with the retrieval probabilities;

[0137] Determine the retrieval probability according to the first-level vector and the second-level vector.

[0138] Optionally, the above program further includes instructions for performing the following steps:

[0139] In the training stage, train the network model of the first feature layer through the first loss function, and the first loss function is used to optimize the type, location, and object weight features of the objects in the image;

[0140] And / or,

[0141] During the training phase, the first fully-connected layer is connected to the first normalization layer, and the REID feature network model of the first fully-connected layer is optimized and trained through a second loss function, where the second loss function is used to indicate the difference between the sample image and the output result of the second normalization layer;

[0142] and / or,

[0143] During the training phase, the second fully-connected layer is connected to the second normalization layer, and the REID feature network model of the second fully-connected layer is optimized and trained through a third loss function, where the third loss function is used to indicate the difference between the sample image and the third normalization layer.

[0144] Optionally, the above program further includes instructions for performing the following steps:

[0145] Obtain target hardware parameters;

[0146] Determine target network configuration parameters corresponding to the target hardware parameters;

[0147] Configure the network models of the first feature extraction layer and the second feature extraction layer according to the target network configuration parameters.

[0148] It can be seen that for the electronic device described in the embodiments of the present application, a target image is obtained, the target image is input into the first feature extraction layer for feature extraction to obtain a first feature image, the first feature image is input into the convolutional layer to obtain a second feature image, the second feature image is input into the object information layer to obtain an object information feature map, where the object information feature map includes: the type, position, and object weight feature of the object, the object information feature map is input into the data analysis layer to parse out the type, position, and object weight feature of the object, the object weight feature is constructed into a two-dimensional weight matrix, the ROI image where the object position is located is compressed to a fixed size and convolved with the two-dimensional weight matrix to obtain an object REID feature map, the object REID feature map is input into the second feature extraction layer to obtain a third feature image, and the third feature image is sequentially input into the first fully-connected layer, the second fully-connected layer, and the first normalization layer to obtain the object re-identification feature after compressing the feature dimension. On the one hand, since a linear model is adopted and an end-to-end network is used for both feature retrieval and object re-identification feature extraction, the object retrieval efficiency can be improved. On the other hand, two fully-connected layers are used to achieve dimensionality reduction, which can achieve a faster dimensionality reduction speed and also has higher accuracy. Therefore, the present application can achieve efficient object positioning and retrieval.

[0149] Figure 4It is a functional unit composition block diagram of the object retrieval device 400 involved in the embodiments of the present application. The object retrieval device 400 is applied to an electronic device. The device 400 includes: an acquisition unit 401, a first extraction unit 402, an analysis unit 403, a second extraction unit 404, and an output unit 405. Among them,

[0150] The acquisition unit 401 is configured to acquire a target image;

[0151] The first extraction unit 402 is configured to input the target image into a first feature extraction layer for feature extraction to obtain a first feature image; input the first feature image into a convolutional layer to obtain a second feature image; and input the second feature image into an object information layer to obtain an object information feature map. The object information feature map includes: the type, position, and object weight feature of the object;

[0152] The analysis unit 403 is configured to input the object information feature map into a data analysis layer, parse out the type, position, and object weight feature of the object, construct the object weight feature into a two-dimensional weight matrix, compress the ROI image where the object is located to a fixed size, and perform a convolutional calculation with the two-dimensional weight matrix to obtain an object REID feature map;

[0153] The second extraction unit 404 is configured to input the object REID feature map into a second feature extraction layer to obtain a third feature image;

[0154] The output unit 405 is configured to sequentially input the third feature image into a first fully connected layer, a second fully connected layer, and a first normalization layer to obtain an object re-identification feature after compressing the feature dimension.

[0155] Optionally, in terms of inputting the object information feature map into a data analysis layer, parsing out the type, position, and weight feature of the object, constructing the weight feature into a two-dimensional weight matrix, compressing the ROI image where the object is located to a fixed size, and performing a convolutional calculation with the two-dimensional weight matrix to obtain an object REID feature map, the analysis unit 403 is specifically configured to:

[0156] Parse out the object type, position, and weight feature from the object information feature map; parse out the position of the object in the image, obtain the object ROI, and compress the object ROI to a fixed size;

[0157] Construct the weight vector corresponding to the weight feature into a two-dimensional weight matrix, and perform a convolutional calculation on the compressed object ROI and the object weight matrix feature to obtain an n*n network feature map;

[0158] Input the n*n network feature map into the activation function to obtain the n*n object REID feature map.

[0159] Optionally, when there are P objects in the target image and searching is performed, where P is a positive integer greater than 1, the apparatus 400 is further specifically configured to:

[0160] Determine respective similarity probabilities of the P objects in their respective bottom libraries according to the object re-identification features, obtaining Q probabilities, where Q is a positive integer;

[0161] Construct a first-level vector of the query image according to the Q probabilities;

[0162] Obtain the retrieval probabilities of the respective objects in the query image output by the object information layer, and construct a second-level vector with the retrieval probabilities;

[0163] Determine the retrieval probability according to the first-level vector and the second-level vector.

[0164] Optionally, the apparatus 400 is further specifically configured to:

[0165] In the training phase, train the network model of the first feature layer through a first loss function, where the first loss function is used to optimize the type, position, and object weight features of the objects in the image;

[0166] And / or,

[0167] In the training phase, the first fully connected layer is connected to the first normalization layer, and the REID feature network model of the first fully connected layer is optimized and trained through a second loss function, where the second loss function is used to indicate the difference between the sample image and the output result of the second normalization layer;

[0168] And / or,

[0169] In the training phase, the second fully connected layer is connected to the second normalization layer, and the REID feature network model of the second fully connected layer is optimized and trained through a third loss function, where the third loss function is used to indicate the difference between the sample image and the third normalization layer.

[0170] Optionally, the apparatus 400 is further specifically configured to:

[0171] Obtain target hardware parameters;

[0172] Determine target network configuration parameters corresponding to the target hardware parameters;

[0173] Configure the network models of the first feature extraction layer and the second feature extraction layer according to the target network configuration parameters.

[0174] It can be seen that the object retrieval device described in the embodiments of the present application acquires a target image, inputs the target image into a first feature extraction layer for feature extraction to obtain a first feature image, inputs the first feature image into a convolutional layer to obtain a second feature image, inputs the second feature image into an object information layer to obtain an object information feature map. The object information feature map includes: the type, location, and object weight feature of the object. Input the object information feature map into a data analysis layer to parse out the type, location, and object weight feature of the object, construct the object weight feature into a two-dimensional weight matrix, compress the ROI image where the object is located to a fixed size and perform convolution calculation with the two-dimensional weight matrix to obtain an object REID feature map, input the object REID feature map into a second feature extraction layer to obtain a third feature image, and input the third feature image into a first fully connected layer, a second fully connected layer, and a first normalization layer in sequence to obtain an object re-identification feature after compressing the feature dimension. On the one hand, since a linear model is adopted and an end-to-end network is used in both feature retrieval and object re-identification feature extraction, the object retrieval efficiency can be improved. On the other hand, two fully connected layers are used to achieve dimensionality reduction, which can achieve a faster dimensionality reduction speed and also have higher accuracy. Therefore, the present application can achieve efficient object positioning and retrieval.

[0175] It can be understood that the functions of the respective program modules of the object retrieval device in this embodiment can be specifically implemented according to the method in the above method embodiment, and the specific implementation process can refer to the relevant description of the above method embodiment, which will not be elaborated here.

[0176] The embodiments of the present application further provide a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, and the computer program enables a computer to execute part or all of the steps of any method recorded in the above method embodiment, and the above computer includes an electronic device.

[0177] The embodiments of the present application further provide a computer program product. The above computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the above computer program can operate to enable a computer to execute part or all of the steps of any method recorded in the above method embodiment. The computer program product can be a software installation package, and the above computer includes an electronic device.

[0178] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0179] In the above embodiments, the descriptions of the various embodiments have their own focuses. For parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0180] In several embodiments provided by the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the above division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical or other form.

[0181] The units described above as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0182] In addition, the functional units in the various embodiments of the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0183] If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the above methods in the various embodiments of the present application. And the aforementioned memory includes: USB flash drives, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), mobile hard disks, magnetic disks, or optical discs and other various media that can store program codes.

[0184] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable memory, which may include: a flash drive, a read-only memory (abbreviation: ROM), a random access memory (abbreviation: RAM), a magnetic disk, an optical disk, etc.

[0185] The above has introduced the embodiments of the present application in detail. Specific examples are used in this article to elaborate on the principles and implementation manners of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A method for object positioning and retrieval, characterized in that, The method includes: Obtain a target image; Input the target image into a first feature extraction layer for feature extraction to obtain a first feature image; Input the first feature image into a convolutional layer to obtain a second feature image; Input the second feature image into an object information layer to obtain an object information feature map, where the object information feature map includes: the type, location, and object weight feature of the object; Input the object information feature map into a data analysis layer, parse out the type, location, and object weight feature of the object, construct the object weight feature into a two-dimensional weight matrix, compress the ROI image where the object is located to a fixed size and perform convolution calculation with the two-dimensional weight matrix to obtain an object REID feature map; Input the object REID feature map into a second feature extraction layer to obtain a third feature image; Input the third feature image into a first fully connected layer, a second fully connected layer, and a first normalization layer in sequence to obtain an object re-identification feature after compressing the feature dimension; the dimension of the first fully connected layer is greater than the dimension of the second fully connected layer; Among them, the step of inputting the object information feature map into the data analysis layer, parsing out the type, location, and weight feature of the object, constructing the weight feature into a two-dimensional weight matrix, compressing the ROI image where the object is located to a fixed size and performing convolution calculation with the two-dimensional weight matrix to obtain an object REID feature map includes: Parse out the object type, location, and weight feature from the object information feature map; parse out the location of the object in the image, obtain the object ROI, and compress the object ROI to a fixed size; Construct the weight vector corresponding to the weight feature into a two-dimensional weight matrix, and perform convolution calculation on the compressed object ROI and the object weight matrix feature to obtain an n*n network feature map; Input the n*n network feature map into an activation function to obtain the n*n object REID feature map.

2. The method according to claim 1, characterized in that, There are P objects in the target image, and when performing a search, P is a positive integer greater than 1. The method further includes: Determine the respective similarity probabilities of the P objects in their respective bottom libraries according to the object re-identification feature to obtain Q probabilities, where Q is a positive integer; Construct a first-level vector of the query image according to the Q probabilities; Obtain the retrieval probability of each object in the query image output by the object information layer, and construct the retrieval probability into a second-level vector; Determine the retrieval probability according to the first-level vector and the second-level vector.

3. The method according to claim 1 or 2, characterized in that, The method further includes: In the training stage, train the network model of the first feature layer through a first loss function, and the first loss function is used to optimize the type, location, and object weight feature of the object in the image; And / or, In the training stage, the first fully connected layer is connected to the first normalization layer, and the REID feature network model of the first normalization layer is optimized and trained through a second loss function, and the second loss function is used to indicate the difference between the sample image and the output result of the first normalization layer; And / or, During the training phase, the second fully connected layer is connected to the second normalization layer, and the second normalized REID feature network model is optimized and trained using a third loss function, which is used to indicate the difference between the sample image and the second normalization layer.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Obtaining target hardware parameters; Determining target network configuration parameters corresponding to the target hardware parameters; Configuring the network models of the first feature extraction layer and the second feature extraction layer according to the target network configuration parameters.

5. An object retrieval device, characterized in that, The apparatus includes: an acquisition unit, a first extraction unit, an analysis unit, a second extraction unit, and an output unit, where The acquisition unit is configured to acquire a target image; The first extraction unit is configured to input the target image into a first feature extraction layer for feature extraction to obtain a first feature image; input the first feature image into a convolutional layer to obtain a second feature image; and input the second feature image into an object information layer to obtain an object information feature map, where the object information feature map includes: the type, position, and object weight feature of the object; The analysis unit is configured to input the object information feature map into a data analysis layer, parse out the type, position, and object weight feature of the object, construct the object weight feature into a two-dimensional weight matrix, compress the ROI image where the object position is located to a fixed size, and perform a convolutional calculation with the two-dimensional weight matrix to obtain an object REID feature map; The second extraction unit is configured to input the object REID feature map into a second feature extraction layer to obtain a third feature image; The output unit is configured to sequentially input the third feature image into a first fully connected layer, a second fully connected layer, and a first normalization layer to obtain an object re-identification feature after compressing the feature dimension; the dimension of the first fully connected layer is greater than the dimension of the second fully connected layer; Wherein, in the aspect of inputting the object information feature map into the data analysis layer, parsing out the type, position, and weight feature of the object, constructing the weight feature into a two-dimensional weight matrix, compressing the ROI image where the object position is located to a fixed size, and performing a convolutional calculation with the two-dimensional weight matrix to obtain an object REID feature map, the analysis unit is specifically configured to: Parse out the object type, position, and weight feature from the object information feature map; parse out the position of the object in the image, obtain the object ROI, and compress the object ROI to a fixed size; Construct the weight vector corresponding to the weight feature into a two-dimensional weight matrix, and perform a convolutional calculation on the compressed object ROI and the object weight matrix feature to obtain an n*n network feature map; Input the n*n network feature map into an activation function to obtain the n*n object REID feature map.

6. The device according to claim 5, characterized in that, When there are P objects in the target image and during the search, where P is a positive integer greater than 1, the apparatus is further specifically configured to: Determine the respective similarity probabilities of the P objects in their respective bottom libraries according to the object re-identification feature to obtain Q probabilities, where Q is a positive integer; Construct a first-level vector of the query image according to the Q probabilities; Obtain the retrieval probabilities of each object in the query image output by the object information layer, and construct a second-level vector from the retrieval probabilities. Determine the retrieval probability according to the first-level vector and the second-level vector.

7. An electronic device, characterized in that, It includes a processor and a memory. The memory is used to store one or more programs and is configured to be executed by the processor. The programs include instructions for performing the steps in the method according to any one of claims 1-4.

8. A computer-readable storage medium, characterized in that, Store a computer program for electronic data exchange, wherein the computer program causes a computer to execute the method according to any one of claims 1-4.

Images