Image feature extraction method, model training method and image retrieval method and device

Abstract

The invention provides an image feature extraction method, a model training method and an image retrieval method and device. The main technical scheme comprises the steps of obtaining a target image; extracting a first feature representation of the target image, wherein the first feature representation comprises a visual angle feature; determining at least one target expert network in the hybrid expert network and a processing weight corresponding to the target expert network based on the view angle features; obtaining a second feature representation of the target image predicted by the at least one target expert network based on the first feature representation; based on the processing weight corresponding to the at least one target expert network, weighting the second feature representation obtained by prediction of the at least one target expert network to obtain a third feature representation of the target image; and correspondingly storing the target image and the third feature representation in an image library for image retrieval. According to the method, the image retrieval accuracy can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an image feature extraction method, a model training method, and an image retrieval method and device. Background Art

[0002] Text-to-object retrieval technology plays a vital role in fields such as intelligent security. Current mainstream text-to-image object retrieval methods often construct image retrieval models based on single-view surveillance data, assuming that all target images are captured from the same viewing angle. However, real-world security scenarios are complex, with data from multiple sources and heterogeneous perspectives, such as drone inspections and oblique capture from wearable devices. The appearance of objects varies significantly from one viewpoint to another, significantly reducing the retrieval accuracy of traditional methods in practical applications. Summary of the Invention

[0003] In view of this, the present application provides an image feature extraction method, a model training method, an image retrieval method and a device to improve the accuracy of image retrieval.

[0004] This application provides the following solutions:

[0005] In a first aspect, a method for extracting image features is provided, comprising:

[0006] Acquire the target image;

[0007] extracting a first feature representation of the target image, wherein the first feature representation includes a viewing angle feature;

[0008] Determining at least one target expert network in the hybrid expert network and a processing weight corresponding to each of the at least one target expert network based on the perspective feature;

[0009] Obtaining second feature representations of the target image predicted by the at least one target expert network based on the first feature representation;

[0010] performing weighted processing on the second feature representations respectively predicted by the at least one target expert network based on the processing weights respectively corresponding to the at least one target expert network to obtain a third feature representation of the target image;

[0011] The target image and the third feature representation are correspondingly stored in an image library for image retrieval.

[0012] Optionally, extracting a first feature representation of the target image includes:

[0013] extracting an embedded representation of the target image;

[0014] Additional perspective tokens are added based on the embedded representation of the target image, where the perspective tokens are pre-trained and used to reflect the perspective features of the target image;

[0015] The embedded representation with the perspective word added is subjected to self-attention processing to obtain a first feature representation of the target image.

[0016] Optionally, the hybrid expert network is composed of multiple expert networks;

[0017] The determining, based on the view feature, at least one target expert network in the hybrid expert network and the processing weights corresponding to the at least one target expert network respectively includes:

[0018] generating, using a routing network, processing probability distributions of the plurality of expert networks based on the view features;

[0019] selecting at least one target expert network from the plurality of expert networks based on the processing probability distribution;

[0020] Based on the processing probability distribution of the selected at least one target expert network, processing weights corresponding to the at least one target expert network are determined.

[0021] Optionally, the routing network includes image-level routing and word-level routing;

[0022] The generating of the processing probability distribution of the plurality of expert networks based on the view features by using a routing network includes:

[0023] The image-level routing is used to predict the view category of the target image based on the view feature; a mask is generated based on the view category, wherein elements in the mask correspond one-to-one to the multiple expert networks, and the values ​​of the elements represent whether the corresponding expert network is a candidate expert network; the word-level routing is used to generate initial probability distributions of the multiple expert networks based on each word in the first feature representation; and based on the mask and the initial probability distributions, processing probability distributions corresponding to the candidate expert networks are obtained.

[0024] The selecting at least one target expert network from the plurality of expert networks based on the processing probability distribution includes:

[0025] At least one target expert network is selected from the candidate expert networks based on the processing probability distributions respectively corresponding to the candidate expert networks.

[0026] Optionally, the expert network includes: private experts and shared experts;

[0027] The obtaining of the second feature representation of the target image predicted by the at least one target expert network based on the first feature representation includes:

[0028] When the target expert network is a private expert, obtaining a second feature representation of the target image predicted by the private expert based on the view feature in the first feature representation;

[0029] When the target expert network is a shared expert, a second feature representation of the target image predicted by the shared expert based on the non-view features in the first feature representation is obtained.

[0030] In a second aspect, an image retrieval method is provided, the method comprising:

[0031] Obtaining image description text and extracting text features of the image description text;

[0032] Matching the text features with third feature representations of multiple images stored in an image library respectively; wherein the third feature representations are obtained based on the image feature extraction method according to any one of the first aspects;

[0033] Based on the matching result, a retrieved image is determined from the plurality of images.

[0034] In a third aspect, a method for training an image retrieval model is provided, comprising:

[0035] Acquire training data including a plurality of training samples, wherein the training samples include: image samples and image description texts corresponding to the image samples;

[0036] Training an image retrieval model based on the training data, wherein the training comprises:

[0037] Extracting text features of the image description text;

[0038] Inputting the image sample into an image retrieval model, extracting a first feature representation of the image sample by the image retrieval model, wherein the first feature representation includes a view feature; determining at least one target expert network in the hybrid expert network and processing weights corresponding to the at least one target expert network based on the view feature; obtaining a second feature representation of the image sample predicted by the at least one target expert network based on the first feature representation; performing weighted processing on the second feature representations predicted by the at least one target expert network based on the processing weights corresponding to the at least one target expert network to obtain a third feature representation of the image sample;

[0039] The model parameters of the image retrieval model are updated using the loss function value corresponding to the first training objective, where the first training objective includes: minimizing the distance between the third feature representation and the text feature.

[0040] Optionally, the image retrieval model includes an embedding layer network and a self-attention layer network;

[0041] The extracting a first feature representation of the image sample includes:

[0042] Extracting an embedded representation of the image sample using the embedding layer network, and adding a perspective word based on the embedded representation of the image sample, wherein the perspective word is used to reflect the perspective feature of the target image;

[0043] The self-attention layer network is used to perform self-attention processing on the embedded representation with the perspective word added thereto to obtain a first feature representation of the image sample.

[0044] Optionally, the image retrieval model further includes a routing network and the hybrid expert network, and the hybrid expert network is composed of multiple expert networks;

[0045] The determining, based on the view feature, at least one target expert network in the hybrid expert network and the processing weights corresponding to the at least one target expert network respectively includes:

[0046] The routing network is used to generate a processing probability distribution of the multiple expert networks based on the perspective features; based on the processing probability distribution, at least one target expert network is selected from the multiple expert networks; based on the processing probability distribution of the selected at least one target expert network, the processing weights corresponding to the at least one target expert network are determined.

[0047] Optionally, the routing network includes image-level routing and word-level routing;

[0048] The generating, by using the routing network, the processing probability distribution of the plurality of expert networks based on the viewpoint features comprises:

[0049] The image-level routing is used to predict the view category of the image sample based on the view feature; a mask is generated based on the view category, wherein the elements in the mask correspond one-to-one to the multiple expert networks, and the values ​​of the elements represent whether the corresponding expert network is a candidate expert network; the word-level routing is used to generate initial probability distributions of the multiple expert networks based on each word in the first feature representation; and based on the mask and the initial probability distribution, processing probability distributions corresponding to the candidate expert networks are obtained.

[0050] The selecting at least one target expert network from the plurality of expert networks based on the processing probability distribution includes:

[0051] At least one target expert network is selected from the candidate expert networks based on the processing probability distributions respectively corresponding to the candidate expert networks.

[0052] Optionally, the expert network includes: a private expert network and a shared expert network;

[0053] The obtaining of second feature representations of the image samples respectively predicted by the at least one target expert network based on the first feature representation includes:

[0054] When the target expert network is a private expert network, obtaining a second feature representation of the image sample predicted by the private expert network based on the view feature in the first feature representation;

[0055] When the target expert network is a shared expert network, a second feature representation of the image sample predicted by the shared expert network based on the non-view features in the first feature representation is obtained.

[0056] Optionally, the training sample further includes: a real viewing angle category corresponding to the image sample;

[0057] The training also includes:

[0058] The perspective word is updated using a loss function value corresponding to a second training objective, wherein the second training objective includes minimizing the difference between the perspective category predicted by the image-level routing and the true perspective category.

[0059] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the first aspect are implemented.

[0060] According to a fifth aspect, an electronic device is provided, including:

[0061] one or more processors; and

[0062] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the first aspects above.

[0063] In a sixth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0064] The embodiment of the present application obtains a target image and extracts a first feature representation containing a perspective feature. Based on the perspective feature, the target expert network in the hybrid expert network and its corresponding processing weight are determined. The target expert network processes the first feature representation, predicts the second feature representation, and then weights the third feature representation to obtain a third feature representation. The third feature representation is stored in the image library together with the target image for subsequent use in image retrieval. This process implements the selection and processing of a perspective-driven hybrid expert network, determines the target expert network in the hybrid expert network and its corresponding processing weight based on the perspective feature of the target image, significantly improves the accuracy of feature representation extraction of images in complex perspective scenes, and thus improves the matching accuracy of text and images. On the one hand, it can effectively improve the accuracy of image retrieval, and on the other hand, it can significantly improve the robustness to scenes with different perspectives. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0066] Figure 1 Flowchart of the image feature extraction method provided in the embodiment of the present application;

[0067] Figure 2 A schematic diagram of a hybrid expert network provided in an embodiment of the present application;

[0068] Figure 3 A flowchart of the image retrieval method provided in an embodiment of the present application;

[0069] Figure 4 A schematic diagram of the principle of the image retrieval method provided in an embodiment of the present application;

[0070] Figure 5 A flowchart of a method for training an image retrieval model provided in an embodiment of the present application;

[0071] Figure 6 A schematic block diagram of an image feature extraction device provided in an embodiment of the present application;

[0072] Figure 7 A schematic block diagram of an image retrieval device provided in an embodiment of the present application;

[0073] Figure 8 A schematic block diagram of a training device for an image retrieval model provided in an embodiment of the present application;

[0074] Figure 9 A schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0075] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0076] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "an", "the" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0077] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0078] Currently, some image retrieval technologies build image retrieval models based on single-view surveillance data. However, real-world scenarios are complex, with data from multiple sources and heterogeneous perspectives, such as drones inspecting from above and wearable devices capturing data from angled angles. The appearance of objects varies significantly from one perspective to another, significantly reducing the retrieval accuracy of traditional methods in practical applications.

[0079] In view of this, the present application provides a new idea, an image feature extraction method, an image retrieval method, an image retrieval model training method and related devices. Figure 1 This is a flowchart of the image feature extraction method provided in an embodiment of the present application. This method can be performed by an image feature extraction device, which is equipped with a trained image retrieval model. The image feature extraction device can be installed in any computer device with data storage and processing capabilities, such as a server or a computer terminal with strong data processing capabilities. The method may include the following steps:

[0080] Step 101: Acquire a target image;

[0081] Step 102: extracting a first feature representation of the target image, where the first feature representation includes a viewing angle feature.

[0082] Step 103: Based on the viewpoint feature, determine at least one target expert network in the hybrid expert network and the processing weights corresponding to the at least one target expert network.

[0083] Step 104: Obtain a second feature representation of the target image predicted by at least one target expert network based on the first feature representation.

[0084] Step 105: Based on the processing weights corresponding to the at least one target expert network, weighted processing is performed on the second feature representations respectively predicted by the at least one target expert network to obtain a third feature representation of the target image.

[0085] Step 106: Store the target image and the third feature representation in a corresponding manner in an image library for image retrieval.

[0086] It can be seen that the present application obtains the target image and extracts the first feature representation containing the perspective feature, determines the target expert network in the hybrid expert network and its corresponding processing weight based on the perspective feature, processes the first feature representation by the target expert network, predicts the second feature representation, and then weights it to obtain the third feature representation, which is then stored in the image library with the target image for subsequent use in image retrieval. This process implements the selection and processing of the perspective-driven hybrid expert network, determines the target expert network in the hybrid expert network and its corresponding processing weight based on the perspective feature of the target image, and significantly improves the accuracy of feature representation extraction of images in complex perspective scenes, thereby improving the matching accuracy of text and images. On the one hand, it can effectively improve the accuracy of image retrieval, and on the other hand, it can significantly improve the robustness to scenes with different perspectives.

[0087] The following describes in detail the steps in the above process and the effects that can be further produced in conjunction with the embodiments.

[0088] First, the above step 101, namely “obtaining a target image”, is described in detail with reference to the embodiment.

[0089] In the embodiment of the present application, the target image may be, but is not limited to, an image containing a preset type of target, such as an image containing a pedestrian. These images may be taken from different perspectives, such as a horizontal perspective, a top-down perspective, an oblique perspective, etc.

[0090] In actual application scenarios, target images are acquired from a variety of sources. For example, in intelligent security systems, target images can be acquired through surveillance cameras deployed in different locations. These cameras may include conventional surveillance cameras shooting horizontally, overhead cameras mounted on drones, and oblique snapshot cameras carried by wearable devices worn by personnel. For example, in a monitoring scenario, when a specific target needs to be tracked, the system will acquire images containing the target in real time from image acquisition devices distributed in different areas and of different types. These images will present a variety of different perspectives, such as horizontal, overhead, and oblique, depending on the position and angle of the shooting devices. Furthermore, the acquired target image can be a single frame or a frame in a video stream.

[0091] The above step 102, namely "extracting a first feature representation of the target image, wherein the first feature representation includes a viewing angle feature", is described in detail below in conjunction with an embodiment.

[0092] In an embodiment of the present application, in order to extract the first feature representation of the target image, the embedded representation of the target image can be extracted first, and then the perspective word unit is added based on the embedded representation of the target image. Finally, the embedded representation with the perspective word unit is self-attention processed to obtain the first feature representation of the target image, wherein the perspective word unit in the first feature representation is pre-trained and can reflect the perspective characteristics of the target image.

[0093] To extract an embedded representation of a target image, the image is first preprocessed. For example, this involves segmenting the image into multiple non-overlapping patches and adding category tokens to these patches. These tokens act as carriers of global semantic information. Next, the segmented patches and category tokens are processed to obtain an embedded representation of the target image. This embedded representation is essentially a fixed-length sequence of visual tokens that contains the basic feature information of the target image.

[0094] Take a target image with a resolution of 224×224 as an example, split it into S image blocks of size 16×16, S=196. Then add the category word V cls , after being processed by the embedding layer network, the embedded representation E of the target image is obtained v At this time, the embedding representation of the target image is E v is a visual word sequence of length S+1, namely E v ={V cls , V1, V2...V s}.

[0095] In addition to the above extraction methods, other methods such as convolutional neural networks and autoencoders can be used to extract the embedded representation of the target image according to actual needs and specific application scenarios. The embodiments of this application do not impose specific restrictions on the method of extracting the embedded representation.

[0096] Based on the embedding representation of the target image obtained above, the pre-trained perspective word V is added view , we get the embedded representation with the perspective word attached, namely E' v ={V cls , V1……V s , V view}.

[0097] Viewpoint tokens are trained using a large amount of image data from different perspectives. They can represent the image's perspective information, such as shooting angle and direction. As learnable parameters, these tokens have pre-set initial values ​​and are continuously optimized and updated during model training to better represent the image's perspective characteristics.

[0098] It should be noted that in the field of natural language processing and deep learning, tokens are the basic units of model processing, and are usually the smallest semantic or information units obtained after word segmentation, block segmentation or other forms of preprocessing of input data. In the embodiment of the present application, after the target image is preprocessed and split into multiple image blocks, and category tokens and perspective tokens are added, each element in the formed visual token sequence (such as category tokens, tokens corresponding to image blocks, perspective tokens, etc.) can be regarded as tokens. These tokens together constitute the basic units of model processing, carrying the semantic information and perspective information of the image.

[0099] The embedded representation with the perspective token is then subjected to self-attention. The self-attention mechanism captures the correlation between different positions in the image. This process yields the first feature representation of the target image, where the perspective token reflects the perspective characteristics of the target image, such as horizontal, top-down, or tilted perspectives.

[0100] The above step 103, namely "determining at least one target expert network in the hybrid expert network and the processing weights corresponding to the at least one target expert network based on the viewpoint features", is described in detail below in conjunction with an embodiment.

[0101] Based on the viewpoint features in the obtained first feature representation, at least one target expert network in the hybrid expert network is determined, along with corresponding processing weights, for processing the current target image. The hybrid expert network can be composed of multiple expert networks, which can be multiple parallel neural networks, such as feedforward neural networks and convolutional neural networks, each responsible for processing a specific type of input.

[0102] Optionally, the hybrid expert network in the embodiment of the present application can construct an expert grouping system based on predefined perspective categories. For example, if there are N predefined perspective categories and E expert networks, then these expert networks can be divided into N expert groups (e1, e2...e N ).

[0103] As one of the feasible ways, each expert group includes private experts and shared experts. The private experts are unique to the expert group and are responsible for processing features related to the viewpoint, that is, the viewpoint features in the first feature representation; the shared experts are shared by all expert groups and are responsible for processing common features that are not related to the viewpoint, that is, the non-viewpoint features in the first feature representation.

[0104] Take the division of two expert groups as an example, Figure 2 As shown, expert group e1 includes private experts and shared experts, where private experts specialize in processing features related to the group's perspective, and shared experts process common features that are irrelevant to the perspective; the same is true for expert group e2.

[0105] In this way, it is possible to achieve diversion and targeted processing of different perspective features and common features, avoid the interference of perspective changes in a single feature space on image semantic understanding, significantly improve the accuracy of image feature extraction, and thus improve the robustness of image retrieval in multi-perspective image scenarios.

[0106] It should be noted that, in addition to the above-mentioned achievable methods, it is also possible not to distinguish between private experts and shared experts.

[0107] In an embodiment of the present application, in order to determine the target expert network and its corresponding weight, the routing network can be used to analyze the perspective features to generate the processing probability distribution of multiple expert networks, and then based on the processing probability distribution of these expert networks, at least one target expert network is selected from the multiple expert networks. Finally, based on the processing probability distribution of the selected at least one target expert network, the processing weight corresponding to the at least one target expert network is determined.

[0108] Furthermore, the routing network in the embodiment of the present application may adopt a hierarchical routing mechanism, including image-level routing and word-level routing. Figure 2 ,The image-level routing is responsible for identifying the view category of the target image and selecting the corresponding expert group based on the view category, while the word-level routing selects the target expert network from the corresponding expert group for each word.

[0109] Image-level routing first predicts the target image's viewpoint category based on viewpoint features, such as distinguishing between horizontal, top-down, and oblique viewpoints. A mask is then generated based on the predicted viewpoint category. Elements in the mask correspond one-to-one to multiple expert networks in the hybrid expert network. The value of each element indicates whether the corresponding expert network is a candidate expert network—that is, whether it is likely to be selected as the target expert network to process the current image features.

[0110] For example, image-level routing can predict the view category of the target image based on the view word in the first feature representation as shown in formula (1):

[0111] z=argmax(R img (v view )) (1)

[0112] Wherein, z represents the predicted view category of the target image, argmax is a mathematical function used to find the input parameter that maximizes the function value in a given function output. In the embodiment of the present application, it represents determining the view category with the highest probability from the output results of the image-level routing; R img () represents the processing function of image-level routing; v view Represents the perspective tokens input to the routing network.

[0113] Then, as shown in formula (2), a mask is generated according to the predicted view category:

[0114] M j =0, if e j ∈e z

[0115] M j =-∞,otherwise (2)

[0116] Among them, M represents the mask, which is a vector of dimension E, used to limit the range of experts involved in feature processing; M j Represents the jth element in the mask vector, corresponding to the jth expert network e j , and e z Represents the expert group corresponding to perspective category z, including private experts and shared experts.

[0117] That is, if the expert network e j The expert group e corresponding to perspective category z z , then M j Assigning a value of 0 indicates that the expert network is a candidate expert network and can participate in subsequent calculations; otherwise, M jAssigning a value of -∞ indicates that the expert network is not a candidate expert network and will not participate in subsequent calculations. In this way, based on the predicted view category z, the mask M can be used to filter out expert networks that are not suitable for processing the view, retaining only the expert networks of the corresponding expert group, thus achieving preliminary screening of expert networks under view constraints.

[0118] Furthermore, word-level routing generates initial probability distributions for multiple expert networks based on each word in the first feature representation. This initial probability distribution is then combined with the mask to generate the processing probability distributions corresponding to the candidate expert networks. Based on this processing probability distribution, at least one target expert network is selected from the candidate expert networks, and the corresponding processing weights are determined for each target expert network. The processing weights for each target expert network reflect the importance and contribution of each target expert network in processing the current image features.

[0119] For example, word-level routing can be as shown in formula (3), first generating the initial probability distribution of multiple expert networks:

[0120] P=R tok (v i ) (3)

[0121] Among them, P represents each word v in the first feature representation i The corresponding initial probability distribution; R tok Indicates word-level routing; v i Represents the i-th word in the first feature representation, that is, the embedded representation E' with the perspective word attached v Class word V in cls , image block word V1……V s and perspective word V view .

[0122] That is, the word-level routing R tok With a single word v i As input, the initial probability distribution P of the word unit on all expert networks is calculated and output, thereby reflecting the adaptability of each expert network to the word unit.

[0123] Then, as shown in formula (4), the initial probability distribution and the mask are combined to obtain the processing probability distribution corresponding to the candidate expert networks.

[0124] P'=Softmax(P+M) (4)

[0125] Where P' represents the processed probability distribution after adding mask constraints and smoothing by Softmax; P is the initial probability distribution output by word-level routing; M is the mask vector generated by image-level routing; Softmax is the activation function used to convert the input vector into a probability distribution.

[0126] That is to say, the initial probability distribution P is added to the mask M. At this time, the probability of the corresponding position of the expert network in the non-target expert group is reduced by M. j =-∞ and approaches 0, and then the Softmax function is applied to the added vector to obtain the processing probability distribution P' of the expert network (ie, candidate expert network) that only contains the expert network in the target expert group.

[0127] After determining the processing probability distribution P' of the candidate expert networks, we can further select the top K expert networks as the target expert networks, denoted as e1, e2...e k .

[0128] Finally, as shown in formula (5), based on the processing probability distribution of the above K expert networks, their processing weights are determined.

[0129]

[0130] Among them, w ek represents the kth selected target expert network e k Corresponding processing weight; P'e k Represents the target expert network e in the processing probability distribution P' of the candidate expert network k The corresponding probability value; K represents the number of selected target expert networks.

[0131] That is to say, the K expert networks with the highest probability are selected from the processing probability distribution P' of the candidate expert networks as the target expert networks, and the probability value of each target expert network is divided by the sum of the probabilities of these K target expert networks to obtain the normalized weight w ek , to ensure that the sum of the processing weights is 1.

[0132] The above step 104, i.e., "obtaining second feature representations of the target image predicted by at least one target expert network based on the first feature representation," is described in detail below in conjunction with an embodiment.

[0133] Once the target expert networks are determined, the second feature representations predicted by these target expert networks based on the first feature representations can be obtained.

[0134] Optionally, the expert network in the embodiment of the present application may include private experts and shared experts, the private experts are specifically used to process features related to a specific perspective, that is, the perspective features in the first feature representation, while the shared experts are used to process general features that are not related to the perspective, that is, the non-perspective features in the first feature representation.

[0135] When the target expert network is a private expert, the second feature representation of the target image predicted by the private expert based on the perspective feature in the first feature representation is obtained; when the target expert network is a shared expert, the second feature representation of the target image predicted by the shared expert based on the non-perspective feature in the first feature representation is obtained.

[0136] Through different processing of private experts and shared experts, it is possible to achieve diversion processing of viewpoint features and non-viewpoint features, avoid the interference of viewpoint changes in a single feature space on image semantic understanding, significantly improve the accuracy of image feature extraction, and thus improve the robustness of image retrieval in multi-viewpoint image scenarios.

[0137] The above-mentioned step 105, namely "based on the processing weights corresponding to the at least one target expert network, performing weighted processing on the second feature representations respectively predicted by the at least one target expert network to obtain a third feature representation of the target image" is described in detail below in conjunction with an embodiment.

[0138] After obtaining the second feature representations predicted by each target expert network, these second feature representations are weighted according to the processing weights corresponding to each target expert network. The purpose of weighted processing is to fuse the outputs of multiple expert networks to obtain a comprehensive and more representative third feature representation.

[0139] Specifically, each second feature representation is multiplied by the processing weight of its corresponding target expert network. The processing weight is determined based on the routing network's analysis of viewpoint features and the expert network's selection process. It reflects the importance and credibility of each expert network in the current image feature processing task. The weighted second feature representations are aggregated and integrated, for example through summation, to obtain the final third feature representation of the target image.

[0140] Through this weighted processing method, the processing results of multiple target expert networks are integrated to obtain the third feature representation of the target image. This third feature representation fully integrates the processing advantages of different expert networks on image features and can more accurately characterize the target image.

[0141] For example, the weighted processing process can be shown as formula (6):

[0142]

[0143] Among them, v' i represents the final representation of the i-th word after being processed by multiple target expert networks; represents the kth selected target expert network e k corresponding processing weights; Represents the k-th target expert network for word v i The processed output (i.e., the second feature representation); K is the number of target expert networks.

[0144] The final representations of all word units are concatenated to obtain the third feature representation in the embodiment of the present application, namely, E" = {V' cls , V'1……V' s , V' view}.

[0145] The above step 106, namely "storing the target image and the third feature representation in the image library in correspondence with each other for image retrieval", is described in detail below with reference to an embodiment.

[0146] After obtaining the third feature representation of the target image, the target image and the corresponding third feature representation can be stored in an image repository for subsequent applications such as image retrieval. An image repository is typically a data storage structure specifically designed to store a large number of images and their corresponding feature representations. It can be a database, file system, or other suitable data storage solution.

[0147] Furthermore, the third feature representation is composed of a weighted aggregated word sequence, including the category word V' cls , image block word V'1...V' s and perspective word V' view Among them, the category word V' cls It is a global semantic carrier that represents the overall characteristics of the pedestrian in the target image (such as clothing color, body shape, etc.). Therefore, in the embodiment of the present application, only the category word V' in the third feature representation can be stored. cls , to reduce storage overhead.

[0148] In this case, during storage, it is only necessary to establish a correspondence between the target image and the category terms in its third feature representation, for example by assigning a unique identifier to each image and then storing the identifier in association with the corresponding category terms. This allows subsequent image retrieval to be performed when a text description is given. A text encoder can then be used to extract the text features of the image description text, and then these text features can be matched and compared with the category terms of multiple images stored in the image library. By calculating similarity and other methods, images that are most similar and relevant to the image description text can be quickly and accurately retrieved from the image library, thereby achieving efficient image retrieval capabilities.

[0149] For example, in smart security and other scenarios requiring cross-camera tracking, when searching for a target pedestrian, the system can quickly locate the target pedestrian by matching the target image and its category tokens stored in the image library with the textual features of the image description. Because the stored category tokens effectively process features from different viewpoints, they provide accurate feature support for image retrieval in complex viewpoint scenarios, improving retrieval accuracy and efficiency.

[0150] All word units included in the third feature representation may also be stored in correspondence with the target image.

[0151] Based on the above image feature extraction method, the embodiment of the present application also provides an image retrieval method. Figure 3 This is a flowchart of an image retrieval method provided in an embodiment of the present application. This method can be performed by an image retrieval device. The image retrieval device can be installed on any computer device with data storage and processing capabilities, such as a server or a computer terminal with strong data processing capabilities. The method may include the following steps:

[0152] Step 301: Obtain image description text and extract text features of the image description text.

[0153] Step 302: Match the text features with the third feature representations of multiple images stored in the image library respectively.

[0154] The third feature representation is obtained based on the above-mentioned image feature extraction method.

[0155] Step 303: Based on the matching result, determine the retrieved image from the multiple images.

[0156] As can be seen, the present embodiment extracts text features and matches them with the third feature representation in the image library, then determines the retrieved image from multiple images based on the matching results. Because the third feature representation incorporates multi-view information, it can significantly improve the accuracy and robustness of retrieval, effectively addressing the shortcomings of traditional methods with multi-source, heterogeneous view data.

[0157] The following describes in detail the steps in the above process and the effects that can be further produced in conjunction with the embodiments.

[0158] First, the above step 301, namely "obtaining image description text and extracting text features of the image description text", is described in detail with reference to the embodiment.

[0159] In image retrieval tasks, users typically express their retrieval requirements through natural language descriptions. For example, they might enter image descriptions such as "pedestrians wearing red coats" or "people running in a park." To convert these image descriptions into a form that can be matched with image features, feature extraction is required.

[0160] The process of text feature extraction usually includes the following steps:

[0161] Text preprocessing: Perform operations such as word segmentation and stop word removal on the input image description text to extract key semantic information.

[0162] Word embedding: Mapping each word or phrase in the image description text into a high-dimensional vector space. These vectors can capture the semantic relationship between words.

[0163] Text feature aggregation: The embedding vectors of all words in the image description are aggregated to obtain a feature vector that represents the entire image description. The aggregation method can be simple average pooling or a more complex attention mechanism to highlight the more important semantic information in the text.

[0164] Through the above steps, the image description text input by the user can be converted into a fixed-size text feature vector, which can effectively represent the semantic content of the text and provide a basis for further image matching.

[0165] In actual applications, the image description text entered by the user is a description of a preset target type, and the retrieved images are images containing the preset target type. For example, in pedestrian retrieval, if the user enters "male wearing a red coat", the retrieval process will focus on the words "wearing", "red", "coat", and "male", thereby retrieving images containing males wearing red coats from the image library.

[0166] In this way, the user's retrieval intention can be understood more accurately, and the image that best matches the image description text can be retrieved from the image library.

[0167] The above step 302, ie, "matching the text features with the third feature representations of the plurality of images stored in the image library respectively", is described in detail with reference to the embodiment.

[0168] After extracting text features, they can be matched with image features stored in the image library. The image features stored in the image library are third-party feature representations obtained through the aforementioned image feature extraction method. These feature representations integrate image information from different perspectives, providing a more comprehensive description of the image content.

[0169] The matching process can generally include the following steps:

[0170] Feature alignment: Ensure that text features and image features are in the same feature space for effective comparison. If the dimensions of text features and image features are inconsistent, alignment may need to be performed through dimensionality reduction or dimensionality increase.

[0171] Similarity calculation: Calculate the similarity between the text features and the third feature representation of each image in the image library. Common similarity calculation methods include cosine similarity, Euclidean distance, or Hamming distance. Cosine similarity measures the directional similarity between two vectors and is suitable for high-dimensional feature spaces; Euclidean distance measures the actual distance between two vectors in space and is suitable for low-dimensional feature spaces.

[0172] Sorting and screening: The images in the image library are sorted based on the calculated similarity. Typically, several images with the highest similarity are selected as candidate search results, and further screening and verification are performed to remove possible false matches.

[0173] Through the above matching steps, the image that best matches the image description text entered by the user can be quickly found from the image library, providing the user with accurate retrieval results.

[0174] The above step 303 , namely “determining a retrieved image from multiple images based on the matching result”, is described in detail with reference to the embodiments.

[0175] After completing the matching of text features and image features, the final retrieval image is determined from the image library based on the matching results. The specific steps are as follows:

[0176] Determine the retrieval threshold: Set a similarity threshold based on the needs of the actual application scenario. An image is considered a valid retrieval result only when the similarity between the image and the text description exceeds this threshold. The threshold can be determined empirically or experimentally to balance retrieval precision and recall.

[0177] Select search results: From the sorted candidate images, select images with a similarity above the threshold as the final search results. If multiple images have similarities above the threshold, select the images with the highest similarity as the search results, or perform further screening and optimization based on specific needs.

[0178] Results display: The determined search images are displayed to the user, who can further adjust the search requirements or perform other operations based on the search results. At the same time, the system can also record user feedback information for subsequent model optimization and improvement.

[0179] Through the above steps, the image retrieval method of the embodiment of the present application can effectively find the image that best matches the text description entered by the user from a large number of images, significantly improving the accuracy and efficiency of image retrieval, and providing users with a more convenient and efficient retrieval experience.

[0180] like Figure 4 The figure shows a schematic diagram of the principle of the image retrieval method provided by an embodiment of the present application. In the image feature extraction part, the embedding layer network, the self-attention layer network, the image-level routing, the word-level routing and the hybrid expert network work together to generate a third feature representation of the target image, and store the category words and the target image in the third feature representation in the image library. In the image retrieval part, the text encoder extracts text features from the input image description text, and matches the text features with the category words of multiple images stored in the image library respectively, and based on the matching results, the required image is accurately retrieved from the image library.

[0181] The image retrieval method of the embodiment of the present application has a wide range of application scenarios, and is particularly suitable for the following fields:

[0182] Intelligent security: By inputting a description of the suspect, the target person's image can be quickly retrieved from the surveillance video, improving the efficiency and accuracy of security monitoring.

[0183] Intelligent transportation: In traffic monitoring systems, images of relevant vehicles can be quickly retrieved by describing license plate numbers or vehicle characteristics, providing support for traffic management and law enforcement.

[0184] E-commerce: In product image retrieval, users can enter a product description and quickly find product images that match the description, improving the shopping experience.

[0185] The image retrieval model in the above process can be pre-trained. Figure 5 This is a flowchart of a method for training an image retrieval model provided in an embodiment of the present application. This method can be performed by an image retrieval model training device. The image retrieval model training device can be installed on any computer device with data storage and processing capabilities, such as a server or a computer terminal with strong data processing capabilities. The method may include the following steps:

[0186] Step 501: Acquire training data including a plurality of training samples, where the training samples include: image samples and image description texts corresponding to the image samples.

[0187] Step 502: Training an image retrieval model based on the training data, wherein the training includes:

[0188] Extract text features of image description text;

[0189] An image sample is input into an image retrieval model, and the image retrieval model extracts a first feature representation of the image sample, where the first feature representation includes a view feature; based on the view feature, at least one target expert network in the hybrid expert network and processing weights corresponding to at least one target expert network are determined; a second feature representation of the image sample is obtained, which is predicted by at least one target expert network based on the first feature representation; and based on the processing weights corresponding to at least one target expert network, a weighted processing is performed on the second feature representation predicted by at least one target expert network to obtain a third feature representation of the image sample;

[0190] The model parameters of the image retrieval model are updated using the loss function value corresponding to the first training objective, where the first training objective includes minimizing the distance between the third feature representation and the text feature.

[0191] It can be seen from the above process that the training method of the image retrieval model proposed in the embodiment of the present application combines a hybrid expert network and a hierarchical routing mechanism. Based on the characteristics of multi-view images, it realizes the effective processing of different view features and precise matching of cross-modal features, which helps to improve the matching accuracy of text and images of the model in complex view scenes, thereby improving the accuracy and reliability of retrieval.

[0192] The following describes in detail the steps in the above process and the effects that can be further produced in conjunction with the embodiments.

[0193] First, the above step 501, namely "obtaining training data including multiple training samples", is described in detail with reference to the embodiment.

[0194] In the embodiments of the present application, image samples may include, but are not limited to, images containing preset types of objects, such as images containing pedestrians. These images may be captured from different perspectives, such as horizontal, top-down, and oblique perspectives. These image samples are also annotated with image description text.

[0195] Furthermore, since the embodiment of the present application introduces perspective features, these image samples can also be labeled with real perspective categories.

[0196] Next, the above step 502, namely “training the image retrieval model based on the training data”, is described in detail with reference to an embodiment.

[0197] In an embodiment of the present application, text features of an image description text are extracted; and an image sample is input into an image retrieval model, and a first feature representation of the image sample is extracted by the image retrieval model, the first feature representation including a view feature; based on the view feature, at least one target expert network in the hybrid expert network and processing weights corresponding to at least one target expert network are determined; a second feature representation of the image sample predicted by at least one target expert network based on the first feature representation is obtained; based on the processing weights corresponding to at least one target expert network, a weighted processing is performed on the second feature representation predicted by at least one target expert network to obtain a third feature representation of the image sample; and the model parameters of the image retrieval model are updated using the loss function value corresponding to the first training objective, the first training objective including: minimizing the distance between the third feature representation and the text feature.

[0198] The image retrieval model consists of an embedding layer network and a self-attention layer network. During training, the embedding layer network extracts an embedded representation of the image sample, and then adds perspective tokens to the embedded representation of the image sample. The self-attention layer network then performs self-attention processing on the embedded representation with the perspective tokens, resulting in a first feature representation of the image sample. The perspective tokens in the first feature representation reflect the perspective features.

[0199] Furthermore, the image retrieval model also includes a routing network and a hybrid expert network, which is composed of multiple expert networks. During training, the routing network is used to generate processing probability distributions for the multiple expert networks based on viewpoint features. Based on the processing probability distributions, at least one target expert network is selected from the multiple expert networks. Based on the processing probability distributions of the selected at least one target expert network, processing weights corresponding to the at least one target expert network are determined.

[0200] Furthermore, the routing network can also include image-level routing and word-level routing. During the training process, image-level routing is used to predict the view category of an image sample based on view features; a mask is generated based on the view category, with elements in the mask corresponding one-to-one to the multiple expert networks, and the value of the element indicates whether the corresponding expert network is a candidate expert network; word-level routing is used to generate initial probability distributions of multiple expert networks based on each word in the first feature representation; based on the mask and the initial probability distribution, the corresponding processing probability distributions of the candidate expert networks are obtained; finally, based on the corresponding processing probability distributions of the candidate expert networks, at least one target expert network is selected from the candidate expert networks.

[0201] Furthermore, the expert network may also include: a private expert network and a shared expert network. During the training process, when the target expert network is a private expert network, the second feature representation of the image sample predicted by the private expert network based on the viewpoint features in the first feature representation is obtained; when the target expert network is a shared expert network, the second feature representation of the image sample predicted by the shared expert network based on the non-viewpoint features in the first feature representation is obtained.

[0202] It should also be noted that in this embodiment of the present application, the training samples also include the actual viewpoint categories corresponding to the image samples. Therefore, when training the image retrieval model, in addition to the first training objective described above, the viewpoint tokens can also be updated using the loss function corresponding to the second training objective: minimizing the difference between the viewpoint categories predicted by image-level routing and the actual viewpoint categories.

[0203] based on Figure 5 The training method can effectively train the image retrieval model and use the trained image retrieval model to achieve Figure 1 The method shown enables the image retrieval model to generate a third feature representation of the target image.

[0204] The specific structure of the image retrieval model mentioned above (including the embedding layer network, self-attention layer network, routing network and hybrid expert network) has been described in detail in the previous article and will not be repeated here.

[0205] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0206] According to another embodiment, an image feature extraction device is provided. Figure 6 A schematic block diagram of the image feature extraction device according to an embodiment is shown as follows: Figure 6 As shown, the apparatus 600 includes:

[0207] An acquisition unit 601 is configured to acquire a target image;

[0208] A first feature extraction unit 602 is configured to extract a first feature representation of the target image, wherein the first feature representation includes a viewing angle feature;

[0209] The expert network determining unit 603 is configured to determine at least one target expert network in the hybrid expert network and a processing weight corresponding to each of the at least one target expert network based on the view feature;

[0210] A second feature extraction unit 604 is configured to obtain a second feature representation of the target image predicted by the at least one target expert network based on the first feature representation;

[0211] A third feature extraction unit 605 is configured to perform weighted processing on the second feature representations respectively predicted by the at least one target expert network based on the processing weights respectively corresponding to the at least one target expert network to obtain a third feature representation of the target image;

[0212] The storage unit 606 is configured to store the target image and the third feature representation in an image library in correspondence with each other for image retrieval.

[0213] Optionally, the first feature extraction unit 602 is specifically configured to:

[0214] extracting an embedded representation of the target image;

[0215] Additional perspective tokens are added based on the embedded representation of the target image, where the perspective tokens are pre-trained and used to reflect the perspective features of the target image;

[0216] The embedded representation with the perspective word added is subjected to self-attention processing to obtain a first feature representation of the target image.

[0217] Optionally, the hybrid expert network is composed of multiple expert networks;

[0218] Then the expert network determining unit 603 is specifically configured as follows:

[0219] generating, using a routing network, processing probability distributions of the plurality of expert networks based on the view features;

[0220] selecting at least one target expert network from the plurality of expert networks based on the processing probability distribution;

[0221] Based on the processing probability distribution of the selected at least one target expert network, processing weights corresponding to the at least one target expert network are determined.

[0222] Optionally, the routing network includes image-level routing and word-level routing;

[0223] When the expert network determining unit 603 generates the processing probability distribution of the plurality of expert networks based on the viewing angle feature using the routing network, it is configured as follows:

[0224] The image-level routing is used to predict the view category of the target image based on the view feature; a mask is generated based on the view category, wherein elements in the mask correspond one-to-one to the multiple expert networks, and the values ​​of the elements represent whether the corresponding expert network is a candidate expert network; the word-level routing is used to generate initial probability distributions of the multiple expert networks based on each word in the first feature representation; and based on the mask and the initial probability distributions, processing probability distributions corresponding to the candidate expert networks are obtained.

[0225] When the expert network determining unit 603 selects at least one target expert network from the multiple expert networks based on the processing probability distribution, it is configured to:

[0226] At least one target expert network is selected from the candidate expert networks based on the processing probability distributions respectively corresponding to the candidate expert networks.

[0227] Optionally, the expert network includes: private experts and shared experts;

[0228] The second feature extraction unit 604 is specifically configured to:

[0229] When the target expert network is a private expert, obtaining a second feature representation of the target image predicted by the private expert based on the view feature in the first feature representation;

[0230] When the target expert network is a shared expert, a second feature representation of the target image predicted by the shared expert based on the non-view features in the first feature representation is obtained.

[0231] According to another embodiment, an image retrieval device is provided. Figure 7 A schematic block diagram of the image retrieval device according to an embodiment is shown as follows. Figure 7 As shown, the apparatus 700 includes:

[0232] The text processing unit 701 is configured to obtain image description text and extract text features of the image description text;

[0233] A matching unit 702 is configured to match the text feature with third feature representations of multiple images stored in an image library, respectively; wherein the third feature representation is obtained based on any of the above-mentioned image feature extraction methods;

[0234] The retrieval unit 703 is configured to determine a retrieved image from the multiple images based on the matching result.

[0235] Optionally, the image description text is a description text for a preset type of target;

[0236] The retrieved image is an image containing the preset type of target.

[0237] According to another embodiment, a training device for an image retrieval model is provided. Figure 8 A schematic block diagram of a training device for the image retrieval model according to one embodiment is shown, Figure 8 As shown, the apparatus 800 includes:

[0238] The training data acquisition unit 801 is configured to acquire training data including a plurality of training samples, wherein the training samples include: image samples and image description texts corresponding to the image samples;

[0239] The training unit 802 is configured to train the image retrieval model based on the training data, wherein the training includes:

[0240] Extracting text features of the image description text;

[0241] Inputting the image sample into an image retrieval model, extracting a first feature representation of the image sample by the image retrieval model, wherein the first feature representation includes a view feature; determining at least one target expert network in the hybrid expert network and processing weights corresponding to the at least one target expert network based on the view feature; obtaining a second feature representation of the image sample predicted by the at least one target expert network based on the first feature representation; performing weighted processing on the second feature representations predicted by the at least one target expert network based on the processing weights corresponding to the at least one target expert network to obtain a third feature representation of the image sample;

[0242] The model parameters of the image retrieval model are updated using the loss function value corresponding to the first training objective, where the first training objective includes: minimizing the distance between the third feature representation and the text feature.

[0243] Optionally, the image retrieval model includes an embedding layer network and a self-attention layer network;

[0244] When the training unit 802 extracts the first feature representation of the image sample, it is configured to:

[0245] Extracting an embedding representation of the image sample using the embedding layer network, and adding a perspective word based on the embedding representation of the image sample;

[0246] The self-attention layer network is used to perform self-attention processing on the embedded representation with the perspective word-unit attached to obtain a first feature representation of the image sample, wherein the perspective word-unit in the first feature representation reflects the perspective feature.

[0247] Optionally, the image retrieval model further includes a routing network and the hybrid expert network, and the hybrid expert network is composed of multiple expert networks;

[0248] The training unit 802 determines at least one target expert network in the hybrid expert network and the processing weights corresponding to the at least one target expert network based on the view feature, and is configured as follows:

[0249] The routing network is used to generate a processing probability distribution of the multiple expert networks based on the perspective features; based on the processing probability distribution, at least one target expert network is selected from the multiple expert networks; based on the processing probability distribution of the selected at least one target expert network, the processing weights corresponding to the at least one target expert network are determined.

[0250] Optionally, the routing network includes image-level routing and word-level routing;

[0251] The training unit 802 generates processing probability distributions of the multiple expert networks based on the view features using the routing network, and is configured to:

[0252] The image-level routing is used to predict the view category of the image sample based on the view feature; a mask is generated based on the view category, wherein the elements in the mask correspond one-to-one to the multiple expert networks, and the values ​​of the elements represent whether the corresponding expert network is a candidate expert network; the word-level routing is used to generate initial probability distributions of the multiple expert networks based on each word in the first feature representation; and based on the mask and the initial probability distribution, processing probability distributions corresponding to the candidate expert networks are obtained.

[0253] The selecting at least one target expert network from the plurality of expert networks based on the processing probability distribution includes:

[0254] At least one target expert network is selected from the candidate expert networks based on the processing probability distributions respectively corresponding to the candidate expert networks.

[0255] Optionally, the expert network includes: a private expert network and a shared expert network;

[0256] The training unit 802 obtains the second feature representations of the image samples respectively predicted by the at least one target expert network based on the first feature representation, and is configured to:

[0257] When the target expert network is a private expert network, obtaining a second feature representation of the image sample predicted by the private expert network based on the view feature in the first feature representation;

[0258] When the target expert network is a shared expert network, a second feature representation of the image sample predicted by the shared expert network based on the non-view features in the first feature representation is obtained.

[0259] Optionally, the training sample further includes: a real viewing angle category corresponding to the image sample;

[0260] The training unit 802 is further configured to:

[0261] The perspective word is updated using a loss function value corresponding to a second training objective, wherein the second training objective includes minimizing the difference between the perspective category predicted by the image-level routing and the true perspective category.

[0262] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the data batch processing device embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For the relevant parts, refer to the partial description of the method embodiment. The data batch processing device embodiment described above is only exemplary, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0263] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0264] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0265] And an electronic device comprising:

[0266] one or more processors; and

[0267] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0268] The present application also provides a computer program product, comprising a computer program, which implements the steps of any one of the methods described in the aforementioned method embodiments when executed by a processor.

[0269] in, Figure 9 The electronic device architecture is shown as an example, and may include a processor 910, a video display adapter 911, a disk drive 912, an input / output interface 913, a network interface 914, and a memory 920. The processor 910, the video display adapter 911, the disk drive 912, the input / output interface 913, the network interface 914, and the memory 920 may be communicatively connected via a communication bus 930.

[0270] Among them, the processor 910 can be implemented by a general CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solutions provided in this application.

[0271] The memory 920 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 920 can store an operating system 921 for controlling the operation of the electronic device 900, and a basic input and output system (BIOS) 922 for controlling the low-level operations of the electronic device 900. In addition, a web browser 923, a data storage management system 924, as well as an image feature extraction device 600, an image retrieval device 700, an image retrieval model training device 800, etc. can also be stored. The above-mentioned image feature extraction device 600, image retrieval device 700, and image retrieval model training device 800 can be the application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided by the present application is implemented by software or firmware, the relevant program code is stored in the memory 920 and is called and executed by the processor 910.

[0272] The input / output interface 913 is used to connect input / output modules to implement information input and output. The input / output modules can be configured as components in the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0273] The network interface 914 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0274] The bus 930 comprises a pathway for transmitting information between the various components of the device (eg, the processor 910 , the video display adapter 911 , the disk drive 912 , the input / output interface 913 , the network interface 914 , and the memory 920 ).

[0275] It should be noted that although the above device only shows the processor 910, video display adapter 911, disk drive 912, input / output interface 913, network interface 914, memory 920, bus 930, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may also include only the components necessary to implement the solution of the present application, and does not necessarily include all the components shown in the figure.

[0276] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer program product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application or certain parts of the embodiments.

[0277] The above is a detailed introduction to the technical solutions provided by this application. Specific examples are used herein to illustrate the principles and implementation methods of this application. The description of the above embodiments is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the contents of this specification should not be understood as limiting this application.

Claims

1. A method for extracting image features, characterized in that: include: Acquire the target image; extracting a first feature representation of the target image, wherein the first feature representation includes a viewing angle feature; Determining at least one target expert network in the hybrid expert network and a processing weight corresponding to each of the at least one target expert network based on the perspective feature; Obtaining second feature representations of the target image predicted by the at least one target expert network based on the first feature representation; performing weighted processing on the second feature representations respectively predicted by the at least one target expert network based on the processing weights respectively corresponding to the at least one target expert network to obtain a third feature representation of the target image; The target image and the third feature representation are correspondingly stored in an image library for image retrieval.

2. The method according to claim 1, characterized in that The extracting a first feature representation of the target image includes: extracting an embedded representation of the target image; Additional perspective tokens are added based on the embedded representation of the target image, where the perspective tokens are pre-trained and used to reflect the perspective features of the target image; The embedded representation with the perspective word added is subjected to self-attention processing to obtain a first feature representation of the target image.

3. The method according to claim 1, characterized in that The hybrid expert network is composed of multiple expert networks; The determining, based on the view feature, at least one target expert network in the hybrid expert network and the processing weights corresponding to the at least one target expert network respectively includes: generating, using a routing network, processing probability distributions of the plurality of expert networks based on the view features; selecting at least one target expert network from the plurality of expert networks based on the processing probability distribution; Based on the processing probability distribution of the selected at least one target expert network, processing weights corresponding to the at least one target expert network are determined.

4. The method according to claim 3, characterized in that The routing network includes image-level routing and word-level routing; The generating of the processing probability distribution of the plurality of expert networks based on the view features by using a routing network includes: The image-level routing is used to predict the view category of the target image based on the view feature; a mask is generated based on the view category, wherein elements in the mask correspond one-to-one to the multiple expert networks, and the values ​​of the elements represent whether the corresponding expert network is a candidate expert network; the word-level routing is used to generate initial probability distributions of the multiple expert networks based on each word in the first feature representation; and based on the mask and the initial probability distributions, processing probability distributions corresponding to the candidate expert networks are obtained. The selecting at least one target expert network from the plurality of expert networks based on the processing probability distribution includes: At least one target expert network is selected from the candidate expert networks based on the processing probability distributions respectively corresponding to the candidate expert networks.

5. The method according to claim 3 or 4, characterized in that The expert network includes: private experts and shared experts; The obtaining of the second feature representation of the target image predicted by the at least one target expert network based on the first feature representation includes: When the target expert network is a private expert, obtaining a second feature representation of the target image predicted by the private expert based on the view feature in the first feature representation; When the target expert network is a shared expert, a second feature representation of the target image predicted by the shared expert based on the non-view features in the first feature representation is obtained.

6. An image retrieval method, characterized in that: The method comprises: Obtaining image description text and extracting text features of the image description text; Matching the text features with third feature representations of multiple images stored in an image library respectively; wherein the third feature representations are obtained based on the image feature extraction method according to any one of claims 1 to 5; Based on the matching result, a retrieved image is determined from the plurality of images.

7. A method for training an image retrieval model, characterized in that: include: Acquire training data including a plurality of training samples, wherein the training samples include: image samples and image description texts corresponding to the image samples; Training an image retrieval model based on the training data, wherein the training comprises: Extracting text features of the image description text; Inputting the image sample into an image retrieval model, extracting a first feature representation of the image sample by the image retrieval model, wherein the first feature representation includes a view feature; determining at least one target expert network in the hybrid expert network and processing weights corresponding to the at least one target expert network based on the view feature; obtaining a second feature representation of the image sample predicted by the at least one target expert network based on the first feature representation; performing weighted processing on the second feature representations predicted by the at least one target expert network based on the processing weights corresponding to the at least one target expert network to obtain a third feature representation of the image sample; The model parameters of the image retrieval model are updated using the loss function value corresponding to the first training objective, where the first training objective includes: minimizing the distance between the third feature representation and the text feature.

8. The method according to claim 7, characterized in that The image retrieval model includes an embedding layer network and a self-attention layer network; The extracting a first feature representation of the image sample includes: Extracting an embedded representation of the image sample using the embedding layer network, and adding a perspective word based on the embedded representation of the image sample, wherein the perspective word is used to reflect the perspective feature of the target image; The self-attention layer network is used to perform self-attention processing on the embedded representation with the perspective word added thereto to obtain a first feature representation of the image sample.

9. The method according to claim 8, characterized in that The image retrieval model further includes a routing network and the hybrid expert network, wherein the hybrid expert network is composed of multiple expert networks; The determining, based on the view feature, at least one target expert network in the hybrid expert network and the processing weights corresponding to the at least one target expert network respectively includes: The routing network is used to generate a processing probability distribution of the multiple expert networks based on the perspective features; based on the processing probability distribution, at least one target expert network is selected from the multiple expert networks; based on the processing probability distribution of the selected at least one target expert network, the processing weights corresponding to the at least one target expert network are determined.

10. The method according to claim 9, characterized in that The routing network includes image-level routing and word-level routing; The generating, by using the routing network, the processing probability distribution of the plurality of expert networks based on the viewpoint features comprises: The image-level routing is used to predict the view category of the image sample based on the view feature; a mask is generated based on the view category, wherein the elements in the mask correspond one-to-one to the multiple expert networks, and the values ​​of the elements represent whether the corresponding expert network is a candidate expert network; the word-level routing is used to generate initial probability distributions of the multiple expert networks based on each word in the first feature representation; and based on the mask and the initial probability distribution, processing probability distributions corresponding to the candidate expert networks are obtained. The selecting at least one target expert network from the plurality of expert networks based on the processing probability distribution includes: At least one target expert network is selected from the candidate expert networks based on the processing probability distributions respectively corresponding to the candidate expert networks.

11. The method according to claim 9 or 10, characterized in that The expert network includes: a private expert network and a shared expert network; The obtaining of second feature representations of the image samples respectively predicted by the at least one target expert network based on the first feature representation includes: When the target expert network is a private expert network, obtaining a second feature representation of the image sample predicted by the private expert network based on the view feature in the first feature representation; When the target expert network is a shared expert network, a second feature representation of the image sample predicted by the shared expert network based on the non-view features in the first feature representation is obtained.

12. The method according to claim 10, characterized in that The training samples also include: the real viewing angle category corresponding to the image sample; The training also includes: The perspective word is updated using a loss function value corresponding to a second training objective, wherein the second training objective includes minimizing the difference between the perspective category predicted by the image-level routing and the true perspective category.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

14. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 12.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images