Search methods, devices, electronic equipment and readable storage media

Abstract

This application discloses a retrieval method, apparatus, electronic device, and readable storage medium, belonging to the field of information technology. The retrieval method includes: obtaining a retrieval object and extracting its initial semantic features; generating a semantic feature set of the retrieval object based on its initial semantic features; generating a semantic feature set corresponding to each of N target objects, where N is an integer greater than 1, based on the initial semantic features of each target object and the semantic feature set of the retrieval object; determining the matching degree between each target object and the retrieval object based on the matching degree between each target object and the retrieval object; and determining the target object as the retrieval result from the N target objects based on the matching degree of each target object and the retrieval object. This application determines the retrieval result based on two diverse semantic feature sets, thus making the retrieval results more diverse.

Description

Technical Field

[0001] This application relates to the field of information technology, and more specifically, to a retrieval method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] The process of finding target objects related to the search object based on the search object proposed by the user, using certain technical means, is usually called retrieval or search.

[0003] Currently, while the search results obtained using related technologies can achieve relatively high accuracy, the diversity of the search results is low. For example, when a user enters the query text "pet," the user's actual intention may be to search for images of pet supplies or pet markets. However, after performing the search using related technologies, the results usually only return images of dogs or cats, resulting in low diversity of search results. Summary of the Invention

[0004] In view of the above problems, embodiments of this application provide a retrieval method, apparatus, electronic device, and readable storage medium, aiming to improve the diversity of retrieval results.

[0005] In a first aspect, embodiments of this application provide a retrieval method, the method comprising: obtaining a retrieval object and extracting initial semantic features of the retrieval object; generating a semantic feature set of the retrieval object based on the initial semantic features of the retrieval object; generating a semantic feature set corresponding to each of the N target objects based on the initial semantic features of each target object, where N is an integer greater than 1; determining the matching degree between each target object and the retrieval object based on the semantic feature set of each target object and the semantic feature set of the retrieval object; and determining a target object as a retrieval result from the N target objects based on the matching degree between each target object and the retrieval object.

[0006] Secondly, embodiments of this application provide a retrieval device, comprising an initial semantic feature extraction module, a semantic feature set generation module, a matching degree determination module, and a retrieval result determination module. The initial semantic feature extraction module obtains a retrieval object and extracts its initial semantic features. The semantic feature set generation module generates a semantic feature set of the retrieval object based on its initial semantic features, and further generates a semantic feature set corresponding to each of N target objects based on the initial semantic features of each target object, where N is an integer greater than 1. The matching degree determination module determines the matching degree between each target object and the retrieval object based on the semantic feature set of each target object and the semantic feature set of the retrieval object. The retrieval result determination module determines the target object as the retrieval result from the N target objects based on the matching degree between each target object and the retrieval object.

[0007] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory; one or more programs are stored in the memory and configured to be executed by the processor to implement the above-described method.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing program code, wherein the above-described method is executed when the program code is run by a processor.

[0009] Fifthly, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method described above.

[0010] The technical solution provided in this application can bring the following beneficial effects:

[0011] A semantic feature set for the retrieval object is generated based on the initial semantic features of the retrieval object, and a semantic feature set for the target object is generated based on the initial semantic features of the target object. This means that the retrieval object and the target object are no longer represented by a single initial semantic feature, but by a variety of semantic features. Then, based on the semantic feature sets of the target object and the retrieval object, the matching degree between the target object and the retrieval object is determined, and the retrieval results are determined based on the matching degrees of multiple target objects with the retrieval object. In this application, the matching degree for the retrieval object and the target object is not determined based on two single initial semantic features, but rather on two sets of diverse semantic features, thereby determining the retrieval results and making the retrieval results more diverse. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of an implementation environment provided in one embodiment of this application;

[0014] Figure 2 This is a schematic diagram of an implementation environment provided in another embodiment of this application;

[0015] Figure 3 This is a flowchart of a retrieval method proposed in an embodiment of this application;

[0016] Figure 4 This is a schematic diagram of a plurality of first intermediate features proposed in an embodiment of this application;

[0017] Figure 5 This is a schematic diagram illustrating the use of a model to determine the matching degree according to an embodiment of this application;

[0018] Figure 6 This is a flowchart of the training process of a semantic generation model proposed in one embodiment of this application;

[0019] Figure 7 This is a schematic diagram of a training semantic generation model proposed in an embodiment of this application;

[0020] Figure 8 This is a flowchart illustrating the determination of distance penalty loss components according to an embodiment of this application;

[0021] Figure 9 This is a flowchart of a retrieval method proposed in another embodiment of this application;

[0022] Figure 10 This is a flowchart of a retrieval method proposed in another embodiment of this application;

[0023] Figure 11 This is a schematic diagram of a training semantic generation model proposed in another embodiment of this application;

[0024] Figure 12 This is an embodiment of the present application based on Figure 9 A schematic diagram illustrating the semantic generation model performing a retrieval task.

[0025] Figure 13 This is a schematic diagram of a retrieval device 1100 according to an embodiment of this application;

[0026] Figure 14 This is a schematic diagram of a retrieval device 1200 according to another embodiment of this application;

[0027] Figure 15 A structural block diagram of an electronic device for performing a retrieval method according to an embodiment of this application is shown. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0029] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0030] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0031] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0032] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in multiple fields. The technical solutions provided in the embodiments of this application involve the application of machine learning in the field of data retrieval, specifically, a retrieval method.

[0033] In related technologies, search results typically achieve high accuracy, but the diversity of search results is low. For example, when a user enters the query text "pet," the user's actual intention may be to search for images of pet supplies or pet markets. However, after performing the search using related technologies, the results usually only return images of dogs or cats, resulting in low diversity of search results.

[0034] In view of this, embodiments of this application provide a retrieval method, apparatus, electronic device, and readable storage medium. Specifically, a pre-trained semantic generation model is used to automatically generate diverse semantic features of the retrieval object based on its initial semantic features. Then, diverse semantic features of each target object are automatically generated based on its initial semantic features. Finally, the matching degree between the retrieval object and each target object is determined based on the diverse semantic features of the retrieval object and each target object, and the retrieval result is determined based on the matching degree. The retrieval results determined by the retrieval method provided in this application are more diverse.

[0035] The following describes the implementation environment involved in the retrieval method provided in the embodiments of this application. (Reference) Figure 1 , Figure 1 This is a schematic diagram of an implementation environment provided in one embodiment of this application. For example... Figure 1 As shown, the implementation environment may include server 110, target object library 120, and terminal 130.

[0036] Among them, server 110 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. It can also be a dedicated or platform server that provides services such as vehicle networking, road network collaboration, vehicle-road collaboration, intelligent transportation, autonomous driving, industrial internet services, and data communication (such as 4G, 5G, etc.).

[0037] The target object library 120 can be configured on the server 110 or on other devices; this application does not impose any restrictions on this. Figure 1 In this embodiment, the target object library 120 is schematically configured on the server 110. The target object can be text, image, video, or news; this application does not restrict the data modality of the target object.

[0038] The terminal 130 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. Optionally, the terminal 130 may run a client, such as a browser client, instant messaging client, content interaction client, short video client, or shopping client. The terminal 130 and the server 110 can be directly or indirectly connected via wired or wireless communication, and this application does not impose any restrictions on this.

[0039] In specific implementation, server 110 receives the search object sent by terminal 130. Server 110 then executes the search method provided in this application to select several target objects matching the search object from the target object library 120 as search results. Here, the search object serves as the basis for the search, and the target objects are the objects to be selected. For example, when a user wants to search for images related to pets, the user can input the query text "pet" into terminal 130. Terminal 130 sends the query text "pet" to server 110. Server 110 then executes the search method provided in this application to select several images matching the query text "pet" from the image library as search results. Here, the query text "pet" is the search object, and the images are the target objects.

[0040] refer to Figure 2 , Figure 2 This is a schematic diagram of an implementation environment provided by another embodiment of this application. For example... Figure 2 As shown, the implementation environment may include a retrieval device 210 and a target object library 220.

[0041] The retrieval device 210 can be a computer device, which refers to an electronic device with data computing, processing, and storage capabilities. This computer device can be a terminal device such as a PC (Personal Computer), tablet computer, smartphone, smart speaker, wearable device, smart robot, or in-vehicle terminal. The retrieval device 210 has an information input device, which can be such as a mouse, keyboard, touchscreen, touchpad, microphone, or camera. For example, with a touchscreen, users can input information into the retrieval device 210 through touch and handwriting operations. With a microphone, users can input information into the retrieval device 210 by inputting voice. With a camera, users can input information into the retrieval device 210 by making predetermined gestures in front of the activated camera.

[0042] The target object library 220 can be configured on the retrieval device 210 or on other devices; this application does not impose any restrictions on this. Figure 2 In this embodiment, the target object library 220 is schematically configured on the retrieval device 210. The target object can be text, image, video, or news; this application does not limit the data modality of the target object.

[0043] In practice, the user inputs information into the retrieval device 210 via an information input device, and the retrieval device 210 obtains the retrieval object from the user's input information. The retrieval device 210, by executing the retrieval method provided in this application, selects several target objects matching the retrieval object from the target object library 220 as retrieval results. Here, the retrieval object serves as the basis for the retrieval, and the target object serves as the candidate object. For example, when a user wants to retrieve images related to pets, the user can handwrite the query text "pet" on the touchscreen of the retrieval device 210. The retrieval device 210, by executing the retrieval method provided in this application, selects several images matching the query text "pet" from the image library as retrieval results, and displays them to the user via the touchscreen. Here, the query text "pet" serves as the retrieval object, and the image serves as the target object.

[0044] It should be noted that the above combination Figure 1 and Figure 2 The two implementation environments provided are two examples of the various implementation environments of this application. For the sake of simplicity, the device used to perform the retrieval method of this application (such as the aforementioned server 110 or retrieval device 210) will be uniformly referred to as the execution subject.

[0045] In some implementations, the retrieval object and the target object may belong to the same data modality. For example, both the retrieval object and the target object may be text data, or both may be image data, or both may be video data, or both may be news data. For simplicity, the data modalities of the retrieval object and the target object are not exhaustively listed here. Taking the example that both the retrieval object and the target object are text data, the executing entity obtains the query text (i.e., the retrieval object), and the executing entity searches for relevant target text from multiple target texts (i.e., target objects) by executing the retrieval method in the embodiments of this application.

[0046] In other embodiments, the retrieval object and the target object may belong to different data modalities. For example, the retrieval object may be text data, and the target object may be image data. Or the retrieval object may be text data, and the target data may be video data. Or the retrieval object may be text data, and the target object may be news data. Or the retrieval object may be image data, and the target data may be text data. Or the retrieval object may be image data, and the target data may be news data. For simplicity, the data modalities of the retrieval object and the target object are not exhaustively listed here. Taking the retrieval object being text data and the target objects being all image data as an example, the executing entity obtains the query text (i.e., the retrieval object), and the executing entity searches for relevant target images from multiple target images (i.e., target objects) by executing the retrieval method in the embodiments of this application.

[0047] It should be noted that in some application scenarios, the executing entity can perform the retrieval task in stages. For example, in the first retrieval stage, the executing entity selects 100,000 target objects from 10 million target objects as the retrieval result for the first retrieval stage. In the second retrieval stage, the executing entity selects 10,000 target objects from the 100,000 target objects as the retrieval result for the second retrieval stage. In the third retrieval stage, the executing entity selects 1,000 target objects from the 10,000 target objects as the final retrieval result for this retrieval task. The retrieval method in this embodiment can be applied to each retrieval stage of the retrieval task.

[0048] For example, the retrieval method in this embodiment is applied to a later retrieval stage of the retrieval task. A later retrieval stage refers to a retrieval stage following the first retrieval stage, such as the last retrieval stage. Since the executing entity has already filtered out irrelevant target objects from a massive number of target objects in the earlier retrieval stages, in the later retrieval stages, the executing entity can select multiple relatively relevant and diverse target objects from the relatively relevant target objects as retrieval results by running the retrieval method in this embodiment.

[0049] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0050] refer to Figure 3 , Figure 3 This is a flowchart of a retrieval method proposed in an embodiment of this application, as follows: Figure 3 As shown, the retrieval method includes:

[0051] S310: Obtain the search object and extract the initial semantic features of the search object.

[0052] In this context, the search object serves as the basis for the search task, and the objective of the executing entity is to find target objects related to the search object. For example, in a search task, if a user wants to search for images related to pets and enters the query text "pets" into the search box, then the query text "pets" is the search object for that search task.

[0053] In some implementations, the search object can be proposed by the user. In these implementations, the executing entity obtains the search object directly input by the user, as detailed in the following references. Figure 2 The implementation environment is shown below. For example, when a user wants to search for content related to pets, the user can enter the query text "pet" into the execution entity, which then obtains the query text, which is the search object.

[0054] Alternatively, the executing entity may obtain the search object input by the executing entity at the other end of the network. For details, please refer to... Figure 1 The implementation environment is shown below. For example, when a user wants to search for content related to pets, the user can enter the query text "pet" on their terminal. The terminal sends the query text "pet" to the executing entity via the network. The executing entity obtains the query text, which is the search object.

[0055] In practice, when a user performs a search operation through a terminal, the terminal generates a search log in HTML format. This search log is included in the search request sent by the terminal to the server (the executing entity). Since this search log includes not only the user-input query text but also information such as the query timestamp, username, mobile phone model, search location, or search method, the server reads the query text from the HTML-formatted search log and uses this query text as the search object.

[0056] In addition, since the retrieved query text may also include special characters, such as "+", "-", or "\" characters entered incorrectly by the user, in order to improve the purity of the query text, the server can remove special characters contained in the predefined special character table from the query text and use the query text with special characters removed as the retrieval object.

[0057] In some implementations, the search object can also be generated by the executing entity. For example, when a user needs to search for a certain text, the executing entity can extract keywords from that text and use the extracted keywords as the search object. It should be noted that this application does not limit the method of obtaining the search object.

[0058] In this application, the initial semantic feature and the semantic feature set are two relative concepts. The number of features in the initial semantic feature is less than the number of features in the semantic feature set. For example, the initial semantic feature can be a single feature represented by a vector, while the semantic feature set includes multiple features, each represented by a different vector. In practical applications, if the executing entity generates multiple features based on the features of the retrieval object, then that single feature can be considered the initial semantic feature of the retrieval object, and the generated multiple features can be considered the semantic feature set of the retrieval object.

[0059] As mentioned earlier, the search object can be text data, image data, or video data, etc.

[0060] In applications where the retrieval object is text data, the Doc2Vec algorithm can be used to extract features of the retrieval object, and these extracted features can be used as the initial semantic features of the retrieval object. Alternatively, the Word2vec algorithm can also be used to extract features of the retrieval object, and these extracted features can also be used as the initial semantic features of the retrieval object. Taking the Doc2Vec algorithm as an example, it is an algorithm that converts text (such as sentences or paragraphs) into vectors. The vectors converted by the Doc2Vec algorithm are usually called word vectors, sentence vectors, or paragraph vectors, and these vectors are used to represent the syntactic structure features of the text. Specifically, when text of different lengths is input into the Doc2Vec algorithm, the algorithm outputs vectors of a specified length. In other words, the Doc2Vec algorithm can extract features of a fixed length from text. In practical implementation, the retrieval object (i.e., text data) can be input into a pre-trained Doc2Vec algorithm to obtain the vector output by the Doc2Vec algorithm, and this vector can be used as the initial semantic features of the retrieval object. It should be noted that this application does not limit the method of extracting the initial semantic features.

[0061] In applications where the retrieval object is image data, ResNet (Residual Neural Network) can be used to extract features of the retrieval object, and these extracted features can serve as the initial semantic features of the retrieval object. Alternatively, LeNet, VGG, or DenseNet (Densely Connected Convolutional Networks) can also be used to extract features of the retrieval object, and these extracted features can serve as the initial semantic features of the retrieval object. It should be noted that this application does not limit the method for extracting initial semantic features.

[0062] In applications where the retrieval object is video data, C3D or I3D networks can be used to extract features of the retrieval object, and these extracted features can serve as the initial semantic features of the retrieval object. It should be noted that this application does not limit the method for extracting the initial semantic features.

[0063] S320: Generate a set of semantic features of the search object based on the initial semantic features of the search object.

[0064] As mentioned earlier, the semantic feature set and the initial semantic features are two relative concepts; the semantic feature set has more features than the initial semantic features. Specifically, the multiple features generated based on the initial semantic features can be considered as the semantic feature set.

[0065] In this application, the semantic feature set has more features than the initial semantic feature set, and the semantic feature set has higher diversity than the initial semantic feature set. For ease of understanding, for example, suppose the retrieval object is the query text "rapeseed flowers in a mountain valley". The initial semantic feature extracted by executing S310 is a single feature, and the semantic meaning represented by this feature may be "rapeseed flowers in the mountains". The semantic feature set generated by executing S320 includes multiple features, and the semantic meanings represented by these multiple features may be "in the mountains", "rapeseed flowers", and "mountain flowers", respectively.

[0066] In some implementations, a set of semantic features for the retrieved object can be automatically generated using a pre-trained semantic generation model. In practical applications, the initial semantic features of the retrieved object can be input into the semantic generation model, which will then generate a set of semantic features for the retrieved object based on these initial semantic features.

[0067] The semantic generation model can include a first algorithm model and a third algorithm model. The first algorithm model is used to generate multiple first intermediate features of the search object based on the initial semantic features of the search object. The first intermediate features are a concept relative to the initial semantic features. In practical applications, the executing entity inputs the initial semantic features of the search object into the first algorithm model, and the first algorithm model generates multiple features. The multiple features generated by the first algorithm model can be regarded as multiple first intermediate features.

[0068] Multiple first intermediate features exhibit greater diversity compared to the initial semantic features. For ease of understanding, continuing with the example above, the initial semantic feature of the retrieved object represents "rapeseed flowers in the mountains." Multiple first intermediate features generated by the first algorithm model may represent "mountains," "rapeseed," and "flowers," respectively. It is evident that multiple first intermediate features demonstrate greater diversity compared to the initial semantic features.

[0069] Optionally, the first algorithm model can be a variational autoencoder (VAE), a multi-head attention mechanism, or a topic model. Variational autoencoders, multi-head attention mechanisms, and topic models all have the ability to generate multiple features from a single feature. It should be noted that the first algorithm model can also be other models with the ability to generate multiple features from a single feature, such as autoencoders.

[0070] Taking the first algorithm model using a variational autoencoder as an example, a variational autoencoder is a generative model consisting of an encoder and a decoder. The encoder's role is to encode the distribution of latent variables z based on the input data. The decoder's role is to generate data similar to the input data based on samples of the latent variables z. Specifically, the initial semantic features are input into the encoder of the variational autoencoder, which encodes the initial semantic features into a distribution of latent variables z. Then, multiple latent variables z are sampled from this distribution as multiple intermediate features.

[0071] Taking the first algorithm model that uses a multi-head attention mechanism as an example, when the initial semantic features are input into the multi-head attention mechanism, the multi-head attention mechanism performs multiple rounds of calculations based on the initial semantic features, thereby generating multiple intermediate features.

[0072] The first algorithm model generates multiple first intermediate features of the search object based on its initial semantic features. Although these first intermediate features already possess greater diversity than the initial semantic features, a third algorithm model can be used to optimize them further to enhance their correlation. This optimization transforms each first intermediate feature into a corresponding first semantic feature, which then forms the semantic feature set of the search object. For ease of understanding, using the example above, the multiple first semantic features in the semantic feature set might represent "mountain," "rapeseed flower," and "mountain flower," respectively. It is evident that, compared to multiple intermediate features, the multiple first semantic features in the semantic feature set, while maintaining the same diversity, also possess stronger correlation and are closer to the actual intent.

[0073] The process of optimizing multiple first intermediate features using the third algorithm model includes: the third algorithm model determines similar features for each first intermediate feature from the multiple first intermediate features, and transforms each first intermediate feature into a corresponding first semantic feature based on each first intermediate feature and its similar features. The multiple first intermediate features are transformed into multiple first semantic features, which constitute the semantic feature set of the retrieval object.

[0074] In the above method, since the first semantic feature is obtained based on the first intermediate feature and the similarity feature of the first intermediate feature, the first semantic feature can reflect the correlation between the first intermediate features, and the first semantic feature is closer to the actual retrieval intent.

[0075] Optionally, when determining similar features for each first intermediate feature, the similarity between the first intermediate feature and each of the other first intermediate features can be calculated for the first intermediate feature. If the similarity meets a preset condition, the two are determined to be similar features.

[0076] Specifically, the similarity between two first intermediate features can be measured by cosine similarity. If the similarity between two first intermediate features reaches a preset threshold of 0.6, then the two first intermediate features are considered similar. Alternatively, the similarity between two first intermediate features can also be measured by Euclidean distance, Manhattan distance, or Chebyshev distance. Taking Euclidean distance as an example, if the Euclidean distance between two first intermediate features is below a preset threshold, then the two first intermediate features are considered similar.

[0077] For ease of understanding, refer to the example provided. Figure 4 , Figure 4This is a schematic diagram of a plurality of first intermediate features proposed in an embodiment of this application. Figure 4 This includes six primary intermediate features: a, b, c, d, e, and f. When determining similarity features for 'a', the similarity between 'a' and 'b' is first calculated. If this similarity meets a preset condition, then 'b' is determined to be a similar feature of 'a'. For example... Figure 4 As shown, a line connects a and b, indicating that b is a similar feature of a (or a is a similar feature of b). Next, the similarity between a and c is calculated. If this similarity does not meet a preset condition, c is determined not to be a similar feature of a. For example... Figure 4 As shown, there is no line connecting a and c, indicating that c is not a similar feature of a (or a is not a similar feature of c). By analogy, all similar features are determined for a, such as... Figure 4 As shown, the similar features of a include b, d, and f. Similarly, following the above method, similar features are determined for b, c, d, e, and f, respectively.

[0078] Optionally, when converting each first intermediate feature into a corresponding first semantic feature based on each first intermediate feature and the similar features of each first intermediate feature, specifically, for each first intermediate feature, the most similar feature can first be determined from the similar features of the first intermediate feature, then the determined most similar feature is concatenated with the first intermediate feature, and then a preset activation function is applied to the concatenation result to convert the corresponding first semantic feature.

[0079] In practice, the aggregated feature can be represented by the following formula:

[0080]

[0081] in, Let J represent the j-th similar feature of the first intermediate feature, N represent the total number of similar features that the first intermediate feature has, and MAX_AGGREGATE() represents the function that determines the most similar feature from the N similar features. This represents the most similar feature to the first intermediate feature. For example, when determining the most similar feature from N similar features, the cosine similarity between each of the N similar features and the first intermediate feature can be calculated first, and then the similar feature with the cosine similarity closest to 1 can be determined as the most similar feature.

[0082] In practical implementation, the first semantic feature can be represented by the following formula:

[0083]

[0084] in, The most similar feature to the i-th first intermediate feature is represented by the right-hand side of the equation. This represents the i-th first intermediate feature. CONCAT() represents the feature concatenation function, which is used to concatenate features of different lengths. W t W represents the pre-trained weight vector. t This includes multiple weight values, where σ represents the activation function. σ can be the sigmoid function. The left side of the equation... This represents the first semantic feature transformed from the i-th first intermediate feature.

[0085] For ease of understanding, exemplarily, It is a 1×50 feature, in other words, It is a matrix with a width of 1 and a length of 50. It is also a 1×50 feature, in other words, It is a matrix with a width of 1 and a length of 50. Then... It is a 1×100 feature, in other words, It is a matrix with a width of 1 and a length of 100. W t It is a 100×50 matrix, in other words, W t It is a matrix with a width of 100 and a length of 50. W t and After multiplication, It is a 1×50 feature, in other words, It is a matrix with a width of 1 and a length of 50. It should be noted that the specific values ​​in the above examples (such as 1×50, 1×100, and 100×50, etc.) are only examples and should not be construed as limiting this application.

[0086] The third algorithm model optimizes multiple first intermediate features, thereby transforming them into multiple first semantic features. These multiple first semantic features constitute the semantic feature set of the retrieval object. It should be noted that the aforementioned third algorithm model is actually the algorithm applied in the process of transforming the first intermediate features, such as the calculation described above. and The formula.

[0087] S330: Generate a set of semantic features for each of the N target objects based on the initial semantic features of each target object, where N is an integer greater than 1.

[0088] In this context, the target object is the object being retrieved. For example, in a retrieval task, if a user expects to search for images related to pets, and the executing entity performs this retrieval task by searching for pet-related images from multiple images, then these multiple images are multiple target objects in this retrieval task.

[0089] In some implementations, as described above, the executing entity can perform the retrieval task in stages, and the retrieval method provided in this application embodiment can be applied to the later stages of the retrieval task. In these specific implementations, in the earlier retrieval stage, the executing entity retrieves target objects from the target object database, obtaining N target objects, which serve as the retrieval results for the earlier retrieval stage. In the later retrieval stage, the executing entity further determines the retrieval results from the N target objects by executing the retrieval method provided in this application embodiment.

[0090] In some implementations, the target object library includes N target objects, and the executing entity can also directly determine the search results from the N target objects included in the target object library by executing the search method provided in the embodiments of this application.

[0091] In some implementations, the initial semantic features of each target object have been pre-extracted. When the executing entity generates the set of semantic features corresponding to the target object based on the initial semantic features of the target object, it can directly use the initial semantic features pre-extracted from the target object, thereby shortening the retrieval time.

[0092] In some implementations, when the executing entity generates a set of semantic features of the target object based on the initial semantic features of the target object, it may first extract the initial semantic features of the target object, and then generate a set of semantic features corresponding to the target object based on the initial semantic features.

[0093] As mentioned earlier, the target object can be text data, image data, or video data, etc.

[0094] In application scenarios where the target object is text data, the Doc2Vec algorithm can be used to extract features of the target object, and the extracted features can be used as the initial semantic features of the target object. Alternatively, the Word2vec algorithm can also be used to extract features of the target object, and the extracted features can be used as the initial semantic features of the target object. It should be noted that this application does not limit the method for extracting the initial semantic features.

[0095] In applications where the target object is image data, ResNet (Residual Neural Network) can be used to extract features of the target object, and these extracted features can serve as the initial semantic features of the target object. Alternatively, LeNet, VGG, or DenseNet (Densely Connected Convolutional Networks) can also be used to extract features of the target object, and these extracted features can serve as the initial semantic features of the target object. It should be noted that this application does not limit the method for extracting initial semantic features.

[0096] In application scenarios where the target object is video data, C3D or I3D networks can be used to extract features of the target object, and the extracted features can be used as the initial semantic features of the target object. It should be noted that this application does not limit the method of extracting the initial semantic features.

[0097] As mentioned earlier, in some implementations, a set of semantic features for the retrieved object can be automatically generated using a pre-trained semantic generation model. In these implementations, the semantic generation model can also automatically generate a set of semantic features for the target object. In practical applications, the initial semantic features of the target object can be input into the semantic generation model, which will then generate a corresponding set of semantic features for the target object based on these initial semantic features.

[0098] In this application, for each of the N target objects, a corresponding semantic feature set can be generated in the same way. Therefore, for the sake of simplicity, this application takes a single target object as an example and provides an optional method for generating a semantic feature set for that target object.

[0099] As mentioned earlier, the semantic generation model may include a first algorithm model and a third algorithm model. Furthermore, the semantic generation model may also include a second algorithm model. The second algorithm model is used to generate multiple second intermediate features of the target object based on the initial semantic features of the target object. Second intermediate features are a concept relative to the initial semantic features. In practical applications, the executing entity inputs the initial semantic features of the target object into the second algorithm model, which generates multiple features. These multiple features generated by the second algorithm model can be considered as multiple second intermediate features. Multiple second intermediate features exhibit greater diversity compared to the initial semantic features.

[0100] Optionally, the second algorithm model can be a Variational Autoencoder (VAE), a Conditional Variational Autoencoder (CVAE), a Multi-head Attention mechanism, or a topic model. Variational autoencoders, Conditional Variational Autoencoders, Multi-head Attention mechanisms, and topic models all have the ability to generate multiple features from a single feature. It should be noted that the second algorithm model can also be other models with the ability to generate multiple features from a single feature, such as autoencoders.

[0101] In practical implementation, when the retrieval object and the target object belong to different data modalities, the first algorithm model preferably uses a multi-head attention mechanism, and the second algorithm model preferably uses a variational autoencoder or a conditional variational autoencoder. For example, when the retrieval object is text data and the target object is image data, the first algorithm model uses a multi-head attention mechanism, and the second algorithm model uses a conditional variational autoencoder.

[0102] The Conditional Variational Autoencoder (CVA) is a generative model comprising an encoder and a decoder. The encoder encodes the distribution of latent variables *z* that satisfy the input conditions, based on the input data and input conditions (which can also be understood as labels). The decoder generates data that satisfies the input conditions based on the sampling of latent variables *z* and the input conditions. For example, in a specific implementation of this application, the initial semantic features of the target object can be used as input data, and the initial semantic features of the retrieved object can be used as input conditions. In this application, the input conditions are input to fully connected layers (FC) to obtain transformed features output by the FC. These transformed features are then concatenated with the input data (i.e., the initial semantic features of the target object) to obtain concatenated features. These concatenated features are input to the encoder of the CVA, which encodes the concatenated features into a distribution of latent variables *z* that satisfy the input conditions. Multiple latent variables *z* are then sampled from this distribution as multiple second intermediate features of the target object.

[0103] The second algorithm model generates multiple second intermediate features of the target object based on the initial semantic features of the target object. Although the multiple second intermediate features have higher diversity than the initial semantic features, in order to further enhance the correlation between the multiple second intermediate features, the third algorithm model can be used to optimize the multiple second intermediate features, so that the multiple second intermediate features are transformed into corresponding second semantic features, and the multiple second semantic features serve as the semantic feature set corresponding to the target object.

[0104] The process of optimizing multiple second intermediate features using the third algorithm model includes: the third algorithm model determines similar features for each second intermediate feature from multiple second intermediate features, and transforms each second intermediate feature into a corresponding second semantic feature based on each second intermediate feature and its similar features; the multiple second semantic features transformed from multiple second intermediate features serve as the semantic feature set of the target object.

[0105] In the above method, since the second semantic feature is obtained based on the similarity features of the second intermediate feature and the second intermediate feature, the second semantic feature can reflect the correlation between the second intermediate features and is closer to the actual retrieval intent.

[0106] Optionally, the process of determining similar features for each second intermediate feature can refer to the aforementioned process of determining similar features for each first intermediate feature.

[0107] Optionally, the process of converting each second intermediate feature into a corresponding second semantic feature based on each second intermediate feature and the similarity features of each second intermediate feature can refer to the aforementioned process of converting each first intermediate feature into a corresponding first semantic feature.

[0108] The third algorithm model optimizes multiple second intermediate features, thereby transforming them into multiple second semantic features, which constitute the semantic feature set of the target object. It should be noted that the aforementioned third algorithm model is actually the algorithm applied in the process of transforming the first and second intermediate features.

[0109] S340: Determine the matching degree between each target object and the search object based on the semantic feature set of each target object and the semantic feature set of the search object.

[0110] In this application, the semantic feature set has higher diversity than the initial semantic features. Instead of determining the matching degree based on two single initial semantic features, this application determines the matching degree based on two sets of diverse semantic features (i.e., two semantic feature sets) for the retrieval object and the target object.

[0111] For simplicity, the features included in the semantic feature set of the retrieval object are called first semantic features, and the features included in the semantic feature set of the target object are called second semantic features. The semantic feature set of the retrieval object includes p first semantic features, where p is an integer greater than 1. The semantic feature set of each target object includes q second semantic features, where q is an integer greater than 1. It should be noted that for each of the N target objects, its matching degree with the retrieval object can be determined in the same way. For simplicity, the J-th target object is used as an example, where J is a positive integer less than or equal to N.

[0112] In some implementations, in order to determine the matching degree between the Jth target object and the search object, the difference degree between the Jth target object and the i-th first semantic feature can be determined based on the q second semantic features of the Jth target object and the i-th first semantic feature of the search object, where i is a positive integer less than or equal to p; then, the minimum difference degree is determined from the difference degree between the Jth target object and each first semantic feature, and the matching degree between the Jth target object and the search object is determined based on the minimum difference degree.

[0113] For ease of understanding, exemplarily, assume that the semantic feature set of the J-th target object includes 4 second semantic features, and the semantic feature set of the retrieved object includes 6 first semantic features. When determining the matching degree between the target object and the retrieved object: based on the 4 second semantic features and the 1st first semantic feature, the difference between the target object and the 1st first semantic feature is determined to be 0.3; based on the 4 second semantic features and the 2nd first semantic feature, the difference between the target object and the 2nd first semantic feature is determined to be 0.1; and so on, finally determining the difference between the target object and the 6 first semantic features to be 0.3, 0.1, 0.6, 0.4, 0.8, and 0.9 respectively, where the minimum difference is equal to 0.1. Therefore, the matching degree between the target object and the retrieved object is determined based on the minimum difference of 0.1.

[0114] Following this method, if a target object closely matches one aspect of the search object, regardless of whether the target object closely matches other aspects of the search object, the difference between the target object and the search object will be relatively small, and the target object is very likely to be included in the search results. Thus, the resulting multiple search results may each closely match different aspects of the search object, exhibiting a certain degree of diversity among them.

[0115] Optionally, when determining the degree of difference between the Jth target object and the i-th first semantic feature based on the q-th second semantic features of the J-th target object and the i-th first semantic feature of the retrieved object, it can be determined in the following way:

[0116] First, based on each second semantic feature of the J-th target object and the attention weight of each second semantic feature, a weighted sum of q second semantic features of the J-th target object is determined, where the attention weight of each second semantic feature is the attention weight of the second semantic feature corresponding to the i-th first semantic feature. Next, the vector difference between the i-th first semantic feature and the weighted sum is determined, and the vector difference includes multiple vector element values. Finally, the minimum vector element value among the multiple vector element values ​​is determined as the difference between the J-th target object and the i-th first semantic feature.

[0117] In practice, the difference between the target object and the i-th first semantic feature can be expressed by the following formula:

[0118]

[0119] in, α represents the j-th second semantic feature of the target object. ij α represents the attention weight of the j-th second semantic feature corresponding to the i-th first semantic feature. ij It can be obtained through pre-training. This represents the weighted sum of q second semantic features of the target object. This represents the i-th first semantic feature of the retrieved object. The vector difference is represented by `min()`, which is a function that takes the minimum value of a vector from a set of vector elements. i This represents the degree of difference between the target object and the i-th first semantic feature.

[0120] For ease of understanding, let's assume, by way of example, that but Here, 0.1 is the minimum vector element value, so 0.1 is determined as the difference between the target object and the i-th first semantic feature.

[0121] In the above approach, if a second semantic feature of the target object, multiplied by its corresponding attention weight, is very close to the i-th first semantic feature, regardless of whether other second semantic features of the target object, multiplied by their corresponding attention weights, are also very close to the i-th first semantic feature, the difference between the target object and the i-th first semantic feature will be small. In other words, if one aspect of the target object particularly matches one aspect of the search object, regardless of whether other aspects of the target object particularly match other aspects of the search object, the difference between the target object and the search object will be relatively small, and the target object is very likely to be included as a search result. Thus, the multiple search results ultimately determined may each particularly match different aspects of the search object, and there are certain differences between the multiple search results, resulting in a relatively high degree of diversity.

[0122] Optionally, when determining the matching degree between the target object and the search object based on the minimum difference, the minimum difference can be directly determined as the matching degree. In this case, the smaller the matching degree value, the more closely the target object and the search object match. Alternatively, the reciprocal of the minimum difference can be determined as the matching degree. In this case, the larger the matching degree value, the more closely the target object and the search object match. It should be noted that this application does not limit the process of determining the matching degree based on the minimum difference.

[0123] Optionally, the process of determining the matching degree between the target object and the retrieved object can also be performed by a pre-trained fourth algorithm model, where the attention weight α involved in the above formula... ij As a parameter of the fourth algorithm model, the attention weight α is used during the training of the fourth algorithm model. ij The fourth algorithm model gradually converges during training. In practice, the fourth algorithm model can be connected to the output of the aforementioned third algorithm model. During model application, the third algorithm model outputs a set of semantic features of the retrieved object and a set of semantic features of the target object. These two sets of semantic features are input into the fourth algorithm model, which is based on attention weight α. ij Calculate the matching degree between two semantic feature sets, that is, the matching degree between the target object and the retrieved object.

[0124] During model training, the third algorithm model outputs a first set of predicted semantic features for the sample retrieval object and a second set of predicted semantic features for the sample target object. These two sets of predicted semantic features are then input into the fourth algorithm model, which is based on the initial attention weights α. ij The prediction matching degree between two sets of predicted semantic features is calculated, which is the prediction matching degree between the target object of the sample and the retrieved object of the sample. Then, the loss value is determined based on the prediction matching degree, and the fourth algorithm model, the third algorithm model, the second algorithm model, and the first algorithm model are updated based on the loss value.

[0125] For ease of understanding, please refer to Figure 5 , Figure 5 This is a schematic diagram illustrating the method of determining the matching degree using a model, as proposed in an embodiment of this application. Figure 5As shown, in practical applications, the initial semantic features 501 of a target object can be input into the second algorithm model 520 of the semantic generation model 500. Simultaneously, the initial semantic features 503 of the retrieval object 502 are extracted and input into the first algorithm model 510 of the semantic generation model 500. The second algorithm model 520 outputs multiple second intermediate features 504, and the first algorithm model 510 outputs multiple first intermediate features 505. The third algorithm model 530 optimizes the multiple second intermediate features 504 and the multiple first intermediate features 505, thereby outputting the semantic feature set 506 of the target object and the semantic feature set 507 of the retrieval object 502. The fourth algorithm model 540 determines and outputs the matching degree between the target object and the retrieval object 502 based on the two semantic feature sets.

[0126] In some implementations, in order to determine the matching degree between the Jth target object and the search object, the difference degree between the jth second semantic feature of the Jth target object and the search object can be determined based on the jth second semantic feature of the Jth target object and the p first semantic features of the search object, where j is a positive integer less than or equal to q; then, the minimum difference degree is determined from the difference degree between each second semantic feature of the Jth target object and the search object, and the matching degree between the Jth target object and the search object is determined based on the minimum difference degree.

[0127] S350: Based on the matching degree between each target object and the search object, determine the target object as the search result from N target objects.

[0128] In this application, the search results are determined from N target objects based on the matching degree between each target object and the search object, thereby making the search results both highly diverse and highly accurate.

[0129] In some implementations, the search results can be determined by selecting the M target objects corresponding to the M highest matching scores, where M is a positive integer less than or equal to N. For example, if a higher matching score indicates a better match, the top 1000 target objects with the highest matching scores can be selected as search results. Alternatively, if a lower matching score indicates a better match, the top 1000 target objects with the lowest matching scores can be selected as search results.

[0130] In some implementations, the target object can be identified as a search result based on its matching degree with the search object, provided the matching degree value meets a preset condition. For example, if a higher matching degree value indicates a better match between the target object and the search object, and the matching degree value of the target object and the search result exceeds a preset threshold (e.g., 0.6), then the target object is identified as a search result. Alternatively, if a lower matching degree value indicates a better match between the target object and the search object, and the matching degree value of the target object and the search result is below a preset threshold (e.g., 0.3), then the target object is identified as a search result.

[0131] It should be noted that this application does not limit the process of determining search results based on matching degree.

[0132] In this application, a semantic feature set for the retrieval object is generated based on the initial semantic features of the retrieval object, and a semantic feature set for the target object is generated based on the initial semantic features of the target object. This means that the retrieval object and the target object are no longer represented by a single initial semantic feature, but by diverse semantic features. Then, based on the semantic feature sets of the target object and the retrieval object, the matching degree between the target object and the retrieval object is determined, and the retrieval results are determined based on the matching degrees of multiple target objects with the retrieval object. Therefore, this application does not determine the matching degree based on two single initial semantic features, but rather based on two diverse sets of semantic features, thereby determining the retrieval results and making the retrieval results more diverse.

[0133] As mentioned above, in some implementations, steps S320 to S340 can be performed by a semantic generation model. Optionally, the training process of this semantic generation model can refer to... Figure 6 and Figure 7 , Figure 6 This is a flowchart illustrating the training process of a semantic generation model proposed in one embodiment of this application. Figure 7 This is a schematic diagram of a training semantic generation model proposed in an embodiment of this application. For example... Figure 6 As shown, the training process of the semantic generation model includes:

[0134] S610: Obtain the sample retrieval object and its initial semantic features, and obtain multiple sample target objects and their initial semantic features. Each sample target object carries a label, and the label carried by each sample target object is used to characterize whether the sample target object has been retrieved by the sample retrieval object.

[0135] The initial semantic features of the sample retrieval object and the sample target object can be pre-generated, or they can be extracted during training. For specific methods of extracting initial semantic features, please refer to the preceding content.

[0136] Each sample target object carries either a positive or negative label. If a sample target object carries a positive label, it means that the sample target object has been retrieved by the sample retrieval object. If a sample target object carries a negative label, it means that the sample target object has not been retrieved by the sample retrieval object.

[0137] Optionally, the sample retrieval object and some sample target objects can come from historical search data accumulated by the search product. Historical search data can include multiple sets of search data, each set including one retrieval object and multiple search results. The multiple search results are determined by the search product based on the retrieval object. If the sample target object and the sample retrieval object come from the same set of search data, the sample target object carries a positive label, indicating that the sample target object has been retrieved by the sample retrieval object. If the sample target object and the sample retrieval object do not come from the same set of search data, the sample target object carries a negative label, indicating that the sample target object has not been retrieved by the sample retrieval object.

[0138] S620: Input the initial semantic features of the sample retrieval object into the semantic generation model to generate the first predicted semantic feature set of the sample retrieval object.

[0139] S630: Input the initial semantic features of each sample target object into the semantic generation model to generate the second predicted semantic feature set corresponding to each sample target object.

[0140] like Figure 7 As shown, for each sample target object, the initial semantic features 702 of the sample target object and the initial semantic features 701 of the sample retrieval object can be simultaneously input into the semantic generation model 700. Specifically, the initial semantic features 702 of the sample target object are input into the second algorithm model 720, which outputs multiple second intermediate features 704 of the sample target object. The initial semantic features 701 of the sample retrieval object are input into the first algorithm model 710, which outputs multiple first intermediate features 703 of the sample retrieval object. The third algorithm model 730 optimizes the multiple second intermediate features 704 to generate a second predicted semantic feature set 706, and optimizes the multiple first intermediate features 703 to generate a first predicted semantic feature set 705.

[0141] The specific types of the first algorithm model 710 and the second algorithm model 720 can be found in the foregoing. The optimization process of the third algorithm model 730 for multiple first intermediate features 703 and multiple second intermediate features 704 can also be found in the foregoing, i.e., the process by which the third algorithm model optimizes multiple first intermediate features. It should be understood that during training, the model parameters within the third algorithm model 730 (e.g., the aforementioned weight vector W)... t These are not yet the optimal parameters. These model parameters need to be trained and updated during model training. Therefore, during training, the optimization effect of the third algorithm model 730 on multiple first intermediate features 703 and multiple second intermediate features 704 may not be ideal.

[0142] S640: Determine the predicted matching degree between each sample target object and the sample retrieval object based on the second predicted semantic feature set and the first predicted semantic feature set of each sample target object.

[0143] like Figure 7 As shown, the fourth algorithm model 740 can be used to determine the predicted matching degree between each sample target object and the sample retrieval object. Specifically, the third algorithm model 730 outputs a first predicted semantic feature set 705 for the sample retrieval object and a second predicted semantic feature set 706 for the sample target object. These two predicted semantic feature sets are input into the fourth algorithm model 740, which is based on the initial attention weight α. ij Calculate the prediction matching degree between two sets of predicted semantic features, that is, the prediction matching degree between the target object of the sample and the sample retrieval object.

[0144] S650: Determine the loss value based on the label and predicted matching degree of the target object for each sample, and train the semantic generation model based on the loss value.

[0145] Figure 7 This paper only schematically illustrates the processing procedure for a single sample target object, thereby obtaining the predicted matching degree between the sample retrieval object and that sample target object. It should be noted that this application will also process other sample target objects in the same manner to obtain the predicted matching degree between the sample retrieval object and each individual sample target object, such as... Figure 7 The predicted matching degree b to the predicted matching degree x are obtained from the predicted matching degree b. After obtaining the predicted matching degree corresponding to each of the multiple sample target objects, such as... Figure 7 As shown, the loss value can be determined based on the label and predicted matching degree of each sample target object, and the fourth algorithm model, the third algorithm model, the second algorithm model, and the first algorithm model can be updated by backpropagating the loss value.

[0146] In some implementations, the loss value may include a triplet loss component L. milThe loss component of the triplet L mil It can be expressed by the following formula:

[0147]

[0148] Wherein S(t,v) - S(t,v) represents the predicted matching degree between the sample retrieval object and the sample target object carrying the negative label; + ) represents the predicted matching degree between the sample retrieval object and the sample target object carrying a positive label; Δ is the preset distance compensation coefficient; || is a function, which takes the value 0 when the value in || is less than or equal to 0, and takes the value itself when the value in || is greater than 0; C represents the multiple S(t, v) in one round of training. - ) and multiple S(t, v + The total number of pairs of 1 / 2 that can be formed; L mil This indicates the loss component of the ternary set.

[0149] During training, if S(t, v) - ) is relatively large and S(t, v + If the value of L is small, it indicates that the semantic generation model cannot accurately distinguish between positive and negative target objects. In this case, the L calculated using the above formula... mil The value is relatively large, therefore when using this L mil When updating the semantic generation model, the semantic generation model can be updated to a large extent, which helps to accelerate the convergence of the semantic generation model.

[0150] If S(t, v) - ) is relatively small and S(t, v + A larger value for L indicates that the semantic generation model can accurately distinguish between positive and negative target objects. In this case, L calculated using the above formula... mil It is closer to 0, therefore when using this L mil When updating the semantic generation model, the semantic generation model can be updated to a smaller extent, which is conducive to the smooth convergence of the semantic generation model.

[0151] In some implementations, the loss value may include a label classification loss component L. label The process of determining the label classification loss component includes: for each predicted semantic feature set output by the semantic generation model (including the first predicted semantic feature set and the second predicted semantic feature set), a classifier is used to predict the type of the predicted semantic feature set to obtain the type prediction result; then, based on the type prediction result and type label of each predicted semantic feature set, the label classification loss component L is determined. label The type label of the predicted semantic feature set is used to characterize the data modality type of the predicted semantic feature set.

[0152] For ease of understanding, exemplarily, in this training round, the total number of target objects is 100, and the number of retrieval objects is 1. Each time, the initial semantic features of the retrieval object and the initial semantic features of the target object are input into the semantic generation model, resulting in a first predicted semantic feature set and a second predicted semantic feature set output by the semantic generation model. Thus, this training round ultimately yields 100 first predicted semantic feature sets and 100 second predicted semantic feature sets. Each of the 100 first predicted semantic feature sets corresponds to a retrieval object, and each of the 100 second predicted semantic feature sets corresponds to one of the 100 target objects.

[0153] Then, for each of the 100 first predicted semantic features, multiple features in that first predicted semantic feature set are concatenated to obtain the first concatenated feature corresponding to that first predicted semantic feature set. In this way, 100 first concatenated features are obtained. Next, a first label is assigned to each first concatenated feature. The first label is used to represent the data modality type of the first concatenated feature. For example, if the data modality type of the sample retrieval object is text data, then the first label can be in the form of "0".

[0154] Similarly, for each of the 100 second predicted semantic features, multiple features in that set are concatenated to obtain the corresponding second concatenated feature. This results in 100 second concatenated features. Next, a second label is assigned to each second concatenated feature. This second label represents the data modality type of the second concatenated feature. For example, if the data modality type of each target retrieval object is image data, the second label could be in the form of "1".

[0155] Next, these 100 first concatenated features and 100 second concatenated features are used as sample data and input into a preset classifier to obtain 200 classification results. Then, based on these 200 classification results and the label corresponding to each classification result (i.e., the aforementioned first label or second label), the label classification loss component L is determined. label .

[0156] The above examples illustrate how to determine the label classification loss component L. label The specific process has been explained. It should also be noted that the specific values ​​involved in the above examples (such as 100, 200, etc.) are only examples and should not be construed as limitations on this application.

[0157] Furthermore, the labels carried by some sample target objects may deviate from reality. For example, some sample target objects carrying negative labels may not have been retrieved by the sample retrieval object, but these sample target objects are actually quite relevant to the sample retrieval object. However, during model training, these sample target objects still participate in training as those carrying negative labels.

[0158] While it's possible to train a semantic generation model with the necessary capabilities even if the above conditions are ignored, some implementations incorporate a distance penalty loss component in the loss value to further enhance the model's fault tolerance. This distance penalty loss component is determined based on the prior and predicted matching degrees of the negatively labeled target object. The prior matching degree of the target object refers to the degree of matching between the target object and the retrieved object, determined based on their initial semantic features and the initial semantic features of the retrieved object.

[0159] Optionally, refer to Figure 8 , Figure 8 This is a flowchart illustrating the determination of distance penalty loss components according to an embodiment of this application. For example... Figure 8 As shown, the process of determining the distance penalty loss component may include:

[0160] S810: Based on the initial semantic features of each negatively labeled sample target object and the initial semantic features of the sample retrieval object, determine the prior matching degree between each negatively labeled sample target object and the sample retrieval object.

[0161] In determining the prior matching degree between the target object and the retrieved object, the cosine similarity between the initial semantic features of the target object and the retrieved object can be calculated based on their initial semantic features. The calculated cosine similarity is then used as the prior matching degree between the target object and the retrieved object.

[0162] Alternatively, when determining the prior matching degree between the target object and the retrieved object, the Euclidean distance, Manhattan distance, or Chebyshev distance between the initial semantic features of the target object and the retrieved object can be calculated, and the result can be used as the prior matching degree between the target object and the retrieved object. It should be noted that this application does not limit the specific method for determining the prior matching degree.

[0163] S820: Based on the prior matching degree of each sample target object carrying a negative label, determine the prior matching degree ranking of multiple prior matching degrees.

[0164] When determining the order of prior matching degrees among multiple prior matching degrees, the sorting can be performed according to a preset sorting rule. For example, the preset sorting rule could be sorting from largest to smallest, or from smallest to largest, or other sorting methods. This application does not limit the preset sorting rule.

[0165] For example, taking the sorting from largest to smallest as an example, suppose the target objects carrying negative labels in this round of training are target object 1, target object 2, ..., target object 10, and the prior matching degrees of target objects 1 to 10 are 0.15, 0.26, 0.71, 0.25, 0.16, 0.22, 0.07, 0.55, 0.42, and 0.12, respectively. Then the prior matching degrees of the 10 target objects are sorted as follows: target object 3, target object 8, target object 9, target object 2, target object 4, target object 6, target object 5, target object 1, target object 10, and target object 7.

[0166] S830: Based on the predicted matching degree of each sample target object carrying a negative label, determine the ranking of the predicted matching degrees among multiple predicted matching degrees.

[0167] When determining the ranking of multiple predicted matching degrees, the ranking can be done according to the aforementioned preset ranking rules.

[0168] For example, taking the sorting from largest to smallest as an example, suppose the target objects carrying negative labels in this round of training are target object 1, target object 2, ..., target object 10, and the predicted matching scores of target objects 1 to 10 are 0.12, 0.28, 0.63, 0.31, 0.10, 0.25, 0.02, 0.51, 0.39, and 0.07, respectively. Then the predicted matching scores of the 10 target objects are sorted as follows: target object 3, target object 8, target object 9, target object 4, target object 2, target object 6, target object 1, target object 5, target object 10, and target object 7.

[0169] S840: Determine the distance penalty loss component based on the similarity between the prior matching degree ranking and the predicted matching degree ranking.

[0170] The similarity between the prior matching ranking and the predicted matching ranking can be represented by the KL (Kullback-Leibler) divergence of the two rankings, and the KL divergence of the two rankings can be determined as the distance penalty loss component. Specifically, the distance penalty loss component can be expressed by the following formula:

[0171] L dis =KL(dist(t, v) - )||S(t,v - ))

[0172] Where, dist(t, v) - S(t, v) represents the prior matching ranking of multiple negatively labeled sample target objects. - KL(dist(t, v)) represents the ranking of predicted match scores for multiple negatively labeled sample target objects. - )||S(t,v - )) represents the KL divergence between the prior matching ranking and the predicted matching ranking, L dis This represents the distance penalty loss component.

[0173] In the above approach, a distance penalty loss component is determined based on the similarity between the prior matching score ranking and the predicted matching score ranking. This distance penalty loss component is then used to train the semantic generation model, enabling the model to learn the differences between negative examples (i.e., target sample objects carrying negative labels). The predicted matching scores of multiple negative examples can also reflect the distance differences between them. Thus, even when some labels may deviate from reality, the trained semantic generation model exhibits stronger fault tolerance.

[0174] refer to Figure 9 , Figure 9 This is a flowchart of a retrieval method proposed in another embodiment of this application. For example... Figure 9 As shown, the retrieval method includes:

[0175] S910: Obtain the retrieval object and extract the initial semantic features of the retrieval object.

[0176] S920: Generate a set of semantic features of the search object based on the initial semantic features of the search object.

[0177] S930: Based on the initial semantic features of each of the N target objects and the initial semantic features of the retrieved object, generate a set of semantic features corresponding to each target object. Here, N is an integer greater than 1.

[0178] In this application, since the initial semantic features of the search object can reflect the user's search intent, the set of semantic features generated based on the initial semantic features of the search object and the initial semantic features of the target object is closer to the user's search intent.

[0179] It should be noted that for each of the N target objects, the corresponding set of semantic features can be determined in the same way. To simplify the explanation, we will take the Kth target object as an example, where K is a positive integer less than or equal to N.

[0180] Optionally, a pre-trained conditional variational autoencoder can be used to generate a set of semantic features for the target object. For example, the initial semantic features of the Kth target object and the initial semantic features of the retrieved object can be input into the pre-trained conditional variational autoencoder to obtain latent variables generated by the conditional variational autoencoder. Then, multiple latent variables are collected from the latent variables generated by the conditional variational autoencoder, and the set of semantic features for the Kth target object is generated based on the multiple latent variables.

[0181] In this system, the conditional variational autoencoder uses the initial semantic features of the retrieved object as the encoding guidance condition. In other words, the initial semantic features of the retrieved object serve as the input condition (the input condition can also be understood as a label) of the conditional variational autoencoder, making the latent variables generated by the conditional variational autoencoder closer to the initial semantic features of the retrieved object.

[0182] In specific implementation, as mentioned above, a semantic generation model can be used to generate a set of semantic features for the retrieved object and a set of semantic features for the target object. The semantic generation model includes a first algorithm model, a second algorithm model, and a third algorithm model. Specifically, the second algorithm model and the third algorithm model generate the set of semantic features for the target object, and the second algorithm model uses a conditional variational autoencoder.

[0183] For example, when generating a semantic feature set of a target object using a conditional variational autoencoder and a third algorithm model, the initial semantic features of the target object can be used as input data, and the initial semantic features of the retrieved object can be used as input conditions. The input conditions are input to a fully connected layer to obtain the transformed features output by the fully connected layer. These transformed features are then concatenated with the input data (i.e., the initial semantic features of the target object) to obtain concatenated features. These concatenated features are input to the encoder of the conditional variational autoencoder, which encodes the concatenated features into a distribution of latent variables z that conform to the input conditions. Multiple latent variables z are then sampled from the distribution of latent variables z as multiple second intermediate features of the target object. The third algorithm model optimizes these multiple second intermediate features to generate a semantic feature set of the target object.

[0184] S940: Determine the matching degree between each target object and the search object based on the semantic feature set of each target object and the semantic feature set of the search object.

[0185] S950: Based on the matching degree between each target object and the search object, determine the target object as the search result from N target objects.

[0186] For specific implementations of S910, S920, S940 and S950, please refer to the aforementioned specific implementations of S310, S320, S340 and S350 respectively. To avoid repetition, they will not be described again here.

[0187] In this application, it can be in accordance with Figure 6 The conditional variational autoencoder (i.e., the second algorithm model) in the semantic generation model is trained in the manner shown. When according to... Figure 6 After obtaining the loss value as shown, this loss value is used to train various algorithm models in the semantic generation model, including the conditional variational autoencoder. The loss value may include an encoding loss component, which can be represented by the following formula:

[0188]

[0189] Where p(z) represents the distribution of the latent variable z. q(z|f v f t () represents the distribution of the latent variable z under the first condition, and the initial semantic features f of the retrieved object. t and the initial semantic features f of the target object v As the first condition. p(f) v |f t z) represents the initial semantic features f of the target object under the second condition. v The distribution of the initial semantic features f of the retrieved object t And the latent variable z is used as the second condition. KL(q(z|f) v f t )||p(z)) represents p(z) and q(z|f v f t KL divergence between ). Represents logp(f) v |f t The expectation of z). L qvae This represents the encoding loss component.

[0190] In some implementations, according to Figure 6 The loss value obtained in the manner shown includes: the triplet loss component L. mil Label classification loss component L label Distance penalty loss component L dis and encoding loss component L qvae The loss value can be expressed by the following formula:

[0191] L = L label +λ1L qvae +λ2L mil +λ3L dis

[0192] Where λ1, λ2, and λ3 are all preset weighting coefficients, and L represents the loss value.

[0193] refer to Figure 10 , Figure 10 This is a flowchart of a retrieval method proposed in another embodiment of this application. For example... Figure 8 As shown, the retrieval method includes:

[0194] S1010: Obtain the search object and extract the initial semantic features of the search object.

[0195] S1020: Based on a pre-trained projection strategy, the initial semantic features of the retrieved object are projected onto the target vector space corresponding to the projection strategy to obtain the first projection features of the retrieved object; based on the first projection features, a set of semantic features of the retrieved object is generated.

[0196] S1030: Based on the projection strategy, the initial semantic features of each of the N target objects are projected onto the target vector space to obtain the second projection features of each target object; based on the second projection features of each target object, a set of semantic features corresponding to each target object is generated.

[0197] As mentioned earlier, in some embodiments, the retrieval object and the target object may belong to different data modalities. In these embodiments, to make the retrieval object and the target object from different data modalities more comparable, the initial semantic features of the retrieval object and the initial semantic features of the target object can be projected onto the same vector space, i.e., the target vector space. Thus, the first projected feature and the second projected feature obtained by projection are features located in the same vector space. Consequently, the semantic feature set generated based on the first projected feature and the semantic feature set generated based on the second projected feature are more comparable.

[0198] The projection strategy includes multiple projection parameters, among which are spatial parameters describing the target vector space. This projection strategy can be pre-trained; by training the projection strategy, the various projection parameters converge to suitable values, thereby determining a suitable target vector space.

[0199] The specific method for generating the semantic feature set based on the first projection feature can be found in the foregoing. For example, the first projection feature can be input into the semantic generation model, thereby generating the semantic feature set of the retrieval object through the first and third algorithm models of the semantic generation model.

[0200] The specific method for generating the semantic feature set based on the second projection features can also be found in the foregoing. For example, the second projection features can be input into the semantic generation model, thereby generating the semantic feature set of the target object through the second and third algorithm models of the semantic generation model. Alternatively, the second projection features and the first projection features can be concatenated and input into the semantic generation model, thereby generating the semantic feature set of the target object through the second and third algorithm models of the semantic generation model, where the second algorithm model is a conditional variational autoencoder.

[0201] S1040: Determine the matching degree between each target object and the search object based on the semantic feature set of each target object and the semantic feature set of the search object.

[0202] S1050: Based on the matching degree between each target object and the search object, determine the target object as the search result from N target objects.

[0203] For specific implementations of S1010, S1040 and S1050, please refer to the aforementioned specific implementations of S310, S340 and S350 respectively. To avoid repetition, they will not be described again here.

[0204] In some implementations, steps S1020 to S1030 can be performed by a pre-trained semantic generation model. During the training of the semantic generation model, a projection strategy can be trained simultaneously. Specifically, the sample retrieval object and the sample target object belong to different data modalities. To generate a first predicted semantic feature set for the sample retrieval object, the initial semantic features of the sample retrieval object can be projected onto a preset vector space corresponding to the preset projection strategy, based on a preset projection strategy, to obtain the first sample projection features of the sample retrieval object; then, the first sample projection features are input into the semantic generation model to generate the first predicted semantic feature set of the sample retrieval object.

[0205] To generate a second predicted semantic feature set corresponding to each sample target object, the initial semantic features of each sample target object can be projected onto a preset vector space based on a preset projection strategy to obtain the second sample projection features of each sample target object; then, the second sample projection features of each sample target object are input into the semantic generation model to generate a second predicted semantic feature set corresponding to each sample target object.

[0206] By using the above method, the initial semantic features of the sample retrieval object and the initial semantic features of the sample target object are projected onto the same vector space, i.e., the preset vector space, thereby obtaining the first sample projection features and the second sample projection features located in the same vector space. Furthermore, the first predicted semantic feature set generated based on the first sample projection features and the second predicted semantic feature set generated based on the second sample projection features are more comparable.

[0207] Furthermore, after determining the loss value, a preset projection strategy can be trained based on the loss value, so that the projection parameters in the preset projection strategy are updated, thereby also updating the spatial parameters of the preset vector space.

[0208] refer to Figure 11 , Figure 11 This is a schematic diagram of model training proposed in another embodiment of this application. The model training process adopts an end-to-end learning approach, simultaneously training the projection strategy, semantic feature extractor, and fourth algorithm model. Figure 11 As shown, the semantic generation model 1100 includes a conditional variational autoencoder 1120, a multi-head attention mechanism 1110, and a multi-instance learning model 1130. For simplicity, the accompanying figures... Figure 11 The projection strategy is not shown.

[0209] like Figure 11 As shown, the sample retrieval object is the query text "Golden Knight," and the first sample target object is an image of a knight (referred to as Figure a). During training, the Doc2Vec algorithm can be used to extract the initial semantic features 1101 of the sample retrieval object. Then, based on a projection strategy, the initial semantic features 1101 are projected onto a preset vector space to obtain the first projected feature 1102. ResNet can be used to extract the initial semantic features 1103 of Figure a. Then, based on a projection strategy, the initial semantic features 1103 are projected onto a preset vector space to obtain the second projected feature 1104.

[0210] like Figure 11 As shown, the first projection feature 1102 is input into the multi-head attention mechanism 1110 of the semantic generation model 1100 to obtain multiple first intermediate features 1105 output by the multi-head attention mechanism 1110. The second projection feature 1104 and the first projection feature 1102 are input into the conditional variational autoencoder 1120 to obtain multiple second intermediate features 1106 (i.e., multiple sampled latent variables z) output by the conditional variational autoencoder 1120.

[0211] like Figure 11 As shown, the multi-instance learning model 1130 generates a first example image 1107 based on multiple first intermediate features 1105. In the first example image 1107, some of the first intermediate features 1105 are connected by lines, and two first intermediate features 1105 connected by lines are similar features. The multi-instance learning model 1130 transforms each first intermediate feature 1105 into a corresponding first semantic feature based on each first intermediate feature 1105 and its similar features, thereby obtaining a first predicted semantic feature set of the sample retrieval object.

[0212] Similarly, the multi-instance learning model 1130 generates a second example graph 1108 based on multiple second intermediate features 1106. In the second example graph 1108, some of the second intermediate features 1106 are connected by lines, and two second intermediate features 1106 connected by lines are similar features. The multi-instance learning model 1130 transforms each second intermediate feature 1106 into a corresponding second semantic feature based on each second intermediate feature 1106 and its similar features, thereby obtaining the second predicted semantic feature set of graph a.

[0213] The fourth algorithm model 1140 is based on the initial attention weight α ij Calculate the prediction matching degree 'a' between two sets of predicted semantic features, which is the prediction matching degree 'a' between the sample retrieval object and graph 'a'.

[0214] Figure 11 The processing procedure for Figure a is only schematically illustrated, thus obtaining the predicted matching degree 'a' between the sample retrieval object and Figure a. It should be noted that this application will also process the sample target objects for Figures b, c…x in the same way to obtain the predicted matching degree 'b' between the sample retrieval object and Figure b, the predicted matching degree 'c' between the sample retrieval object and Figure c,…the predicted matching degree 'x' between the sample retrieval object and Figure x. Among the sample target objects in Figures a to x, some sample target objects carry positive labels, indicating that the sample target object has been retrieved by the sample retrieval object. Some sample target objects carry negative labels, indicating that the sample target object has not been retrieved by the sample retrieval object.

[0215] After obtaining the predicted matching scores for multiple sample target objects, a loss value can be determined based on the label and predicted matching score of each sample target object. The fourth algorithm model 1140, the semantic generation model, and the projection strategy are then trained based on this loss value. In other words, the fourth algorithm model 1140, the multi-instance learning model 1130, the conditional variational autoencoder 1120, the multi-head attention mechanism 1110, and the projection strategy are trained based on this loss value. Specifically, by training the fourth algorithm model 1140, the parameters of the fourth algorithm model 1140 (e.g., attention weight α) are adjusted. ij The projection policy is updated. By training the projection policy, the parameters of the projection policy (e.g., the preset vector space) are updated.

[0216] The loss value can include at least one of the following components: triplet loss component L mil Label classification loss component L label Distance penalty loss component L dis Or encoding loss component L qvae For details on how to determine these loss components, please refer to the foregoing content.

[0217] refer to Figure 12 , Figure 12 This is an embodiment of the present application based on Figure 11 The diagram illustrates how each model performs a retrieval task.

[0218] like Figure 12 As shown, the search object is the query text "rapeseed flowers in a mountain valley". Furthermore, the first target object is an image of rapeseed flowers (referred to as Figure A). Figure 12 (The target object is not shown in the figure). During the retrieval task, the Doc2Vec algorithm can be used to extract the initial semantic features 1201 of the retrieval object. Then, based on the trained projection strategy, the initial semantic features 1201 are projected onto the target vector space to obtain the first projected feature 1202. The initial semantic features 1203 of Figure A can be obtained, and then based on the trained projection strategy, the initial semantic features 1203 are projected onto the target vector space to obtain the second projected feature 1204. Here, the target vector space is the parameter in the trained projection strategy, that is, the parameter that the aforementioned preset vector space gradually converges to during training. The initial semantic features 1203 of Figure A can be extracted before the retrieval task is executed.

[0219] like Figure 12 As shown, the first projection feature 1202 and the second projection feature 1204 are input into the pre-trained semantic generation model 1210, thereby obtaining two semantic feature sets output by the semantic generation model 1210. These two semantic feature sets are the semantic feature set 1205 of the retrieved object and the semantic feature set 1206 of Figure A, respectively. The fourth algorithm model 1220 is based on the pre-trained attention weight α. ij Calculate the matching degree 'a' between the two semantic feature sets. The matching degree 'a' is also the matching degree between the retrieved object and graph A. Figure 12 The processing procedure for Figure A is only schematically illustrated, thus obtaining the matching degree 'a' between the search object and Figure A. It should be noted that this application will also process Figures B, C…N, etc., in the same manner to obtain the matching degree 'b' between the search object and Figure B, the matching degree 'b' between the search object and Figure C, and so on, the matching degree 'n' between the search object and Figure N. Figure 12 As shown, the target object for search result 1207 is finally determined from N target objects based on the matching degree between each target object and the search object.

[0220] refer to Figure 13 , Figure 13 This is a schematic diagram of a retrieval device 1300 according to an embodiment of this application. Figure 13 As shown, the retrieval device 1300 includes:

[0221] The initial semantic feature extraction module 1310 is used to obtain the retrieval object and extract the initial semantic features of the retrieval object.

[0222] The semantic feature set generation module 1320 is used to generate a semantic feature set of the search object based on the initial semantic features of the search object, and is also used to generate a semantic feature set corresponding to each target object based on the initial semantic features of each of the N target objects, where N is an integer greater than 1.

[0223] The matching degree determination module 1330 is used to determine the matching degree between each target object and the search object based on the semantic feature set of each target object and the semantic feature set of the search object.

[0224] The retrieval result determination module 1340 is used to determine the target object as the retrieval result from N target objects based on the matching degree between each target object and the retrieval object.

[0225] Optionally, the semantic feature set of the retrieved object includes p first semantic features, where p is an integer greater than 1; the semantic feature set of each target object includes q second semantic features, where q is an integer greater than 1.

[0226] The matching degree determination module 1330 is specifically used for: determining the difference between the J-th target object and the i-th first semantic feature based on the q-th second semantic features of the J-th target object and the i-th first semantic feature of the retrieved object; determining the minimum difference from the difference between the J-th target object and each first semantic feature; and determining the matching degree between the J-th target object and the retrieved object based on the minimum difference. Here, J is a positive integer less than or equal to N, and i is a positive integer less than or equal to p.

[0227] Optionally, when determining the difference between the J-th target object and the i-th first semantic feature, the matching degree determination module 1330 is specifically configured to: determine the weighted sum of q second semantic features of the J-th target object based on each second semantic feature of the J-th target object and the attention weight of each second semantic feature; determine the vector difference between the i-th first semantic feature and the weighted sum, wherein the vector difference includes multiple vector element values; and determine the minimum vector element value among the multiple vector element values ​​as the difference between the J-th target object and the i-th first semantic feature. Here, the attention weight of each second semantic feature is the attention weight of that second semantic feature corresponding to the i-th first semantic feature.

[0228] Optionally, when generating the semantic feature set generation module 1320 for each target object, it is specifically used to: generate the semantic feature set for each target object based on the initial semantic features of each target object and the initial semantic features of the retrieved object.

[0229] Optionally, when generating the semantic feature set generation module 1320 for each target object, it specifically performs the following steps: inputting the initial semantic features of the Kth target object and the initial semantic features of the retrieved object into a pre-trained conditional variational autoencoder to obtain latent variables generated by the conditional variational autoencoder; collecting multiple latent variables from the latent variables generated by the conditional variational autoencoder, and generating the semantic feature set of the Kth target object based on the multiple latent variables. Here, the conditional variational autoencoder uses the initial semantic features of the retrieved object as the encoding guidance condition, and K is a positive integer less than or equal to N.

[0230] Optionally, the retrieval object and the target object belong to different data modalities. When generating the semantic feature set of the retrieval object, the semantic feature set generation module 1320 specifically performs the following steps: based on a pre-trained projection strategy, it projects the initial semantic features of the retrieval object onto the target vector space corresponding to the projection strategy to obtain the first projection feature of the retrieval object; and generates the semantic feature set of the retrieval object based on the first projection feature.

[0231] When generating the semantic feature set generation module 1320 for each target object, it specifically performs the following steps: based on a projection strategy, it projects the initial semantic features of each target object onto the target vector space to obtain the second projection features of each target object; and generates the semantic feature set corresponding to each target object based on the second projection features of each target object.

[0232] Optionally, the semantic feature set of the retrieved object and the semantic feature set of each target object are both generated through a semantic generation model.

[0233] Optionally, the semantic generation model includes a first algorithm model, a second algorithm model, and a third algorithm model. The first algorithm model generates multiple first intermediate features of the search object based on its initial semantic features. The third algorithm model determines similar features for each of the multiple first intermediate features and transforms each first intermediate feature into a corresponding first semantic feature based on each first intermediate feature and its similar features. The multiple first semantic features transformed from the multiple first intermediate features serve as the semantic feature set of the search object.

[0234] The second algorithm model is used to generate multiple second intermediate features of the target object based on the initial semantic features of the target object; the third algorithm model is also used to determine similar features for each second intermediate feature from the multiple second intermediate features, and to transform each second intermediate feature into a corresponding second semantic feature based on each second intermediate feature and the similar features of each second intermediate feature; the multiple second semantic features transformed from the multiple second intermediate features serve as the semantic feature set of the target object.

[0235] Optionally, the retrieval object and the target object belong to different data modalities. The first algorithm model is a multi-head attention mechanism, and the second algorithm model is a variational autoencoder or a conditional variational autoencoder.

[0236] Optionally, refer to Figure 14 , Figure 14 This is a schematic diagram of a retrieval device 1400 according to another embodiment of this application. (See diagram below.) Figure 14 As shown, the retrieval device 1400 includes not only: an initial semantic feature extraction module 1410, a semantic feature set generation module 1420, a matching degree determination module 1430, and a retrieval result determination module 1440, but also:

[0237] The sample acquisition module 1401 is used to obtain the sample retrieval object and the initial semantic features of the sample retrieval object, and to obtain multiple sample target objects and the initial semantic features of each sample target object. Each sample target object carries a label, and the label carried by each sample target object is used to characterize whether the sample target object has been retrieved by the sample retrieval object.

[0238] The predictive semantic feature set generation module 1402 is used to input the initial semantic features of the sample retrieval object into the semantic generation model to generate the first predictive semantic feature set of the sample retrieval object; and to input the initial semantic features of each sample target object into the semantic generation model to generate the second predictive semantic feature set corresponding to each sample target object.

[0239] The prediction matching degree determination module 1403 is used to determine the prediction matching degree between each sample target object and the sample retrieval object based on the second prediction semantic feature set and the first prediction semantic feature set of each sample target object.

[0240] The model training module 1404 is used to determine the loss value based on the label and predicted matching degree of the target object for each sample, and to train the semantic generation model based on the loss value.

[0241] Optionally, each sample target object carries a label that can be positive or negative. If a sample target object carries a positive label, it means that the sample target object has been retrieved by the sample retrieval object. If a sample target object carries a negative label, it means that the sample target object has not been retrieved by the sample retrieval object.

[0242] The loss value includes a distance penalty loss component. When determining the distance penalty loss component, the model training module 1404 specifically performs the following: based on the initial semantic features of each negatively labeled sample target object and the initial semantic features of the sample retrieval object, determine the prior matching degree between each negatively labeled sample target object and the sample retrieval object; based on the prior matching degree of each negatively labeled sample target object, determine the prior matching degree ranking of multiple prior matching degrees; based on the predicted matching degree of each negatively labeled sample target object, determine the predicted matching degree ranking of multiple predicted matching degrees; and based on the similarity between the prior matching degree ranking and the predicted matching degree ranking, determine the distance penalty loss component.

[0243] Optionally, the sample retrieval object and the sample target object belong to different data modalities. When generating the first predicted semantic feature set, the predicted semantic feature set generation module 1402 is specifically used to: project the initial semantic features of the sample retrieval object onto the preset vector space corresponding to the preset projection strategy based on the preset projection strategy to obtain the first sample projection features of the sample retrieval object; input the first sample projection features into the semantic generation model to generate the first predicted semantic feature set of the sample retrieval object.

[0244] When generating the second predicted semantic feature set, the predicted semantic feature set generation module 1402 is specifically used to: project the initial semantic features of each sample target object onto a preset vector space based on a preset projection strategy to obtain the second sample projection features of each sample target object; and input the second sample projection features of each sample target object into the semantic generation model to generate the second predicted semantic feature set corresponding to each sample target object.

[0245] The model training module 1404 is also used to: train a preset projection strategy based on the loss value.

[0246] The retrieval apparatus provided in this application generates a set of semantic features for the retrieval object based on the initial semantic features of the retrieval object, and generates a set of semantic features for the target object based on the initial semantic features of the target object. This means that the retrieval object and the target object are no longer represented by a single initial semantic feature, but by diverse semantic features. Then, based on the set of semantic features of the target object and the set of semantic features of the retrieval object, the matching degree between the target object and the retrieval object is determined, and the retrieval results are determined based on the matching degrees of multiple target objects with the retrieval object. Therefore, this application does not determine the matching degree based on two single initial semantic features, but rather based on two sets of diverse semantic features, thereby determining the retrieval results and making the retrieval results more diverse.

[0247] It should be noted that the device embodiments in this application correspond to the aforementioned method embodiments. The specific principles in the device embodiments can be found in the content of the aforementioned method embodiments, and will not be repeated here.

[0248] The following will combine Figure 15 This application describes an electronic device.

[0249] Figure 15 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0250] It should be noted that, Figure 15 The computer system of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0251] like Figure 15 As shown, the computer system includes a Central Processing Unit (CPU) 1501, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 1502 or programs loaded from Storage Unit 1508 into Random Access Memory (RAM) 1503, such as performing the methods described in the above embodiments. The RAM 1503 also stores various programs and data required for system operation. The CPU 1501, ROM 1502, and RAM 1503 are interconnected via a bus 1504. An Input / Output (I / O) interface 1505 is also connected to the bus 1504.

[0252] The following components are connected to I / O interface 1505: an input section 1506 including a keyboard, mouse, etc.; an output section 1507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1508 including a hard disk, etc.; and a communication section 1509 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1509 performs communication processing via a network such as the Internet. A drive 1510 is also connected to I / O interface 1505 as needed. Removable media 1511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1510 as needed so that computer programs read from them can be installed into storage section 1508 as needed.

[0253] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1509, and / or installed from removable medium 1511. When the computer program is executed by central processing unit (CPU) 1501, it performs various functions defined in the system of this application.

[0254] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0255] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0256] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0257] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0258] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0259] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0260] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0261] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A retrieval method, characterized in that, The method includes: Obtain the search object and extract the initial semantic features of the search object; Based on the initial semantic features of the search object, a set of semantic features of the search object is generated; Based on the initial semantic features of each of the N target objects, a semantic feature set corresponding to each target object is generated; the semantic feature set of the retrieved object includes p first semantic features; the semantic feature set of each target object includes q second semantic features, where N, p, and q are all integers greater than 1; Determining the matching degree between each target object and the search object based on the semantic feature set of each target object and the semantic feature set of the search object includes: determining a weighted sum of q second semantic features of the Jth target object based on each second semantic feature and the attention weight of each second semantic feature, wherein the attention weight of each second semantic feature is the attention weight of the second semantic feature corresponding to the i-th first semantic feature of the search object; determining the vector difference between the i-th first semantic feature and the weighted sum, wherein the vector difference includes multiple vector element values; determining the minimum vector element value among the multiple vector element values ​​as the difference degree between the Jth target object and the i-th first semantic feature; wherein J is a positive integer less than or equal to N, and i is a positive integer less than or equal to p; determining the minimum difference degree from the difference degrees between the Jth target object and each first semantic feature, and determining the matching degree between the Jth target object and the search object based on the minimum difference degree; Based on the matching degree between each target object and the search object, the target object is determined from the N target objects as the search result.

2. The method according to claim 1, characterized in that, The step of generating a semantic feature set corresponding to each of the N target objects based on the initial semantic features of each target object includes: Based on the initial semantic features of each target object and the initial semantic features of the retrieved object, a set of semantic features corresponding to each target object is generated.

3. The method according to claim 2, characterized in that, The step of generating a semantic feature set corresponding to each target object based on the initial semantic features of each target object and the initial semantic features of the retrieved object includes: The initial semantic features of the Kth target object and the initial semantic features of the retrieved object are input into a pre-trained conditional variational autoencoder to obtain the latent variables generated by the conditional variational autoencoder. The conditional variational autoencoder uses the initial semantic features of the retrieved object as the encoding guidance condition, and K is a positive integer less than or equal to N. Multiple latent variables are collected from the latent variables generated by the conditional variational autoencoder, and the semantic feature set of the Kth target object is generated based on the multiple latent variables.

4. The method according to claim 1, characterized in that, The search object and the target object belong to different data modalities; The step of generating a semantic feature set for the search object based on its initial semantic features includes: Based on a pre-trained projection strategy, the initial semantic features of the search object are projected onto the target vector space corresponding to the projection strategy to obtain the first projection features of the search object; based on the first projection features, a set of semantic features of the search object is generated. The step of generating a semantic feature set corresponding to each of the N target objects based on the initial semantic features of each target object includes: Based on the projection strategy, the initial semantic features of each target object are projected onto the target vector space to obtain the second projection features of each target object; and the semantic feature set corresponding to each target object is generated according to the second projection features of each target object.

5. The method according to claim 1, characterized in that, The semantic feature set of the search object and the semantic feature set of each target object are both generated through a semantic generation model.

6. The method according to claim 5, characterized in that, The semantic generation model includes a first algorithm model, a second algorithm model, and a third algorithm model; The first algorithm model is used to generate multiple first intermediate features of the search object based on the initial semantic features of the search object; The third algorithm model is used to determine similar features for each of the plurality of first intermediate features, and to transform each first intermediate feature into a corresponding first semantic feature based on each first intermediate feature and the similar features of each first intermediate feature; the plurality of first semantic features transformed from the plurality of first intermediate features serve as the semantic feature set of the retrieval object; The second algorithm model is used to generate multiple second intermediate features of the target object based on the initial semantic features of the target object; The third algorithm model is further used to determine similar features for each of the plurality of second intermediate features, and to transform each second intermediate feature into a corresponding second semantic feature based on each second intermediate feature and the similar features of each second intermediate feature; the plurality of second semantic features transformed from the plurality of second intermediate features serve as the semantic feature set of the target object.

7. The method according to claim 6, characterized in that, The retrieval object and the target object belong to different data modalities; the first algorithm model is a multi-head attention mechanism, and the second algorithm model is a variational autoencoder or a conditional variational autoencoder.

8. The method according to claim 5, characterized in that, The training process of the semantic generation model includes: Obtain the sample retrieval object and its initial semantic features, and obtain multiple sample target objects and their initial semantic features. Each sample target object carries a tag, and the tag carried by each sample target object is used to characterize whether the sample target object has been retrieved by the sample retrieval object. The initial semantic features of the sample retrieval object are input into the semantic generation model to generate the first predicted semantic feature set of the sample retrieval object; The initial semantic features of each sample target object are input into the semantic generation model to generate a second predicted semantic feature set corresponding to each sample target object; Based on the second predicted semantic feature set and the first predicted semantic feature set of each sample target object, the predicted matching degree between each sample target object and the sample retrieval object is determined; Based on the label and predicted matching degree of the target object for each sample, a loss value is determined, and the semantic generation model is trained based on the loss value.

9. The method according to claim 8, characterized in that, Each sample target object carries a label that can be either positive or negative. If a sample target object carries a positive label, it means that the sample target object has been retrieved by the sample retrieval object. If a sample target object carries a negative label, it means that the sample target object has not been retrieved by the sample retrieval object. The loss value includes a distance penalty loss component, and the process of determining the distance penalty loss component includes: Based on the initial semantic features of each negatively labeled sample target object and the initial semantic features of the sample retrieval object, the prior matching degree between each negatively labeled sample target object and the sample retrieval object is determined. Based on the prior matching degree of each sample target object carrying a negative label, determine the prior matching degree ranking of multiple prior matching degrees; Based on the predicted matching degree of each sample target object carrying a negative label, determine the ranking of the predicted matching degrees among multiple predicted matching degrees; The distance penalty loss component is determined based on the similarity between the prior matching degree ranking and the predicted matching degree ranking.

10. The method according to claim 8, characterized in that, The sample retrieval object and the sample target object belong to different data modalities; The step of inputting the initial semantic features of the sample retrieval object into the semantic generation model to generate a first predicted semantic feature set of the sample retrieval object includes: Based on a preset projection strategy, the initial semantic features of the sample retrieval object are projected onto a preset vector space corresponding to the preset projection strategy to obtain the first sample projection features of the sample retrieval object; the first sample projection features are input into the semantic generation model to generate the first predicted semantic feature set of the sample retrieval object. The step of inputting the initial semantic features of each sample target object into the semantic generation model to generate a second predicted semantic feature set corresponding to each sample target object includes: Based on the preset projection strategy, the initial semantic features of each sample target object are projected onto the preset vector space to obtain the second sample projection features of each sample target object; the second sample projection features of each sample target object are input into the semantic generation model to generate the second predicted semantic feature set corresponding to each sample target object; The method further includes: The preset projection strategy is trained based on the loss value.

11. A retrieval device, characterized in that, The device includes: An initial semantic feature extraction module is used to obtain the retrieval object and extract the initial semantic features of the retrieval object; The semantic feature set generation module is used to generate a semantic feature set of the search object based on the initial semantic features of the search object, and is also used to generate a semantic feature set corresponding to each of the N target objects based on the initial semantic features of each target object; the semantic feature set of the search object includes p first semantic features; the semantic feature set of each target object includes q second semantic features, where N, p, and q are all integers greater than 1; A matching degree determination module is used to determine the matching degree between each target object and the search object based on the semantic feature set of each target object and the semantic feature set of the search object. This includes: determining a weighted sum of q second semantic features of the J-th target object based on each second semantic feature and its attention weight, wherein the attention weight of each second semantic feature is the attention weight of the i-th first semantic feature of the search object corresponding to that second semantic feature; determining the vector difference between the i-th first semantic feature and the weighted sum, wherein the vector difference includes multiple vector element values; determining the minimum vector element value among the multiple vector element values ​​as the difference degree between the J-th target object and the i-th first semantic feature; where J is a positive integer less than or equal to N, and i is a positive integer less than or equal to p; determining the minimum difference degree from the difference degrees between the J-th target object and each first semantic feature, and determining the matching degree between the J-th target object and the search object based on the minimum difference degree. The retrieval result determination module is used to determine the target object as the retrieval result from the N target objects based on the matching degree between each target object and the retrieval object.

12. The apparatus according to claim 11, characterized in that, The semantic feature set generation module is used to generate a semantic feature set corresponding to each target object based on the initial semantic features of each target object and the initial semantic features of the search object.

13. The apparatus according to claim 12, characterized in that, The semantic feature set generation module is used to: input the initial semantic features of the Kth target object and the initial semantic features of the retrieved object into a pre-trained conditional variational autoencoder to obtain the latent variables generated by the conditional variational autoencoder, wherein the conditional variational autoencoder uses the initial semantic features of the retrieved object as the encoding guidance condition, and K is a positive integer less than or equal to N; Multiple latent variables are collected from the latent variables generated by the conditional variational autoencoder, and the semantic feature set of the Kth target object is generated based on the multiple latent variables.

14. The apparatus according to claim 11, characterized in that, The retrieval object and the target object belong to different data modalities; the semantic feature set generation module is used for: Based on a pre-trained projection strategy, the initial semantic features of the search object are projected onto the target vector space corresponding to the projection strategy to obtain the first projection features of the search object; based on the first projection features, a set of semantic features of the search object is generated. The step of generating a semantic feature set corresponding to each of the N target objects based on the initial semantic features of each target object includes: Based on the projection strategy, the initial semantic features of each target object are projected onto the target vector space to obtain the second projection features of each target object; Based on the second projection features of each target object, a semantic feature set corresponding to each target object is generated.

15. The apparatus according to claim 11, characterized in that, The semantic feature set of the search object and the semantic feature set of each target object are both generated through a semantic generation model.

16. The apparatus according to claim 15, characterized in that, The semantic generation model includes a first algorithm model, a second algorithm model, and a third algorithm model; The first algorithm model is used to generate multiple first intermediate features of the search object based on the initial semantic features of the search object; The third algorithm model is used to determine similar features for each of the plurality of first intermediate features, and to transform each first intermediate feature into a corresponding first semantic feature based on each first intermediate feature and the similar features of each first intermediate feature; the plurality of first semantic features transformed from the plurality of first intermediate features serve as the semantic feature set of the retrieval object; The second algorithm model is used to generate multiple second intermediate features of the target object based on the initial semantic features of the target object; The third algorithm model is further used to determine similar features for each of the plurality of second intermediate features, and to transform each second intermediate feature into a corresponding second semantic feature based on each second intermediate feature and the similar features of each second intermediate feature; the plurality of second semantic features transformed from the plurality of second intermediate features serve as the semantic feature set of the target object.

17. The apparatus according to claim 16, characterized in that, The retrieval object and the target object belong to different data modalities; the first algorithm model is a multi-head attention mechanism, and the second algorithm model is a variational autoencoder or a conditional variational autoencoder.

18. The apparatus according to claim 15, characterized in that, The retrieval device further includes: The sample acquisition module is used to obtain the sample retrieval object and the initial semantic features of the sample retrieval object, and to obtain multiple sample target objects and the initial semantic features of each sample target object. Each sample target object carries a tag, and the tag carried by each sample target object is used to characterize whether the sample target object has been retrieved by the sample retrieval object. The predictive semantic feature set generation module is used to input the initial semantic features of the sample retrieval object into the semantic generation model to generate a first predictive semantic feature set of the sample retrieval object; and is used to input the initial semantic features of each sample target object into the semantic generation model to generate a second predictive semantic feature set corresponding to each sample target object; The prediction matching degree determination module is used to determine the prediction matching degree between each sample target object and the sample retrieval object based on the second prediction semantic feature set and the first prediction semantic feature set of each sample target object; The model training module is used to determine the loss value based on the label and predicted matching degree of the target object of each sample, and to train the semantic generation model based on the loss value.

19. The apparatus according to claim 18, characterized in that, Each sample target object carries a label that can be either positive or negative. If a sample target object carries a positive label, it means that the sample target object has been retrieved by the sample retrieval object. If a sample target object carries a negative label, it means that the sample target object has not been retrieved by the sample retrieval object. The loss value includes a distance penalty loss component, and the model training module is further used to: determine the prior matching degree between each negatively labeled sample target object and the sample retrieval object based on the initial semantic features of each negatively labeled sample target object and the initial semantic features of the sample retrieval object; Based on the prior matching degree of each sample target object carrying a negative label, determine the prior matching degree ranking of multiple prior matching degrees; Based on the predicted matching degree of each sample target object carrying a negative label, determine the ranking of the predicted matching degrees among multiple predicted matching degrees; The distance penalty loss component is determined based on the similarity between the prior matching degree ranking and the predicted matching degree ranking.

20. The apparatus according to claim 18, characterized in that, The sample retrieval object and the sample target object belong to different data modalities; The predictive semantic feature set generation module is further configured to: project the initial semantic features of the sample retrieval object onto the preset vector space corresponding to the preset projection strategy based on a preset projection strategy, thereby obtaining the first sample projection feature of the sample retrieval object; The first sample projection features are input into the semantic generation model to generate a first set of predicted semantic features for the sample retrieval object. Based on the preset projection strategy, the initial semantic features of each sample target object are projected onto the preset vector space to obtain the second sample projection features of each sample target object; The second sample projection features of each sample target object are input into the semantic generation model to generate a second predicted semantic feature set corresponding to each sample target object; The model training module is also used to: train the preset projection strategy based on the loss value.

21. An electronic device, characterized in that, It includes a processor and a memory; one or more programs are stored in the memory and configured to be executed by the processor to implement the method of any one of claims 1-10.

22. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code, wherein the program code, when executed by a processor, performs the method according to any one of claims 1-10.

23. A computer program product, characterized in that, It includes a computer program, which, when executed by a processor, implements the method as described in any one of claims 1-10.

Images