Method and apparatus for training a representation model

Abstract

The embodiments of the present disclosure disclose a method and device for training a representation model. A specific implementation of the method includes: constructing a semantic tree based on the category information of a sample image set; initializing the weight of each edge of the semantic tree; performing the following training steps based on the weight: setting the truncation coefficient of the loss function of the representation model to be trained based on the shortest path distance between each leaf node; optimizing the representation model based on the loss function; calculating the distance between each sample image based on the optimized representation model, converting it into an update amount of the shortest path distance between each leaf node, and updating the edge weights connecting the leaf nodes on the semantic tree according to the update amount; if the predetermined conditions are met, outputting the representation model; otherwise, continuing to perform the above training steps after calculating the shortest path distance between each leaf node based on the updated weight. This implementation can represent images in both visual and semantic dimensions.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a method and apparatus for training a representation model. Background Art

[0002] Feature Learning is the core technology of commercial image retrieval engines and an important technology widely used in the field of computer vision. Feature Learning completes the mapping of images from pixel space to representation space. On the one hand, feature learning can reduce information to ensure the computability of similarity of images in the same space; on the other hand, feature learning can retain the most discriminative information of objects while tolerating intra-class changes such as object deformation, background noise, and shooting angle.

[0003] In recent years, with the rapid development of deep learning, representation learning has also made major breakthroughs in various directions, especially in the task of product image retrieval in e-commerce scenarios, which has a strong demand for representation learning. Currently, representation learning technology for image retrieval tasks mainly focuses on the following two directions:

[0004] 1. Representation Learning Technology for Semantic Retrieval

[0005] Image retrieval for semantic retrieval obtains a representation mapping that describes the type of object information through representation learning. Contrastive Loss is generally used for representation learning training. Its goal is to represent the degree of matching between image pairs at the category level. During the optimization process, the distance between two similar images in the feature space is minimized as much as possible, while the distance between two dissimilar images in the feature space is maximized as much as possible.

[0006] 2. Representation Learning Technology for Fine-Grained Retrieval

[0007] Image retrieval technology in this direction distinguishes objects with small differences through representation learning, and generally uses triplet loss for representation learning training. By constructing a triplet consisting of an anchor sample, a positive sample, and a negative sample, the distance between the anchor sample and the positive sample is reduced and the distance between the anchor sample and the negative sample is increased during the optimization process, thereby optimizing the similarity of the image samples in the feature space.

[0008] There are many variants of the above two technologies, but the following problems still exist:

[0009] 1. Sorting results affect user experience: Current technologies only reflect similarity in one dimension, which cannot meet the needs of similarity in two dimensions (semantic similarity and visual similarity) in actual scenarios;

[0010] 2. Hyperparameter setting is difficult: Hyperparameters need to be set manually and the actual effect cannot be predicted;

[0011] 3. Poor transferability: If there are differences between data sets, the effectiveness of the model training method will be greatly reduced. Summary of the invention

[0012] Embodiments of the present disclosure provide methods and devices for training a representation model.

[0013] In the first aspect, an embodiment of the present disclosure provides a method for training a representation model, comprising: constructing a semantic tree based on category information of a sample image set; initializing the weight of each edge of the semantic tree; performing the following training steps based on the weight: setting the truncation coefficient of the loss function of the representation model to be trained based on the shortest path distance between each leaf node; optimizing the representation model based on the loss function; calculating the distance between each sample image based on the optimized representation model, converting it into an update amount of the shortest path distance between each leaf node, and updating the edge weights connecting the leaf nodes on the semantic tree according to the update amount; if a predetermined condition is met, outputting the representation model; otherwise, continuing to perform the above training steps after calculating the shortest path distance between each leaf node based on the updated weight.

[0014] In some embodiments, a semantic tree is constructed based on the category information of a sample picture set, including: obtaining the category information of the sample picture set as labels of different levels of pictures according to the subordinate relationship of the semantic information; constructing a semantic tree according to multi-level labels, wherein the root node is the item, the next layer of nodes is the first-level label, and so on, the bottom-level leaf node is the deepest-level label.

[0015] In some embodiments, the loss function is as follows:

[0016]

[0017] Among them, i, j, k are the index numbers of sample images, P and N are the sets of positive and negative examples, D() calculates the distance between two sample images, M i，k is the truncation coefficient corresponding to the deepest level label of the two sample images.

[0018] In some embodiments, the truncation coefficients are updated in the following form

[0019]

[0020] Among them, m is the initial value, α is the optimization step size, S i，j is the distance between two sample images.

[0021] In some embodiments, the predetermined condition includes: the shortest path distance order between leaf nodes no longer changes.

[0022] In some embodiments, the method further includes: inputting images in a database into a trained representation model to obtain a feature set; inputting a target image into a trained representation model to obtain a target feature; respectively calculating the similarity between the target feature and the features in the feature set; and outputting a predetermined number of images in order of similarity from large to small.

[0023] In a second aspect, an embodiment of the present disclosure provides a method for outputting information, comprising: inputting an image in a database into a representation model trained according to the method described in the first aspect to obtain a feature set; inputting a target image into the representation model to obtain a target feature; respectively calculating the similarity between the target feature and the features in the feature set; and outputting a predetermined number of images in order of similarity from large to small. In a third aspect, an embodiment of the present disclosure provides a device for training a representation model, comprising: a construction unit, configured to construct a semantic tree based on category information of a sample image set; an initialization unit, configured to initialize the weight of each edge of the semantic tree; a training unit, configured to perform the following training steps based on the weights: setting the truncation coefficient of the loss function of the representation model to be trained based on the shortest path distance between each leaf node; optimizing the representation model based on the loss function; calculating the distance between each sample image based on the optimized representation model, converting it into an update amount of the shortest path distance between each leaf node, and updating the edge weights connecting the leaf nodes on the semantic tree according to the update amount; outputting the representation model if a predetermined condition is met; and a loop unit, configured to continue to perform the above training steps after calculating the shortest path distance between each leaf node based on the updated weight if the predetermined condition is not met.

[0024] In some embodiments, the construction unit is further configured to: obtain category information of the sample picture set as labels of different levels of pictures according to the subordinate relationship of semantic information; construct a semantic tree according to multi-level labels, wherein the root node is the item, the next layer of nodes is the first-level label, and so on, and the bottom-level leaf node is the deepest level label.

[0025] In some embodiments, the loss function is as follows:

[0026]

[0027] Among them, i, j, k are the index numbers of sample images, P and N are the sets of positive and negative examples, D() calculates the distance between two sample images, M i，k is the truncation coefficient corresponding to the deepest level label of the two sample images.

[0028] In some embodiments, the truncation coefficients are updated in the following form:

[0029]

[0030] Among them, m is the initial value, α is the optimization step size, S i，j is the distance between two sample images.

[0031] In some embodiments, the predetermined condition includes: the shortest path distance order between leaf nodes no longer changes.

[0032] In a fourth aspect, an embodiment of the present disclosure provides a device for outputting information, comprising: a database building unit, configured to input images in a database into a representation model trained according to a method according to one of claims 1-5 to obtain a feature set; a query unit, configured to input a target image into the trained representation model to obtain a target feature; a calculation unit, configured to respectively calculate the similarity between the target feature and the features in the feature set; and output a predetermined number of images in order of similarity from large to small.

[0033] In a fifth aspect, an embodiment of the present disclosure provides an electronic device for training a characterization model, comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement a method as described in any one of the first aspect or the second aspect.

[0034] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method of any one of the first aspect or the second aspect is implemented.

[0035] The method and device for training a representation model provided by the embodiments of the present disclosure can solve the problem that current representation learning methods can only reflect the sorting results of one dimension, reduce the difficulty of hyperparameter optimization during model training, and improve the generalization ability of the algorithm on different data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0037] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0038] Figure 2 is a flowchart of an embodiment of a method for training a representation model according to the present disclosure;

[0039] Figure 3 is a schematic diagram of an application scenario of the method for training a representation model according to the present disclosure;

[0040] Figure 4is a flowchart of another embodiment of a method for training a representation model according to the present disclosure;

[0041] Figure 5 is a structural schematic diagram of an embodiment of a device for training a representation model according to the present disclosure;

[0042] Figure 6 It is a structural diagram of a computer system of an electronic device suitable for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0043] The present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0044] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0045] Figure 1 An exemplary system architecture 100 is shown to which an embodiment of a method for training a representation model or an apparatus for training a representation model of the present disclosure can be applied.

[0046] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0047] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as image recognition applications, image retrieval applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0048] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens and supporting image retrieval, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4) players, laptop computers and desktop computers, etc. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0049] The server 105 may be a server that provides various services, such as a background search server that provides search support for images displayed on the terminal devices 101, 102, and 103. The background search server may analyze and process the received search request and other data, and feed back the processing results (such as relevant information of similar or identical images) to the terminal device.

[0050] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, multiple software or software modules for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0051] It should be noted that the method for training the representation model provided in the embodiments of the present disclosure is generally executed by the server 105 , and accordingly, the device for training the representation model is generally disposed in the server 105 .

[0052] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0053] Continue to refer Figure 2 , shows a process 200 of an embodiment of a method for training a representation model according to the present disclosure. The method for training a representation model comprises the following steps:

[0054] Step 201: construct a semantic tree based on the category information of the sample picture set.

[0055] In this embodiment, the execution subject (eg, Figure 1 The server shown in the figure) can obtain the sample picture set from the third-party server through a wired connection or a wireless connection. The sample pictures in the sample picture set have corresponding category information. Taking the picture of shoes as an example, its category information includes multi-level labels. Its first-level label to fourth-level label can be set as "shoes"-"casual sports shoes"-"running shoes"-"a brand of running shoes". A semantic tree is constructed based on the multi-level labels, with the root node being the item, the next layer of nodes being the first-level label, and so on, down to the bottom leaf node being the fourth-level label. In actual applications, it is not limited to four-level labels.

[0056] Step 202: Initialize the weight of each edge of the semantic tree.

[0057] In this embodiment, each edge on the semantic tree is initialized with a weight, and the distance of the shortest path between different leaf nodes is represented by the degree of difference in their semantics (categories). The weights of edges at the same level are set to the same value. For example, all are set to 1.

[0058] Step 203: setting a truncation coefficient of the loss function of the representation model to be trained based on the shortest path distance between each leaf node.

[0059] In this embodiment, the representation model here is a neural network for mapping the image from the pixel space to the representation space, for example, a convolutional neural network. The distance between nodes is the sum of the weights of the edges passing between the nodes. The truncation coefficient (margin) is set based on the shortest path distance between leaf nodes, semantic information is introduced into the similarity matching definition of the image, a hierarchical loss function is constructed, and the model optimization of the degree of visual matching is performed under the constraint of semantic information.

[0060] In order to optimize the model of visual matching degree under the constraint of semantic information, different truncation functions are used for image pairs between different leaf nodes (four-level labels). The present invention designs the following loss function:

[0061]

[0062] Among them, i, j, k are the index numbers of the pictures respectively, and f i represents the features of the i-th image, f j represents the feature of the jth image, f k The features P and N of the kth image are respectively the sets of positive and negative examples. D() calculates the distance between two images, which can be calculated using the Euclidean distance. M i,kis the cutoff coefficient corresponding to the four-level labels of the two images. The first term in the formula represents the penalty term set based on the distance between the anchor sample and all positive samples, and the second term represents the penalty term set based on the distance between the anchor sample and all negative samples. The goal of the optimization model of the present invention is to minimize the penalty value calculated according to this loss function.

[0063] The update of the truncation coefficient is performed in the following form:

[0064]

[0065] Among them, m is the initial value, α is the optimization step size, S i,j is the semantic distance between images. i , f j ) and S i,j The difference is that D(f i , f j ) is the distance calculated by the model during the iterative optimization process, S i,j is the distance calculated after the model is fixed.

[0066] Step 204: Optimize the representation model based on the loss function.

[0067] In this embodiment, according to the loss function requirement of step 203, the parameters of the characterization model are adjusted so that the loss value between the input sample and the output sample meets the loss function requirement.

[0068] Step 205, based on the optimized representation model, the distance between each sample image is calculated, and converted into the update amount of the shortest path distance between each leaf node, and the edge weights connecting the leaf nodes on the semantic tree are updated according to the update amount.

[0069] In this embodiment, each sample image is input into the optimized representation model to obtain the features of each sample image. Then the distance S between each feature is calculated. i,j Then, through formula 2, S i,j Convert it to the updated amount of the shortest path distance between leaf nodes on the semantic tree. Then update the edge weights connecting the leaf nodes to the updated amount of the shortest path distance. This is equivalent to only updating the shortest path distance. There is no need to consider the weights updated for each edge separately.

[0070] Step 206: If the predetermined conditions are met, the representation model is output.

[0071] In this embodiment, the predetermined condition is the convergence condition of the model, which may be that the number of iterations reaches a predetermined value, etc. It may also include that the shortest path distance ranking between leaf nodes no longer changes. After each weight update, the shortest path distance between leaf nodes may change, which will affect the ranking. For example, for leaf node A, the leaf nodes closest to it are B, C, D, E, and F in sequence. After the weight is updated, the nearest leaf nodes are still B, C, D, E, and F, indicating that the training is completed and the representation model can be output to other servers or terminal devices.

[0072] Step 207: If the predetermined condition is not met, the shortest path distance between the leaf nodes is calculated based on the updated weights and then steps 203-207 are continued.

[0073] In this embodiment, as shown in the above example, for leaf node A, the original nearest leaf nodes are B, C, D, E, and F. After the weights are updated, the nearest leaf nodes are C, F, B, D, and E, indicating that the training is not completed and it is necessary to continue to use the updated weights to calculate the shortest path distance between the leaf nodes and continue to execute steps 203-207 until the predetermined conditions are met.

[0074] Continue to see Figure 3 , Figure 3 FIG. 1 is a schematic diagram of an application scenario of the method for training a representation model according to this embodiment. Figure 3 In the application scenario, the server obtains the labels of picture 1: "shoes" - "casual sports shoes" - "running shoes" - "brand A running shoes". The labels of picture 2: "shoes" - "casual sports shoes" - "running shoes" - "brand B running shoes". Each edge has a weight. The shortest path distance between the leaf node "brand A running shoes" and the leaf node "brand B running shoes" is weight 3 + weight 3. The truncation coefficient of the loss function is set according to their sum. The loss function is then used to adjust the parameters of the representation model to obtain the optimized representation model. The representation model is used to calculate the distance between the features of each sample image, and then converted into the shortest path distance update between leaf nodes according to formula 2, the weights of the edges of the leaf nodes are updated, and the above process is repeated until the model converges after the predetermined conditions are met.

[0075] In actual tests, the two test sets CARS196 and CUB-200-2011 have relatively obvious multi-level information. The recall rate (Recall@N) is used as the evaluation criterion on these two test sets (the higher the value, the better the ranking result). The specific results are as follows:

[0076] Table 1. Results on CARS196

[0077]

[0078] Table 2. Results on CUB-200-1011

[0079]

[0080] The method provided by the above embodiments of the present disclosure, under the same data and network structure, the loss function proposed by the present invention can represent the image in both visual and semantic dimensions, and convert the similarity of these two dimensions into the final ranking result, thereby improving the user experience of commercial search engines, especially the commodity search task in the e-commerce scenario. At the same time, by adaptively modifying the hyperparameters, the tuning cost caused by setting the hyperparameters is reduced.

[0081] Further references Figure 4 , which shows a process 400 of another embodiment of a method for training a representation model. The process 400 of the method for training a representation model includes the following steps:

[0082] Step 401: construct a semantic tree based on the category information of the sample picture set.

[0083] Step 402: Initialize the weight of each edge of the semantic tree.

[0084] Step 403: setting a truncation coefficient of the loss function of the representation model to be trained based on the shortest path distance between each leaf node.

[0085] Step 404: Optimize the representation model based on the loss function.

[0086] Step 405, based on the optimized representation model, the distance between each sample image is calculated, and converted into the update amount of the shortest path distance between each leaf node, and the edge weights connecting the leaf nodes on the semantic tree are updated according to the update amount.

[0087] Steps 401-405 are substantially the same as steps 201-205, and thus are not described in detail herein.

[0088] Step 406: If the predetermined condition is not met, the shortest path distance between the leaf nodes is calculated based on the updated weights and then steps 403-406 are continued.

[0089] Step 406 is substantially the same as step 207 and thus will not be described in detail.

[0090] Step 407: If the predetermined conditions are met, the image in the database and the target image are respectively input into the trained representation model to obtain a feature set and a target feature.

[0091] In this embodiment, the trained representation model is used to extract features of the images in the database and the target images to be retrieved.

[0092] Step 408, respectively calculating the similarity between the target feature and the features in the feature set.

[0093] In this embodiment, the images are compared by calculating the similarity between the target feature and other features.

[0094] Step 409: output a predetermined number of pictures in descending order of similarity.

[0095] In this embodiment, a predetermined number of pictures with the highest similarity to the target picture are output as recommended pictures. The relevant information of these pictures can also be output together, thereby realizing picture retrieval.

[0096] from Figure 4 It can be seen that Figure 2 Compared with the corresponding embodiment, the process 400 of the method for training the representation model in this embodiment embodies the steps of using the trained representation model to perform image retrieval, which can meet the requirements of similarity in two dimensions (semantic similarity and visual similarity) in actual scenes, thereby improving the recall rate of retrieval.

[0097] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for training a representation model. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0098] like Figure 5 As shown, the apparatus 500 for training the representation model of this embodiment includes: a construction unit 501, an initialization unit 502, a training unit 503 and a loop unit 504. Among them, the construction unit 501 is configured to construct a semantic tree based on the category information of the sample image set; the initialization unit 502 is configured to initialize the weight of each edge of the semantic tree; the training unit 503 is configured to perform the following training steps based on the weight: set the truncation coefficient of the loss function of the representation model to be trained based on the shortest path distance between each leaf node; optimize the representation model based on the loss function; calculate the distance between each sample image based on the optimized representation model, convert it into the update amount of the shortest path distance between each leaf node, and update the edge weights of the leaf node connection on the semantic tree according to the update amount; if the predetermined condition is met, output the representation model; if the predetermined condition is not met, the loop unit 504 is configured to continue to perform the above training steps after calculating the shortest path distance between each leaf node based on the updated weight.

[0099] In this embodiment, the specific processing of the construction unit 501, the initialization unit 502, the training unit 503 and the circulation unit 504 of the apparatus 500 for training the representation model can be referred to. Figure 2Corresponding to step 201, step 202, step 203, and step 204 in the embodiment.

[0100] In some optional implementations of the present embodiment, the construction unit 501 is further configured to: obtain category information of the sample picture set as labels of different levels of pictures according to the subordinate relationship of the semantic information; construct a semantic tree according to the multi-level labels, wherein the root node is the item, the next layer of nodes is the first-level label, and so on, the bottom-level leaf node is the deepest-level label.

[0101] In some optional implementations of this embodiment, the loss function is shown as follows:

[0102]

[0103] Among them, i, j, k are the index numbers of sample images, P and N are the sets of positive and negative examples, D() calculates the distance between two sample images, M i，k is the truncation coefficient corresponding to the deepest level label of the two sample images.

[0104] In some optional implementations of this embodiment, the truncation coefficient is updated in the following form:

[0105]

[0106] Among them, m is the initial value, α is the optimization step size, S i，j is the distance between two sample images.

[0107] In some optional implementations of this embodiment, the predetermined condition includes: the shortest path distance order between leaf nodes no longer changes.

[0108] In some optional implementations of the present embodiment, the device 500 also includes a query unit (not shown in the drawings), which is configured to: input an image in a database into a trained representation model to obtain a feature set; input a target image into a trained representation model to obtain a target feature; respectively calculate the similarity between the target feature and the features in the feature set; and output a predetermined number of images in order of similarity from large to small.

[0109] Reference below Figure 6 , which shows an electronic device (eg, Figure 1The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The terminal device / server shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0110] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0111] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 6 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0112] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above functions defined in the method of the embodiment of the present disclosure are executed. It should be noted that the computer-readable medium described in the embodiment of the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In an embodiment of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus, or a device. In an embodiment of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, an apparatus, or a device. The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wire, optical cable, RF (radio frequency), etc., or any suitable combination of the foregoing.

[0113] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: constructs a semantic tree based on the category information of the sample image set; initializes the weight of each edge of the semantic tree; performs the following training steps based on the weight: sets the truncation coefficient of the loss function of the representation model to be trained based on the shortest path distance between each leaf node; optimizes the representation model based on the loss function; calculates the distance between each sample image based on the optimized representation model, converts it into the update amount of the shortest path distance between each leaf node, and updates the edge weights of the leaf node connection on the semantic tree according to the update amount; if the predetermined conditions are met, the representation model is output; otherwise, the shortest path distance between each leaf node is calculated based on the updated weight and the above training steps are continued.

[0114] Computer program code for performing the operations of embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as a separate software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0115] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0116] The units involved in the embodiments described in the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor including a construction unit, an initialization unit, a training unit, and a loop unit. Among them, the names of these units do not constitute a limitation on the unit itself in some cases, for example, the construction unit may also be described as a "unit for constructing a semantic tree based on the category information of a sample picture set".

[0117] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present disclosure is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other.

Claims

1. A method for training a representation model, comprising: Construct a semantic tree based on the category information of the sample image set; Initializing the weight of each edge of the semantic tree, wherein the distance of the shortest path between different leaf nodes is expressed as the degree of semantic difference, and the weights of edges at the same level are set to the same value; The following training steps are performed based on the weights: a truncation coefficient of a loss function of a representation model to be trained is set based on the shortest path distance between each leaf node, wherein semantic information is introduced into the definition of similar matching of images, a hierarchical loss function is constructed, and a model optimization of the degree of visual matching is performed under the constraint of semantic information, and different truncation functions are used for image pairs between different leaf nodes; the representation model is optimized based on the loss function; the distance between each sample image is calculated based on the optimized representation model, and converted into an update amount of the shortest path distance between each leaf node, and the edge weights connecting the leaf nodes on the semantic tree are updated according to the update amount; if a predetermined condition is met, the representation model is output, wherein the predetermined condition includes that the sorting of the shortest path distance between the leaf nodes does not change; Otherwise, the shortest path distance between each leaf node is calculated based on the updated weights and the above training steps are continued; The loss function is shown in the following formula: Among them, i, j, k are the index numbers of the sample images, respectively, and f i 、f j 、f k Represent the features of the i-th, j-th, and k-th sample images respectively; P and N are the sets of positive and negative examples respectively, D() calculates the distance between two sample images, and M i，k is the truncation coefficient corresponding to the deepest level label of the two sample images.

2. The method according to claim 1, wherein: The semantic tree is constructed based on the category information of the sample picture set, including: According to the subordinate relationship of semantic information, the category information of the sample image set is obtained as labels of different levels of images; A semantic tree is constructed based on multi-level labels, where the root node is the item, the next layer of nodes is the first-level label, and so on, and the leaf nodes at the bottom are the deepest-level labels.

3. The method according to claim 1, wherein: The truncation coefficient is updated in the following form: Among them, m is the initial value, α is the optimization step size, S i，j is the distance between two sample images.

4. A method for outputting information, comprising: Inputting the images in the database into the representation model trained by the method according to any one of claims 1 to 3 to obtain a feature set; Inputting the target image into the representation model to obtain target features; Calculate the similarity between the target feature and the features in the feature set respectively; Output a predetermined number of pictures in descending order of similarity.

5. A device for training a representation model, comprising: A construction unit is configured to construct a semantic tree based on category information of the sample image set; an initialization unit, configured to initialize the weight of each edge of the semantic tree, wherein the distance of the shortest path between different leaf nodes is represented as the degree of semantic difference, and the weights of edges at the same level are set to the same value; The training unit is configured to perform the following training steps based on the weights: setting a truncation coefficient of a loss function of a representation model to be trained based on the shortest path distance between each leaf node, wherein semantic information is introduced into the definition of similar matching of images, a hierarchical loss function is constructed, and a model optimization of the degree of visual matching is performed under the constraint of semantic information, and different truncation functions are used for image pairs between different leaf nodes; optimizing the representation model based on the loss function; calculating the distance between each sample image based on the optimized representation model, converting it into an update amount of the shortest path distance between each leaf node, and updating the edge weights of the leaf node connections on the semantic tree according to the update amount; outputting the representation model if a predetermined condition is met, wherein the predetermined condition includes that the shortest path distance sorting between leaf nodes no longer changes; The loop unit is configured to continue to perform the above training steps after calculating the shortest path distance between the leaf nodes based on the updated weights if the predetermined condition is not met; The loss function is shown in the following formula: Among them, i, j, k are the index numbers of the sample images, respectively, and f i 、f j 、f k Represent the features of the i-th, j-th, and k-th sample images respectively; P and N are the sets of positive and negative examples respectively, D() calculates the distance between two sample images, and M i，k is the truncation coefficient corresponding to the deepest level label of the two sample images.

6. A device for outputting information, comprising: A database building unit, configured to input the images in the database into the representation model trained by the method according to any one of claims 1 to 3 to obtain a feature set; A query unit, configured to input a target image into the trained representation model to obtain target features; A calculation unit is configured to respectively calculate the similarity between the target feature and the features in the feature set; Output a predetermined number of pictures in descending order of similarity.

7. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

8. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images