Cross-modal hash model training method, encoding method, device and electronic device

Abstract

The present application provides a training method, encoding method, device and electronic device for a cross-modal hash model; it relates to the field of artificial intelligence technology, and the method includes: calling a cross-modal hash model to perform dimensionality reduction hash coding processing on multiple sample pairs obtained to obtain multiple hash coding pairs; for each hash coding pair, in the hash data point pair at each position in the hash coding pair, determine the target data hash point with a larger weight, and determine the binary code of the hash coding pair based on each target hash data point; based on each hash coding pair, the similarity matrix corresponding to the multiple sample pairs, and the difference between each hash coding pair and the corresponding binary code, determine the total quantization loss of the cross-modal hash model; update the parameters of the cross-modal hash model based on the total quantization loss. Through this application, the encoding accuracy of the cross-modal hash model can be improved, thereby saving the computing resources occupied by calculating the similarity between the encoding results of different samples.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular to a training method, encoding method, device and electronic device for a cross-modal hash model. Background Art

[0002] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0003] Cross-modal hashing technology is a hash coding technology that maps high-dimensional data of different modalities to a unified low-dimensional Hamming space, usually implemented through a cross-modal hash model. The cross-modal hash model will generate quantization loss in the process of converting high-dimensional data of samples of different modalities into low-dimensional hash codes. There is currently no more effective way to determine more accurate quantization loss in related technologies, resulting in low coding accuracy based on the trained cross-modal hash model.

[0004] Since the cross-modal hash model reduces the dimension of high-dimensional data into hash codes of the same coding length, and represents the similarity between samples by the distance between hash codes of the same coding length (the distance between hash codes is negatively correlated with the similarity), if the coding accuracy is low, the accuracy of the distance between hash codes in representing the similarity between samples is not high, resulting in higher computing resource consumption in the process of calculating the similarity between samples based on the cross-modal hash model with low coding accuracy. Summary of the invention

[0005] The embodiments of the present application provide a training method, encoding method, device and electronic device for a cross-modal hash model, which can improve the accuracy of calculating the similarity between samples using hash coding, thereby saving related computing resources.

[0006] The technical solution of the embodiment of the present application is implemented as follows:

[0007] The present application provides a method for training a cross-modal hash model, the method comprising:

[0008] Acquire multiple sample pairs, wherein each sample pair includes two samples of different modalities for describing the same content;

[0009] Calling a cross-modal hash model to perform dimensionality reduction hash coding processing on the plurality of sample pairs to obtain a plurality of hash code pairs corresponding one-to-one to the plurality of sample pairs, wherein a distance between two hash codes in the hash code pair is used to characterize the similarity between the two samples corresponding to the hash code pair;

[0010] The following processing is performed for each of the hash code pairs: in the hash data point pair at each position in the hash code pair, a target hash data point with a larger weight is determined, and a binary code of the hash code pair is determined based on the target hash data point corresponding to each position, wherein the weight is used to represent the importance of the hash data point for calculating the similarity of the sample pair corresponding to the hash code pair;

[0011] Determine the total quantization loss of the cross-modal hash model based on each of the hash code pairs, the similarity matrix corresponding to the multiple sample pairs, and the difference between each of the hash code pairs and the binary code corresponding to each of the hash code pairs, wherein each element in the similarity matrix is ​​used to characterize the feature similarity corresponding to samples of different modalities;

[0012] Parameters of the cross-modal hashing model are updated based on the total quantization loss.

[0013] An embodiment of the present application provides a method for encoding a cross-modal hash model, the method comprising:

[0014] Acquire a sample pair to be processed, wherein the sample pair includes two samples for describing the same content and in different modalities;

[0015] The cross-modal hash model is called to perform dimensionality reduction hash coding processing on the sample pair to obtain a hash code pair corresponding to the sample pair one by one, wherein the distance between two hash codes in the hash code pair is used to characterize the similarity between the two samples corresponding to the hash code pair.

[0016] The present application embodiment provides a cross-modal hash model training device, including:

[0017] A sample acquisition module, used to acquire a plurality of sample pairs, wherein each sample pair includes two samples for describing the same content and in different modalities;

[0018] A hash coding module, used to call a cross-modal hash model to perform dimensionality reduction hash coding processing on the plurality of sample pairs, to obtain a plurality of hash coding pairs corresponding one-to-one to the plurality of sample pairs, wherein the distance between two hash codes in the hash coding pair is used to characterize the similarity between the two samples corresponding to the hash coding pair;

[0019] A binary code generation module is used to perform the following processing for each of the hash code pairs: determining a target hash data point with a larger weight in the hash data point pair at each position in the hash code pair, and determining a binary code of the hash code pair based on the target hash data point corresponding to each position, wherein the weight is used to represent the importance of the hash data point for calculating the similarity of the sample pair corresponding to the hash code pair;

[0020] A loss determination module, configured to determine a total quantization loss of the cross-modal hash model based on a similarity matrix corresponding to each of the hash code pairs and the plurality of sample pairs, and a difference between each of the hash code pairs and the binary code corresponding to each of the hash code pairs, wherein each element in the similarity matrix is ​​used to characterize a feature similarity corresponding to samples of different modalities;

[0021] The loss determination module is also used to update the parameters of the cross-modal hash model based on the total quantization loss.

[0022] An embodiment of the present application provides an electronic device for training a cross-modal hash model, the electronic device comprising:

[0023] A memory for storing executable instructions;

[0024] The processor is used to implement the method described in the embodiment of the present application when executing the executable instructions stored in the memory.

[0025] An embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein the executable instructions implement the method described in the embodiment of the present application when executed by a processor.

[0026] An embodiment of the present application provides a computer program product, including a computer program or instructions, characterized in that when the computer program or instructions are executed by a processor, the method described in the embodiment of the present application is implemented.

[0027] The embodiments of the present application have the following beneficial effects:

[0028] The weight is positively correlated with the importance of each hash data point in the hash code. The binary code of the hash code pair determined based on the hash data point with a higher weight is more accurate. The binary code can be used to obtain a more accurate total quantization loss of the cross-modal hash model. The cross-modal hash model is trained based on the total quantization loss, so that the distance between the hash codes output by the cross-modal hash model can more accurately represent the similarity between samples, thereby improving the coding accuracy of the cross-modal hash model. Furthermore, in the process of using hash codes to calculate the similarity between samples, the computing resources occupied are saved and the computing efficiency is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of an application scenario of the training method of the cross-modal hash model provided in an embodiment of the present application;

[0030] Figure 2A It is a structural schematic diagram of an electronic device for training a cross-modal hash model provided in an embodiment of the present application;

[0031] Figure 2B It is a structural schematic diagram of a training device for a cross-modal hash model provided in an embodiment of the present application;

[0032] Figure 3A It is a flowchart of a training method for a cross-modal hash model provided in an embodiment of the present application;

[0033] Figure 3B is a relationship diagram between sample pairs and feature data point pairs provided in an embodiment of the present application;

[0034] Figure 3C is a relationship diagram between feature data point pairs and hash code pairs provided in an embodiment of the present application;

[0035] Figure 3D It is a relationship diagram between characteristic data point pairs, hash code pairs and binary codes provided in the embodiment of the present application;

[0036] Figure 4 It is a flowchart of a training method for a cross-modal hash model provided in an embodiment of the present application;

[0037] Figure 5A It is a flowchart of a training method for a cross-modal hash model provided in an embodiment of the present application;

[0038] Figure 5B It is a flowchart of a training method for a cross-modal hash model provided in an embodiment of the present application;

[0039] Figure 6 It is a flowchart of the encoding method of the cross-modal hash model provided in an embodiment of the present application;

[0040] Fig. 7A It is a structural diagram of a cross-modal hash model provided in an embodiment of the present application;

[0041] Figure 7B It is a structural diagram of a cross-modal hash model provided in an embodiment of the present application;

[0042] Figure 8 It is an optional flowchart of the training method of the cross-modal hash model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0044] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0045] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0047] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0048] 1) Modality: Modality refers to the form in which data exists, such as text, audio, image, video and other file formats. In some cases, data exists in different forms, but describes the same thing or event. For example, an image and the text describing the image content form a cross-modality, but the things they describe are the same. Another example is a call recording and the text corresponding to the call recording, both describe the actual content of the call. Cross-modal retrieval is used to implement retrieval between data of different modalities.

[0049] 2) Hash algorithm, also known as hash algorithm, after the sample is processed by the hash algorithm, a fixed-length hash code is output, and the dimension of the sample is higher than the dimension of the hash code. The process of hash algorithm processing is irreversible, and the sample cannot be inferred from the hash code. Although the hash algorithm is called an algorithm, it is actually more like an idea. There is no fixed formula for the hash algorithm. As long as the algorithm conforms to the hash idea, it can be called a hash algorithm. For example: converting a sample (such as a picture) into a hash code through a model conforms to the hash idea and can be used as a hash algorithm.

[0050] 3) Convolutional Neural Networks (CNN): It is a type of feed forward neural network (FNN) that includes convolution calculations and has a deep structure. It is one of the representative algorithms of deep learning. Convolutional neural networks have representation learning capabilities and can perform shift-invariant classification on input images according to their hierarchical structure.

[0051] 4) The fully connected layer plays the role of a "classifier" in the entire convolutional neural network and can map distributed features to the sample label space.

[0052] 5) Hash layer, used to map the features of the sample into hash codes.

[0053] 6) Hamming similarity refers to the use of Hamming distance to measure the similarity between hash codes. The Hamming distance is the number of characters that need to be replaced to transform one string into another. For example, the Hamming distance between 20210301 and 20220512 is 4.

[0054] 7) Cross-modal hashing technology maps high-dimensional data of different modalities into a unified low-dimensional Hamming space (the vector distance in the Hamming space is defined by the Hamming weight of the vector difference, and the Hamming weight is the number of non-zero elements in the string), that is, encoding data of different modalities into low-dimensional compact binary hash codes of uniform length.

[0055] For example: given two modes x and y, the i-th data of mode x is represented as a d-dimensional feature vector x i ∈R 1×d , the i-th data of the y mode is represented as the e-dimensional feature vector y i ∈R 1×e , the hash algorithm aims to map and Encode the data of the two modes into r-dimensional binary hash codes respectively and Where r<<d. After hash coding, the distance between samples of different modes can be approximated by the Hamming distance between their corresponding hash codes, and the calculation of the Hamming distance (the number of characters required to replace a string to transform it into another string, such as the Hamming distance between 1001 and 0000 is 2) can be supported by the computer's bitwise exclusive OR operation (XOR, binary algorithm), thereby accelerating the distance measurement.

[0056] 8) Retrieval accuracy: Based on the query sample, multiple returned samples are retrieved. The returned samples include samples related to the query sample and samples not related to the query sample. The retrieval accuracy is the ratio of relevant samples to all returned samples.

[0057] In the field of cross-modal hashing research, a quantization loss function is usually used to obtain the quantization error in the process of converting samples into hash codes by a hash model. The relevant technology has proposed a variety of loss functions to better control the quantization error. However, the relevant technology has not yet proposed a more accurate and effective quantization loss determination scheme, which makes the hash code output by the trained cross-modal hash model not very accurate, that is, the distance between hash codes represents the similarity between samples in a low accuracy. As a result, calculating the similarity between samples based on hash codes requires more computing resources, which reduces computing efficiency and affects the retrieval accuracy of cross-modal retrieval using hash codes.

[0058] The embodiments of the present application provide a training method for a cross-modal hash model, an encoding method for a cross-modal hash model, a training device for a cross-modal hash model, an electronic device for training a cross-modal hash model, and a computer-readable storage medium, which can improve the encoding accuracy of the cross-modal hash model, thereby saving computing resources for calculating the similarity between samples based on hash coding and improving computing efficiency. The following is an example of an electronic device as a server (hereinafter referred to as a training server) for explanation.

[0059] See also Figure 1 , Figure 1 It is a schematic diagram of an application scenario of the training method of the cross-modal hash model provided in an embodiment of the present application. Figure 1 The present invention demonstrates the application of the training method of the cross-modal hash model in the scenario of cross-modal item recommendation. Figure 1 The system includes: a training server 401, a recommendation system server 402, a network 300 and a terminal device 200. The training server 401 and the recommendation system server 402 communicate with each other through the network 300, or communicate with each other through other means. The terminal device 200 is connected to the recommendation system server 402 through the network 300, and the network 300 can be a wide area network or a local area network, or a combination of the two.

[0060] The training server 401 is used to execute the training method of the cross-modal hash model of the embodiment of the present application to obtain a trained cross-modal hash model. The training server 401 is also used to synchronize the cross-modal hash model to the recommendation system server 402 through the network 300, or synchronize the cross-modal hash model to the recommendation system server 402 through data transmission between servers. The recommendation system server 402 receives and stores the trained cross-modal hash model. Based on the cross-modal hash model, the recommendation system server 402 can convert the data of multiple items to be recommended in the database (the database contains multiple items to be recommended, and the items to be recommended can be real items or virtual items, for example: real items such as daily necessities, food and electronic devices; virtual items such as game props, electronic books, online learning courses, online consulting services, etc.) into corresponding hash codes. The recommendation system server 402 can also call the cross-modal hash model to convert the feature data corresponding to the user into the corresponding hash code. The user uses the terminal device 200 to send a recommendation request to the recommendation system server 402 through the network 300; the recommendation system server 402 responds to the recommendation request, obtains the information of the recommended item, and sends the information of the recommended item to the terminal device 200, and the user receives the information of the recommended item.

[0061] For example, the recommendation system server 402 can obtain the information of the recommended items in the following way: the user's feature data set and the identifier corresponding to the feature data set (for example, user ID) are pre-stored in the recommendation system server 402, and the feature data set includes data such as user portrait, user account information, and historical browsing records. The feature data set can reflect multiple aspects of interest to the user. The recommendation system server 402 is also used to call the cross-modal hash model to perform dimensionality reduction hash coding processing on the feature data set, obtain the hash code corresponding to each feature data in the feature data set, and store the hash code. The terminal device 200 generates a recommendation request in response to the login operation of the application program or the trigger operation of the page by the user to be recommended, wherein the recommendation request carries the identifier of the user to be recommended, and sends the recommendation request to the recommendation system server 402 through the network 300. The recommendation system server 402 retrieves the hash code of the user to be recommended from the database based on the identifier, and calculates the similarity between the hash code of the user to be recommended and the hash code of multiple items stored in the database. The similarity is represented by the distance between the hash codes, and the closer the distance between the hash codes, the higher the similarity, and the distance between the hash codes is negatively correlated with the similarity. The distance between hash codes can be obtained by a bitwise XOR operation (XOR, binary algorithm, bitwise XOR is also called bitwise XOR operation) of a computer. After obtaining the similarity represented by the distance between hash codes, multiple items with the highest similarity in descending order are selected as the recommended items to be recalled, and the information of the recommended items (such as advertisements displayed in the form of text, pictures, videos, etc.) is sent to the terminal device 200 through the network 300.

[0062] The following is an example of advertisement. For example, the recommendation system server 402 performs dimensionality reduction hash coding on the feature data of each item in the database in advance, obtains the hash code of each item, and stores it in the database; when receiving a recommendation request, the recommendation system server 402 searches the database based on the identifier of the user to be recommended carried in the recommendation request, and obtains the hash code pre-calculated based on the features of the user to be recommended. The distance between the hash code of the item and the hash code of each item is calculated by a bitwise XOR operation, and the similarity between the user's features and different items is represented by the distance between the hash codes. The closer the distance between the hash codes, the higher the similarity. The recommendation system server 402 sorts all similarities in descending order, and takes the advertisements of multiple items with the highest similarity in descending order as advertisements that the user may be interested in. The recommendation system server 402 sends the advertisements to the terminal device 200 through the network 300, and the terminal device 200 displays the advertisements to the user. The user can browse any advertisement on the terminal device 200 according to the needs, or block any advertisement. The terminal device 200 generates corresponding feedback data based on the user's browsing or blocking operation and submits it to the recommendation system server 402. The recommendation system server 402 can improve the user-related data according to the feedback data, and update the stored user portrait, user account information, historical browsing records and other data, so as to provide users with more accurate recommendation services. The similarity between cross-modal data is calculated by hash coding, which improves the speed of calculating similarity, so that the recommendation system server 402 can quickly respond to recommendation requests, and avoid the phenomenon of blank advertising space caused by low efficiency of recommendation information when users receive advertisements.

[0063] In an embodiment of the present application, a cross-modal hash model is trained so that the distance between hash codes output by the cross-modal hash model can more accurately represent the similarity between samples, and the trained cross-modal hash model is used to perform dimensionality reduction hash coding, and the hash code has a lower dimension than the original data of the sample; the similarity between the corresponding cross-modal samples is calculated based on the hash code, which can save the computing resources required for the recommendation system server to calculate the sample similarity and improve the calculation speed.

[0064] In some embodiments, the server 401 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal device 200 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a car terminal, etc., but is not limited thereto. The terminal device 200 and the server 401 may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present invention.

[0065] In some embodiments, the multiple sample pairs used in the process of training the cross-modal hash model may be sent by the terminal device 200 to the server 401 through the network 300, or may be already stored in the server 401. The server 401 calls the cross-modal hash model to perform dimensionality reduction hash coding processing on the sample pairs, and obtains the quantization loss based on the hash coding, and trains the cross-modal hash model by back propagating the quantization loss. The server 401 can also perform hash coding processing based on the trained cross-modal hash model, establish a mapping relationship between the sample and the hash coding, and use the mapping relationship for cross-modal retrieval.

[0066] The embodiments of the present application can also be implemented through machine learning. Machine Learning (ML) is a multi-disciplinary cross-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching.

[0067] The embodiments of the present application can also be implemented through cloud technology. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model application. It can form a resource pool, which is used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, as well as the promotion of search services (such as cross-modal retrieval services), social networks, mobile commerce and open collaboration, each item may have its own hash code identification mark in the future, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately, and all kinds of industry data require strong system backing support, which can only be achieved through cloud computing.

[0068] In some embodiments, multiple servers may be combined into a blockchain, and server 401 is a node on the blockchain. An information connection may exist between each node in the blockchain, and information may be transmitted between nodes through the above information connection.

[0069] Among them, blockchain is a new application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods, each of which contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer. Among them, the data related to the training method of the cross-modal hash model provided in the embodiment of the present application (for example: the algorithm for obtaining the binary code corresponding to the continuous hash code, the hash code corresponding to the different modal samples obtained based on the cross-modal hash model, etc.) can be stored on the blockchain.

[0070] For example, the electronic device is a terminal device, see Figure 2A , Figure 2A It is a structural diagram of an electronic device for training a cross-modal hash model provided by an embodiment of the present application. The electronic device 400 shown in Figure 2 includes: at least one processor 410, a memory 450, and at least one network interface 420. The various components in the electronic device 400 are coupled together through a bus system 440. It can be understood that the bus system 440 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, various buses are labeled as bus system 440 in Figure 2.

[0071] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0072] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0073] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0074] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0075] The operating system 451 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic businesses and process hardware-based tasks.

[0076] The network communication module 452 is used to reach other computing devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include: Bluetooth, Wireless Authentication (WiFi), and Universal Serial Bus (USB).

[0077] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. FIG. 2 shows a training device 455 of a cross-modal hash model stored in a memory 450, which can be software in the form of a program and a plug-in, including the following software modules: a sample acquisition module 4551, a hash encoding module 4552, a binary code generation module 4553, and a loss determination module 4554. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0078] In other embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the training method and encoding method of the cross-modal hash model provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic components.

[0079] refer to Figure 2B , Figure 2B It is a structural diagram of a training device for a cross-modal hash model provided in an embodiment of the present application.

[0080] The training device 455 of the cross-modal hash model includes: a sample acquisition module 4551, a hash coding module 4552 (ie, a cross-modal hash model), a binary code generation module 4553 and a loss determination module 4554.

[0081] The cross-modal hash model includes a first modality hash model 4552A and a second modality hash model 4552B; the first modality and the second modality belong to different types of modalities, such as image and text, text and audio, audio and image, etc. A sample pair includes a sample of the first modality and a sample of the second modality, and the two samples describe the same content.

[0082] The first modality hash model 4552A includes a first extraction layer 521, a first fully connected layer 522 and a first hash layer 523; the first extraction layer 521 is used to receive samples of the first modality from multiple sample pairs, and perform feature extraction on each sample of the first modality, and output the extracted sample feature set of the first modality (the sample feature set contains multiple feature data points, and each feature data point is used to characterize all feature data corresponding to a sample) to the first fully connected layer 522; the first fully connected layer 522 is used to classify the feature data contained in each feature data point in the sample feature set of the first modality, and output the classification result to the first hash layer 523; the first hash layer 523 is used to perform dimensionality reduction hash coding processing according to the classification result to obtain a one-to-one hash code for the samples of the first modality.

[0083] The second modality hash model 4552B includes a second extraction layer 524, a second fully connected layer 525 and a second hash layer 526; the second extraction layer 524 is used to receive samples of the second modality from multiple sample pairs, and perform feature extraction on each sample of the second modality, and output the extracted sample feature set of the second modality to the second fully connected layer 525; the second fully connected layer 525 is used to classify the feature data contained in each feature data point in the sample feature set of the second modality, and output the classification result to the second hash layer 526; the second hash layer 526 is used to perform dimensionality reduction hash coding processing according to the classification result to obtain the hash code corresponding to the second modality sample.

[0084] The binary code generation module 4553 receives hash codes of different modes, obtains the hash data point at each position in each hash code pair, and compares the weights corresponding to the values ​​at the same position (the weight is used to calculate the Hamming similarity between two hash codes. The weight represents the importance of the hash data point in the process of calculating the Hamming similarity, and is positively correlated with the importance. The larger the weight, the higher the importance). The hash data points with larger weights are binarized, and the binarized data points at each position are used to form the binary code of each hash code pair (that is, the binary code of the sample pair corresponding to each hash code pair).

[0085] The quantization loss module 4554 is used to determine the quantization loss of the cross-modal hash model based on the binary code, each hash code, and the feature similarity between different feature data points of different modalities. The quantization loss module 4554 can also back-propagate the obtained quantization loss to the cross-modal hash model, and the parameters of the cross-modal hash model can be updated based on the quantization loss.

[0086] See also Figure 3A , Figure 3A This is a flow chart of the training method of the cross-modal hash model provided in the embodiment of the present application, with the terminal device as the execution subject, and will be combined with Figure 3A The steps shown are explained.

[0087] In step 101, a plurality of sample pairs are obtained.

[0088] Here, each sample pair includes two samples of different modalities for describing the same content; the different modalities include a first modality and a second modality; and each sample pair includes: a sample of the first modality and a sample of the second modality.

[0089] In some embodiments, the first modality is any modality in the modality set, and the second modality is any modality in the modality set except the first modality. The modality set may include: picture modality, text modality, video modality and audio modality. The modality set may also include more different modalities, which is not limited by the embodiments of the present application. Among multiple sample pairs, the samples of the first modality may constitute a sample set of the first modality, and the samples of the second modality may constitute a sample set of the second modality.

[0090] For example, explanations are given for describing the same content. For example, a sample pair consists of a picture and a text. The picture (i.e., the description) is a red umbrella, and the text description is "red umbrella". The two describe the same content. The content described in different sample pairs can be the same or different. For example, among multiple sample pairs, the first sample pair describes white shoes, the second sample pair can also describe white shoes, and the third sample pair describes black shoes.

[0091] In step 102, a cross-modal hash model is called to perform dimensionality reduction hash coding processing on multiple sample pairs to obtain multiple hash code pairs corresponding one-to-one to the multiple sample pairs.

[0092] Here, the distance between two hash codes in a hash code pair is used to characterize the similarity between the two samples corresponding to the hash code pair. The distance between hash codes is negatively correlated with the similarity between the two samples. The closer the distance between hash codes, the higher the similarity between the two samples.

[0093] For example, the levels included in the cross-modal hash model can be referred to Figure 2B In the hash coding module 4552, the dimension reduction hash coding process is to convert the high-dimensional data corresponding to each sample in the sample pair into a low-dimensional hash code, and the cross-modal hash model can realize the function of dimension reduction hash coding process. The similarity between samples represented by the distance between hash codes can be Hamming similarity.

[0094] In some embodiments, the multiple hash code pairs corresponding to the multiple sample pairs include: multiple hash codes corresponding to the multiple samples of the first modality, and multiple hash codes corresponding to the multiple samples of the second modality. Step 101 can be implemented in the following manner: extracting features from the multiple samples of the first modality in the multiple sample pairs to obtain a sample feature set of the first modality, and extracting features from the multiple samples of the second modality in the multiple sample pairs to obtain a sample feature set of the second modality. Performing a dimensionality reduction hash coding process on the sample feature set of the first modality to obtain multiple hash codes corresponding to the multiple samples of the first modality, and performing a dimensionality reduction hash coding process on the sample feature set of the second modality to obtain multiple hash codes corresponding to the multiple samples of the second modality.

[0095] Here, the length of the hash code of the plurality of samples of the first modality is the same as the length of the hash code of the plurality of samples of the second modality. Figure 3B , Figure 3B is a relationship diagram between sample pairs and feature data point pairs provided in an embodiment of the present application. In the embodiment of the present application, the picture mode is used as the first mode and the text mode is used as the second mode for explanation. Figure 3B Used to display the relationship between samples, sample sets, sample feature sets and feature data points of two modalities.

[0096] The sample set 301B of the picture modality includes N sample pictures (sample picture 1, sample picture 2...sample picture N), and the sample set 302B of the text modality includes N segments of sample text (sample text 1, sample text 2...sample text N). Multiple sample texts correspond one-to-one to multiple sample pictures, forming multiple sample pairs (sample pair 1, sample pair 2...sample pair N).

[0097] Feature extraction is performed on multiple sample images in the sample set 301B of the image modality to obtain a sample feature set X of the image modality. The sample feature set X of the image modality includes multiple feature data points (feature data point x1, feature data point x2, ... feature data point x3). N ), a feature data point corresponds to all feature data of a sample image. Sample image i in the sample feature set X of the image modality corresponds to feature data point x i (1≤i≤N).

[0098] Feature extraction is performed on multiple sample texts in the sample set 302B of the text modality to obtain a sample feature set Y of the text modality. The sample feature set Y of the text modality includes multiple feature data points (feature data point y1, feature data point y2, ..., feature data point y N), a feature data point corresponds to all feature data of a sample text. In the sample feature set Y of the text modality, the sample text i corresponds to the feature data point y i (1≤i≤N).

[0099] For example, the dimension of the feature data corresponding to the feature data point is higher than the dimension of the hash code, so the storage space required for the hash code is less than the data before the encoding process; in order to facilitate the representation of the similarity between samples by the distance between hash codes, it is necessary to perform corresponding calculations on the hash data points corresponding to each position of the hash code pair, so the encoding length of each hash code is the same; and for feature data, the extraction process of feature data of different modalities varies according to the modality (for example, in the feature extraction process, keyword recognition is performed on samples of the image modality, while pixel-level feature extraction is performed on samples of the image modality), and the data storage space occupied by different samples is also different. The calculation of the similarity between samples of different modalities is not affected by the data length of the feature data, so the data length corresponding to the feature data of the image modality and the data length corresponding to the feature data of the text modality can be the same or different.

[0100] In some embodiments, a dimensionality reduction hash coding process is performed on a sample feature set of a first modality to obtain a plurality of hash codes corresponding one-to-one to a plurality of samples of the first modality. This can be achieved in the following manner: a plurality of feature data points included in the sample feature set of the first modality are respectively subjected to dimensionality reduction hash coding process to obtain a plurality of hash codes corresponding one-to-one to a plurality of samples of the first modality.

[0101] Here, the multiple feature data points correspond one-to-one to the multiple samples of the first modality.

[0102] In some embodiments, a dimensionality reduction hash coding process is performed on the sample feature set of the second modality to obtain multiple hash codes corresponding one-to-one to multiple samples of the second modality. This can be achieved in the following way: a plurality of feature data points included in the sample feature set of the second modality are respectively subjected to dimensionality reduction hash coding process to obtain hash codes corresponding one-to-one to multiple samples of the second modality.

[0103] Here, the multiple feature data points correspond one-to-one to the multiple samples of the second modality.

[0104] As an example, refer to Figure 2B The cross-modal hash model includes hash models of different modalities. Continuing to explain based on the above image modality and text modality as examples, the cross-modal hash model at least includes a hash model corresponding to the image modality and a hash model corresponding to the text modality. The hash model of each modality performs dimensionality reduction hash coding on the sample feature set of each modality to obtain hash codes corresponding to multiple samples of each modality. Figure 3C , Figure 3C is a relationship diagram between feature data point pairs and hash code pairs provided in an embodiment of the present application, Figure 3C It is used to show the relationship between sample feature sets, feature data points, hash code sets, and hash codes of different modalities. Figure 3B and Figure 3C To explain, the hash codes of multiple samples of the image modality (hash code a1, hash code a2... hash code a N ) form a hash code set A, and the i-th hash code in the hash code set A is represented as hash code a i , hash code a i Corresponding to image sample i, and feature data point x of image sample i i . Hash codes of multiple samples of text modality (hash code b1, hash code b2, ... hash code b N ) form a hash code set B, and the i-th hash code in the hash code set B is represented as hash code b i , hash code b i Corresponding to text sample i, and feature data point y of text sample i i . Hash code a i Can be hashed with b i Composed of hash code pairs i (such as Figure 3C In the example, hash code pair 1, hash code pair 2, ..., hash code pair N) forms a mapping relationship between the sample and the corresponding feature data point and the corresponding hash code.

[0105] As an example, the value space corresponding to the hash code obtained by the dimensionality reduction hash coding process can be discrete or continuous. For example: the hash code [1 -1 1 -1 1], the code length is 5, the value range corresponding to each hash data point in the hash code is 1, -1, and the value space is discrete; the hash code [0.52 0.68 -0.72 -0.82 0.95], the code length is 5, the value range corresponding to each hash data point in the hash code is greater than -1 and less than 1, and the value space is continuous.

[0106] Steps 103 and 104 are performed for each hash code pair. In step 103, a target hash data point with a larger weight is determined in the hash data point pair at each position in the hash code pair.

[0107] Here, the weight is used to characterize the importance of the hash data point for calculating the similarity of the sample pair corresponding to the hash code pair. In the embodiment of the present application, the value space corresponding to the hash code is a continuous value space as an example for explanation; the different modalities include a first modality and a second modality, each hash code pair includes a hash code of a sample of the first modality and a hash code of a sample of the second modality, and each hash code includes multiple hash data points.

[0108] For easy understanding, refer to Figure 3D , Figure 3D is a relationship diagram between feature data point pairs, hash code pairs and binary codes provided by this application, Figure 3D Used to show the relationship between feature data point pairs, hash code pairs, and binary codes, and Figure 3D The relationship diagram shown applies to each hash code pair. Figure 3D The special data point x1 and the feature data point y1 correspond one-to-one to the two samples in a pair of samples. The feature data corresponding to the feature data point is represented as a rectangle composed of multiple grids, and each grid in the rectangle corresponds to data of one dimension. The dimension of the feature data point x1 is m1, and the dimension of the feature data point y1 is m2. m1 and m2 can be equal or unequal. After the dimensionality reduction hash coding process, a hash code pair corresponding to the feature data point pair is obtained. The hash code is also represented as a rectangle composed of multiple grids, and each grid corresponds to a hash data point, that is, data of one dimension. In the hash code pair, each hash code has a one-to-one correspondence with the hash data points of the same dimension (that is, in the hash code pair, the hash data points at the same position correspond one-to-one). The dimension (that is, the coding length) of the hash code a1 and the hash code b1 is m3. m3 is smaller than m1 and m2.

[0109] refer to Figure 4 , Figure 4 is a flow chart of the training method of the cross-modal hash model provided in the embodiment of the present application; step 103 can be implemented through steps 1031 to 1032, which are described in detail below

[0110] In step 1031, each hash code pair is traversed to obtain a target weight matrix of the traversed hash code pair.

[0111] Here, the target weight matrix is ​​a diagonal matrix, and the number of elements on the diagonal of the diagonal matrix is ​​equal to the sum of the code lengths of the two hash codes in the hash code pair.

[0112] For example, the encoding length of each hash code is the same, and the target weight matrix of any hash code pair can be expressed as Each element λ on the diagonal represents a weight. There are 2R elements on the diagonal, where R is the code length of the hash code and 2R is the sum of the code lengths. In each hash code pair, the weights of the hash data points at each position of the hash code corresponding to the sample of the first modality are λ1 to λ R , the weights of the hash data points at each position of the hash code corresponding to the sample of the second modality are λ R+1 To 2R .

[0113] In some embodiments, step 1031 can be implemented in the following manner: obtaining a weight variable matrix of the traversed hash code pairs; determining the variable Hamming similarity of two samples represented by the distance between the hash code pairs based on the weight variable matrix and the hash code pairs, wherein the variable Hamming similarity is represented in matrix form; determining the variable similarity difference between the feature similarity of two feature data points corresponding to the hash code pairs and the variable Hamming similarity; determining the minimum similarity difference corresponding to the variable similarity difference, and determining the target Hamming similarity when the similarity difference is minimum based on the minimum difference value and the feature similarity corresponding to the two feature data points; determining the target weight matrix of the hash code pair when the similarity difference is minimum based on the target Hamming similarity and the hash code pairs.

[0114] For example, the target weight matrix can be obtained by assuming a diagonal matrix, that is, setting a weight variable matrix, and deriving the target weight matrix through the weight variable matrix. The weight variable matrix is ​​set to a matrix with the same format as the target weight matrix, the weight variable matrix is ​​a diagonal matrix, and the number of elements on the diagonal of the diagonal matrix is ​​equal to the sum of the code lengths of the two hash codes in the hash code pair, and the elements on the diagonal are variables.

[0115] For example, the weight for calculating the Hamming similarity can be derived based on the relationship between the feature similarity between the feature data points of two samples in the sample pair and the Hamming similarity. Based on the weight variable matrix and the hash code pair, the variable Hamming similarity represented by the product between the matrices can be obtained.

[0116] In some embodiments, based on the weight variable matrix and the hash code pair, the variable Hamming similarity of two samples represented by the distance between the hash code pairs is determined, including: generating a coding matrix based on the hash code pair, and determining the transposed matrix of the coding matrix; determining the product between the coding matrix, the weight variable matrix and the transposed matrix of the coding matrix; dividing the product by the sum of the coding lengths of the two hash codes in the hash code pair to obtain the variable Hamming similarity.

[0117] For example, if the two hash codes in the hash code pair are [0.86 -0.77 0.93] and [-0.67 -0.97 0.83], the encoding matrix can be expressed as The transpose of the encoding matrix can be expressed as The code length R is 3, and the sum of the code lengths 2R is 6. The variable Hamming similarity can be expressed as the following formula (1):

[0118]

[0119] Among them, λ1 to λ6 are variables used to characterize weights. The feature similarity between the two feature data points corresponding to the sample pair can be expressed as S. The difference between the feature similarity S and the variable Hamming similarity can be determined by obtaining the difference between the two or the F-norm of the difference. In the embodiment of the present application, the F-norm of the difference between the two is used as an example for explanation. The F-norm of the difference between the two can be expressed as the following formula (2):

[0120]

[0121] Set the difference between the two to a minimum value, which can be 0 or a value close to 0.

[0122] For example, the weight variable matrix can be calculated by obtaining the closed-form solution formula of the weight variable matrix. Based on formula (2), the closed-form solution formula of the weight matrix Λ can be derived, and the closed-form solution formula can be expressed as the following formula (3):

[0123]

[0124] Among them, st represents middle Satisfaction Constraints. ⊙ represents the Hadamard Product (basic product) operation of the matrix; g represents the matrix Z T The vector composed of the diagonal elements of SZ. Substituting the hash code and the corresponding feature similarity in the hash code pair into the above formula (3) can obtain the target weight matrix.

[0125] Step 1032 and step 1033 are executed for each position in the traversed hash code pair.

[0126] In step 1032, based on the target weight matrix of the traversed hash code pairs, the weight of the hash data point located at position in the hash code of the sample of the first modality and the weight of the hash data point located at position in the hash code of the sample of the second modality are determined.

[0127] In some embodiments, step 1032 can be implemented by: determining the first column number at position in the hash code pair, and taking the sum of the first column number and the code length as the second column number; taking the elements located on the diagonal of the target weight matrix at the first column number as the weights of the hash data points of the samples of the first modality at position; taking the elements located on the diagonal of the target weight matrix at the second column number as the weights of the hash data points of the samples of the second modality at position.

[0128] For example, the 1st to Rth elements on the diagonal of the target weight variable matrix correspond to the weights of the hash data points at each position in the hash code of the samples of the first modality, and the R+1st to 2Rth elements on the diagonal correspond to the weights of the hash data points at each position in the hash code of the samples of the second modality. That is, if the code length is R, the number of elements on the diagonal is 2R, and the number of columns of the position in the hash code is M, then the weight of the first modality is located in the Mth column in the matrix, and the weight of the second modality is located in the R+Mth column in the matrix.

[0129] In step 1033, the hash data point with a larger weight is determined as the target hash data point corresponding to the position.

[0130] For example, we continue to use the two hash codes [0.86 -0.77 0.93] and [-0.67 -0.97 0.83] in the hash code pair as an example. The hash data points at the first position in the hash code pair are 0.86 and -0.67. The weight corresponding to 0.86 is λ1, and the weight corresponding to -0.67 is λ4. Compare the sizes of λ1 and λ4. If λ1 is larger, 0.86 is used as the target hash data point. If λ4 is larger, -0.67 is used as the target hash data point. Perform the above operations for each hash data point pair in the hash code pair to obtain the target hash data point corresponding to each position.

[0131] In an embodiment of the present application, the importance of hash data points is reflected by weights, and the importance of hash data points is digitized, which is conducive to obtaining more accurate binary codes and quantization losses, thereby improving the output results of the cross-modal hash model and the input results to fit more accurately, so that the hash code output by the trained cross-modal hash model is more accurate, that is, the distance between hash codes can more accurately reflect the similarity between samples.

[0132] In step 104, a binary code of the hash coding pair is determined based on the target hash data point corresponding to each position.

[0133] For example, continue to refer to Figure 3D, by comparing the weights of the hash data points corresponding to each position in the hash code pair, the hash data point with a larger weight is obtained as the target hash data point. The number of hash data point pairs in the hash code pair is m3, so m3 target hash data points can be obtained, and each target hash data point is binarized to obtain a binary data point, and based on the position of the hash data point pair corresponding to the target hash data point, the binary data points are combined in sequence to obtain a binary code 1. The dimension of the binary code 1 is also m3, that is, the encoding length of the binary code of the hash code pair is the same as the encoding length of each hash code in the hash code pair.

[0134] In some embodiments, the target hash data point corresponding to each position is binarized, and the obtained binarized data point at each position is combined into a binary code of a hash code pair. Figure 5A , Figure 5A It is a flow chart of the training method of the cross-modal hash model provided in an embodiment of the present application; step 104 can be implemented through steps 1041 to 1043, which are described in detail below.

[0135] In some embodiments, the value type of the binary hash data point includes a first value and a second value, the first value is greater than the second value; the binary code is a code represented by two different numbers, for example, the first value can be 1 and the second value can be 0. The first value can be 1 and the second value can be -1.

[0136] In step 1041, when the value of the target hash data point at the position is greater than zero, the value corresponding to the target hash data point is binarized to obtain a binarized data point of a first value.

[0137] In step 1042, when the value of the target hash data point at the position is less than zero, the value corresponding to the target hash data point is binarized to obtain a binarized data point of a second value.

[0138] For example, in the embodiment of the present application, the first numerical value is 1 and the second numerical value is -1. Assuming that the encoding length of the hash code pair is 3, the target hash data points obtained are h1, h2, and h3 respectively; h1 and h2 are less than 0, and h3 is greater than 0, then the numerical values ​​corresponding to the binary code data points are -1, -1, and 1 respectively.

[0139] In step 1043, the obtained binary data points at each position are combined into a binary code of a hash code pair.

[0140] For example, the binary data points at each position are combined one by one according to the order of each position in the hash code, and a binary code corresponding to a hash code pair can be obtained. Steps 1031 to 1033 and steps 1041 to 1043 are performed on all hash code pairs to obtain the corresponding binary codes of all hash code pairs.

[0141] In an embodiment of the present application, by binarizing hash data points with larger weights, a binary code with higher precision is obtained. Based on the binary code with higher precision, a quantization loss with higher accuracy can be obtained, which is conducive to better training of the cross-modal hash model.

[0142] In some embodiments, the sample feature set of the first modality includes multiple feature data points, and the sample feature set of the second modality includes multiple feature data points. The feature data points include feature data of the corresponding samples. Before step 105, the similarity matrix corresponding to the multiple sample pairs can be obtained in the following manner: for each feature data point in the sample feature set of the first modality, the following processing is performed: determine the position number of the feature data point of the first modality in the sample feature set, and determine the row corresponding to the position number; according to the position number of each feature data point of the second modality in the sample feature set, obtain the feature similarity of the feature data point of the first modality and each feature data point of the second modality in turn; take each obtained feature similarity as an element in the row in turn, and combine the elements of each row obtained into a similarity matrix.

[0143] For example, the sample feature set of the image modality is represented as X, and the feature data points corresponding to the i images in the sample feature set of the image modality are represented as x i (1≤i≤N). The sample feature set of the text modality is represented as Y, and the feature data point corresponding to the i-th text in the sample feature set of the text modality is represented as y i (1≤i≤N).

[0144] For the feature data point x i , whose position number is i, get x in sequence i and feature data points y1 to y N The feature similarity S between i1 To S iN , and the feature similarity S is calculated in the order of position numbers. i1 To S iN As the element of the i-th row in the similarity matrix S. Perform the above operation on each feature data point in the sample feature set of the image modality to obtain the value of the feature similarity of each row, and combine the feature similarities of each row in the order of position numbers to obtain the feature similarity matrix S.

[0145] Then the similarity matrix S can be expressed as Each element in the similarity matrix can be expressed as S ij , S ij Represents the i-th feature data point xi in the sample feature set of the image modality and the j-th feature data point y in the sample feature set of the text modality j The feature similarity between .

[0146] For example, in the similarity matrix S, since each pair of samples includes two samples with the same description content, S ij =S ji ; When i = j, S ij Located on the diagonal of the similarity matrix S, S ij Represents the feature similarity of feature data points of a pair of samples.

[0147] In the embodiment of the present application, by obtaining the similarity matrix corresponding to multiple sample pairs, factors such as the similarity between two samples in a sample pair and the similarity between samples in different sample pairs are taken into consideration, so as to facilitate obtaining a more accurate quantization loss.

[0148] Continue to refer Figure 3A In step 105, the total quantization loss of the cross-modal hash model is determined based on each hash code pair, the similarity matrix corresponding to the multiple sample pairs, and the difference between each hash code pair and the binary code corresponding to each hash code pair.

[0149] For example, based on the similarity matrix corresponding to each hash code pair and multiple sample pairs, the quantization loss of converting high-dimensional data to low-dimensional data in the process of converting sample pairs to hash code pairs, as well as the quantization loss of similarity between samples of different modalities (similarity is similarity expressed by numerical values, that is, converting abstract similarity into specific numerical values) can be obtained. The binary code can represent the distance required to approximate between two hash codes in a hash code pair. The cross-modal hash model performs dimensionality reduction hash coding on a sample pair with the same description content. Ideally, two identical hash codes are obtained. The binary code has no difference with each hash code, so that the hash code can be used as the index of the sample and cross-modal retrieval can be performed based on the index of the sample. Based on the difference between each hash code pair and the binary code corresponding to each hash code pair, the difference between the ideal hash code and the actual hash code can be determined.

[0150] In some embodiments, the total quantization loss may be obtained by a quantization loss function, and the formula (4) corresponding to the quantization loss function is as follows:

[0151]

[0152] Among them, S is the matrix composed of feature similarities, ‖·‖2 represents the l2 norm of the vector, ‖·‖ F represents the F norm of the matrix, Z x is a continuous hash code set corresponding to the x modality (in the embodiment of the present application, the x modality is the picture modality, and the continuous hash code set of the picture modality can be expressed as in represents one of the consecutive hash codes), Z y is the continuous hash code set corresponding to the y mode (in the embodiment of the present application, the y mode is the text mode, and the continuous hash code set of the text mode can be expressed as in, represents one of the consecutive hash codes). is the jth consecutive hash code corresponding to the x modality, is the transposed continuous hash code of the ith continuous hash code corresponding to the modality of y.

[0153] In some embodiments, the different modalities include a first modality and a second modality, each hash code pair includes a hash code of a sample of the first modality and a hash code of a sample of the second modality, and each hash code includes a plurality of hash data points; Figure 5B FIG. 5B is a flow chart of the training method of the cross-modal hash model provided in an embodiment of the present application; step 105 can be implemented through steps 1051 to 1054, which are described in detail below.

[0154] The following steps 1051 to 1054 are performed for any two hash code pairs:

[0155] In step 1051, the hash code of the sample of the second modality in the first hash code pair is transposed to obtain a transposed hash code.

[0156] For example, the first hash code pair can be any hash code pair. For example, if the hash code of the sample of the second modality obtained in any hash code pair is [0.86 -0.77 0.93], then the transposed hash code is

[0157] In step 1052, the hash code of the sample of the first modality in the second hash code pair is multiplied by the transposed hash code to obtain a hash code product.

[0158] For example, the second hash code pair can be any hash code pair, that is, the second hash code pair can be the first hash code pair, or a hash code pair other than the first hash code pair. Assuming that the hash code of the sample of the first modality is [d1 d2 d3], and the hash code of the sample of the second modality is [e1 e2 e3], the hash code of the sample of the first modality is multiplied by the transposed hash code, and the hash code product obtained is (d1*e1+d2*e2+d3*e3).

[0159] In step 1053, the feature similarity between the one-to-one feature data points of the two samples corresponding to the hash code product is obtained from the similarity matrix.

[0160] For example, the similarity matrix is ​​expressed as Each element in the similarity matrix can be expressed as S ij , taking the first mode as the picture mode and the second mode as the text mode as an example, S ij Represents the i-th feature data point xi in the sample feature set of the image modality and the j-th feature data point y in the sample feature set of the text modality j The feature similarity between .

[0161] In step 1054, a similarity loss is determined based on the difference between the hash code product and the corresponding feature similarity.

[0162] For example, the hash code product is obtained based on any two hash code pairs, and the similarity difference corresponding to any two hash code pairs can be obtained by the formula (4) express.

[0163] The following steps 1055 are performed for each hash code pair: In step 1055, the difference loss of each hash code pair is determined based on the difference between the hash code of the first modality and the corresponding binary code, and the difference between the hash code of the second modality and the corresponding binary code.

[0164] For example, each hash code pair refers to a hash code pair corresponding to a sample pair, and is expressed by part, the sum of each difference loss can be obtained.

[0165] In step 1056, each similarity loss and each difference loss are added together to obtain the total quantization loss of the cross-modal hashing model.

[0166] For example, based on formula (4), each similarity loss and each difference loss are obtained, and each similarity loss and each difference loss are added together to obtain the total quantization loss of the cross-modal hashing model.

[0167] In the embodiment of the present application, the quantization loss of the cross-modal hash model in the hash coding process is measured from multiple aspects such as the similarity between samples and the similarity of two hash codes in a hash code pair, which improves the accuracy of the quantization loss and is conducive to more accurate training of the cross-modal hash model. The trained cross-modal hash model can be used to perform hash coding on samples of different modalities to obtain more accurate hash codes, that is, the distance between hash codes can more accurately represent the similarity between samples. Thus, the computing resources required to calculate the similarity between samples through the distance between hash codes are saved, and the computing speed is improved; the search results obtained by cross-modal search based on hash codes are more accurate.

[0168] Continue to refer Figure 3A , in step 106, the parameters of the cross-modal hashing model are updated based on the total quantization loss.

[0169] In some embodiments, step 106 can be implemented in the following manner: determining the product of the total quantization loss and the learning rate corresponding to the cross-modal hash model, and taking the product as the target loss; taking each hash code pair as the actual output result, and determining the expected output result of the cross-modal hash model based on the target loss and the actual output result; back-propagating the cross-modal hash model based on the expected output result and multiple sample pairs, and determining the update parameters of the cross-modal hash model during the back-propagation process, wherein the update parameters are used to fit the expected output result and multiple sample pairs; replacing the original parameters of the cross-modal hash model with the update parameters to obtain an updated cross-modal hash model.

[0170] For example, the original parameters can be any parameters in the hash layer, the feature extraction layer or the fully connected layer. Assuming that the quantization loss is L and the learning rate of the cross-modal hash model is η, the product of the two is Lη, and Lη is used as the target loss. Each hash code pair is used as the actual output result of the cross-modal hash model. The actual output result and the target loss are calculated to obtain the expected output result. Back propagation can be understood based on the chain derivation rule in higher mathematics. The expected output result is brought into each level of the cross-modal hash model for reverse derivation in turn, so that the target loss is transmitted from back to front in the cross-modal hash model, and finally the reverse derivation result is obtained. If the reverse derivation result is different from the data corresponding to the multiple sample pairs input into the cross-modal hash model, the parameters in the cross-modal hash model are adjusted so that the reverse derivation result is equal to (or approximate to) the data corresponding to the multiple sample pairs, so that the adjusted parameters can make the expected output result and the multiple sample pairs (the data corresponding to the multiple sample pairs) fit each other. The adjusted parameters are also updated parameters. The updated parameters are used to replace the original parameters in the cross-modal hash model to obtain the updated trained cross-modal hash model.

[0171] In an embodiment of the present application, the cross-modal hash model is updated through back-propagation based on the quantization loss, so that the cross-modal hash model can effectively reduce the quantization loss and be optimized.

[0172] In some embodiments, during the process of updating the cross-modal hash model for multiple iterations, a learning rate decay mechanism can be introduced in the process of back propagating the quantization loss based on the learning rate to reduce the impact of abnormal data on parameter updates. The learning rate decay mechanism is implemented in the following ways: 1. Round decay, when the model is trained (updated) for every N rounds, the learning rate is reduced by a preset value; 2. Exponential slowdown, the learning rate decreases exponentially according to the growth of the number of training times; 3. Fractional slowdown, the formula corresponding to the fractional slowdown method is: lη t =lη0 / (1+kt), k represents the control slowdown amplitude, t represents the number of training rounds, and η represents the learning rate.

[0173] In an embodiment of the present application, a learning rate decay mechanism is introduced in the process of training a cross-modal hash model by back-propagation to prevent the parameters from being updated too quickly and to prevent abnormal data from affecting the parameter update, thereby effectively ensuring the accuracy of the model training.

[0174] In an embodiment of the present application, the importance of each hash data point in the hash code is characterized by weight, and binarization is performed according to the hash data points with higher importance, so that a binary code with higher accuracy can be obtained. The total quantization loss of the cross-modal hash model is determined based on the binary code with higher accuracy, thereby improving the accuracy of the quantization loss. The cross-modal hash model is updated based on the more accurate quantization loss, thereby improving the encoding accuracy of the cross-modal hash model, that is, the distance between the hash codes output by the cross-modal hash model can more accurately reflect the similarity between samples. At the same time, the hash code output based on the cross-modal hash model with higher encoding accuracy can save the amount of resources occupied by the computing device in calculating the similarity between samples based on the hash code, thereby improving the computing speed. The trained cross-modal hash model can be used for hash coding processing. An embodiment of the present application also proposes a coding method for a cross-modal hash model, with reference to Figure 6 , Figure 6 It is a flow chart of the encoding method of the cross-modal hash model provided in the embodiment of the present application; the encoding method is based on the updated cross-modal hash model in the embodiment of the present application.

[0175] In step 601, a pair of samples to be processed is obtained.

[0176] The sample pair includes two samples for describing the same content but in different modalities.

[0177] For example, in the embodiment of the present application, the sample pair to be processed is a sample pair consisting of a picture and a text. For example, the picture is a picture of a certain product, and the text content is the product description information of the certain product.

[0178] In step 602, a cross-modal hash model is called to perform dimensionality reduction hash coding processing on the sample pairs to obtain hash coding pairs that correspond one-to-one to the sample pairs.

[0179] Here, the distance between two hash codes in a hash code pair is used to represent the similarity between two samples corresponding to the hash code pair.

[0180] Here, the cross-modal hash model includes a first modality hash model and a second modality hash model, and different modalities include a first modality and a second modality; a sample pair includes: a sample of the first modality and a sample of the second modality; a one-to-one corresponding hash code pair of the sample pair includes: a hash code of the sample of the first modality, and a hash code of the sample of the second modality.

[0181] For example, we will continue to take the sample pair to be processed as a sample pair consisting of a picture and a text as an example, and continue to refer to Figure 2B For example, pictures and texts are both data samples that can be stored in a memory. The pictures and texts are input into a cross-modal hash model (i.e., a hash coding module 4552). The first extraction layer 521 in the first modal hash model 4552A in the cross-modal hash model extracts features from the data corresponding to the picture, and classifies the extracted features through the first fully connected layer 522 to obtain feature data of the picture. The first hash layer 523 performs a dimensionality reduction hash coding process on the feature data of the picture to obtain a hash code corresponding to the picture. The second extraction layer 524 in the second modal hash model 4552B in the cross-modal hash model extracts features from the data corresponding to the text, and classifies the extracted features through the second fully connected layer 525 to obtain feature data of the text. The second hash layer 526 performs a dimensionality reduction hash coding process on the feature data of the text to obtain a hash code corresponding to the text. The hash code of the picture modality corresponds to the hash code of the text modality.

[0182] In some embodiments, step 602 can be implemented by the following steps: calling the first modality hash model, performing feature extraction on the sample of the first modality in the sample pair to obtain feature data of the first modality, and performing dimensionality reduction hash coding on the feature data of the first modality to obtain a hash code corresponding to the sample of the first modality; calling the second modality hash model, performing feature extraction on the sample of the second modality in the sample pair to obtain feature data of the second modality, and performing dimensionality reduction hash coding on the feature data of the second modality to obtain a hash code corresponding to the sample of the second modality.

[0183] Here, the hash code of the sample of the first modality has the same length as the hash code of the sample of the second modality.

[0184] In some embodiments, the modality may be a video modality, an audio modality, a picture modality, and a text modality. For different modalities, the hash model corresponding to the modality may perform corresponding feature extraction processing.

[0185] For example, when the first modality is an audio modality and the second modality is a text modality, assuming that the audio sample in the sample pair is a speech and the text sample is the text content corresponding to the speech; the first modality hash model can perform speech recognition on the audio data, obtain the text content corresponding to the language, and perform feature extraction on the text content. The second modality hash model performs feature extraction on the text sample and generates a hash code of the corresponding text.

[0186] For example, when the first modality is a video modality and the second modality is a text modality, assuming that the video modality in the sample pair is a video and the text sample is a text describing the content of the video; the first modality hash model can extract the video frame by frame, obtain the feature data corresponding to each frame, perform dimensionality reduction hash coding on the feature data of all frames, and obtain the hash code corresponding to the video. The second modality hash model performs feature extraction on the text sample and generates the hash code of the corresponding text.

[0187] For example, after obtaining the hash code of each sample of different modalities, a mapping relationship between each sample and its hash code can be established, and the samples of different modalities and the mapping relationship can be stored in the database. In the following, the recommendation system server is used as an example for explanation. The items to be recommended in the database can be commodities, and the database contains data such as commodity display pictures, display videos, introduction texts or introduction audios. The recommendation system server converts the data related to multiple commodities of different modalities in the database into multiple hash codes with uniform coding lengths in advance.

[0188] The recommendation system server responds to the recommendation request sent by the user through the terminal device, and the recommendation request carries the user's identification. The recommendation system server retrieves the pre-stored user feature data based on the user's identification. The recommendation system server calls the cross-modal hash coding model to convert the user feature data into a target hash code, calculates the distance between the target hash code and the hash codes of the commodities in different modalities in the database, and uses the distance between the hash codes to represent the similarity between the user feature data and other commodity data in the database. It recalls multiple commodity data sorted in descending order of similarity, and uses the recalled commodity data as the target commodity data (for example: commodity-related videos, texts, and pictures, etc.), and pushes the target commodity data to the user's terminal device, so that the user can receive relevant information about the commodity of interest.

[0189] In an embodiment of the present application, cross-modal hash coding processing is performed based on a trained cross-modal hash model, and hash code pairs with one-to-one correspondence between sample pairs of different modalities can be obtained. There is a mapping relationship between the hash code pairs and the sample pairs. The similarity between cross-modal samples is calculated based on the hash coding, which saves computing resources, improves computing efficiency, and can improve cross-modal retrieval efficiency.

[0190] Below, an exemplary application of the embodiment of the present application in an application scenario of recommending advertisements to users will be described.

[0191] refer to Figure 8 , Figure 8 is an optional flow chart of the training method of the cross-modal hash model provided in the embodiment of the present application. Figure 8 Includes steps 801 to 804.

[0192] In step 801: multiple sample pairs are obtained, and a cross-modal hash model is called to perform dimensionality reduction hash coding processing on the multiple sample pairs to obtain hash coding pairs corresponding to each sample pair one by one.

[0193] This application takes the hash code generated by the cross-modal hash model as a hash code in a continuous value space (i.e., continuous hash code) as an example for explanation, and the continuous hash code is used to represent the hash code below. In some embodiments, the hash code generated by the cross-modal hash model can also be a hash code in a discrete value space.

[0194] The present application embodiment takes the cross-modality between the image mode and the text mode, and the hash code is a continuous hash code in a continuous value space as an example for explanation. For ease of explanation, the following is explained in conjunction with the picture. Fig. 7A , Fig. 7A 7 is a schematic diagram of the structure of a high-quality cross-modal hash network provided in an embodiment of the present application (high quality means higher coding accuracy); the high-quality cross-modal hash network includes a cross-modal hash model 700, a high-quality binary code generator 730, and an objective function optimization module 740. The cross-modal hash model 700 is used to output a picture continuous hash code 715 and a text continuous hash code 725 corresponding to a paired picture sample and a text sample.

[0195] refer to Figure 7B , Figure 7B7 is a schematic diagram of the structure of the cross-modal hash model provided in an embodiment of the present application. The cross-modal hash model 700 includes a picture modality hash model 710 and a text modality hash model 720. The picture modality hash model 710 includes a picture convolution network layer 712, a picture fully connected layer 713 and a picture hash layer 714. The text modality hash model 720 includes a text sample 721, a text feature extraction layer 722, a text fully connected layer 723 and a text hash layer 724.

[0196] For example, a picture sample set is input into a picture modal hash model 710, and a text sample set is input into a text modal hash model 720. Picture sample 711 is an example of a picture sample in the picture sample set input into the picture modal hash model 710. Text sample 721 is an example of a text sample in the text sample set input into the text modal hash model 720. Picture sample 711 shows a picture of "a woman wearing a yellow raincoat is riding on a horse on a bridge in a brown mountain landscape with a few trees...", and the picture of picture sample 711 corresponds to the description of text sample 721 "a woman-wearing a yellow raincoat is riding on a horse on a bridge in a brown mountain landscape with a few trees...". A sample pair is formed between picture sample 711 and text sample 721.

[0197] For example, a picture sample set is input into a picture convolutional network layer 712, which extracts features from each sample in the picture sample set to obtain a picture sample feature set, and outputs the picture sample feature set to a picture fully connected layer 713 (the picture convolutional network layer 712 may include multiple layers), which is used to classify the picture sample feature set and output the classified picture sample feature set to a picture hash layer 714, which converts the classified picture sample feature set into a picture continuous hash code 715.

[0198] For example, the sample feature set is a set of feature data (points) of multiple samples of the same modality, where each feature data point includes feature data of a sample in the modality. The classified image sample feature set can be set as Among them, the feature data point x i Represents the feature data of the i-th image in the image modality.

[0199] For example, a text sample set is input into a text feature extraction layer 722, and the text feature extraction layer 722 extracts keywords from the text sample and converts the keywords into corresponding text feature codes. In the embodiment of the present application, the format of the feature code is a binary code as an example for explanation. For example, the content of the text sample 721 is "a woman-wearing a yellow raincoat is riding on a horse on a bridge in a brown mountain landscape with a few trees", and the keywords extracted by the text feature extraction layer 722 are "woman, horse, bridge, tree, yellow" (cat, bird and other words are keywords corresponding to other samples, which are only examples here), and the text feature extraction layer 722 binarizes the above keywords, and the binary codes obtained are "1, 1, 1, 0, 1" in sequence. The text fully connected layer 723 receives the text feature code, and classifies the text feature code, and outputs the classification result to the text hash layer 724, and the text hash layer 724 determines the text continuous hash code 725 of the text sample according to the classification result.

[0200] For example, the feature set of the classified text samples can be set to Feature data point y i represents the feature data of the i-th text in the text modality, and x i and i The description is the same (assuming x i Corresponding to the image sample 711 of a horse rider, y i It is the text that describes the content of the picture of a person riding a horse, text sample 721).

[0201] For example, based on the sample feature sets of different modalities, a similarity matrix consisting of feature similarities between feature data points of different modalities can also be obtained. The similarity matrix can be expressed as a matrix S = [-1, 1] n×d , in the similarity matrix S ij Represents the feature data point x i and j The feature similarity between them. Since the feature data point x i With the characteristic data point y i The description content is the same, the feature data point x j and feature data point y j The description content is also the same, so the similarity matrix S ij =S ji The elements on the diagonal of the similarity matrix correspond to the feature similarities between the feature data points of each sample pair.

[0202] For example, the contents described by different samples in the same sample set may be the same or different. For example, a sample set of images may contain multiple cat images and multiple dog images. Dog images are different from cat images, and cat images are the same or different from cat images.

[0203] Each feature data point in the sample feature set of the image modality is processed through the hash layer in the image modality hash model to obtain a continuous hash code set of the image modality. in Represents one of the continuous hash codes; each feature data point in the sample feature set of the text modality is processed through the hash layer in the text modality hash model to obtain a continuous hash code set of the text modality in, Represents one of the consecutive hash codes. r is the code length of the hash code, and n is the number of hash codes in the set of consecutive hash codes.

[0204] In step 802: a binary code of each hash code pair is obtained.

[0205] Continue to refer Fig. 7A , Fig. 7A , a group of continuous hash codes of images are shown as (0.86 -0.77...0.93-0.89) and a group of continuous hash codes of texts are shown as (-0.67 -0.97...0.83 -0.75). The continuous hash codes of images 715 and the continuous hash codes of texts 725 are input into the high-quality binary code generator 730, which outputs the corresponding binary code according to the input pairs of continuous hash codes. Fig. 7A A set of binary codes (-1 -1...1-1) is shown as an example.

[0206] Here, the binary code is the distance required to approximate the hash data points at the same position of two consecutive hash codes in each hash code pair.

[0207] For example, the high-quality binary code generator 730 generates a high-quality binary code set based on each consecutive hash code pair. where c i Represents the feature data point x i and i Continuous hash code of and The binary code required to be approximated between them. High quality means higher importance, and the importance can be represented by weight. In the embodiment of the present application, the weight can be the corresponding weight of the hash data point when calculating the sample Hamming similarity. The sample Hamming similarity is equal to the sum of the products of the two hash data points in each position of the same sample pair in the two hash codes and the weight.

[0208] For example, Hamming similarity calculation can be performed based on weights, and the weights used to calculate Hamming similarity can be derived based on the relationship between the feature similarity of the samples and the Hamming similarity. Refer to the following formula (5):

[0209]

[0210] Formula (5) is a minimization formula, where S represents the feature similarity of a pair of feature data points; Z is a matrix of continuous hash code pairs (i.e., encoding matrix). T is the transposed matrix of Z (that is, the transposed matrix of the encoding matrix); the weight matrix Represents a diagonal matrix, and the elements on the diagonal correspond to the variables λ k (1≤k≤r) represents continuous hash coding The weight corresponding to the kth hash data point in ; the variable λ corresponding to the elements on the diagonal k (r+1≤k≤2r) represents continuous hash coding The weight corresponding to the krth hash data point in . The larger the weight of the hash data point, the higher the importance, that is, the higher the quality. The meaning of formula (5) is to obtain the minimum difference value between the feature similarity and the Hamming similarity. The minimum difference value can be set according to the actual situation. For example, the minimum difference value can also be 0. The embodiment of the present application does not limit the value of the minimum difference value.

[0211] Based on formula (5), the following formula (6) can be obtained:

[0212]

[0213] Formula (6) is a closed-form solution formula, where st represents middle Satisfaction Constraints. ⊙ represents the Hadamard Product (basic product) operation of the matrix; g represents the matrix Z T SZ is the vector consisting of the diagonal elements.

[0214] Based on formula (6), we can get the weight matrix. Each element on the diagonal of the weight matrix is ​​the weight corresponding to each hash data point in the continuous hash code pair. The weight of each hash data point can be expressed as a weight vector When When , it means that the quality of the i-th hash data point of the continuous hash code corresponding to the x mode is better than the quality of the i-th hash data point of the continuous hash code corresponding to the y mode, then the i-th hash data point of the continuous hash code corresponding to the x mode is binarized to obtain the i-th binarized data point of the binary code. Otherwise, the i-th hash data point of the y mode hash code is binarized. Each binarized data point of the binary code can be obtained by formula (7), and a complete binary code is formed based on each binarized data point. Formula (7) is as follows:

[0215]

[0216] Among them C *i , Represents the continuous hash code z corresponding to the binary code c and x modality respectively x , the continuous hash code z corresponding to the y mode y The ith column (ith position) element of ; sgn(·) is the element-level threshold function, see Fig. 7A , the binary code can be composed of 1 and -1. When the element is greater than 0, formula (7) returns 1, otherwise it returns -1. Alternatively, the binary code can be composed of 0 and 1. When the element is greater than 0, formula (7) returns 1, otherwise it returns 0.

[0217] In step 803: based on the binary code, each hash code and the feature similarity of each sample pair, a quantization loss is determined.

[0218] Here, the quantization loss is the value obtained by quantizing the loss of similarity caused by converting the original data corresponding to the sample into hash code by the hash model. The similarity between samples can be represented by feature similarity and the distance required to be approximated between hash codes (which can be represented by binary codes). The quantization loss can be obtained based on parameters such as binary codes and feature similarity.

[0219] For example, the quantization loss function corresponding to the quantization loss L can be expressed as the following formula (8):

[0220]

[0221] Among them, S is the matrix composed of feature similarities, ‖·‖2 represents the l2 norm of the vector, ‖·‖ F represents the F norm of the matrix, Z xis a continuous hash code set corresponding to the x modality (in the embodiment of the present application, a continuous hash code set corresponding to the picture modality), Z y It is a set of continuous hash codes corresponding to the y mode (in the embodiment of the present application, it is a set of continuous hash codes corresponding to the text mode). Can be used to characterize continuous hash codes With continuous hash encoding The similarity of The similarity loss during the conversion of feature data into continuous hash codes can be characterized. Can characterize continuous hash codes The difference loss between the corresponding binary code, Can characterize continuous hash codes The difference loss between and the corresponding binary code. Based on formula (8), the value of quantization loss L can be obtained.

[0222] For example, after obtaining the binary code of each continuous hash code pair, the binary code is substituted into formula (8), and formula (8) is optimized by the gradient random descent algorithm to obtain the optimized quantization loss.

[0223] In step 804: the cross-modal hash model is iteratively updated based on the quantization loss and the learning rate.

[0224] Continue to refer Fig. 7A The binary code, the image continuous hash code 715 and the text continuous hash code 725 are input into the objective function optimization module 740. The objective function optimization module 740 obtains the quantization loss L based on the above parameters and optimizes formula (8) (quantization loss function).

[0225] For example, the learning rate η is used to characterize the amount of quantization loss used in the back-propagation process. The expected quantization loss required in the back-propagation process can be obtained according to the product of the quantization loss L and the learning rate. The expected output result of the hash model is obtained based on the hash code of the expected quantization loss and the actual output. The feature data corresponding to each sample is obtained by back-propagation based on the expected output result and the hash model. The parameters that can fit the expected output result and the feature data corresponding to each sample are determined as update parameters, and the original parameters in the hash model are replaced with the update parameters to implement the update operation of the hash model. Steps 801 to 804 are looped to implement the iterative update operation of the hash model.

[0226] For example, based on the cross-modal hash model that completes iterative updates, the samples in the database can be hash-coded to obtain the hash code corresponding to each sample. The samples in the database can be feature data of items to be recommended. After obtaining the hash code of each item to be recommended in different modalities, a mapping relationship between each item to be recommended and its hash code is established, and the hash codes and mapping relationships of the items to be recommended in different modalities are stored in the database together. The samples in the database can also be the user's feature data (user portrait, account information, browsing history, preference settings, etc.), each piece of feature data has the corresponding user's identifier, and the feature data is hash-coded based on the cross-modal hash model to obtain a hash code corresponding to each feature data. When a recommendation request carrying the identifier of the user to be recommended is received from the user to be recommended through the terminal device, the feature data of the user to be recommended and the hash code corresponding to the feature data are retrieved from the database based on the identifier. Calculate the distance between the hash code corresponding to the feature data and the hash code of each recommended item stored in the database, and use the distance between hash codes to represent the similarity between the user's features and different items. The distance between hash codes is negatively correlated with the similarity. The closer the distance between hash codes, the higher the similarity. Sorting advertisements of multiple items in descending order of similarity as the ones that the user may be interested in, push advertisements that may be of interest to the user (for example, in the form of text, pictures, videos, etc.).

[0227] In the embodiment of the present application, the updated cross-modal hash model is called to perform dimensionality reduction hash coding processing on samples of different modalities, and a hash code that can more accurately characterize the similarity between samples through the distance between hash codes can be obtained. The updated cross-modal hash model can be applied to perform dimensionality reduction hash coding on data of different modalities in the database, and the hash code can be used as a label or index of the data corresponding to the hash code. The hash codes of data of different modalities are used for data retrieval, which reduces the amount of calculation required to calculate the similarity between the query sample and the returned sample in the cross-modal retrieval, saves computing resources, and improves the retrieval effect.

[0228] It is understandable that in the embodiments of the present application, related data such as user information and user feedback data are involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0229] The following continues to describe the exemplary structure of the cross-modal hash model training device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 2AAs shown, the software modules in the training device 455 of the cross-modal hash model stored in the memory 440 may include: a sample acquisition module 4551, configured to acquire multiple sample pairs, wherein each sample pair includes two samples for describing the same content and different modalities; a hash coding module 4552, configured to call the cross-modal hash model to perform dimensionality reduction hash coding processing on the multiple sample pairs, and obtain multiple hash coding pairs corresponding to the multiple sample pairs one by one, wherein the distance between two hash codes in the hash coding pair is used to characterize the similarity between the two samples corresponding to the hash coding pair; a binary code generation module 4553, configured to perform the following processing on each hash coding pair Reasoning: In the hash data point pair at each position in the hash code pair, determine the target hash data point with a larger weight, and determine the binary code of the hash code pair based on the target hash data point corresponding to each position, wherein the weight is used to characterize the importance of the hash data point for calculating the similarity of the sample pair corresponding to the hash code pair; the loss determination module 4554 is configured to determine the total quantization loss of the cross-modal hash model based on each hash code pair, the similarity matrix corresponding to multiple sample pairs, and the difference between each hash code pair and the binary code corresponding to each hash code pair; the loss determination module 4554 is also configured to update the parameters of the cross-modal hash model based on the total quantization loss.

[0230] In some embodiments, the different modalities include a first modality and a second modality; each sample pair includes: a sample of the first modality and a sample of the second modality; the multiple hash code pairs corresponding to the multiple sample pairs include: multiple hash codes corresponding to the multiple samples of the first modality, and multiple hash codes corresponding to the multiple samples of the second modality;

[0231] The hash coding module 4552 is configured to perform feature extraction on multiple samples of the first modality in the multiple sample pairs to obtain a sample feature set of the first modality, and perform feature extraction on multiple samples of the second modality in the multiple sample pairs to obtain a sample feature set of the second modality;

[0232] A dimensionality reduction hash coding process is performed on the sample feature set of the first modality to obtain multiple hash codes corresponding one-to-one to multiple samples of the first modality, and a dimensionality reduction hash coding process is performed on the sample feature set of the second modality to obtain multiple hash codes corresponding one-to-one to multiple samples of the second modality, wherein the hash codes of the multiple samples of the first modality have the same length as the hash codes of the multiple samples of the second modality.

[0233] The hash coding module 4552 is configured to perform dimensionality reduction hash coding processing on the multiple feature data points included in the sample feature set of the first modality, respectively, to obtain multiple hash codes corresponding to the multiple samples of the first modality, wherein the multiple feature data points correspond to the multiple samples of the first modality one by one;

[0234] The hash coding module 4552 is configured to perform dimensionality reduction hash coding processing on the multiple feature data points included in the sample feature set of the second modality, and obtain hash codes that correspond one-to-one to the multiple samples of the second modality, wherein the multiple feature data points correspond one-to-one to the multiple samples of the second modality.

[0235] In some embodiments, the sample feature set of the first modality includes multiple feature data points, wherein the multiple feature data points correspond one-to-one to the multiple samples of the first modality; the sample feature set of the second modality includes multiple feature data points, wherein the multiple feature data points correspond one-to-one to the multiple samples of the second modality.

[0236] The hash coding module 4552 is configured to obtain the similarity matrix corresponding to multiple sample pairs in the following manner: perform the following processing on each feature data point in the sample feature set of the first modality: determine the position number of the feature data point of the first modality in the sample feature set, and determine the row corresponding to the position number; obtain the feature similarity between the feature data point of the first modality and each feature data point of the second modality in turn according to the position number of the feature data point of each second modality in the sample feature set; use each obtained feature similarity as an element in the row in turn, and combine the obtained elements of each row into a similarity matrix.

[0237] In some embodiments, the different modalities include a first modality and a second modality, each hash code pair includes a hash code of a sample of the first modality and a hash code of a sample of the second modality, and each hash code includes a plurality of hash data points.

[0238] The binary code generation module 4553 is configured to traverse each hash code pair, obtain the target weight matrix of the traversed hash code pair, and perform the following processing for each position in the traversed hash code pair: determine the weight of the hash data point located at position in the hash code of the sample of the first modality and the weight of the hash data point located at position in the hash code of the sample of the second modality based on the target weight matrix of the traversed hash code pair; determine the hash data point with a larger weight as the target hash data point corresponding to the position.

[0239] In some embodiments, the target weight matrix is ​​a diagonal matrix, and the number of elements on the diagonal of the diagonal matrix is ​​equal to the sum of the code lengths of the two hash codes in the hash code pair;

[0240] The binary code generation module 4553 is configured to determine the first column number at the position in the hash code pair, and use the sum of the first column number and the code length as the second column number; use the elements located on the diagonal of the target weight matrix at the first column number as the weights of the hash data points of the samples of the first mode at the position; and use the elements located on the diagonal of the target weight matrix at the second column number as the weights of the hash data points of the samples of the second mode at the position.

[0241] The binary code generation module 4553 is configured to obtain the weight variable matrix of the traversed hash code pair, wherein the weight variable matrix is ​​a diagonal matrix, and the number of elements on the diagonal of the diagonal matrix is ​​equal to the sum of the code lengths of the two hash codes in the hash code pair, and the elements on the diagonal are variables; based on the weight variable matrix and the hash code pair, determine the variable Hamming similarity of two samples represented by the distance between the hash code pairs, wherein the variable Hamming similarity is represented in matrix form; determine the variable similarity difference between the feature similarity of two feature data points corresponding to the hash code pair and the variable Hamming similarity; determine the minimum similarity difference corresponding to the variable similarity difference, and determine the target Hamming similarity when the similarity difference is minimum based on the minimum difference value and the feature similarity corresponding to the two feature data points; based on the target Hamming similarity and the hash code pair, determine the target weight matrix of the hash code pair when the similarity difference is minimum.

[0242] The binary code generation module 4553 is configured to generate a coding matrix based on the hash code pair and determine the transposed matrix of the coding matrix; determine the product between the coding matrix, the weight variable matrix and the transposed matrix of the coding matrix; divide the product by the sum of the coding lengths of the two hash codes in the hash code pair to obtain the variable Hamming similarity.

[0243] The binary code generation module 4553 is configured to perform binarization processing on the target hash data point corresponding to each position, and combine the obtained binary data points at each position into a binary code of a hash code pair.

[0244] In some embodiments, the value type of the binary hash data point includes a first value and a second value, and the first value is greater than the second value.

[0245] The binary code generation module 4553 is configured to, when the value of the target hash data point at the position is greater than zero, binarize the value corresponding to the target hash data point to obtain a binary data point of a first value; when the value of the target hash data point at the position is less than zero, binarize the value corresponding to the target hash data point to obtain a binary data point of a second value; and combine the obtained binary data points at each position into a binary code of a hash code pair.

[0246] In some embodiments, the different modalities include a first modality and a second modality, each hash code pair includes a hash code of a sample of the first modality and a hash code of a sample of the second modality, and each hash code includes a plurality of hash data points.

[0247] The loss determination module 4554 is configured to perform the following processing on any two hash code pairs: transpose the hash code of the sample of the second modality in the first hash code pair to obtain a transposed hash code; multiply the hash code of the sample of the first modality in the second hash code pair by the transposed hash code to obtain a hash code product; obtain the feature similarity between the one-to-one feature data points of the two samples corresponding to the hash code product from the similarity matrix; determine the similarity loss based on the difference between the hash code product and the corresponding feature similarity; perform the following processing on each hash code pair: determine the difference loss of each hash code pair based on the difference between the hash code of the first modality and the corresponding binary code, and the difference between the hash code of the second modality and the corresponding binary code; add each similarity loss and each difference loss to obtain the total quantization loss of the cross-modal hash model.

[0248] The loss determination module 4554 is configured to determine the product of the total quantization loss and the learning rate corresponding to the cross-modal hash model, and use the product as the target loss; use each hash code pair as the actual output result, and determine the expected output result of the cross-modal hash model based on the target loss and the actual output result; based on the expected output result and multiple sample pairs, back-propagate the cross-modal hash model, and determine the update parameters of the cross-modal hash model during the back-propagation process, wherein the update parameters are used to fit the expected output result and multiple sample pairs; replace the original parameters of the cross-modal hash model with the update parameters to obtain an updated cross-modal hash model.

[0249] In some embodiments, the first modality is any modality in a modality set, and the second modality is any modality in the modality set except the first modality; the modality set includes: picture modality, text modality, video modality and audio modality.

[0250] The hash coding module 4552 is further configured to obtain a sample pair to be processed, wherein the sample pair includes two samples for describing the same content but in different modalities; calling a cross-modal hash model to perform dimensionality reduction hash coding processing on the sample pair to obtain a hash coding pair corresponding one-to-one to the sample pair, wherein the distance between two hash codes in the hash coding pair is used to characterize the similarity between the two samples corresponding to the hash coding pair.

[0251] In some embodiments, the cross-modal hash model includes a first modality hash model and a second modality hash model, and different modalities include a first modality and a second modality; sample pairs include: samples of the first modality and samples of the second modality; hash code pairs corresponding one to one to the sample pairs include: hash codes of samples of the first modality, and hash codes of samples of the second modality.

[0252] The hash coding module 4552 is further configured to call the first modality hash model, perform feature extraction on the sample of the first modality in the sample pair to obtain feature data of the first modality, and perform dimensionality reduction hash coding on the feature data of the first modality to obtain a hash code corresponding to the sample of the first modality; call the second modality hash model, perform feature extraction on the sample of the second modality in the sample pair to obtain feature data of the second modality, and perform dimensionality reduction hash coding on the feature data of the second modality to obtain a hash code corresponding to the sample of the second modality, wherein the hash code of the sample of the first modality has the same length as the hash code of the sample of the second modality.

[0253] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the training method of the cross-modal hash model described in the embodiment of the present application.

[0254] The present application embodiment provides a computer-readable storage medium storing executable instructions, wherein executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the training method of the cross-modal hash model provided by the present application embodiment, for example, Figure 3A The training method of the cross-modal hashing model is shown.

[0255] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0256] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0257] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0258] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0259] In summary, through the embodiments of the present application, the importance of each hash data point in the hash code is characterized by weights, and the weights are positively correlated with the importance. The binary codes of the hash code pairs determined based on the hash data points with higher weights have higher accuracy, and a more accurate total quantization loss of the cross-modal hash model can be obtained using the binary codes. The cross-modal hash model is trained based on the total quantization loss, so that the distance between the hash codes output by the cross-modal hash model can more accurately characterize the similarity between samples, thereby improving the encoding accuracy of the cross-modal hash model. Furthermore, in the process of calculating the similarity between samples using hash codes, the occupied computing resources are saved and the computing efficiency is improved; in the process of cross-modal retrieval using hash codes, retrieval results that better meet the retrieval requirements can be obtained.

[0260] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A training method for a cross-modal hashing model, characterized in that: The method comprises: Acquire multiple sample pairs, wherein each sample pair includes two samples for describing the same content but in different modalities; the modality of the sample is one of text, audio, image, and video; Calling a cross-modal hash model to perform dimensionality reduction hash coding processing on the plurality of sample pairs to obtain a plurality of hash code pairs corresponding one-to-one to the plurality of sample pairs, wherein a distance between two hash codes in the hash code pair is used to characterize the similarity between the two samples corresponding to the hash code pair; The different modalities include a first modality and a second modality, each of the hash code pairs includes a hash code of a sample of the first modality and a hash code of a sample of the second modality, and each of the hash codes includes a plurality of hash data points; The following processing is performed for each of the hash code pairs: in the hash data point pair at each position in the hash code pair, a target hash data point with a larger weight is determined, and a binary code of the hash code pair is determined based on the target hash data point corresponding to each position, wherein the weight is used to represent the importance of the hash data point for calculating the similarity of the sample pair corresponding to the hash code pair; The following processing is performed for any two of the hash code pairs: the hash code of the sample of the second modality in the first hash code pair is transposed to obtain a transposed hash code; the hash code of the sample of the first modality in the second hash code pair is multiplied by the transposed hash code to obtain a hash code product; from a similarity matrix, the feature similarity between the feature data points of the two samples corresponding to the hash code product is obtained; based on the difference between the hash code product and the corresponding feature similarity, the similarity loss is determined; Performing the following processing on each of the hash code pairs: determining a difference loss of each of the hash code pairs based on a difference between the hash code of the first modality and the corresponding binary code, and a difference between the hash code of the second modality and the corresponding binary code; Adding each of the similarity losses and each of the difference losses to obtain a total quantization loss of the cross-modal hash model; Parameters of the cross-modal hashing model are updated based on the total quantization loss.

2. The method according to claim 1, characterized in that Each of the sample pairs includes: a sample of the first modality and a sample of the second modality; the multiple hash code pairs corresponding to the multiple sample pairs include: multiple hash codes corresponding to the multiple samples of the first modality, and multiple hash codes corresponding to the multiple samples of the second modality; The step of performing dimensionality reduction hash coding processing on the plurality of sample pairs to obtain a plurality of hash coding pairs corresponding one-to-one to the plurality of sample pairs includes: Performing feature extraction on multiple samples of the first modality in the multiple sample pairs to obtain a sample feature set of the first modality, and performing feature extraction on multiple samples of the second modality in the multiple sample pairs to obtain a sample feature set of the second modality; Perform dimensionality reduction hash coding on the sample feature set of the first modality to obtain multiple hash codes corresponding one-to-one to the multiple samples of the first modality, and perform dimensionality reduction hash coding on the sample feature set of the second modality to obtain multiple hash codes corresponding one-to-one to the multiple samples of the second modality, wherein the hash codes of the multiple samples of the first modality have the same length as the hash codes of the multiple samples of the second modality.

3. The method according to claim 2, characterized in that The performing dimensionality reduction hash coding processing on the sample feature set of the first modality to obtain a plurality of hash codes corresponding one-to-one to a plurality of samples of the first modality includes: Performing dimensionality reduction hash coding processing on a plurality of feature data points included in the sample feature set of the first modality, respectively, to obtain a plurality of hash codes corresponding one-to-one to a plurality of samples of the first modality, wherein the plurality of feature data points correspond one-to-one to the plurality of samples of the first modality; The performing dimensionality reduction hash coding processing on the sample feature set of the second modality to obtain a plurality of hash codes corresponding one-to-one to a plurality of samples of the second modality includes: Perform dimensionality reduction hash coding processing on multiple feature data points included in the sample feature set of the second modality to obtain hash codes that correspond one-to-one to the multiple samples of the second modality, wherein the multiple feature data points correspond one-to-one to the multiple samples of the second modality.

4. The method according to claim 2, characterized in that The sample feature set of the first modality includes a plurality of feature data points, wherein the plurality of feature data points correspond one-to-one to a plurality of samples of the first modality; The sample feature set of the second modality includes a plurality of feature data points, wherein the plurality of feature data points correspond one-to-one to a plurality of samples of the second modality; Before determining the total quantization loss of the cross-modal hashing model, the method further includes: The similarity matrix corresponding to multiple sample pairs is obtained in the following way: The following processing is performed on each feature data point in the sample feature set of the first modality: Determine the position number of the feature data point of the first modality in the sample feature set, and determine the row corresponding to the position number; According to the position number of each feature data point of the second modality in the sample feature set, sequentially obtaining the feature similarity between the feature data point of the first modality and each feature data point of the second modality; Each obtained feature similarity is sequentially used as an element in the row, and the obtained elements of each row are combined into the similarity matrix.

5. The method according to claim 1, characterized in that The step of determining a target hash data point with a larger weight in the hash data point pair at each position in the hash code pair includes: Traverse each of the hash code pairs, obtain the target weight matrix of the traversed hash code pair, and perform the following processing for each position in the traversed hash code pair: Determine, based on the target weight matrix of the traversed hash code pairs, the weight of the hash data point located at the position in the hash code of the sample of the first modality and the weight of the hash data point located at the position in the hash code of the sample of the second modality; The hash data point with a larger weight is determined as the target hash data point corresponding to the position.

6. The method according to claim 5, characterized in that The target weight matrix is ​​a diagonal matrix, and the number of elements on the diagonal of the diagonal matrix is ​​equal to the sum of the code lengths of the two hash codes in the hash code pair; The determining, based on the target weight matrix of the traversed hash code pairs, the weight of the hash data point located at the position in the hash code of the sample of the first modality and the weight of the hash data point located at the position in the hash code of the sample of the second modality comprises: Determine a first column number of the position in the hash code pair, and use the sum of the first column number and the code length as a second column number; Using the elements located in the first column on the diagonal of the target weight matrix as the weights of the hash data points of the samples of the first modality located at the positions; The elements on the diagonal of the target weight matrix and located in the second column are used as the weights of the hash data points of the samples of the second modality located at the positions.

7. The method according to claim 5, characterized in that The obtaining of the target weight matrix of the traversed hash code pairs includes: Obtaining a weight variable matrix of the traversed hash code pair, wherein the weight variable matrix is ​​a diagonal matrix, and the number of elements on the diagonal of the diagonal matrix is ​​equal to the sum of the code lengths of the two hash codes in the hash code pair, and the elements on the diagonal are variables; Based on the weight variable matrix and the hash code pair, determining the variable Hamming similarity of the two samples represented by the distance between the hash code pair, wherein the variable Hamming similarity is represented in matrix form; Determine a variable similarity difference between a feature similarity of two feature data points corresponding to the hash code pair and the variable Hamming similarity; Determine a minimum similarity difference corresponding to the variable similarity difference, and determine a target Hamming similarity when the similarity difference is minimum based on the minimum similarity difference and the feature similarities corresponding to the two feature data points; Based on the target Hamming similarity and the hash code pair, a target weight matrix of the hash code pair when the similarity difference is minimized is determined.

8. The method according to claim 7, characterized in that The determining, based on the weight variable matrix and the hash code pair, the variable Hamming similarity of the two samples represented by the distance between the hash code pairs comprises: Generate a coding matrix based on the hash code pair, and determine a transposed matrix of the coding matrix; Determining the product of the encoding matrix, the weight variable matrix and the transposed matrix of the encoding matrix; The variable Hamming similarity is obtained by dividing the product by the sum of the code lengths of the two hash codes in the hash code pair.

9. The method according to claim 1, characterized in that The determining the binary code of the hash code pair based on the target hash data point corresponding to each position includes: The target hash data point corresponding to each position is binarized, and the obtained binarized data points at each position are combined into a binary code of the hash code pair.

10. The method according to claim 9, characterized in that The value type of the binary data point includes a first value and a second value, and the first value is greater than the second value; The binarization processing is performed on the target hash data point corresponding to each position, and the obtained binarized data point of each position is combined into a binary code of the hash code pair, including: When the value of the target hash data point at the position is greater than zero, binarizing the value corresponding to the target hash data point to obtain a binarized data point of a first value; When the value of the target hash data point at the position is less than zero, binarizing the value corresponding to the target hash data point to obtain a binarized data point of a second value; The obtained binary data points at each position are combined into a binary code of the hash code pair.

11. The method according to claim 1, characterized in that The updating of the parameters of the cross-modal hash model based on the total quantization loss includes: Determine a product of the total quantization loss and a learning rate corresponding to the cross-modal hash model, and use the product as a target loss; Taking each of the hash code pairs as an actual output result, and determining an expected output result of the cross-modal hash model based on the target loss and the actual output result; Based on the expected output result and the multiple sample pairs, back-propagating the cross-modal hash model, and determining update parameters of the cross-modal hash model during the back-propagation process, wherein the update parameters are used to fit the expected output result and the multiple sample pairs; The original parameters of the cross-modal hash model are replaced with the updated parameters to obtain an updated cross-modal hash model.

12. The method according to claim 1, characterized in that The first modality is any modality in a modality set, and the second modality is any modality in the modality set except the first modality; The modality set includes: picture modality, text modality, video modality and audio modality.

13. A method for encoding a cross-modal hash model, characterized in that: The method is based on the cross-modal hash model of any one of claims 1 to 12; the method comprises: Obtain a sample pair to be processed, wherein the sample pair includes two samples for describing the same content but in different modalities; the modality of the sample is one of text, audio, image, and video; The cross-modal hash model is called to perform dimensionality reduction hash coding processing on the sample pair to obtain a hash code pair corresponding to the sample pair one by one, wherein the distance between two hash codes in the hash code pair is used to characterize the similarity between the two samples corresponding to the hash code pair.

14. The method according to claim 13, characterized in that The cross-modal hash model includes a first modality hash model and a second modality hash model; the sample pair includes: a sample of the first modality and a sample of the second modality; the hash code pair corresponding to the sample pair includes: a hash code of the sample of the first modality, and a hash code of the sample of the second modality; The calling of the cross-modal hash model to perform dimensionality reduction hash coding processing on the sample pairs to obtain hash coding pairs corresponding to the sample pairs one by one includes: Calling the first modality hash model, performing feature extraction on the sample of the first modality in the sample pair to obtain feature data of the first modality, and performing dimensionality reduction hash coding processing on the feature data of the first modality to obtain a hash code corresponding to the sample of the first modality; Call the second modality hash model, perform feature extraction on the samples of the second modality in the sample pair to obtain feature data of the second modality, and perform dimensionality reduction hash coding on the feature data of the second modality to obtain a hash code of the samples of the second modality, wherein the hash code of the samples of the first modality has the same length as the hash code of the samples of the second modality.

15. A training device for a cross-modal hashing model, characterized in that: The device comprises: A sample acquisition module, used to acquire a plurality of sample pairs, wherein each sample pair includes two samples for describing the same content but in different modalities; the modality of the sample is one of text, audio, image, and video; A hash coding module, used for calling a cross-modal hash model to perform dimensionality reduction hash coding processing on the plurality of sample pairs, to obtain a plurality of hash coding pairs corresponding one-to-one to the plurality of sample pairs, wherein the distance between two hash codes in the hash coding pair is used to characterize the similarity between the two samples corresponding to the hash coding pair; the different modalities include a first modality and a second modality, each of the hash coding pairs includes a hash code of a sample of the first modality and a hash code of a sample of the second modality, and each of the hash codes includes a plurality of hash data points; A binary code generation module is used to perform the following processing for each of the hash code pairs: determining a target hash data point with a larger weight in the hash data point pair at each position in the hash code pair, and determining a binary code of the hash code pair based on the target hash data point corresponding to each position, wherein the weight is used to represent the importance of the hash data point for calculating the similarity of the sample pair corresponding to the hash code pair; A loss determination module is used to perform the following processing on any two of the hash code pairs: transpose the hash code of the sample of the second modality in the first hash code pair to obtain a transposed hash code; multiply the hash code of the sample of the first modality in the second hash code pair by the transposed hash code to obtain a hash code product; obtain the feature similarity between the one-to-one feature data points of the two samples corresponding to the hash code product from the similarity matrix; determine the similarity loss based on the difference between the hash code product and the corresponding feature similarity; perform the following processing on each of the hash code pairs: determine the difference loss of each of the hash code pairs based on the difference between the hash code of the first modality and the corresponding binary code, and the difference between the hash code of the second modality and the corresponding binary code; add each of the similarity losses and each of the difference losses to obtain the total quantization loss of the cross-modal hash model; The loss determination module is also used to update the parameters of the cross-modal hash model based on the total quantization loss.

16. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable instructions; A processor, configured to implement the method according to any one of claims 1 to 14 when executing the executable instructions stored in the memory.

17. A computer-readable storage medium storing executable instructions, characterized in that: When the executable instructions are executed by a processor, the method according to any one of claims 1 to 14 is implemented.

18. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

Images