Material retrieval method and device, computer equipment and storage medium

Abstract

The invention belongs to the field of artificial intelligence, and relates to a material retrieval method which comprises the following steps: acquiring a material sample set, and performing multi-modal feature extraction on each material picture of the material sample set to obtain text features and image features; splicing the text feature and the image feature of each material picture to obtain a plurality of multi-modal fusion features; performing Hash coding on the multi-modal fusion features to obtain material Hash codes of all the multi-modal fusion features, and constructing a retrieval database based on all the material Hash codes; obtaining a multi-modal fusion feature of the to-be-detected image to obtain a query vector, and calculating a hash value of the query vector to obtain a query hash value; and matching a material hash code most similar to the query hash value from a retrieval database to obtain a target hash code, and taking a material corresponding to the target hash code as a retrieval result. The invention further provides a material retrieval device, computer equipment and a storage medium. According to the method, quick retrieval of the material library is realized, and the accuracy of material retrieval is improved.

Description

Technical Field

[0001] The present application relates to the field of digital medicine and material retrieval technology, and in particular to a material retrieval method, apparatus, computer equipment and storage medium. Background Art

[0002] Traditional material retrieval methods mainly rely on structured information such as material codes and keywords. However, unstructured information such as material description texts and pictures often contains richer semantic information. How to effectively use this information to improve the precision and recall of material retrieval is the main challenge currently faced. Similarity-based material retrieval methods can solve the above problems to a certain extent, but there are still some limitations in practical applications. First, the similarity calculation of materials often relies on manually defined features, such as keyword frequency, text length, etc. The selection of these features requires a lot of domain knowledge and experience, and it is difficult to characterize the deep semantic information of the material. Secondly, for non-text information such as pictures, traditional feature representation methods such as color histograms and texture features cannot accurately characterize the semantic information of the picture content, resulting in low accuracy of similarity calculation. Summary of the invention

[0003] The purpose of the embodiments of the present application is to propose a material retrieval method, device, computer equipment and storage medium to solve the technical problem of low material identification accuracy in traditional technologies.

[0004] In order to solve the above technical problems, the present application embodiment provides a material retrieval method, which adopts the following technical solution:

[0005] Obtain a material sample set, perform multimodal feature extraction on each material image in the material sample set, and obtain text features and image features;

[0006] The text features and image features of each material image are spliced ​​to obtain multiple multimodal fusion features;

[0007] Hash coding is performed on the multimodal fusion features to obtain material hash codes of all multimodal fusion features, and a retrieval database is constructed based on all material hash codes;

[0008] Acquire the multimodal fusion features of the image to be detected to obtain a query vector, and calculate the hash value of the query vector to obtain a query hash value;

[0009] The material hash code that is most similar to the query hash value is matched from the search database to obtain a target hash code, and the material corresponding to the target hash code is used as the search result.

[0010] Furthermore, the multimodal feature extraction is performed on each material image in the material sample set to obtain text features, including:

[0011] Performing text recognition on the material image to obtain text data;

[0012] Preprocessing the text data to obtain a processed text containing a plurality of segmented words;

[0013] A preset word embedding model is used to perform feature extraction on the processed text to obtain the text features.

[0014] Furthermore, the image features are obtained by:

[0015] Performing image enhancement on each material image in the material sample set to obtain an enhanced sample set;

[0016] Using a pre-trained convolutional neural network to perform multi-dimensional feature extraction on the material images in the enhanced sample set to obtain multi-dimensional image features;

[0017] The image features of all dimensions are fused by weighted fusion to obtain the image features.

[0018] Furthermore, the text features and image features of each material picture are spliced ​​to obtain multiple multimodal fusion features, including:

[0019] Calling a preset multimodal feature fusion network, wherein the multimodal feature fusion network includes an attention mechanism module and a fully connected module;

[0020] Inputting the text features and the image features into the multimodal feature fusion network, and extracting attention features of different dimensions from the text features and the image features based on a preset attention mechanism module, to obtain text attention features and image attention features of multiple dimensions;

[0021] The text attention features and image attention features of multiple dimensions are input into the fully connected module to obtain the multimodal fusion features.

[0022] Further, matching the material hash code most similar to the query hash value from the search database to obtain the target hash code includes:

[0023] Calculate the Hamming distance between the query hash value and each material hash code in the retrieval database;

[0024] Count all the material hash codes whose Hamming distance exceeds the preset threshold to obtain the candidate material set;

[0025] The cosine similarity between the multimodal fusion feature corresponding to each material hash code in the candidate material set and the query vector is calculated, and the material hash codes in the candidate material set are sorted from large to small according to the cosine similarity, and the optimal material hash code is selected from the sorted candidate material set as the target hash code.

[0026] Furthermore, after sorting the material hash codes in the candidate material set from large to small according to the cosine similarity, the method further includes:

[0027] Obtaining auxiliary information of the material corresponding to the hash code of each material in the candidate material set, wherein the auxiliary information at least includes material quality and supplier credibility;

[0028] Performing weighted scoring on the sorted material hash codes in the candidate material set based on the auxiliary information;

[0029] The ranking of the candidate material set is optimized according to the scores to obtain a comprehensively ranked candidate material set.

[0030] Furthermore, after taking the material corresponding to the target hash code as the search result, the method further comprises:

[0031] Recording the search results corresponding to the query vector and generating a query log;

[0032] According to the query log, the retrieval database is updated in real time according to incremental learning.

[0033] In order to solve the above technical problems, the embodiment of the present application further provides a material retrieval device, which adopts the following technical solution:

[0034] A feature extraction module is used to obtain a material sample set, perform multimodal feature extraction on each material image in the material sample set, and obtain text features and image features;

[0035] A feature fusion module, used to combine the text features and image features of each material image to obtain multiple multimodal fusion features;

[0036] A construction module, used to perform hash coding on the multimodal fusion features, obtain material hash codes of all multimodal fusion features, and construct a retrieval database based on all material hash codes;

[0037] A calculation module, used to obtain the multimodal fusion features of the image to be detected, obtain a query vector, and calculate a hash value of the query vector to obtain a query hash value;

[0038] The matching module is used to match the material hash code most similar to the query hash value from the search database to obtain a target hash code, and use the material corresponding to the target hash code as the search result.

[0039] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0040] The computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the material retrieval method as described in any one of the above items are implemented.

[0041] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0042] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the material retrieval method as described in any one of the above items are implemented.

[0043] Compared with the prior art, the embodiments of the present application have the following beneficial effects: in view of the fact that material library images contain both descriptive text and image information, the multimodal features of material images are extracted to obtain the text features and image features of the material images, and the two modal features are fused to form multimodal fusion features. On this basis, the hash coding method is used to map the multimodal fusion features into binary codes, so as to realize the rapid retrieval of massive material libraries, which significantly improves the accuracy and efficiency of material retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0046] Figure 2 is a flow chart of an embodiment of a material retrieval method according to the present application;

[0047] Figure 3 yes Figure 2 A flowchart of a specific implementation method of obtaining text features in step S201;

[0048] Figure 4 yes Figure 2A flowchart of a specific implementation method of obtaining image features in step S201;

[0049] Figure 5 yes Figure 2 A flowchart of a specific implementation of step S202;

[0050] Figure 6 yes Figure 2 A flowchart of a specific implementation of step S205;

[0051] Figure 7 yes Figure 6 A flowchart of a specific implementation method after step S603;

[0052] Figure 8 yes Figure 2 A flowchart of a specific implementation method after step S205;

[0053] Fig. 9 is a structural schematic diagram of an embodiment of a material retrieval device according to the present application;

[0054] Fig.10 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0056] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0057] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0058] like Figure 1As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0059] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0060] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0061] The server 103 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal device 101 .

[0062] It should be noted that the material retrieval method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the material retrieval device is generally arranged in the server / terminal device.

[0063] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0064] Continue to refer Figure 2 , shows a flow chart of an embodiment of a material retrieval method according to the present application. The material retrieval method comprises the following steps:

[0065] Step S201, obtaining a material sample set, performing multimodal feature extraction on each material image in the material sample set, and obtaining text features and image features;

[0066] In this embodiment, the electronic device (eg, Figure 1 The server / terminal device shown in the figure) can receive a request for obtaining a pair of sample images and description texts through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection and other wireless connection methods currently known or to be developed in the future. Among them, the material sample set can be called or downloaded from the local database of the hospital or the Internet.

[0067] Step S202, combining the text features and image features of each material image to obtain a plurality of multimodal fusion features;

[0068] Step S203, performing hash coding on the multimodal fusion features to obtain material hash codes of all multimodal fusion features, and constructing a retrieval database based on all material hash codes;

[0069] In this embodiment, a simple feature splicing method can be used, or a feature fusion method based on an attention mechanism can be used to adaptively learn features of different modalities so that the obtained multimodal fusion features have material features with higher accuracy. Based on the multimodal fusion features of all material images, a hash coding method is used to encode all multimodal fusion features into binary material hash codes, and a retrieval database is established with all material hash codes. Compared with storing multimodal fusion features of material images, storing material hash codes can greatly save storage resources.

[0070] Step S204, obtaining a multimodal fusion feature of the image to be detected, obtaining a query vector, and calculating a hash value of the query vector to obtain a query hash value;

[0071] In this embodiment, an image to be detected is obtained, and multimodal features of the image to be detected are extracted and spliced ​​according to the aforementioned method to obtain multimodal fusion features of the image to be detected. The multimodal fusion features of the image to be detected are used as a query vector, and the query vector is hash encoded to obtain a hash value of the query vector. The hash value of the query vector is used as the query hash value.

[0072] Step S205, matching the material hash code most similar to the query hash value from the search database to obtain a target hash code, and taking the material corresponding to the target hash code as the search result.

[0073] In this embodiment, based on the query vector, the most similar material hash code is matched from the retrieval database. The matching method can adopt similarity measurement methods such as cosine similarity, Hamming distance, and Euclidean distance to calculate the similarity between the query vector and each material hash code in the retrieval database. The material hash code with the highest similarity is used as the target hash code, and the material corresponding to the target hash code is used as the retrieval result.

[0074] This embodiment extracts the multimodal features of each material image in the material sample set, obtains the text features and image features of each material image, and splices the text features and image features of each material image to obtain the multimodal fusion features of each material image, and then obtains a more accurate representation vector of the material image; the hash coding method is used to map the multimodal fusion features into binary codes, thereby realizing rapid retrieval of massive material libraries, significantly improving the accuracy and efficiency of material retrieval.

[0075] like Figure 3 As shown, in some optional implementations of this embodiment, the step 201 of performing multimodal feature extraction on each material image of the material sample set to obtain text features includes the following steps:

[0076] Step S301, performing text recognition on the material image to obtain text data;

[0077] Step S302, preprocessing the text data to obtain a processed text containing a plurality of segmented words;

[0078] Step S303: Use a preset word embedding model to perform feature extraction on the processed text to obtain the text features.

[0079] In this embodiment, the text recognition of the material image can be performed through OCR recognition technology to obtain the text data of the material image; the text data is preprocessed, including text cleaning, word segmentation, and stop word removal to obtain the processed text; the text data is converted into a low-dimensional dense vector representation using a word embedding model such as Word2Vec to obtain the text features of the material text. Furthermore, in order to ensure the accuracy of the text features, the word vector model can be incrementally trained and optimized regularly. Extracting the text features of the text data through the word embedding model is conducive to obtaining the semantic information of the material image and providing information support for material retrieval.

[0080] like Figure 4 As shown, in some optional implementations of this embodiment, the image features in step 201 are obtained by:

[0081] Step S401, performing image enhancement on each material image in the material sample set to obtain an enhanced sample set;

[0082] Step S402, using a pre-trained convolutional neural network to perform multi-dimensional feature extraction on the material images in the enhanced sample set to obtain multi-dimensional image features;

[0083] Step S403: weighted fusion is used to fuse the image features of all dimensions to obtain the image features.

[0084] In this embodiment, in order to extract as many material features as possible, each material image in the material sample set is enhanced, such as improving the contrast and brightness of the material image, improving the quality of the material image, which is conducive to extracting more material image features when extracting features from the material image in the subsequent process. A pre-trained convolutional neural network is used to perform multi-dimensional feature extraction on the material images in the enhanced sample set to obtain multi-dimensional image features, and the image features of each dimension are fused through weighted fusion, and the features of the material image in different dimensions are integrated, so that the obtained image features have more material information.

[0085] like Figure 5 As shown, in some optional implementations of this embodiment, step 202 includes the following steps:

[0086] Step S501, calling a preset multimodal feature fusion network, wherein the multimodal feature fusion network includes an attention mechanism module and a fully connected module;

[0087] Step S502, inputting the text features and the image features into the multimodal feature fusion network, and extracting attention features of different dimensions from the text features and the image features based on a preset attention mechanism module, to obtain text attention features and image attention features of multiple dimensions;

[0088] Step S503: input the text attention features and image attention features of multiple dimensions into the fully connected module to obtain the multimodal fusion features.

[0089] In this embodiment, the attention mechanism is an important technology in deep learning. Through the attention mechanism, the user can allow the model to selectively learn the key parts of image features and text features, improve the model's attention to important features, and then mine features. For image features and text features, a multimodal feature fusion network is constructed, and the semantic association between the two features is learned through the attention mechanism, and the complementary information between the features is mined to generate a fused multimodal feature representation. Among them, for text features, by setting different convolution kernel sizes such as 3, 4, 5, etc., text features of different dimensions are extracted to obtain a multi-scale text feature vector with a dimension of 512. For image features, by setting different pooling layer sizes, image features of different dimensions such as 64x64, 128x128, 256x256 are extracted. The extracted text features are input into the attention mechanism, and the text attention features of different dimensions are obtained by calculating the similarity between features of different dimensions. Similarly, the image features are input into the attention mechanism to obtain image attention features of different dimensions.

[0090] This embodiment uses a multimodal feature fusion network to learn the semantic association and complementarity between image features and text features, thereby enhancing the representation of the material image and the corresponding text possessed by the fused features.

[0091] like Figure 6 As shown, in some optional implementations of this embodiment, step 205 includes the following steps:

[0092] Step S601, calculating the Hamming distance between the query hash value and each material hash code in the retrieval database;

[0093] Step S602, counting the hash codes of all materials whose Hamming distance exceeds a preset threshold to obtain a candidate material set;

[0094] Step S603, calculating the cosine similarity between the multimodal fusion feature corresponding to each material hash code in the candidate material set and the query vector, and sorting the material hash codes in the candidate material set from large to small according to the cosine similarity, and selecting the optimal material hash code from the sorted candidate material set as the target hash code.

[0095] In this embodiment, a hash index is constructed based on a search database. When a search request from a user is received, the query hash value of the request is obtained, and the Hamming distance between the query hash value and each material hash code in the search database is calculated one by one. The similarity between the material and the query is determined according to the size of the Hamming distance, and the material corresponding to the material hash code whose Hamming distance exceeds a preset threshold is used as a candidate material to obtain a candidate material set; in the candidate material set, the material is sorted according to the cosine similarity between the multimodal fusion feature and the query vector, wherein the material hash code corresponding to the multimodal fusion feature with the highest cosine similarity is the target hash code. Among them, the Hamming distance is a method to measure the difference between two equal-length strings, which represents the number of different characters at corresponding positions of the two strings.

[0096] This embodiment uses hash retrieval to quickly match a set of candidate materials from a retrieval database, thereby quickly narrowing the range of candidate materials; based on cosine similarity calculation, the cosine similarity between the multimodal fusion features of each candidate material and the query vector is calculated to achieve further similarity comparison, thereby greatly improving the retrieval accuracy.

[0097] like Figure 7 As shown, in some optional implementations of this embodiment, after step 603, the material retrieval method disclosed in this application further includes the following steps:

[0098] Step S701, obtaining auxiliary information of the material corresponding to each material hash code in the candidate material set, wherein the auxiliary information at least includes material quality and supplier credibility;

[0099] Step S702, performing weighted scoring on the sorted material hash codes in the candidate material set based on the auxiliary information;

[0100] Step S703, optimizing the ranking of the candidate material set according to the scores to obtain a comprehensively ranked candidate material set.

[0101] In this embodiment, in order to further improve the retrieval accuracy, after sorting the material hash codes in the candidate material set according to the cosine similarity, the material hash codes in the sorted candidate material set can be weighted and scored in combination with the auxiliary information of the material, such as material quality and supplier reputation, and the sorting in the candidate material set can be optimized according to the final score. By optimizing the sorting in the candidate material set in combination with the auxiliary information, the accuracy and practicality of the retrieval results can be ensured.

[0102] like Figure 8 As shown, in some optional implementations of this embodiment, after step S205, the following steps are included:

[0103] Step S801, recording the search results corresponding to the query vector and generating a query log;

[0104] Step S802: updating the retrieval database in real time according to the query log by incremental learning.

[0105] In this embodiment, in order to continuously optimize the index and query strategy and improve the performance and user experience of material retrieval, by recording the retrieval results corresponding to the query vector, a query log is generated, and incremental learning is used to update the retrieval database in real time to ensure the real-time and accuracy of the retrieval results.

[0106] Furthermore, during the material retrieval process, real-time feedback information from users is obtained, and the user intent understanding model and similarity calculation model are optimized and updated in real time using an online learning algorithm. The user's retrieval request is processed based on the optimized user intent understanding model and similarity calculation model, and the cosine similarity between the multimodal fusion feature corresponding to the material hash code and the query vector is calculated, and materials with high cosine similarity are returned to the user as retrieval results. User feedback information on the retrieval results is continuously collected, the feedback information is analyzed, and user satisfaction evaluation on the retrieval results is obtained. The retrieval algorithm is further optimized and improved based on the satisfaction evaluation, so as to continuously improve the performance of the material retrieval system and user experience.

[0107] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0108] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0109] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0110] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0111] Further references Fig. 9 , as a response to the above Figure 2 The present application provides an embodiment of a material retrieval device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0112] like Fig. 9 As shown, the material search device 900 described in this embodiment includes: a feature extraction module 901, a feature fusion module 902, a construction module 903, a calculation module 904, and a matching module 905. Among them:

[0113] The feature extraction module 901 is used to obtain a material sample set, perform multimodal feature extraction on each material image in the material sample set, and obtain text features and image features;

[0114] A feature fusion module 902 is used to combine the text features and image features of each material image to obtain multiple multimodal fusion features;

[0115] A construction module 903 is used to perform hash coding on the multimodal fusion features to obtain material hash codes of all multimodal fusion features, and to construct a retrieval database based on all material hash codes;

[0116] The calculation module 904 is used to obtain the multimodal fusion features of the image to be detected, obtain a query vector, and calculate the hash value of the query vector to obtain a query hash value;

[0117] The matching module 905 is used to match the material hash code most similar to the query hash value from the search database to obtain a target hash code, and use the material corresponding to the target hash code as the search result.

[0118] In some optional implementations of this embodiment, the feature extraction module 901 includes:

[0119] A recognition unit, used for performing text recognition on the material image to obtain text data;

[0120] A preprocessing unit, used for preprocessing the text data to obtain a processed text containing a plurality of segmented words;

[0121] The text feature extraction unit is used to extract features from the processed text using a preset word embedding model to obtain the text features.

[0122] In some optional implementations of this embodiment, the feature extraction module 901 includes:

[0123] An enhancement unit, configured to perform image enhancement on each material image in the material sample set to obtain an enhanced sample set;

[0124] An image feature extraction unit, used to perform multi-dimensional feature extraction on the material images in the enhanced sample set using a pre-trained convolutional neural network to obtain multi-dimensional image features;

[0125] The multi-dimensional feature fusion unit is used to fuse the image features of all dimensions by using weighted fusion to obtain the image features.

[0126] In some optional implementations of this embodiment, the feature fusion module 902 includes:

[0127] A calling unit, used to call a preset multimodal feature fusion network, wherein the multimodal feature fusion network includes an attention mechanism module and a fully connected module;

[0128] An attention feature extraction unit, used for inputting the text features and the image features into the multimodal feature fusion network, and extracting attention features of different dimensions from the text features and the image features based on a preset attention mechanism module, so as to obtain text attention features and image attention features of multiple dimensions;

[0129] A connection unit is used to input the text attention features and image attention features of multiple dimensions into the fully connected module to obtain the multimodal fusion features.

[0130] In some optional implementations of this embodiment, the matching module 905 includes:

[0131] A distance calculation unit, used to calculate the Hamming distance between the query hash value and each material hash code in the retrieval database;

[0132] A statistical unit is used to count the hash codes of all materials whose Hamming distance exceeds a preset threshold to obtain a candidate material set;

[0133] A sorting unit is used to calculate the cosine similarity between the multimodal fusion feature corresponding to each material hash code in the candidate material set and the query vector, and sort the material hash codes in the candidate material set from large to small according to the cosine similarity, and select the optimal material hash code from the sorted candidate material set as the target hash code.

[0134] In some optional implementations of this embodiment, the material retrieval device 900 further includes:

[0135] An acquisition unit, used to acquire auxiliary information of a material corresponding to each material hash code in the candidate material set, wherein the auxiliary information at least includes material quality and supplier credibility;

[0136] A scoring unit, configured to perform weighted scoring on the sorted material hash codes in the candidate material set based on the auxiliary information;

[0137] The scoring and sorting unit is used to optimize the sorting of the candidate material set according to the scoring to obtain a sorted candidate material set.

[0138] In some optional implementations of this embodiment, the material retrieval device 900 further includes:

[0139] A recording module, used to record the search results corresponding to the query vector and generate a query log;

[0140] An updating module is used to update the retrieval database in real time according to the query log and incremental learning.

[0141] To solve the above technical problems, the present application also provides a computer device. Fig.10 , Fig.10 This is a basic structural block diagram of the computer device in this embodiment.

[0142] The computer device 10 includes a memory 1001, a processor 1002, and a network interface 1003 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 10 having a memory 1001, a processor 1002, and a network interface 1003, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.

[0143] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0144] The memory 1001 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 1001 can be an internal storage unit of the computer device 10, such as a hard disk or memory of the computer device 10. In other embodiments, the memory 1001 can also be an external storage device of the computer device 8, such as a plug-in hard disk equipped on the computer device 10, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 1001 can also include both the internal storage unit of the computer device 10 and its external storage device. In this embodiment, the memory 1001 is generally used to store the operating system and various application software installed on the computer device 10, such as computer-readable instructions of the material retrieval method, etc. In addition, the memory 1001 can also be used to temporarily store various types of data that have been output or are to be output.

[0145] The processor 1002 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 1002 is generally used to control the overall operation of the computer device 10. In this embodiment, the processor 1002 is used to run the computer-readable instructions stored in the memory 1001 or process data, such as computer-readable instructions for running the material retrieval method.

[0146] The network interface 1003 may include a wireless network interface or a wired network interface. The network interface 1003 is generally used to establish a communication connection between the computer device 10 and other electronic devices.

[0147] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the material retrieval method as described above.

[0148] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0149] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

[0150] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

Claims

1. A material retrieval method, characterized in that: The steps include: Obtain a material sample set, perform multimodal feature extraction on each material image in the material sample set, and obtain text features and image features; The text features and image features of each material image are spliced ​​to obtain multiple multimodal fusion features; Hash coding is performed on the multimodal fusion features to obtain material hash codes of all multimodal fusion features, and a retrieval database is constructed based on all material hash codes; Acquire the multimodal fusion features of the image to be detected to obtain a query vector, and calculate the hash value of the query vector to obtain a query hash value; The material hash code that is most similar to the query hash value is matched from the search database to obtain a target hash code, and the material corresponding to the target hash code is used as the search result.

2. The material retrieval method according to claim 1, characterized in that: The multimodal feature extraction is performed on each material image in the material sample set to obtain text features, including: Performing text recognition on the material image to obtain text data; Preprocessing the text data to obtain a processed text containing a plurality of segmented words; A preset word embedding model is used to perform feature extraction on the processed text to obtain the text features.

3. The material retrieval method according to claim 1, characterized in that: The image features are obtained in the following way: Performing image enhancement on each material image in the material sample set to obtain an enhanced sample set; Using a pre-trained convolutional neural network to perform multi-dimensional feature extraction on the material images in the enhanced sample set to obtain multi-dimensional image features; The image features of all dimensions are fused by weighted fusion to obtain the image features.

4. The material retrieval method according to claim 1, characterized in that: The text features and image features of each material picture are spliced ​​to obtain multiple multimodal fusion features, including: Calling a preset multimodal feature fusion network, wherein the multimodal feature fusion network includes an attention mechanism module and a fully connected module; Inputting the text features and the image features into the multimodal feature fusion network, and extracting attention features of different dimensions from the text features and the image features based on a preset attention mechanism module, to obtain text attention features and image attention features of multiple dimensions; The text attention features and image attention features of multiple dimensions are input into the fully connected module to obtain the multimodal fusion features.

5. The material retrieval method according to claim 1, characterized in that: The step of matching the material hash code most similar to the query hash value from the search database to obtain a target hash code includes: Calculate the Hamming distance between the query hash value and each material hash code in the retrieval database; Count all the material hash codes whose Hamming distance exceeds the preset threshold to obtain the candidate material set; The cosine similarity between the multimodal fusion feature corresponding to each material hash code in the candidate material set and the query vector is calculated, and the material hash codes in the candidate material set are sorted from large to small according to the cosine similarity, and the optimal material hash code is selected from the sorted candidate material set as the target hash code.

6. The material retrieval method according to claim 5, characterized in that: After sorting the material hash codes in the candidate material set from large to small according to the cosine similarity, the method further includes: Obtaining auxiliary information of the material corresponding to the hash code of each material in the candidate material set, wherein the auxiliary information at least includes material quality and supplier credibility; Performing weighted scoring on the sorted material hash codes in the candidate material set based on the auxiliary information; The ranking of the candidate material set is optimized according to the scores to obtain a comprehensively ranked candidate material set.

7. The material retrieval method according to claim 1, characterized in that: After the material corresponding to the target hash code is used as the search result, the method includes: Recording the search results corresponding to the query vector and generating a query log; According to the query log, the retrieval database is updated in real time according to incremental learning.

8. A material retrieval device, characterized in that: include: A feature extraction module is used to obtain a material sample set, perform multimodal feature extraction on each material image in the material sample set, and obtain text features and image features; A feature fusion module, used to combine the text features and image features of each material image to obtain multiple multimodal fusion features; A construction module, used to perform hash coding on the multimodal fusion features, obtain material hash codes of all multimodal fusion features, and construct a retrieval database based on all material hash codes; A calculation module, used to obtain the multimodal fusion features of the image to be detected, obtain a query vector, and calculate a hash value of the query vector to obtain a query hash value; The matching module is used to match the material hash code most similar to the query hash value from the search database to obtain a target hash code, and use the material corresponding to the target hash code as the search result.

9. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the material retrieval method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the material retrieval method according to any one of claims 1 to 7 are implemented.

Images