Multimodal search method, apparatus, storage medium, and device

Abstract

The application provides a multimodal search method, device, storage medium and equipment, the method comprises the following steps: respectively extracting deep semantic representation vectors and shallow semantic representation contents from multimodal data and to-be-queried data, calculating the first similarity of the deep semantic representation vector of the to-be-queried data and the deep semantic representation vector of the multimodal data, obtaining a first preset number of multimodal data, calculating the second similarity of the shallow semantic representation content of the to-be-queried data and the shallow semantic representation content of the multimodal data, obtaining a second preset number of multimodal data, combining the first preset number of multimodal data and the second preset number of multimodal data to obtain recall data, inputting the features of the recall data into a feature combination model after feature extraction, outputting a matching score, and obtaining a multimodal search result according to the matching score. The application can find corresponding content from multimodal data for to-be-queried content at one time, and the search speed is fast and the search accuracy is high.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a multi-modal search method and device, a storage medium and equipment. BACKGROUND

[0002] The current era is one of information explosion, and users are surrounded by various data, but it takes more time to carefully screen the data they want.

[0003] Most search platforms retrieve data in a single mode, for example, to search for a picture, the user needs to input text related to the picture to search in the search platform; to search for a video, the user needs to input text related to the video to search in the search platform; to search for a document or a news, the user needs to input text related to the document or news to search in the search platform. The search method used in the prior art can only search in a single mode of data, and the user needs to search for each mode of data to obtain the desired content, which is slow. At the same time, when the user input text does not match the content they want to search for, it will also lead to inaccurate search results. SUMMARY

[0004] Therefore, the present application provides a multi-modal search method, device, storage medium and equipment, which can find the corresponding content from the multi-modal data for the to-be-queried content at one time, and has a fast search speed and high search accuracy.

[0005] In a first aspect, the present application provides a multi-modal search method, characterized in that the method comprises:

[0006] extracting multi-modal text information from multi-modal data to obtain multi-modal deep semantic representation vectors and multi-modal shallow semantic representation content;

[0007] extracting to-be-queried data information from to-be-queried data to obtain deep semantic representation vectors of the to-be-queried data and shallow semantic representation content of the to-be-queried text;

[0008] calculating a first similarity between the deep semantic representation vectors of the to-be-queried data and the multi-modal deep semantic representation vectors, and obtaining a first preset number of multi-modal data according to the first similarity score;

[0009] calculating a second similarity between the shallow semantic representation content of the to-be-queried data and the multi-modal shallow semantic representation content, and obtaining a second preset number of multi-modal data according to the second similarity score;

[0010] combining the first preset number of multi-modal data and the second preset number of multi-modal data to obtain recall data;

[0011] The recall data is input into a feature combination model after feature extraction, and a matching score of the to-be-queried data and the multi-modal data is output, and a multi-modal search result of the to-be-queried data is obtained according to the matching score.

[0012] Further, the multi-modal data includes various types of document texts, pictures, audios, and videos, the multi-modal text information is extracted from the multi-modal data, and the multi-modal deep semantic representation vector and the multi-modal shallow semantic representation content are obtained, which includes:

[0013] The document texts, pictures, audios, and videos are parsed respectively, and the titles, keywords, abstracts, and text contents in the document texts, pictures, audios, and videos are extracted respectively;

[0014] The titles, keywords, and abstracts are vectorized to obtain the multi-modal deep semantic representation vector, which is stored in a deep semantic representation database;

[0015] The titles and text contents are stored as multi-modal shallow semantic representation vectors in a shallow semantic representation database.

[0016] Further, the document texts are parsed by a document parser to extract the titles, keywords, abstracts, and text contents in the document texts.

[0017] Further, the pictures are parsed to extract the titles, keywords, abstracts, and text contents in the pictures, which includes:

[0018] The pictures are classified by using a picture classification model to obtain the classification labels of the pictures;

[0019] The document texts are extracted from the pictures by using a character recognition method;

[0020] The extracted document texts in the pictures are parsed by using a document editor to obtain the titles, keywords, abstracts, and text contents corresponding to the pictures.

[0021] Further, the method further includes:

[0022] It is detected whether the title corresponding to the picture exists;

[0023] If the title exists, the title corresponding to the picture and the classification label of the picture are used as one of the extraction features of the recall data;

[0024] If the title does not exist, the classification label of the picture is used as one of the extraction features of the recall data.

[0025] Further, the audios and videos are parsed to extract the titles, keywords, abstracts, and text contents in the audios and videos, which includes:

[0026] detecting whether there is a subtitle in the audio and video;

[0027] if yes, extracting the subtitle text in the audio and video, using a document parser to extract the title, keywords, abstract and text content in the subtitle text;

[0028] if no, converting the speech text in the audio and video into document text, using a document parser to extract the title, keywords, abstract and text content in the document text.

[0029] Further, the features extracted from the recall data include:

[0030] normalizing the second similarity score to obtain a normalized similarity score;

[0031] calculating a third similarity score of the deep semantic representation vector of the to-be-queried data and the deep semantic representation vector of the recall data;

[0032] an editable distance of the to-be-queried data and the recall data;

[0033] the number of characters in the to-be-queried data;

[0034] the total number of character content in the recall data;

[0035] obtaining a first matching ratio according to the ratio of the deep semantic representation vector of the recall data and the third similarity score;

[0036] obtaining a second matching ratio according to the ratio of the third similarity score and the editable distance;

[0037] the content type label, update date and comment number of the multi-modal data.

[0038] In a second aspect, an embodiment of the present application provides a multi-modal search device, the device comprising:

[0039] a multi-modal representation content extraction module for extracting multi-modal text information from multi-modal data, obtaining a multi-modal deep semantic representation vector and a multi-modal shallow semantic representation content;

[0040] a to-be-queried data representation content extraction module for extracting to-be-queried data information from to-be-queried data, obtaining a deep semantic representation vector of the to-be-queried data and a shallow semantic representation content of the to-be-queried text;

[0041] a first similarity calculation module for calculating a first similarity of the deep semantic representation vector of the to-be-queried data and the multi-modal deep semantic representation vector, and obtaining a first preset number of multi-modal data according to the first similarity score;

[0042] The second similarity calculation module is used to calculate the second similarity between the shallow semantic representation content of the data to be queried and the multimodal shallow semantic representation content, and to obtain a second preset number of multimodal data based on the second similarity score.

[0043] The combination module is used to combine the first preset number of multimodal data and the second preset number of multimodal data to obtain recall data;

[0044] The matching module is used to extract features from the recalled data and input them into the feature combination model, output the matching score between the query data and the multimodal data, and obtain the multimodal search results of the query data based on the matching score.

[0045] Thirdly, embodiments of the present invention provide a storage medium storing a computer program, wherein the computer program is configured to execute the method described in any one of the first aspects when running.

[0046] Fourthly, embodiments of the present invention provide an apparatus including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the method described in any one of the first aspects.

[0047] The technical solution provided by this invention extracts multimodal text information from multimodal data to obtain multimodal deep semantic representation vectors and multimodal shallow semantic representation content. It also extracts deep semantic representation vectors and shallow semantic representation content of the query data from the query data. Next, it calculates the first similarity between the deep semantic representation vectors and multimodal semantic representation vectors of the query data, obtains a first preset number of multimodal data based on the first similarity score, calculates the second similarity between the shallow semantic representation content of the query data and the shallow semantic representation content of the multimodal data, obtains a second preset number of multimodal data based on the second similarity score, combines the first preset number of multimodal data and the second preset number of multimodal data to obtain recall data, extracts features from the recall data and inputs it into a feature combination model, outputs a matching score between the query data and the multimodal data, and obtains the search results for the query data based on the matching score. Therefore, this application extracts deep semantic representation vectors and shallow semantic representation content from multimodal data. Then, it calculates the similarity between the deep semantic representation vectors and shallow semantic representation content of the query data and the deep semantic representation vectors and shallow semantic representation content of the multimodal data. Based on the similarity calculation results, it obtains initially filtered multimodal data. Next, it combines the multimodal data obtained by the two similarity calculation methods to obtain recall data. Finally, it extracts features from the recall data and inputs them into a feature combination model, outputting a matching score. Based on the matching score, search results can be obtained. Therefore, this application can obtain documents, images, audio, and video content related to the query content in a single search, improving search speed. Furthermore, since users do not need to manually input text related to the search content, it improves search accuracy.

[0048] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0049] Figure 1 This is a flowchart of a multimodal search method provided in an embodiment of the present invention;

[0050] Figure 2 This is a structural diagram of a multimodal search device provided in an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0053] See Figure 1 , Figure 1 This is a flowchart of a multimodal search method provided in an embodiment of the present invention, the method comprising the following steps:

[0054] Step 101: Extract multimodal text information from multimodal data to obtain multimodal deep semantic representation vectors and multimodal shallow semantic representation content.

[0055] In this step, the multimodal data is first processed by extracting text information and then vectorizing it. The multimodal data can be any one or more of the following: document text, images, audio, and video.

[0056] In some embodiments, extracting multimodal text information from the modal data to obtain a multimodal semantic representation vector and multimodal shallow semantic representation content may include:

[0057] Step 1011: Parse the document text, images, audio, and video respectively, and extract the title, keywords, summary, and text content from the document text, images, audio, and video respectively;

[0058] Step 1012: Vectorize the title, keywords and abstract to obtain multimodal deep semantic representation vectors and store them in the deep semantic representation database;

[0059] Step 1013: Store the title and text content as multimodal shallow semantic representation vectors in the shallow semantic representation database.

[0060] Specifically, for document text, parsing the document text and extracting the title, keywords, summary, and text content can be achieved in the following ways:

[0061] The document parser is used to parse various types of documents in the multimodal data to obtain the title, keywords, abstract and full text content of each document.

[0062] In this application, after parsing each document, the title, keywords, abstract, and full text content of each document can be obtained. The title is divided into at least two levels of subheadings, and each subheading is expanded. The parent title and the current subheading are used as title indexes, and the paragraph content corresponding to the current subheading is used as a content index. Both the title index and the content index are vectorized, and the vectorized title index and content index are stored in the sentence pair index.

[0063] For images in multimodal data, the first step is to use an image classification model to classify the images and obtain classification labels. For example, after inputting an image into the classification model, content classification labels such as entertainment, military, society, and children can be obtained. At the same time, text recognition methods (such as Optical Character Recognition, or OCR) are used to extract the text within the image. Then, a document parser is used to extract keywords and summary information from the text content. The image title, keywords, and summary information are used as the document representation content, and after vectorization, deep sentence representation vectors are generated and stored in the deep semantic representation database. Finally, the text title, image description, and image text recognition content are used as shallow semantic representation content and stored in the shallow semantic representation database.

[0064] In some embodiments, after parsing the document text extracted from the image using a document editor, the method further includes: detecting whether a title corresponding to the image exists; if a title exists, the title corresponding to the image and the category label of the image are used as one of the extraction features of the recall data; if not, the category label corresponding to the image is used as one of the extraction features of the recall data. Here, recall data refers to a combination of multimodal data obtained based on the similarity between the deep semantic representation vector and shallow semantic representation content of the query data and the multimodal deep semantic representation vector and multimodal shallow semantic representation content, respectively. Recall data will be described in detail below.

[0065] For audio and video documents in multimodal data, the process first determines whether there are subtitles in the audio and video content. If there are subtitles, the original subtitles are used; otherwise, speech-to-text conversion is used to extract the subtitles from the audio and video. The subtitle content is then processed by a document parser to extract the title, keywords, and summary information of the text content. The title, keywords, and summary information of the audio and video are used as the representation content of the document. After vectorization, deep sentence representation vectors are generated and stored in a deep semantic representation database. Finally, the text information such as the title and subtitles are stored and processed as shallow semantic representation content.

[0066] Therefore, the multimodal data of this application includes document text, images, audio and video. The deep semantic representation database stores the titles, keywords and summary information of the document text, images, audio and video, while the shallow semantic representation database stores the titles, keywords and image content of the document text, images, audio and video.

[0067] Step 102: Extract the information of the data to be queried from the data to be queried, and obtain the deep semantic representation vector and the shallow semantic representation content of the data to be queried.

[0068] In this step, the data to be queried is processed to extract the information of the data to be queried, and to obtain the deep semantic representation vector and shallow semantic representation content of the data to be queried.

[0069] Specifically, when the data to be queried is a text document, the document is parsed to extract the text. A document parser is used to extract the title, keywords, and summary information of the text content. The title, keywords, and summary information of the document to be queried are then vectorized to generate a deep semantic representation vector of the data to be queried. Finally, the title and content of the document to be queried are used as the shallow semantic representation content of the data to be queried.

[0070] When the data to be queried is an image, the image classification model is first used to classify the image and obtain category tags. For example, after inputting the image to be queried into the classification model, the content category tags of the image to be queried can be obtained, such as entertainment, military, society, children, etc. The category tags of the image are used as the title of the image. At the same time, the text extraction function of the text recognition method (such as optical character recognition, or OCR) is used to extract the text in the image to be queried. Then, the document parser is used to extract the keywords and summary information of the text content. The title, keywords and summary information of the image to be queried are used as the representation content of the document. After vectorization processing, a deep sentence representation vector of the image to be queried is generated. Finally, the text information such as the title, image description and image text recognition content are used as the shallow semantic representation content of the image to be queried.

[0071] When the data to be queried is an audio or video document, the system first determines whether the audio or video contains subtitles. If subtitles are present, the original subtitles are used; otherwise, speech-to-text conversion is used to extract the subtitles from the audio or video. The document parser then extracts the title, keywords, and summary information of the text content from the subtitles. The title, keywords, and summary information of the audio or video are used as the representation content of the document. After vectorization, a deep sentence representation vector of the data to be queried is generated. Finally, the text information, such as the title and subtitles, is used as the shallow semantic representation content of the data to be queried.

[0072] Step 103: Calculate the first similarity between the deep semantic representation vector of the data to be queried and the multimodal deep semantic representation vector, and obtain a first preset number of multimodal data based on the first similarity score.

[0073] In this step, the first similarity between the deep semantic representation vector generated from the query data and the deep semantic representation vector generated from the multimodal data is calculated. For example, cosine similarity can be used to calculate the first similarity between the deep semantic representation vector generated from the query data and the deep semantic representation vector generated from the multimodal data. Based on the score of the first similarity, the top N multimodal data are then sorted and obtained. The value of the first preset number N can be selected based on experience. In addition to cosine similarity, other similarity calculation methods can be used, and this application does not limit the specific methods used.

[0074] Step 104: Calculate the second similarity between the shallow semantic representation content of the data to be queried and the multimodal shallow semantic representation content, and obtain a second preset number of multimodal data based on the second similarity score.

[0075] In this step, the BM25 algorithm, a text similarity calculation method, can be used to calculate the second similarity between the shallow semantic representation content and the multimodal shallow semantic representation content of the data to be queried. Based on the second similarity score, the second similarity scores are sorted, and the top M multimodal data are selected. The value of M can be determined by empirical values, and the values ​​of M and N can be the same or different.

[0076] Step 105: Combine the first preset number of multimodal data and the second preset number of multimodal data to obtain recall data.

[0077] In this step, a first preset number of multimodal data and a second preset number of multimodal data are combined, that is, the first N multimodal data obtained based on the first similarity calculation result and the first M multimodal data obtained based on the second similarity calculation result are combined.

[0078] Step 106: Extract features from the recalled data and input them into the feature combination model. Output the matching score between the query data and the multimodal data. Obtain the multimodal search results for the query data based on the matching score.

[0079] In this step, extracting features from the recalled data may include:

[0080] Step 1061: Normalize the second similarity score to obtain the normalized similarity score;

[0081] Step 1062: Calculate the third similarity score between the deep semantic representation vector of the data to be queried and the deep semantic representation vector of the recalled data;

[0082] Step 1063: The editable distance between the data to be queried and the recalled data;

[0083] Step 1064: Number of characters in the data to be queried;

[0084] Step 1065: The total number of text contents in the recalled data;

[0085] Step 1066: Obtain the first matching ratio based on the ratio of the deep semantic representation vector of the recalled data to the third similarity score;

[0086] Step 1067: Obtain the second matching ratio based on the ratio of the third similarity score and the editable distance;

[0087] Step 1068: The content type tags, update date, and number of comments of the multimodal data.

[0088] In this step, after extracting the above features from the recalled data, the features are input into a feature combination model, which can be an xgboost+LR model, to calculate the matching score between the query data and the recalled data. Then, the matching scores are sorted in descending order of score, and the multimodal data with the highest scores can be selected as the multimodal search results.

[0089] Please refer to Figure 2 , Figure 2 This is a structural diagram of a multimodal search device provided in an embodiment of the present invention. The device includes:

[0090] The multimodal representation content extraction module 21 is used to extract multimodal text information from multimodal data to obtain multimodal deep semantic representation vectors and multimodal shallow semantic representation content;

[0091] The data representation content extraction module 22 is used to extract the data information to be queried from the data to be queried, and obtain the deep semantic representation vector of the data to be queried and the shallow semantic representation content with query text.

[0092] The first similarity calculation module 23 is used to calculate the first similarity between the deep semantic representation vector of the data to be queried and the multimodal deep semantic representation vector, and to obtain a first preset number of multimodal data based on the first similarity score.

[0093] The second similarity calculation module 24 is used to calculate the second similarity between the shallow semantic representation content of the data to be queried and the multimodal shallow semantic representation content, and obtain a second preset number of multimodal data based on the second similarity score.

[0094] The combination module 25 is used to combine the first preset number of multimodal data and the second preset number of multimodal data to obtain recall data.

[0095] The matching module 26 is used to extract features from the recalled data and input them into the feature combination model, output the matching score between the query data and the multimodal data, and obtain the multimodal search results of the query data based on the matching score.

[0096] In some embodiments, the multimodal data includes various types of document text, images, audio, and video.

[0097] In some embodiments, the multimodal representation content extraction module 21 may include:

[0098] The parsing unit 211 is used to parse the document text, images, audio and video respectively, and extract the title, keywords, summary and text content from the document text, images, audio and video respectively;

[0099] The deep semantic representation unit 212 is used to vectorize the title, keywords and abstract to obtain multimodal deep semantic representation vectors and store them in the deep semantic representation database.

[0100] The shallow semantic representation unit 213 is used to store the title and text content as multimodal shallow semantic representation vectors in the shallow semantic representation database.

[0101] In some embodiments, a document parser is used to parse the document text and extract the title, keywords, summary, and text content from the document text.

[0102] In some embodiments, parsing the image and extracting the title, keywords, summary, and text content from the image includes:

[0103] Use an image classification model to classify images and obtain their classification labels;

[0104] The document text is extracted from the image using an optical character recognition method.

[0105] The document text extracted from the image is parsed using a document editor to obtain the keywords, summary, and text content corresponding to the image.

[0106] In some embodiments, the apparatus further includes:

[0107] The detection module is used to detect whether the title corresponding to the image exists;

[0108] The first processing module is used to extract the title corresponding to the image and the category label of the image as one of the features of the recall data when the detection module detects that the title of the image exists.

[0109] The second processing module is used to extract the category label corresponding to the image as one of the features of the recall data when the detection module detects that the title of the image does not exist.

[0110] In some embodiments, parsing the audio and video to extract titles, keywords, summaries, and text content from the audio and video includes:

[0111] Detect whether there are subtitles in the audio and video;

[0112] If so, extract the subtitle text from the audio and video, and use a document parser to extract the title, keywords, summary and text content from the subtitle text;

[0113] If not, the speech text in the audio and video is converted into document text, and the title, keywords, summary and text content in the document text are extracted using a document parser.

[0114] The technical solution provided by this invention extracts multimodal text information from multimodal data to obtain multimodal deep semantic representation vectors and multimodal shallow semantic representation content. It also extracts deep semantic representation vectors and shallow semantic representation content of the query data from the query data. Next, it calculates the first similarity between the deep semantic representation vectors and multimodal semantic representation vectors of the query data, obtains a first preset number of multimodal data based on the first similarity score, calculates the second similarity between the shallow semantic representation content of the query data and the shallow semantic representation content of the multimodal data, obtains a second preset number of multimodal data based on the second similarity score, combines the first preset number of multimodal data and the second preset number of multimodal data to obtain recall data, extracts features from the recall data and inputs it into a feature combination model, outputs a matching score between the query data and the multimodal data, and obtains the search results for the query data based on the matching score. Therefore, this application extracts deep semantic representation vectors and shallow semantic representation content from multimodal data. Then, it calculates the similarity between the deep semantic representation vectors and shallow semantic representation content of the query data and the deep semantic representation vectors and shallow semantic representation content of the multimodal data. Based on the similarity calculation results, it obtains initially filtered multimodal data. Next, it combines the multimodal data obtained by the two similarity calculation methods to obtain recall data. Finally, it extracts features from the recall data and inputs them into a feature combination model, outputting a matching score. Based on the matching score, search results can be obtained. Therefore, this application can obtain documents, images, audio, and video content related to the query content in a single search, improving search speed. Furthermore, since users do not need to manually input text related to the search content, it improves search accuracy.

[0115] It should be noted that the multimodal search device in this embodiment of the invention and the multimodal search method in the above embodiments belong to the same inventive concept. Technical details not described in detail in this device can be found in the previous description of the method, and will not be repeated here.

[0116] Furthermore, embodiments of the present invention also provide a storage medium storing a computer program, wherein the computer program is configured to execute the aforementioned method at runtime.

[0117] Figure 3A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0118] like Figure 3 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0119] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0120] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the idle detection method.

[0121] In some embodiments, the idle detection method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the idle detection method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the idle detection method by any other suitable means (e.g., by means of firmware).

[0122] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0123] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0124] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0125] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0126] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0127] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0128] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0129] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A multimodal search method, characterized in that, The method includes: Extract multimodal text information from multimodal data to obtain multimodal deep semantic representation vectors and multimodal shallow semantic representation content; Extract information from the data to be queried to obtain the deep semantic representation vector and the shallow semantic representation content of the data to be queried. Calculate the first similarity between the deep semantic representation vector of the data to be queried and the multimodal deep semantic representation vector, and obtain a first preset number of multimodal data based on the first similarity score; Calculate the second similarity between the shallow semantic representation content of the data to be queried and the multimodal shallow semantic representation content, and obtain a second preset number of multimodal data based on the second similarity score; The first preset number of multimodal data and the second preset number of multimodal data are combined to obtain recall data; After extracting features from the recalled data, the data is input into the XGBoost+LR model, which outputs a matching score between the query data and the multi-recall data. Based on the matching score, the multimodal search results for the query data are obtained. The multimodal data includes various types of document text, images, audio, and video. Extracting multimodal text information from the multimodal data to obtain multimodal deep semantic representation vectors and multimodal shallow semantic representation content includes: The document text, images, audio, and video are parsed separately, and the titles, keywords, summaries, and text content are extracted from each of them. The title, keywords, and abstract are vectorized to obtain multimodal deep semantic representation vectors, which are then stored in a deep semantic representation database. The title and text content are stored as multimodal shallow semantic representation vectors in the shallow semantic representation database.

2. The method according to claim 1, characterized in that, The document text is parsed using a document parser to extract the title, keywords, abstract, and text content.

3. The method according to claim 1, characterized in that, The image is parsed to extract its title, keywords, abstract, and text content, including: Use an image classification model to classify images and obtain their classification labels; The document text is extracted from the image using an optical character recognition method. The document text extracted from the image is parsed using a document editor to obtain the title, keywords, abstract, and text content corresponding to the image.

4. The method according to claim 3, characterized in that, The method further includes: Check if the title corresponding to the image exists; If they exist, the title corresponding to the image and the category label of the image will be used as one of the extracted features of the recall data; If the image does not exist, the category label corresponding to the image will be used as one of the extracted features of the recall data.

5. The method according to claim 1, characterized in that, The audio and video are parsed to extract the title, keywords, summary, and text content, including: Detect whether there are subtitles in the audio and video; If so, extract the subtitle text from the audio and video, and use a document parser to extract the title, keywords, summary and text content from the subtitle text; If not, the speech text in the audio and video is converted into document text, and the title, keywords, summary and text content in the document text are extracted using a document parser.

6. The method according to claim 1, characterized in that, The features extracted from the recalled data include: The second similarity score is normalized to obtain the normalized similarity score; Calculate the third similarity score between the deep semantic representation vector of the query data and the deep semantic representation vector of the recalled data; The editable distance between the data to be queried and the recalled data; The number of characters in the data to be queried; The total number of text contents in the recalled data; The first matching ratio is obtained based on the ratio of the deep semantic representation vector of the recalled data to the third similarity score; A second matching ratio is obtained based on the ratio of the third similarity score to the editable distance; The multimodal data includes content type tags, update dates, and the number of comments.

7. A multimodal search device, characterized in that, The device includes: The multimodal representation content extraction module is used to extract multimodal text information from multimodal data to obtain multimodal deep semantic representation vectors and multimodal shallow semantic representation content. The data representation content extraction module is used to extract the data information to be queried from the data to be queried, and obtain the deep semantic representation vector of the data to be queried and the shallow semantic representation content with query text. The first similarity calculation module is used to calculate the first similarity between the deep semantic representation vector of the data to be queried and the multimodal deep semantic representation vector, and to obtain a first preset number of multimodal data based on the first similarity score. The second similarity calculation module is used to calculate the second similarity between the shallow semantic representation content of the data to be queried and the multimodal shallow semantic representation content, and to obtain a second preset number of multimodal data based on the second similarity score. The combination module is used to combine the first preset number of multimodal data and the second preset number of multimodal data to obtain recall data; The matching module is used to extract features from the recalled data and input them into a feature combination model, outputting a matching score between the query data and the multimodal data, and obtaining multimodal search results for the query data based on the matching score. The multimodal data includes various types of document text, images, audio, and video. The multimodal representation content extraction module is specifically used for: The document text, images, audio, and video are parsed separately, and the titles, keywords, summaries, and text content are extracted from each of them. The title, keywords, and abstract are vectorized to obtain multimodal deep semantic representation vectors, which are then stored in a deep semantic representation database. The title and text content are stored as multimodal shallow semantic representation vectors in the shallow semantic representation database.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method of any one of claims 1 to 6 when it is run.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method of any one of claims 1 to 6.

Images